Intel Physical AI Online Challenge · Bimanual VLA Manipulation · Team META FETA

Two SO-101 arms set a dinner table from a language command

MuJoCo simulation. An OpenVINO INT4 language model plans, an OpenVINO INT8 camera network localises objects, and inverse-kinematics skills drive both arms, including a spoon hand-off between them.

GitHub repositoryDownload video

10/10tuning seeds
10/10hold-out seeds
10/10left-handed layout
2.3 msperception, INT8 CPU
12/14valid LLM plans

Evaluation across 10 randomized seeds

Narrated walkthrough (2:39): one full run of seed 0 with its hand-off, then all 10 randomized seeds sped up, then the final table state for every seed.

Pipeline

1 · Understand

Qwen2.5-1.5B-Instruct, INT4, served with OpenVINO GenAI, turns the instruction into a JSON plan: items, arm assignment, handedness. A left-handed request mirrors the layout.

2 · Observe

An overhead camera feeds a small U-Net segmentation model (OpenVINO INT8). Object poses come from the image, not from simulator state.

3 · Act

Damped-least-squares IK skills for both arms: open drawer, retrieve cutlery, hand off through a shared staging zone, place plate, fork and mug.

4 · Optimize

INT4 planner and INT8 perception. The bench script reports latency, throughput, device and precision for CPU, iGPU and NPU.

Per-seed results: seeds 0–9, as shown in the video

SeedMassFrictionPlate rMug rCutleryLight zFinal mug placement errorSim timeOutcome
0×1.74×0.895.0 cm1.4 cm9.6 cm1.64 m7 mm49.9 s✓ complete
1×0.75×0.904.9 cm1.7 cm9.7 cm1.36 m5 mm50.0 s✓ complete
2×1.70×0.994.7 cm1.4 cm9.9 cm1.25 m5 mm50.3 s✓ complete
3×0.84×0.744.5 cm1.4 cm9.6 cm1.45 m7 mm49.9 s✓ complete
4×0.66×0.955.3 cm1.5 cm9.3 cm1.03 m5 mm50.2 s✓ complete
5×0.75×0.815.2 cm1.6 cm9.1 cm1.08 m5 mm50.0 s✓ complete
6×1.69×1.054.9 cm1.4 cm9.8 cm1.60 m5 mm50.0 s✓ complete
7×1.71×0.825.0 cm1.7 cm9.7 cm1.06 m4 mm50.4 s✓ complete
8×1.09×1.274.7 cm1.7 cm9.1 cm1.31 m5 mm50.3 s✓ complete
9×1.58×0.785.3 cm1.4 cm9.7 cm1.40 m5 mm50.0 s✓ complete

Table colour and lighting also change per seed. Values are read from the video overlays.

Final table state, every seed

Seed 0 final table state
Seed 0 · final state
Seed 1 final table state
Seed 1 · final state
Seed 2 final table state
Seed 2 · final state
Seed 3 final table state
Seed 3 · final state
Seed 4 final table state
Seed 4 · final state
Seed 5 final table state
Seed 5 · final state
Seed 6 final table state
Seed 6 · final state
Seed 7 final table state
Seed 7 · final state
Seed 8 final table state
Seed 8 · final state
Seed 9 final table state
Seed 9 · final state

Scope

Learned

  • LLM task planner (INT4)
  • Camera segmentation network (INT8)

Scripted

  • IK manipulation skills and hand-off

Not yet done

  • Benchmark on Intel Core Ultra (numbers so far are from a cloud CPU)
  • Distilled imitation policy
  • Pouring

The simulation and models run locally (MuJoCo + OpenVINO). This page presents the recorded evaluation.