READ: What Makes a Pretrained VLA Teachable by Demonstration?

Write the demonstrations once, read them at every step

READ adds a demonstration encoder and a few gated readers to π_0.5. The encoder turns five demonstrations (camera video, robot states and actions) into a fixed memory. π_0.5 keeps its usual inputs: camera images, robot state and the instruction. While it generates each action chunk, readers in its action expert query the memory.

Trained on some tasks, tested on others

Every benchmark is split by task. READ is trained on the training tasks only, and is tested mainly on tasks that are not among them. At test time it receives five demonstrations that were never trained on, and each rollout starts from an initial state not used in training.

Trained tasks are also tested, from new demonstrations and initial states, as a reference.

The demonstrations say what the instruction can't

In our Direction suite every task of an object shares one scene and one instruction, move the bread. What differs is the goal: one of twelve directions, 30° apart, 11 cm away. Text never says which. Each clip below is one demonstration, placed at the direction it shows.

Does the policy actually follow them?

Every task of an object starts from the same scene and instruction, so a policy could ignore the demonstrations, always move the object in one default direction, and still succeed on the tasks that ask for that direction. Two controls separate that from following them, both from the same initial state and with the same instruction.

Swap the demonstrations. READ gets demonstrations of a different direction. If it reads them, the object should go where the new demonstrations point.

Remove the demonstrations. The multi-task LoRA baseline is READ without its encoder and readers: the same π_0.5 with a LoRA trained on the same tasks, from instructions alone. Here the instruction is again move the bread (or move the coffee box), the same for every direction, so this baseline shows what the scene and the instruction can do on their own.

Object classes it never manipulated in training

The 37 training objects of the Direction suite come from 11 object classes: mouse, stapler, bell, toy car, Rubik's cube, bread, phone, deck of cards, soap, tea box and coffee box. Here the robot is asked to move objects from six other classes, none of which it ever manipulated in training: a remote control, a can, a seal, a hamburger, a small speaker and a sauce can. The instruction only names the object, move the seal; the five demonstrations show which way it should go.

Success per new object class: two objects of each class, three directions, 50 test episodes each (300 per class; 1,800 in all). Grey: the same policy without demonstrations. Dashed line: READ on new objects of the 11 trained classes in the same three directions.

What in the training data makes this possible?

Two READ models train each of 37 objects in five directions, with nearly the same number of episodes (3,680 and 3,700). The shared split trains every object in the same five directions (30, 180, 210, 270 and 330°). The scattered split gives each object a different five of nine directions: all nine are trained, but each with only 19–22 of the 37 objects. Neither model has trained the bread or the coffee box below at 240°; only the scattered model has seen other objects move that way.

success on new pairings of trained objects and directions, shared split vs scattered split, with nearly the same data.

Beyond the covered directions: carrying helps, reaching doesn't

No training task of any object goes to 90–150°. There the landings follow the demonstrated goal but are pulled toward the neighboring trained directions, 60° and 180°. Can more data close the gap, and which data? We add 79 tasks at 75, 105, 135 and 165° on other objects, so the tested goals stay untrained.

The added data comes in one of two forms. Carrying episodes practice moving an object. Reaching episodes are cheaper to collect: the open gripper approaches the object and sweeps to just above the goal without touching it. They cover the same directions with the arm alone, so comparing the two separates practice in moving objects from practice in moving the arm.

More kinds of behavior in training, more that demonstrations can teach

In our Behavior suite every task of an object again shares one instruction, manipulate the bread, and the demonstrations decide how to move it. Two READ models train on the same 37 objects: a narrow set (hold, translate, rotate) and a broad set that adds tilt, place, drag, push and sequences. The broad set has fewer episodes (3,088 vs 3,626), so it tests richer behavior coverage, not more data.

Where it stops

Shown circling, dropping or shaking, the broad-set model picks the object up and falls back to behaviors it was trained on. Scattering directions across objects and adding behavior kinds both widened what demonstrations could teach; these failures mark what training data should cover next.

Five demonstrations match per-task fine-tuning

On ten held-out LIBERO-Gen tasks, which pair familiar pick-up places and destinations in new ways, READ reaches 92.8% from five demonstrations; a LoRA fine-tuned on each task with all 36–49 of its demonstrations reaches 92.5%. The scene has two identical black bowls, so an instruction must say which bowl to move and where to put it.

Against the same policy without demonstrations (the multi-task LoRA, 85.5%), READ is close on most of the ten tasks and helps most on the one whose instruction gives both places relative to other objects, “pick up the black bowl next to the cookie box and place it in the black bowl next to the ramekin” (T2 in the chart below): 98.8% vs 54.0%.

Full instructions; mean over ten tasks ± std over five evaluation runs of 50 episodes per task. ^†Released checkpoints without a pretrained VLA, five sampling seeds.

(a) Per task. (b) Swapping in another task's demonstrations, with the correct instruction kept, lowers READ's success more than the weight-generating baseline's, so READ relies more on the demonstrations.

The paper in brief

Teaching a pretrained robot policy a new task usually requires another fine-tuning run. We introduce READ, which instead adapts a vision-language-action (VLA) policy from five demonstrations without updating its parameters. A learned encoder turns videos, states and actions into memory tokens that the policy queries while generating actions. On ten held-out LIBERO-Gen tasks with full instructions, READ achieves 92.8% success, matching per-task LoRA fine-tuning on all available demonstrations (92.5%). Two controlled benchmarks test which training experience enables this adaptation. They test whether demonstrations can teach the robot to move familiar objects to new goals, handle new objects, or perform new kinds of manipulation. With nearly the same data, training on a wider range of movement goals raises success on new pairings of trained objects and goals from 1% to 79%. Demonstrations also teach small changes in translation distance, direction and rotation angle. Adding behaviors to training enables the policy to perform them from demonstrations, while six behaviors absent from training remain difficult at this data scale. These findings motivate scaling data to cover more objects, movement goals and behavior kinds.

Architecture

READ adds three components to π_0.5: a demonstration encoder, gated cross-attention readers in the action expert, and one LoRA shared by all tasks. For each demonstration, a trajectory encoder fuses frozen video features with robot states and actions; twenty learned queries summarize it together with its instruction, and an aggregator combines the five demonstrations into the memory. A reader after each of the last six of the action expert's 18 blocks lets the hidden state attend to the memory through a gate that starts at zero, so training starts from π_0.5 with the shared LoRA.

(a) A trajectory encoder, summary queries and an aggregator build the memory once per task. (b) A reader after each of the last six expert blocks lets the hidden state attend to the memory through a zero-initialized gate. Dark bars mark the shared LoRA.

Everything is trained jointly, end to end, with the flow-matching objective used to fine-tune π_0.5 and no auxiliary loss. Each update builds the memory from five episodes of a task and predicts the actions of a different episode of the same task. Because that episode starts from a different initial state, the policy cannot copy a demonstration step by step; it must carry the demonstrated task over to the scene it observes.

Three benchmarks, one question each

LIBERO-Gen: can reading demonstrations replace fine-tuning? Ten held-out tasks recombine pick-up places and destinations seen in 164 training tasks. Its held-out tasks only recombine trained elements and its training set is fixed, so we build two benchmarks whose training sets we can rearrange. Direction suite: 37 objects, each moved 11 cm in one of twelve directions; we control exactly which directions are trained with which objects. Behavior suite: the behavior itself varies, and we ask whether adding behavior kinds widens what demonstrations can teach, as adding directions does. In both suites every task of an object has the same instruction (move the coffee box, manipulate the coffee box), so only the demonstrations show which direction or behavior is requested.

Left, Direction suite: from the same scene, the robot moves the ringed object 11 cm toward one of twelve goals. Right, Behavior suite: tasks differ in how the object is manipulated; dashed boxes mark held-out test behaviors.

Direction suite, by kind of test

Scattered-split model. Swap controls replace the demonstrations with three trained directions of the same object. “Goes there” is a landing within 12° of the swapped-in direction; “original task” uses the full success check. Base π_0.5 succeeds in 0 of 630 episodes.

When the instruction says nothing

With one generic instruction for every LIBERO-Gen task, pick up the black bowl and place it, the demonstrations alone must identify the bowl, its destination and the movement. READ and Demo2LoRA recover part of the task and fail when given another task's demonstrations, but both stay below BPP and below a per-task LoRA trained on five demonstrations.

Separate models are trained for this setting. The last column replaces each task's demonstrations with another task's.

Deployment cost

One L40S, wall time over 360 fresh action chunks per configuration, 100-token LIBERO-Gen memory. The readers query a small cached memory; video is never re-encoded during a rollout.

What comes next

Reliable transfer to the six held-out behaviors remains unresolved at our data scale, and goals in untrained directions are hit far more often once carrying data is added nearby (49% vs 11%). We plan to vary data volume and task diversity separately, using behavior programs with continuous directions, distances and segment orders. We will also test video-pretrained motion encoders and objectives that represent demonstrated object motion, and diagnose whether failures arise in encoding the demonstrations, retrieving the memory or generating actions.