Projects

Experimental investigations, robotic policy evaluations, and open-source implementations.

Open Codebase Rigorous Evaluation LIBERO Closed-Loop Suite

SmolVLA–LIBERO: Language Sensitivity & Robustness in VLA Models

Completed Experimental Pipeline & Open Codebase (2026) · Lead Researcher & Engineer

GitHub Repository Method Documentation Full Results & Logs Multi-Task Pilot Report

Research Question & Motivation

Two natural language instructions can describe the exact same physical robot task (e.g., "pick up the black bowl between the plate and the ramekin and place it on the plate" vs. "grasp the dark dish and set it onto the plate"). Yet vision-language-action policies frequently predict divergent actions solely due to instruction phrasing.

This project investigates: How sensitive is SmolVLA's behavioral policy to semantically equivalent instruction phrasing, and can paired language-training interventions improve robustness without sacrificing canonical task performance?

Experimental Setup & Protocol Isolation

  • Policy & Framework: Official SmolVLA LIBERO checkpoint adapted via Hugging Face LeRobot pipeline. Evaluated on LIBERO-Spatial Task 0.
  • Rollout Protocol: 280-step closed-loop rollouts, single worker, deterministic execution (seed 0). Primary metric is goal_ever.
  • Demonstrations: 36 official training demonstrations; 1,000 fine-tuning steps per experimental arm.
  • Strict Partitioning: Exploratory States 0–2; Confirmatory States 3–22; Model-selection States 23–32; Frozen Final Evaluation States 33–49 (153 rollouts total).

Evaluated Methods

  1. Canonical-Only Fine-Tuning (original_language): Standard imitation learning fine-tuning SmolVLA exclusively on canonical training instructions.
  2. Language Augmentation (augmented_language): Randomly substituting canonical instructions with paraphrased variants during training.
  3. Paired Behavior Cloning (paired_bc): Simultaneously evaluating paired canonical and paraphrased instructions on identical robot states, enforcing aligned action distribution predictions.
  4. Frozen-Teacher Flow Consistency: Penalizing intermediate flow representation drift between paraphrased inputs and a frozen canonical teacher model.

Frozen Final Evaluation Results (Task 0, States 33–49, 153 Rollouts)

Training Arm Canonical Success Two Paraphrases Pooled Balanced Score H
Canonical-only (original_language) 12/17 · 70.6% 5/34 · 14.7% 0.2434
Language Augmentation (augmented_language) 12/17 · 70.6% 7/34 · 20.6% 0.3188
Paired Training (paired_bc) 16/17 · 94.1% 4/34 · 11.8% 0.2092

Critical Empirical Finding (Negative Robustness Result):
Paired language training looked promising on development prompts, but did not improve unseen-paraphrase success in the frozen final evaluation. Registered paired differences for pooled paraphrase success were −2.9 percentage points vs. canonical-only (95% CI [−14.7, +8.8]) and −8.8 points vs. augmentation (95% CI [−20.6, +2.9]). Both intervals include zero. Strong development performance did not predict generalization to held-out prompt structures.

Development vs Final Success Rates
Figure 1. Development and final success rates across the three final models. Strict state and prompt separation revealed that development gains did not transfer to unseen language conditions.

Five-Task Diagnostic Pilot (75 Rollouts)

To examine whether wording sensitivity is isolated to Task 0, the untuned official SmolVLA policy was evaluated across five competence-screened LIBERO tasks (States 5–9, 2 paraphrases each). Canonical success was 21/25 (84%) while pooled paraphrase success dropped to 4/50 (8%).

Five-task Diagnostic Pilot Success Rates
Figure 2. Task-by-task success rates in the 5-task pilot. Across distinct robotic manipulation tasks, performance consistently plummets on paraphrased descriptions.
Early Action Divergence
Figure 3. First-action divergence across 50 matched canonical/paraphrase rollout pairs. Trajectory divergence begins immediately at Step 1 for all pairs, and larger early Euclidean divergence correlates with rollout failure.

Scientific Takeaways

  • Immediate linguistic divergence: Wording differences induce significant policy action divergence from the very first timestep, even with identical visual observations.
  • Development vs. Final discrepancy: Strict separation of development states/prompts from frozen final evaluations is vital in robot learning to prevent false claims of generalization.
  • Negative result value: Documenting where paired imitation fails provides essential grounding for architectural changes in multimodal action representations.