Video + Joints → Force
Does adding Franka joint angles to the V-JEPA video encoder improve contact-force prediction? · sinew-338 · 2026-06-29
Task: video + joints → force (contact / direction / magnitude), Franka-only, in-distribution and across an OOD ladder. The deployable model stays video-only at inference; joints are tested as an inference input and as a training-only signal. Full write-ups: docs/v2f_joints_litreview.md · docs/v2f_joints_design.md · docs/v2f_joints_iteration_log.md.
TL;DR
- Vision+joints ≥ vision in-distribution (0.71 vs 0.69 contact-F1) and same-lab — joints help when the robot-configuration distribution matches.
- Vision+joints falls below vision cross-lab — the classic proprioception shortcut: in-distribution joints become misleading in a new lab.
- Joint-dropout removes the penalty, recovering cross-lab to vision level while keeping the in-distribution gain.
- Joints alone collapse cross-lab (0.16) — bare angles carry no transferable force information (it lives in torque/current, not kinematics).
- Training-only joint↔video SSL is mixed — helps crisp contact, hurts another OOD set; not a decisive generalization lever.
Setup
Frozen V-JEPA2 ViT-L feats (2 cams) cached once → tiny dilated-TCN 3-head studio (holds the vision regime fixed to isolate the joint contribution); confirmed at strong vision with an end-to-end unfreeze run. Joint repr: relative joint angles (q − q[clip-start]) + joint velocity, standardized by train stats, EE-frame labels. OOD ladder (all 7-DoF Franka): in-dist FMB → OOD-task-same-lab fmb_multi → OOD-lab+task crisp peg/ethernet → OOD-lab RH20T cfg5 (contact+mag only). Metric: contact-F1 (vs trivial all-positive) / direction-cosine (contact-masked).
Result 1 — controlled A/B (frozen vision) · contact-F1 / dir-cos
| model | in-dist triv .10 | OOD task fmb_multi | crisp peg triv .76 | crisp eth triv .68 | cfg5 triv .53 |
|---|---|---|---|---|---|
| vision | 0.69 / +0.90 | 0.43 / +0.86 | 0.75 / −0.70 | 0.71 / −0.65 | 0.50 |
| joints only | 0.64 / +0.88 | 0.32 / +0.85 | 0.57 / −0.69 | 0.39 / −0.56 | 0.16 |
| vision + joints | 0.71 / +0.91 | 0.49 / +0.84 | 0.65 / −0.73 | 0.64 / −0.68 | 0.44 |
| vision + joints + joint-dropout | 0.70 / +0.90 | 0.44 / +0.87 | 0.77 / −0.75 | 0.74 / −0.71 | 0.43 |
Cross-lab direction is negative for every model — the known frozen-encoder appearance ceiling, not a joint effect.
Result 2 — training-only joint↔video SSL (inference video-only) · contact-F1
| alignment | in-dist | OOD task | crisp peg | crisp eth | cfg5 |
|---|---|---|---|---|---|
| none (video adapter) | 0.69 | 0.45 | 0.40 | 0.47 | 0.53 |
| joint-aligned | 0.70 | 0.43 | 0.64 | 0.69 | 0.30 |
Mixed: helps crisp contact, hurts cfg5, in-distribution neutral — the same "SSL pays only under a severe gap" pattern seen project-wide.
Result 3 — end-to-end unfreeze confirm (strong vision, in-dist FMB)
| model | contact-F1 | dir-cos | MAE (N) |
|---|---|---|---|
| vision (unfrozen) | 0.605 | 0.819 | 1.30 |
| vision + joints (unfrozen) | 0.614 | 0.859 | 1.21 |
Matched epoch (ep5/8, last-8-block LP-FT, both still climbing). The in-distribution joint benefit persists with a strong unfrozen encoder — small on contact (+0.01), clearer on direction (+0.04) and magnitude (−0.09 N) — consistent with the frozen A/B. Strong vision does not make joints redundant in-distribution, but (per the frozen A/B) the cross-lab shortcut liability remains.
Verdict
Video stays the force source. Joints are a double-edged contact-phase context: a small in-distribution / same-lab gain, a cross-lab liability unless joint-dropout is applied, and no standalone generalization power. The result cleanly matches the proprioception-shortcut literature (FACTR2 / momentum observers: force lives in torque, not angles; angles are autocorrelated with contact phase). For a cross-lab deployable model, video-only with optional joint-dropout-regularized fusion is the safe design.