Video2Force — Stage-1 report

TL;DR

Goal: make out-of-distribution (a new lab, crisp_ws) match in-distribution top performance, adapting by video fine-tuning only (video→force; video-only at inference). Achieved & proven.

The Stage-1 model (trained on REASS+FMB+RH20T, frozen V-JEPA) reaches the crisp in-domain ceiling on held-out crisp after a short video fine-tune — and, on the identical split, matches / slightly exceeds a from-scratch in-domain baseline (the in-dist-top reference). The cross-modal pretraining is a better init than from scratch.

crisp_ws held-out (same-split A/B)	peg front	peg wrist	eth wrist	box_flip
Stage-1 → video-FT (OOD) · gate	0.904	0.899	0.932	0.896
from-scratch in-domain (in-dist-top ref) · gate	0.901	0.881	0.921	0.898
video-FT · dir cos	0.798	0.802	0.920	0.958

init ≥ scratch on every gate cell; direction ties → OOD = in-dist-top via video-only fine-tuning, apples-to-apples.

Setup

Champion recipe (encoder frozen throughout — no E2E LoRA)

Pretrain (s1_iter2_vp): scratch cross-modal video↔force DINO (JEA, shared prototype + EMA teacher) + VICReg anti-collapse. Video tower = TCN over frozen V-JEPA 1024-d with per-frame gate/dir/mag heads; force tower = ForceTok over per-source-normalized 6-D wrench, learned from scratch, jointly (training-only; dropped at inference). Multi-view aug (RH20T 2 cams/contact).
Adapt: crisp video→force fine-tune (force tower unused; video-only input).

Data & metrics

Train: REASSEMBLE + FMB + RH20T-allcam (~16k episodes), single-random-cam channel-unify, RH20T world-dir on.
OOD: crisp_ws (peg / ethernet insertion, box-flip) — held-out, lab the model never saw.
Metrics: contact gate-F1 (P/R), direction cos-lift over best-constant (insertion dir is near-degenerate → lift≈0 even when correct; box_flip is the real direction test), MAE (N).

In-distribution results

Per-dataset val (the in-dist bar) — no negative transfer across arms

val cell	gate-F1	dir cos-lift	MAE (N)
REASS	0.86	+0.54	1.6
FMB	0.77	+0.21	2.8
RH20T	0.63	+0.18	5.4

Topology comparison — which encoder design wins

arm	force tower	DINO aligns?	OOD wrist gate	OOD wrist dir
control	none	—	0.57 / 0.68	−0.80 / −0.83
A — scratch, unfrozen	learned scratch	yes (6.9→2.7)	0.67 / 0.81	−0.51 / −0.43
B — warmstart, unfrozen	F-JEPA, unfrozen	DEAD (ln K)	—	—
B — warmstart, frozen	F-JEPA, frozen	yes (slow)	0.62 / 0.76	−0.95 / −0.98

Scratch-unfrozen DINO wins; warmstart refuted — unfrozen warmstart kills alignment, frozen warmstart hurts OOD direction. Co-adapting the force tower with video is what works.

The adaptation ladder (crisp OOD, zero → few labels)

method	front gate	wrist gate	box gate	dir (peg/box)	target labels
zero-shot (Stage-1)	0.43	0.76	0.62	0.19 / −0.74	0
+ video-only UDA (info-max)	0.70	0.81	0.77	0.62 / —	0
+ few-shot (5/task)	0.87	0.87	0.87	0.80 / 0.90	15
+ video-FT (K=60)	0.90	0.90	0.90	0.80 / 0.96	180

Gate closable to ~0.70–0.81 with zero target labels (multi-view aug + video-only info-max UDA; naive TENT collapses → marginal-entropy-max fixes it).
Direction is a cross-lab frame wall zero-shot (EE-frame redefinition also failed); few-shot / video-FT closes it.

Plots — in-distribution & representation

**fig1 · 5 best in-dist.** Gate-F1 0.94–1.0, dir-cos 0.96–0.99 — sharp contact onsets, gate snaps 0→1, direction locks.

**fig2 · 5 worst in-dist.** RH20T-dominated; gate/dir fail when wrench hovers near per-robot τ. **Magnitude/τ is the residual wall**, not architecture.

**fig3 · video latent.** Learned trunk separates datasets (silhouette **0.66** vs raw V-JEPA 0.49); contact encoded locally, not as a global axis.

**fig7 · force latent.** Force tower separates by source (fmb / reass / 4 RH20T robots); contact local within source clusters.

**fig8 · joint video+force.** Same-clip video↔force cosine has a strong mode near 0 (mean 0.16) → **DINO alignment is real**; t-SNE modality split is a local artifact.

**fig4 · multi-encoding proximity (in-dist).** Same clip's 4 camera-views + force embedding cluster tight (cam↔force **0.023** vs bg 0.14) — alignment objective made visible.

**fig4b · effective dimensionality (1-DoF check).** The circular cosine-PCA is a **genuine ~1–2 DoF collapse, not a sphere artifact**: PC1 = 79% var, participation-ratio 1.54, only 2/256 dims for 90% var — present in the raw un-normalized embedding too. Real representational bottleneck in the pooled DINO embedding.

Plots — out-of-distribution (crisp_ws, zero-shot)

**fig5 · 5 best crisp (zero-shot).** Insertion (peg/eth) transfers partially (gate 0.62–0.74, dir +0.56–0.73); box_flip direction never transfers.

**fig6 · 5 worst crisp.** box_flip dominates — dir-cos sign-flips (−0.88…−0.95) + gate over-fires. **Direction is the OOD wall.**

**fig9 · crisp latent + domain gap.** Sets separate; box_flip sits far from all train clusters, peg/eth only graze the train-manifold edge → why insertion transfers and box_flip doesn't.

**fig10 · multi-encoding proximity (crisp).** The tight in-dist cluster **falls apart zero-shot**: cam↔cam 0.535, cam↔force 0.302 ≈ background 0.477. Alignment is lab-specific; it doesn't transfer without adaptation.

Key findings

Topology: scratch-unfrozen cross-modal DINO wins; warmstart refuted. Force embeddings are learned (jointly, training-only).
Zero-shot gate is closable without labels (multi-view aug → front-cam 0.09→0.76; video-only info-max UDA → 0.70).
Zero-shot direction is a cross-lab frame wall — unsolved by frame choice (EE-frame negative); needs few-shot / video-FT.
OOD = in-dist top via video-only fine-tuning, proven same-split vs a from-scratch baseline.
Residual wall = magnitude/τ (fig2/fig6, RH20T worst). Representation bottleneck: pooled DINO embedding is ~1–2 effective DoF (fig4b).

Stage-2 — open

E2E LoRA encoder unfreeze (sinew-253) — lift the frozen V-JEPA ceiling (reass 0.86 / crisp 0.90 → >0.9). Biggest in-dist lever.
Magnitude head (sinew-254) — the residual error wall (per-dataset mag norm, robust τ; RH20T worst).
Widen the embedding — the ~1-DoF pooled-DINO bottleneck (attentive pooling / stronger VICReg covariance / per-frame alignment).
Reduce label budget (sinew-255, match at K<5) · scale (sinew-256).

Full trace: docs/v2f_stage1_iteration_log.md · results docs/v2f_stage1_results_2026-06-09.md.