v2f Stage-5 — Closing the Video↔Force Modality Gap
epic sinew-285 · frozen V-JEPA 2.1 trunk · embedding-quality primary, deployment no-regress · 2026-06-12
The problem. Stage-4's "aligned" embedding was a mirage: it reported paired-cosine 0.998, yet video and force
sat in separate regions of latent space — a classic modality gap. The instance-alignment loss was
cosine, which is blind to a constant offset between the two clouds, so the optimizer stacked all vectors on a shared
common-mode (centered paired-cos collapsed to 0.014) while the clouds never overlapped.
The fix. Make the alignment offset-aware and prevent the trivial collapse. The winning recipe: cosine-instance binding + CORAL covariance-matching (turns the common-mode collapse into genuine alignment) on a raw-V-JEPA embedding, with a hard pull (inst_w 6) and a force-side CORAL. Result: a real, measurable alignment — not a cosine artifact.
Bottom line. Goal-1 (video↔force distance) — the whole point of the project — is decisively fixed: CKA 0.07→0.51, centered-cos 0.014→0.51, MMD 1.30→0.11. Goals 2 (force intermix) and 3 (visual intermix) hit frozen-feature ceilings documented below; both need the excluded encoder-unfreeze lever.
The fix. Make the alignment offset-aware and prevent the trivial collapse. The winning recipe: cosine-instance binding + CORAL covariance-matching (turns the common-mode collapse into genuine alignment) on a raw-V-JEPA embedding, with a hard pull (inst_w 6) and a force-side CORAL. Result: a real, measurable alignment — not a cosine artifact.
Bottom line. Goal-1 (video↔force distance) — the whole point of the project — is decisively fixed: CKA 0.07→0.51, centered-cos 0.014→0.51, MMD 1.30→0.11. Goals 2 (force intermix) and 3 (visual intermix) hit frozen-feature ceilings documented below; both need the excluded encoder-unfreeze lever.
The gap, before vs after
Champion video↔force joint latent (real euclidean UMAP). Left: colored by MODALITY — the two clouds now
interpenetrate (CKA 0.51) instead of sitting on parallel cones. Right: the same points by DATASET.
Gap-closing across iterations. The Stage-4 champion (cosine only) was a common-mode collapse (centered-cos 0.014);
adding CORAL + a hard raw-embed pull lifts genuine alignment to centered-cos 0.51 / CKA 0.51.
Does video→force still work? (deployment, video-only)
Prediction examples on held-out val. Champion's per-frame gate / magnitude / direction vs ground truth — the
deployed v2f heads are untouched by the SSL geometry work (embed-detach), gate-F1-sum preserved at 2.14.
Four-goal scorecard
| goal | metric | s4_iter1 (old) | champion | verdict |
|---|---|---|---|---|
| 1 — video↔force dist | xmodal CKA | 0.07 | 0.51 | WON |
| centered paired-cos | 0.014 | 0.51 | WON — genuine, not common-mode | |
| xmodal MMD | 1.30 | 0.11 | WON | |
| instance retrieval r1 | ~0 | ~0 | physics ceiling (+Z low-entropy force) | |
| 2 — force intermix | force resid-kNN (→chance 0.42) | 0.92 | 0.97 | NOT achieved — sensor-signature, frozen |
| 3 — visual intermix | vid silhouette (↓ better) | 0.04 | 0.33 | TRADEOFF — see below |
| guardrail | deployed gate-F1-sum | 2.23 | 2.14 | preserved |
Per-camera / per-dataset embedding structure
Video embed by dataset (raw-embed champion). Real-UMAP; raw-embed retains V-JEPA appearance →
datasets stay locally separable (the goal-3 cost of goal-1).
Force embed by dataset. Force tower organizes by physics + dataset; the per-dataset separation is a
sensor signature (resid-kNN 0.97), not a force-regime difference — and is not removable on frozen features.
What we learned (the honest map)
Goal-1 — solved.
- Cosine was the bug. It is invariant to a constant offset → Stage-4's paired-cos 0.998 with separate clouds. The diagnostic (centered paired-cos 0.014, CKA 0.07) proved the "alignment" was a common-mode collapse.
- CORAL is the cure. Covariance-matching converts the cosine collapse into real alignment (CKA 0.43→0.51, centered-cos 0.37→0.51). A harder pull (inst_w 6) maxes it; a force-side CORAL stabilizes the target.
- squared-L2 / VICReg-MSE failed — it collapses (magnitude → 0, rank → ~2) regardless of variance weight; the asymmetric stop-grad variant avoids collapse but under-aligns (CKA 0.15). Fahim cross-modal uniformity also collapsed it.
- Force intermix (goal-2): force_resid-kNN stuck at ~0.96 across every lever incl. force-CORAL — the per-dataset force separation is a sensor signature, not force-regime, and isn't removable while the encoder is frozen.
- Visual intermix (goal-3): a hard tradeoff — the raw-embed required for goal-1 alignment keeps V-JEPA appearance (silhouette 0.33), while a trunk-embed intermixes datasets (silhouette ~0) but kills the cross-modal alignment (CKA → 0.06). You cannot have both on frozen V-JEPA.
- Instance retrieval is floored (~0) everywhere — contact forces are mostly +Z (low entropy) → windows are mutually indistinguishable. A physics ceiling, not a representation failure.
Champion recipe & artifacts
ckpt
cosine-instance binding + CORAL (anti-common-mode + covariance align) on a raw-V-JEPA embedding, hard pull, force-side CORAL; embed decoupled from the supervised heads so the geometry work never regresses deployed v2f.
harness
~/out_stage5/s5_champion.pt (= iter6M1)train_joint_stage1.py --align-mode instance --inst-w 6 --coral-w 1 --force-coral-w 1 --embed-src raw --align-mw mag
--embed-detach --cross-view --cv-w 1 --rh20t-ncam 2 --vic-w 0.5 --vic-cov-w 1 --bs 32
cosine-instance binding + CORAL (anti-common-mode + covariance align) on a raw-V-JEPA embedding, hard pull, force-side CORAL; embed decoupled from the supervised heads so the geometry work never regresses deployed v2f.
harness
scripts/stage5_diag.py (windowed scorecard: modality_gap / centered-paired-cos /
xmodal MMD+CKA / force resid-kNN / v2f-decode; real euclidean UMAP via isolated env)
log docs/v2f_stage5_iteration_log.md
spec docs/superpowers/specs/2026-06-12-v2f-stage5-xmodal-align-design.md
v2f Stage-5 · frozen-trunk SSL · all metrics on windowed projected embeddings (video-only at inference).