v2f Stage-5.6 — Aligning Force to a Frozen Visual Encoder

epic sinew-304 · bind a force-time-series student to frozen V-JEPA (video untouched) · 4 iterations, research→implement→train→robust-eval · 2026-06-19

Goal. Train a force encoder whose per-frame embedding aligns to the frozen V-JEPA video embedding as closely as possible. Success = the cross-modal SWAP test: a linear probe trained on frozen-visual embeddings, applied to the force student (contact-AUC, chance 0.5; direction-cosine, chance 0.0), contact-masked, episode-group-split.

Headline. Contact transfer is SOLVED to the teacher ceiling via a new Sinkhorn-OT binding (champion D2: contact-swap 0.831 ≈ within-vis ceiling 0.847) — this refutes the Stage-5.5 "swap fails" verdict, which was the old two-trainable-tower setup. Direction transfer is frozen-V-JEPA-bound: the frozen pooled feature only weakly encodes world-direction (within-vis ceiling 0.575 vs force's 0.90), and no objective / target / direction-head crosses ~0.35 — the lever is encoder-unfreeze. And latent distance does not predict transfer: the best arm has the largest modality gap.

Swap scorecard (robust, 3-seed mean, identical val windows)

armobjective × targetcontact-SWAPdir-SWAPverdict
D2 ★Sinkhorn-OT × raw1024 + dir-head0.831 ± .0350.301 ± .11champion — contact ≈ ceiling
D3regress × raw1024 + dir-head0.776 ± .010.319 ± .07close 2nd
B1Sinkhorn-OT × ZCA0.787 ± .060.105 ± .22ZCA hurts dir
A4InfoNCE × raw1024 (iter-1)0.726 ± .010.345 ± .06iter-1 baseline
D1Sinkhorn-OT × raw1024 (no dir-head)0.388 ± .04-0.133collapses w/o dir-head
teacher ceilings (within-vis linear probe): contact 0.847 · direction 0.575 · chance: contact 0.5, dir 0.0
0.00 0.25 0.50 0.75 1.00 contact teacher ceiling 0.847 dir teacher ceiling 0.575 contact chance 0.5 0.73 0.34 A4 infonce·raw1024 (iter-1) 0.79 0.10 B1 sinkhorn·zca 0.78 0.32 D3 regress·raw1024·dir 0.83 0.30 D2 sinkhorn·raw1024·dir ★ 0.39 -0.13 D1 sinkhorn·raw1024 (no dir) contact-swap direction-swap

Direction is bounded by the frozen encoder, not by alignment

The decisive, draw-independent number is dir-within-vis ≈ 0.575 (Ridge trained on the frozen video target, tested on the frozen video target — no cross-modal gap). That is the best any linear direction probe can read out of the frozen pooled V-JEPA feature. Force, by contrast, encodes direction at 0.90 within-modal. So cross-modal direction-swap is hard-capped at ~0.575 by the teacher, and sits at ~0.32–0.35 below even that. No objective (InfoNCE / Sinkhorn-OT / regression), target (frz256 / raw1024 / ZCA), or shared cross-modal direction-head moved it. Raising it requires the video encoder to actually encode world-direction → encoder-unfreeze (E2E/LoRA, sinew-193/253). ZCA whitening additionally lowered the dir ceiling (0.575→0.49) by discarding dir-carrying variance.

Latent distance ≠ transfer

The intuitive objective — "make the force and video embeddings close" — is the wrong one. Sinkhorn-OT keeps the two clouds geometrically far apart (centroid gap_l2 ≈ 87 on ZCA / 39 on raw1024, mean vectors orthogonal: cos_means ≈ 0, MMD ≈ 1.0, CKA ≈ 0.15) — yet it has the best contact-swap. Regression / InfoNCE pull the clouds together (gap 12–45, mean vectors nearly parallel: cos_means ≈ 0.99, CKA ≈ 0.32) and transfer worse. OT aligns the per-instance discriminative subspace the swap probe reads, while leaving global geometry separate. Minimizing modality distance optimizes cloud overlap, not the per-sample map.
0.00 0.25 0.50 0.75 1.00 0.40 0.50 0.60 0.70 0.80 cos(mean_force, mean_video) — 0 = orthogonal clouds, 1 = parallel contact-swap AUC OT: far apart, best transfer regress/InfoNCE: close, worse A4 (0.73) B1 (0.79) D3 (0.78) D2 (0.83)

The four iterations

iterhypothesisoutcome
1(recover never-computed baseline) does binding move the swap?contact-swap 0.32→0.80 via InfoNCE binding (vs no-bind); direction stuck ~0.35
2ZCA-whitened target + Sinkhorn-OT + regression objectivesSinkhorn-OT new-best contact (0.82); direction uncracked; ZCA hurt dir
3shared cross-modal direction head closes the dir alignment gapno clean dir lift; exposed a noise-dominated dir-swap eval (fixed: sample-once + 3-seed)
4best-of-both: Sinkhorn-OT × raw1024 (best dir target) + dir-headchampion D2 (contact 0.831 ≈ ceiling); dir-head also stabilizes raw1024 OT
Full log: docs/v2f_stage56_iteration_log.md · code: branch stage56-xmodal-align-2026-06-19 (sinew_video2force)

Verdict & next

Contact cross-modal transfer is done — Sinkhorn-OT-to-frozen-video is a genuine new capability (the old two-tower setup could not bind it). Direction is a quantified frozen ceiling (within-vis 0.575): the banked stop-rule is reached, and the only remaining lever is unfreezing the video encoder so it can encode world-direction (sinew-193 / sinew-253). New code shipped: --vanchor-obj {sinkhorn,regress}, --vanchor-tgt zca, --dir-xmodal-w, plus a fair (sample-once, multi-seed) swap eval.

v2f · Stage-5.6 · 2026-06-19