v2f Stage-5.5 — Do the SSL Embeddings Earn Their Keep?

epic sinew-292 · validation ablation on the Stage-5 champion (frozen V-JEPA) · best-to-best, in-dist + OOD · 2026-06-12

Why. Stage-5 produced a video↔force embedding that is geometrically aligned (CKA 0.07→0.51). This stage asks the hard question: does that embedding actually improve downstream performance, or is the alignment cosmetic? Five ablations, all best-to-best vs honest baselines (raw V-JEPA, raw force, trivial predictors).

Headline. The Stage-5 (raw-embed) SSL embedding does not earn its keep downstream. It is a redundant projection of pooled raw V-JEPA (R²=0.96), never beats raw for force-prediction or behavior-cloning, and its alignment does not transfer task semantics across modalities. Three nuances from post-review re-runs (A5/A5b/A6): (1) v2f does generalize OOD under the proper recipe — A1's OOD collapse was a probe artifact; (2) a trunk-embed (Stage-4) preserves temporal structure that the raw-embed lacks and gives a real (task-split) OOD augment win — the prior good results were genuine; (3) the force embedding is non-trivial and can drive a policy. But no embed adds information beyond the frozen encoder — the consistent lever is encoder-unfreeze, not a raw-derived SSL projection.

Scorecard — five questions

#	question	result	verdict
A1	SSL embed vs raw V-JEPA for v2f (in/OOD)	in-dist: S edges gate-F1 (+0.04) but loses direction (0.33→0.13). OOD: A1's apparent collapse was a probe artifact (pooled-window + zero-shot + in-dist threshold) — see A6.	NO — raw ≥ SSL (in-dist)
A5	does the SSL embed add NEW info? (redundancy + concat)	champion embed = proj(pooled-raw): MLP R²(S from raw)=0.963 ⇒ ~0 new info; concat[raw‖S] never meaningfully beats raw	NO — redundant
A5b	same check on Stage-1 / Stage-4 trunk-embeds	Stage-4 trunk-embed R² 0.854 (less redundant — temporal) ⇒ box_flip OOD augment win; Stage-1 (0.990) & Stage-5 raw (0.963) degenerate	trunk-embed adds temporal (Stage-4), raw-embed doesn't
A6	FAIR v2f OOD (full trunk + multi-cam + target-FT)	OOD is strong (peg/eth/box gateF1 0.87–0.90, dirCos 0.76–0.94 — reproduces Stage-3); SSL-augment ties BASE (no OOD help)	v2f OOD strong; SSL-augment neutral
A2b	task ID from force embed vs raw force	RH20T 7-way all 0.14–0.27 (chance 0.14) ≈ contact-only control; raw ≥ embed	NO — contact-dominated, not semantic
A2c	cross-modal swap (train video → test force)	7-way swap at chance; insert/release swap 0.54 vs within-modal 0.75	NO — alignment geometric, not task-semantic
A3a	SSL embed vs raw V-JEPA for BC (ACT)	raw MSE 5.98e-5 > force 7.74e-5 > SSL 8.76e-5; all beat trivial	NO — raw ≥ SSL
A3b	BC policy preserved under video→force swap	force beats trivial baselines, ties SSL video embed, ~30% worse than raw video	PARTIAL — force can drive a policy
A4	is the force embed trivial?	per-dim R² mean 0.45 (0% dims>0.9, 65%<0.5); 52% of variance unexplained by 181 hand-crafted features; contact-AUC 0.97	NO — force embed is non-trivial

A1 — SSL embed vs raw V-JEPA for v2f

Window-level v2f (where the embed lives; the deployed per-frame path detaches it). Raw V-JEPA (R, 1024-d) vs SSL embed (S, 256-d) vs concat (C). S loses direction and collapses OOD; C ≈ R — the augment effect (sinew-228) does not reproduce for v2f.

split	rep	gateF1	gateAUC	dirCos
in-dist	raw	0.63	0.90	0.33
in-dist	SSL	0.68	0.89	0.13
OOD	raw	0.63	0.62	0.44
OOD	SSL	0.00*	0.42	0.06

*in-dist-tuned threshold mis-calibrated to crisp's positive rate; read AUC for separability.

A2 — task identity from force + the cross-modal swap

Left: 7-way RH20T action classification — raw force / force-embed / video / contact-only, in-dist + leave-one-robot-out. Everything hugs chance + the contact control → the force signal is contact-presence, not task semantics. Right: the train-modality × test-modality swap matrices — off-diagonals at (7-way) or barely above (binary) chance → the aligned latent does not carry task labels across modalities. A2 task classification + cross-modal swap matrices

A2 task classification + cross-modal swap matrices

A3 — behavior cloning (ACT) on FMB delta-pose

Minimal ACT-style chunk regressor, history-as-context. raw V-JEPA > force > SSL embed, all well above trivial baselines. Notably the force-driven policy ties the SSL video embed — a policy can be driven from force alone (3b). A3 BC action-MSE + predicted-vs-GT traces

A3 BC action-MSE + predicted-vs-GT traces

A4 — is the force embed trivial?

Per-dim R² of the 256-d force embed regressed on 181 hand-crafted force features (left), and the top CCA correlations (right). A handful of canonical directions capture the obvious |F|/contact axis (CCA top-5 ≈ 0.99), but 52% of embed variance and 65% of dims are not linearly explainable by even a comprehensive statistic bank → the force tower learned non-trivial structure (it cleanly encodes contact, AUC 0.97, plus real residual). A4 force-embed triviality: per-dim R2 + CCA

A4 force-embed triviality: per-dim R2 + CCA

Do the embeddings add new information? (A5 / A5b / A6 — added after review)

A reviewer flagged that A1's OOD numbers contradicted the prior stages' OOD wins. Re-checking exposed two things:

A1's OOD "collapse" was a recipe artifact. A1 pooled the window (bypassing the temporal trunk), went zero-shot, and used an in-dist-tuned threshold. A6 re-ran v2f OOD with the matched Stage-3 recipe (full trunk + multi-cam + per-domain threshold + few-shot target-FT) → OOD is strong (peg/eth/box gate-F1 0.87–0.90, dir-cos 0.76–0.94), reproducing the Stage-3 wins. v2f generalizes; A1 just measured a weaker path.
Two senses of "new info". Beyond ALL of raw V-JEPA: none — every embed (Stage-1/4/5) is a deterministic function of the frozen encoder's output; it invents no signal V-JEPA didn't extract (frozen ceiling). Beyond POOLED raw: only the trunk's temporal structure, if not pooled away — this is where prior gains live.

encoder	embed_src	R²(S from pooled-raw)	concat helps v2f?
Stage-1 (s1_viewinv)	trunk	0.990	no (temporal pooled away)
Stage-4 (s4_iter1)	trunk	0.854	box_flip OOD win (ΔAUC +0.29); peg/eth tie
Stage-5 champion	raw	0.963	no — degenerate redundant

Reconciliation. The prior augment success is reproduced only by Stage-4's trunk-embed (R² 0.854 ⇒ it preserved window-dynamics that mean_t(raw) discards ⇒ box_flip OOD win) — confirming the mechanism and that the prior results were real. Stage-5's embed_src=raw (R² 0.963) is the degenerate redundant case where concat is dead-weight. Two wrinkles: a trunk is not automatically non-redundant (Stage-1, also trunk, re-pooled to R² 0.990 and fails like Stage-5), and even Stage-4's win is task-split, not uniform. (OOD: read gate-AUC, not gate-F1 — the F1 zeros on peg/ethernet are a compressed-sigmoid threshold artifact.)

What this means

The frozen SSL embedding is alignment-specialized, not a richer state. Optimized for video↔force geometry, the 256-d projection discards direction/action information that raw 1024-d V-JEPA retains — so it loses on v2f and BC, and collapses OOD.
The Stage-5 alignment is geometric, not semantic. CKA 0.51 buys cloud overlap but not task-label transfer across modalities (A2c ≈ chance). The shared structure is coarse (contact / magnitude), consistent with the +Z low-entropy force.
The force tower is real, though. Non-trivial (A4) and able to drive a policy (A3b) — its content is genuine, just contact-dominated rather than task-discriminative.
The lever is unchanged. Every ablation points the same way: gains require raw V-JEPA / supervised heads / encoder-unfreeze, not the frozen SSL embed. This is the empirical close on the "SSL-for-v2f is frozen-bound" thesis — the next real investment is unfreezing the video encoder.

v2f Stage-5.5 · validation ablation · frozen champion s5_champion.pt · all probes best-to-best vs honest baselines.