v2f Stage-4 — High-Quality SSL Embeddings

epic sinew-276 · frozen V-JEPA 2.1 trunk · embedding-quality primary, downstream no-regress · 2026-06-10

Goal. Make one unified video+force SSL embedding that is (a) multi-cam-advantaged, (b) FMB-intermixed, (c) per-camera-close, (d) force-meaningful — measured by a new windowed scorecard (scripts/stage4_metrics.py), all on a frozen backbone (no encoder unfreeze).

Bottom line. 4/4 axes answered. Frozen SSL delivers global intermix + meaningful force embeds + a quantified multi-cam law + same-view cohesion. Two limits are intrinsic to the frozen regime (local-appearance kNN) and to force physics (+Z-entropy retrieval) — both need levers excluded here (encoder-unfreeze / higher-entropy data).

Scorecard figure

SSL representation — latent-space figures (champion s4_iter1.pt, in-dist VAL)

Same figure family as the Stage-1/Stage-2 reports, re-run on the new Stage-4 SSL embedding. (titles read "Stage-1" — the make_v2f_figs template — but all are computed on the Stage-4 champion.)

Dataset intermix — learned net.vid.embed (a) vs RAW frozen V-JEPA (c). Dataset silhouette 0.16 learned vs 0.49 raw → the SSL embedding pulls datasets together that the frozen features keep apart (axis-b, global). latent intermix umap

Per-camera closeness (axis-c, NEW) — every camera embedded, colored by VIEWPOINT type (wrist vs external, left) vs by DATASET (right). NMI_viewpoint 0.03 ≪ NMI_dataset 0.58 → embeddings cluster by dataset, not by physical viewpoint; frozen appearance dominates viewpoint identity. viewpoint umap

Video↔force joint space (axis-d) — shared proto space by modality (a) and dataset (b); same-clip video↔force cosine distance (c) is very tight (paired cos ≈ 0.998) yet instances are mutually indistinguishable → cluster- aligned, instance-retrieval floored (+Z low-entropy forces). joint video-force umap

Force-tower latent (axis-d) — net.frc embedding by source/robot and by contact state; the activated force tower organizes by contact + dataset (force embeds encode physics: probe AUC 0.98). force latent umap

Embedding effective rank — PR / RankMe on raw vs L2-normalized embed: not collapsed (genuine multi-dimensional structure, the high rankme is real not a 1-DoF artifact).

Same-fragment multi-encoding proximity — one clip's 4 REASS cameras + its force embed in a shared space (per-cam cohesion: same-clip cams sit close, xview_cos 0.998).

crisp_ws OOD video latent + domain gap — zero-shot crisp embeds by set / contact, overlaid on train in-dist (the OOD gap that target-FT closes, per Stage-3). ood latent

Four axes — verdict

axis	metric	vp base	champion (iter1)	verdict
(a) multi-cam	contact-AUC reass single→fuse	0.89→0.96	0.90→0.96	WIN — +0.07 on informative views; naive-fuse hurts heterogeneous rh20t
(b) FMB intermix	fmb_mmd_vs_reass (global)	1.254	0.206	WIN — global distributions pulled together
(b) FMB intermix	fmb_knn5 (local)	1.0	1.0	WALL — frozen appearance, all recipes
(c) per-cam close	xview_cos (same-ep)	0.986	0.998	WIN — same-view cohesion
(c) per-cam close	vp_minus_ds (cross-dataset view)	−0.730	−0.400	PARTIAL — still dataset>viewpoint
(d) force meaning	force_probe AUC / magR² / dirCos	.885/.46/.96	.977/.53/.98	WIN — force embeds encode physics
(d) v↔f align	vf_retrieval r1 (instance)	0.004	0.0006	WALL — +Z low-entropy contact forces
guardrail	in-dist gateF1-sum	2.23	2.22	preserved

What moved, what didn't

Frozen SSL achieved

Global FMB intermix: MMD(FMB,REASS) 1.25→0.21, dataset-silhouette 0.66→0.16, DANN/logreg linear-invariant.
Meaningful force embeds: force tower activated (was inert) → linprobe contact-AUC 0.98, dir-cos 0.98; rankme_f 51.
Multi-cam law: advantage is view-quality-dependent, not count — complementary informative views (reass wrist+externals) fuse to +0.07 AUC; rh20t's wild externals add ~0 and naive decision-fusion hurts.
Same-episode cross-view cohesion: xview_cos 0.998 (cross-view consistency loss).
Embedding rank expanded 10× (rankme_v 11.7→115, not collapsed); downstream preserved.

Frozen / physics walls (robust across instance · DANN · cross-view · raw · CORAL)

Local FMB kNN = 1.0: frozen V-JEPA window-level appearance fingerprint; FMB peg-insertion is a genuinely different visual domain. Only encoder-unfreeze (LoRA, sinew-253) can cross it.
Instance v↔f retrieval floored: contact forces are mostly +Z (low entropy) → contact embeds mutually indistinguishable (not a collapse — rankme high).
Cross-dataset same-viewpoint clustering not reached (frozen appearance > viewpoint identity).

Champion recipe & artifacts

ckpt ~/out_stage4/s4_iter1.pt

train_joint_stage1.py --dino-w 1 --align-mode instance --inst-w 1 --cross-view --cv-w 1 --rh20t-ncam 2 --vic-w .5 --vic-cov-w 1 --embed-detach

trunk-embed + instance cross-modal align (activates force tower) + cross-view consistency + VICReg-covariance (anti-collapse), embed decoupled from the supervised heads so intermix never costs downstream.

harness scripts/stage4_metrics.py log docs/v2f_stage4_iteration_log.md spec docs/superpowers/specs/2026-06-10-v2f-stage4-ssl-embeddings-design.md

Next lever

The two walls both point to the same escalation the frozen-only constraint excluded: encoder-unfreeze (V-JEPA LoRA, sinew-253) for local-appearance intermix, and higher-entropy contact data (non-+Z forces) for instance v↔f retrieval. FMB 4-cam re-decode (sinew-277, GCS drip ~20h) folds into the multi-cam & per-cam-closeness axes via --fmb4 when ready. Deployment note: the supervised gate head overfits OOD under long SSL — early-stop or re-fine-tune the head on target (Stage-3 result); the SSL embeddings are unaffected.

v2f Stage-4 · frozen-trunk SSL · all metrics on windowed projected embeddings (video-only at inference).