Video2Force — Stage-2 report

TL;DR

Goal: the Stage-1 latent was ill-trained — it collapsed (~1.5 effective dims) and kept the three training datasets in separate clusters. Stage-2 = make it healthy: collapse-free, datasets intermixed, no downstream cost.

Collapse decisively solved; global dataset-separation collapsed too. The pooled video embedding went from a near-1-D line to a genuinely high-rank manifold, and the datasets went from cleanly separated to broadly overlapping — at zero cost to the supervised task. The strict local intermix (kNN) only partly moved: that residual is appearance-bound on the frozen encoder (see the wall).

latent health (in-dist val, video embed)	BEFORE (Stage-1)	AFTER (Stage-2 P)	goal
RankMe (effective rank)	9.8	138.9	↑↑
participation ratio	1.52	49.4	↑↑
PC1 variance	79%	8%	↓↓
dataset silhouette	0.661	0.062	→0
dataset kNN-5 acc	1.000	0.932	→.41
in-dist task (F1 sum)	5.539	5.527	preserved

Collapse + global intermix solved; downstream preserved. Local kNN is the residual appearance floor.

The disease — quantified on the Stage-1 champion

Diagnosed on s1_iter2_vp (729 in-dist + 114 crisp episodes):

pathology	metric	value	meaning
collapse	RankMe / PR / PC1	9.8 / 1.5 / 79%	~1–2 effective dims of 256 (persists on L2-norm → genuine, not a sphere artifact)
datasets separated	kNN / silhouette	1.000 / 0.659	the model trivially knows which dataset a clip is from — zero intermix
retrieval floored	video→force R@1	0.003	paired cosine is high but a clip can't find its own force → 1-D cone, no instance discrimination

Root cause (code-confirmed): the Stage-1 anti-collapse term was a variance hinge only — it pins each dimension's std≈1 but never decorrelates them, so all dims carry the same signal and effective rank stays ~1.5. The surviving 1–2 axes encode the nuisance factors (dataset-ID + contact), which is exactly why datasets separate and retrieval floors.

The fix — 7 iterations, two clean wins

1 · Collapse → add decorrelation

Adding the missing covariance off-diagonal term to VICReg (decorrelate, not just rescale) lifts RankMe 9.8→100+ immediately, with downstream untouched. Barlow Twins helped less (batch-limited); MixStyle killed rank. Covariance decorrelation is the collapse cure.

2 · Intermix → the right tool (after the wrong ones)

Domain-adversarial DANN and CORAL/moment-matching both failed — DANN destroyed the cross-modal alignment and diverged; CORAL left kNN at 1.0. A history check confirmed why: classification domain-adaptation distorts feature scale and hurts regression (force is a regression target). The working lever is the opposite — bind the embedding to physics:

raw-feat embed — project the pooled embedding straight from the frozen V-JEPA features, bypassing the supervised trunk (whose features are baked dataset-specific by the per-dataset heads).
force-anchor — align that embedding (cosine) to a fixed, dataset-agnostic force target [dir ‖ log(1+|F|)]: force direction+magnitude is the same physical quantity in every lab, so pulling video onto it makes the embedding encode what force is happening, not which lab.

Clean ablations: raw-feats beat trunk (kNN .89 vs .98), force-anchor beats cluster-DINO (1.0). This is the same mechanism that previously drove camera-NMI 1.0→0.009.

Champion s2_armP: --embed-src raw --align-mode anchor --vic-cov-w 0.5 --rh20t-ncam 2 --bs 48 (frozen V-JEPA throughout).

Before / After

**Headline · video latent, BEFORE vs AFTER (t-SNE).** Top (Stage-1): three cleanly separated dataset blobs — silhouette 0.66, kNN 1.00. Bottom (Stage-2 P): the datasets broadly interleave — silhouette **0.062**. (t-SNE preserves *local* neighborhoods, so the residual micro-clusters that keep kNN at 0.93 are still visible — the appearance floor.)

**Collapse fixed · effective dimensionality.** BEFORE (red) reaches 95% of its variance in 2 PCs (a near-1-D line); AFTER (blue) climbs slowly, only ~41% by PC15 — variance now spread across 68 dims. RankMe 9.8→138.9, PR 1.5→49.4, PC1 79%→8%.

Plots — task quality preserved

The latent fix is decoupled from the supervised heads, so task quality is unchanged. Examples:

**5 best in-dist.** Sharp contact onsets, gate snaps 0→1, direction locks — across all three datasets.

**5 worst in-dist.** RH20T-dominated; failures where the wrench hovers near per-robot τ. Magnitude/τ is the residual task wall, unchanged from Stage-1.

**crisp OOD (zero-shot).** Held-out lab, no adaptation — insertion gate transfers partially, box-flip direction does not (the Stage-1 OOD story is intact; the Stage-2 latent fix neither helps nor hurts zero-shot).

The wall — why local kNN floors at ~0.93

Every intermix method (gentle stat-matching, adversarial DANN, instance-anchor, even decoupling the embed from the trunk) drives global separation down (silhouette 0.66→0.06) and makes the embedding linearly domain-unpredictable — but the local kNN-5 purity sticks near 0.9. The reason is a genuine tension, not an unsolved hyperparameter:

Rank ⊥ local-intermix on frozen features. On frozen V-JEPA the only cross-dataset-invariant signal is the low-dimensional force-physics (~4-D direction+magnitude). A high-rank embedding must therefore retain dataset-specific appearance dimensions to fill the other ~60 dims — and those appearance dims locally cluster by dataset. You can have high rank or full local intermix, not both, until the encoder is allowed to learn richer dataset-invariant features. (Retrieval is separately data-limited: contact forces are mostly +Z, so forces are mutually near-indistinguishable.)

Key findings

Collapse was a missing decorrelation term — variance-hinge-only VICReg. The covariance term fixes it decisively (RankMe 9.8→139, PR 1.5→49), free, no downstream cost.
Classification domain-adaptation (DANN/CORAL) hurts regression — confirmed live (DANN killed the cross-modal alignment, moment-matching diverged). The right intermixer is physics-anchoring, not adversarial deletion.
Global dataset intermix is achievable on frozen feats (silhouette 0.66→0.06) via raw-feat embed + fixed force-anchor — the dataset-agnostic-physics mechanism.
Local intermix + instance retrieval are frozen-encoder-bound — the rank⊥intermix tension above; the next lever is the encoder, not another loss.

Stage-3 — open

E2E LoRA encoder unfreeze (sinew-253) — lets V-JEPA learn dataset-invariant features, the only way past the local-kNN floor and the in-dist ceiling. The single biggest remaining lever.
Learned force tower in the shared space — the champion aligns video to a fixed anchor (force tower inert); co-embedding a learned force tower would restore a genuine modality-retrieval axis (where the data allows it).
Magnitude / τ head (sinew-254) — the residual task wall, unchanged since Stage-1.

Full trace: docs/v2f_stage2_iteration_log.md · research docs/v2f_stage2_latent_research.md · champion ~/out_stage2/s2_armP.pt.