Video2Force

SSL vs Supervised — a Stage 1–5 synthesis

① SSL approaches we tried ② Train / test datasets ③ Validation dataset ④ Supervised — architecture & in/out-dist

⑤ SSL latent — video↔force · cameras · datasets ⑥ Other target tasks — BC · force-swap · task-ID ⑦ SSL vs Supervised — who wins, why

Ivan Domrachev · sinew · 2026 · not self-contained · → chapters · ↓ detail

Setup

The task — climb a contact hierarchy from video

multi-cam video → frozen V-JEPA 2.1 → mean-pool → trainable TCN → contact · direction · magnitude

Contact (gate) → Direction → Magnitude
Encoder frozen through Stages 1–5; only the trunk / SSL heads train.
Inference is video-only.

① Approaches

Self-supervised learning — three families

Joint-Embedding — contrastive (CLIP/DINO)

Generative — reconstruct (MAE)

JEPA — predict the latent

X = video (context), Y = force (target), Z = latent, D(·,·) = energy. We keep the frozen V-JEPA video encoder and learn a force/alignment head on top.

① Approaches

What we actually tried — five stages

Stage 1cross-modal DINO: align video↔force on a shared prototype head (EMA teacher, no negatives).

Stage 2anti-collapse + intermix: VICReg-covariance + force-anchor → kill 1-DoF collapse, mix datasets.

Stage 3augment & predict: concat aligned-trunk ‖ raw V-JEPA → TCN with force labels.

Stage 4high-quality trunk-embed: multi-cam advantage, per-camera closeness, global intermix.

Stage 5close the modality gap: cosine + Deep CORAL bind on raw-V-JEPA embed.

One spine throughout: a frozen video encoder + a trainable head whose target is force (or force-aligned). Diagrams ↓.

① Approaches · diagrams

Stage 2 — cross-modal DINO alignment

Idea

Force teacher is scene-blind → pulls a robot-/camera-agnostic video embedding.
Shared weight-norm prototypes (K=1024), EMA teacher, centering+sharpening.
VICReg covariance term added in Stage 2 to stop collapse.

① Approaches · diagrams

Stage 3 — augment & predict (the only labelled SSL use)

concat[ raw V-JEPA ‖ force-aligned SSL embed ] → TCN 3-head. Augment, not replace.

② Data

Train / test corpus — three datasets, ~6.4M frames

RH20T 5.70M (89%) · FMB 0.39M · REASSEMBLE 0.29M ≈ 14× Sparsh

dataset	task	cams	force	role
FMB	peg/board insertion	2–4 ext	wrist F/T, biased	train+test
REASSEMBLE	NIST board insert/remove	4	clean <1 N, 100 Hz	train+test
RH20T	147 tasks · 4 robots	8–11	base-frame, mixed	train+test

≈16.8k rollouts · ~178 h. Force harmonized to a common world/EE frame; per-source normalization. No wrist-force frame in RH20T; in-hand cam tcp dead-zeroed.

② Data · FMB

FMB — Functional Manipulation Benchmark

3 boards · 54 pegs · single- & multi-object insertion.
Multi-camera (4 external); 2 cached on server.
Wrist F/T — force is biased (no gravity-comp) → magnitude noisy, contact clean.
Contact τ ≈ 8–10 N (valley of the force histogram).

② Data · REASSEMBLE

REASSEMBLE — NIST task board

Insert / remove on a NIST board; 4 cameras.
Cleanest force — sub-Newton, 100 Hz (down-sampled to 10).
Real direction signal → the dataset where dir-cosine shines.
Cross-dataset label overlap: insert / release.

② Data · RH20T

RH20T — diverse, multi-robot (the scale + the mess)

rollout on Panda — the held-out OOD robot

7 configs · 4 robots (flexiv/kuka/ur5/panda) · mixed F/T sensors

89% of the corpus · sole multi-task source (7-way) · no wrist-force frame · per-robot τ. The generalization stress-test.

③ Validation

Validation = crisp_ws — a held-out OOD workspace

EASY

peg-insert · ethernet-insert

real — 4-camera workspace

axial +Z push, low-entropy direction
contact onset is sharp & repeatable

MEDIUM

box-flip

LeRobot wrist feed — "Flip Box"

lateral, rotating contact → direction stress
off-axis vs the insertion prior

Why this set

Different lab, robot, cameras → a true OOD check.
Clean EE-frame force (lab-invariant direction).
Real + Isaac-sim twin (same 4 views).
In-domain ceiling (train-on-crisp): gate-F1 ≈ 0.90 every cell → gap is domain shift, not a feature ceiling.

④ Supervised

Supervised feasibility

architecture → in-distribution → out-of-distribution

④ Supervised · arch

Frozen V-JEPA → dilated TCN → 3 heads

dilation 1·2·4·8·… → a window sees 2⁶ past steps; causal, cheap, frozen-feature input

Recipe (frozen ablation winner)

Encoder: V-JEPA 2.1 ViT-L/16@384, frozen, spatial mean-pool → 1024-d / cam.
Trunk: dilated TCN, 7 blocks (dil 1→64), hid 384.
Heads: gate (BCE) · direction (cos) · magnitude (smooth-L1).
Stage 3 adds a multi-cam view-attention fuser.

④ Supervised · in-dist

In-distribution — contact & direction work

REASSEMBLE — direction cos 0.93 vs trivial 0.57 (lift +0.36)

dataset	gate-F1	dir-cos (lift)
REASSEMBLE	0.874	0.93 (+0.34)
FMB	0.78	biased
RH20T	0.55–0.71	frame-dependent

H1 (contact) ✓ across all. H2 (direction) ✓ where force is clean (REASSEMBLE). Magnitude is the residual wall.

④ Supervised · OOD

Out-of-distribution — pretrained-OOD ≥ in-domain

A/B: pretrained→OOD-FT (K=60) matches from-scratch in-domain on F1 & MAE

adaptation ladder — multi-cam reaches the in-domain ceiling with few target demos

crisp_ws (K=20 FT, multi-cam)		gate-F1	AUC	dir-cos
peg-insert	EASY	0.874	0.867	0.765
ethernet-insert	EASY	0.888	0.914	0.828
box-flip	MEDIUM	0.867	0.959	0.935

④ Supervised · ablation

The encoder is not the lever

What we swept

Encoder: V-JEPA-L vs ViG vs E2E ViT-L
Head: TCN vs MLP vs 3-head
Pooling: mean vs attentive
Frames: world vs EE-canonical

What we learned

Frozen V-JEPA-L + dilated TCN + 3-head + mean-pool = best frozen recipe.
Bigger / different frozen backbones cap at the same ceiling (F1 ≈ 0.78 FMB).
Ceiling is label- / sensor-bound, not capacity-bound.
Direction is frame- & viewpoint-locked → multi-cam + frame-unification matter most.

⑤ SSL latent

The SSL latent space

video↔force · across cameras · across datasets

⑤ Latent · video↔force

Video and force, brought into one space

joint video+force on a shared prototype space — by modality (a), by dataset (b), paired V↔F cos-dist histogram (mean 0.16)

Closing the gap

iter	CKA	centered cos
Stage 4 (naïve)	0.07	0.014
mid	0.43	0.37
Stage 5 champ	0.51	0.51

Plain cosine is offset-blind (a “modality gap”) → cosine + Deep CORAL covariance-match converts a common-mode collapse into genuine alignment.

⑤ Latent · video↔force

…but a residual modality gap remains

joint [Zv;Zf] by modality — video (blue) and force (red) still occupy distinct regions

…by dataset — the force region intermixes across datasets; video arcs stay separated

⑤ Latent · camera

The latent forgets the camera

REASSEMBLE by camera — raw V-JEPA = 4 clean clusters (NMI 1.0); force-aligned = intermixed (NMI 0.009)

Raw V-JEPA is camera-separable — viewpoint dominates appearance.
The force-aligned embed erases camera ID while keeping contact structure.
Stage 4: per-viewpoint NMI 0.03 (view-invariant) vs dataset NMI 0.58 (kept).

⑤ Latent · dataset / robot

Video keeps appearance · force is robot-agnostic

video Zv by dataset — separate manifolds

force Zf by dataset — intermixed

RH20T by robot — the force-aligned embed is robot-agnostic (NMI 0.10)

Force is dataset-/robot-agnostic but sensor-signature-bound (residual-kNN ~0.96); video retains lab/appearance on frozen features.

⑥ Other target tasks

Beyond video→force

what else we asked the representation to do

⑥ Other tasks

Three downstream probes on the same features

probe	question	input → output	best representation
Behaviour cloning	can the visual encoder drive a policy?	embed → action chunk (ACT)	raw V-JEPA
Replace vision w/ force	can force stand in for vision?	train on video embed → test on force embed	partial (contact only)
Task-type prediction	is the embed task-semantic?	embed → task-ID (RH20T 7-way · insert/release)	≈ chance from force

Same frozen V-JEPA / SSL / force embeddings, three new heads. The recurring answer: raw frozen V-JEPA ≥ any SSL re-encoding, and force is a real-but-narrow (contact) signal.

⑥ Probe · behaviour cloning

BC with visual encoders — raw V-JEPA wins

FMB delta-pose action-chunk MSE (ACT head) — per-dimension breakdown

Result

encoder → action	MSE ↓
raw V-JEPA	5.98e-5
force encoder	7.74e-5
SSL embed	8.76e-5
trivial	—

Why raw wins: frozen V-JEPA feats are already action-decodable; the SSL embed re-encodes and loses info.
Force can drive a policy (beats trivial, ties the SSL video embed) — but ~30% worse than raw.

⑥ Probe · force-swap & task-ID

Replace vision with force? Only for contact.

cross-modal swap confusion matrices (train V/F × test V/F) + RH20T 7-way task-ID

Replace vision w/ force (train head on video embed → test on force): insert/release swap 0.54 vs within-modal 0.75 → force partially substitutes for contact, not task.
Task-type prediction: RH20T 7-way from the force embed ≈ chance (0.14–0.27, chance 0.14) — same as a contact-only control.
Why: force is contact-dominated, not task-semantic; the cross-modal alignment is geometric, so task structure does not transfer across modalities.

⑦ SSL vs Supervised

Who wins, and why

⑦ vs · scoreboard

Where each tool wins

question	Supervised (trunk + FT)	SSL embedding
in-dist contact / direction	best	ties raw V-JEPA
OOD task (crisp_ws, K-shot FT)	0.87–0.89 F1	no extra lift
unseen camera (zero target)	0.28	0.70
cross-dataset direction (UDA)	negative	restored (0.40→0.85)
robot-agnostic intermix	—	yes (NMI 0.10)
new info beyond frozen V-JEPA	none — every embed is a deterministic function of V-JEPA

Supervised wins the task; SSL wins cross-domain transfer. Complementary, not competing.

⑦ vs · the ceiling

Why — the frozen-feature ceiling

frozen-trunk SSL scorecard — multi-cam AUC ✓ · global intermix MMD 1.25→0.21 ✓ · but FMB-kNN frozen-appearance wall

Every SSL embed is a deterministic function of the frozen encoder → no new information (confirmed 3 ways: pooled-align, augment, temporal).
Alignment bought geometry (CKA 0.51), not shared task structure (the swap ≈ chance — §⑥).
Force is real but sensor-signature-bound; contact is the one shared axis (AUC ~0.97).
The only step-change left = unfreeze the encoder (LoRA). Current work: Stage 5.6 binds force → frozen V-JEPA to test exactly this boundary.

Conclusions

Synthesis

What works

Contact from video — yes, every dataset (H1 ✓).
Direction — yes where force is clean (H2 ✓).
OOD — supervised pretrain + few-shot FT reaches the in-domain ceiling.
SSL — earns its keep on cross-domain transfer (camera / robot / dataset invariance).
Downstream probes — raw V-JEPA drives BC best; force can substitute for vision only on contact, not task.

What's bounded

Magnitude — residual wall.
Frozen-feature ceiling — SSL adds no info beyond V-JEPA.
Geometric alignment ≠ semantic sharing.
Next lever: encoder unfreeze (LoRA) + force→frozen-V binding (Stage 5.6).

Detail per stage in the hosted reports: /report/v2f-stage1 … stage5 · stage5-findings.md. Appendix ↓.

Appendix

extra plots · latent diagnostics · SSL alternatives

Appendix · collapse check

Aligned and diverse — not collapsed

singular-value spectrum + alignment-uniformity plane

embed	eff-rank
raw V-JEPA	396
DINO-JEA (naïve)	33 — collapse
Force-aligned (ours)	61

“Alignment without diversity is a lie” → every invariance claim paired with a collapse check. Stage 2 fixed PR 1.5→49.

Appendix · rollouts

Supervised rollouts — best / worst

best in-distribution

worst OOD zero-shot — direction collapses to ≈ −1 (frame/viewpoint lock)

Appendix · OOD latent

OOD domain gap in the latent

crisp_ws OOD video latent — box-flip sits apart from train; clear domain shift

Train-on-crisp (in-domain) hits gate-F1 ≈ 0.90 on every cell, including the front-cam that collapses zero-shot → the features carry the info; the failure is domain shift, addressable by target adaptation.

Appendix · Stage 2

Collapse, cured

before (collapsed) → after (champion): silhouette 0.66→0.06

effective-dim fix: PR 1.5→49.4, RankMe 10→139

Appendix · force embed

Is the force embedding trivial? No.

non-trivial (agg R² 0.48, no dim R²>0.9) yet strongly correlated with hand-crafted force features

52% of variance unexplained by 181 hand-crafted features.
Contact-presence AUC ~0.97 — the dominant axis.
But it does not intermix across sensors on frozen features.

Appendix · alternatives

SSL alternatives we ruled out

tried	outcome
vision-only JEPA	ruled out — label-bound F1
Force-JEPA / F-SSL (force-only)	narrow cross-robot win; no in-dist edge
squared-L2 / VICReg-MSE bind	collapses (rank→2)
MixStyle / Fahim x-uniformity	hurts alignment
temporal binding (per-frame)	non-redundant but no v2f gain
x-modal force augment (concat)	OOD camera 0.28→0.70 · cross-dataset dir restored

Full record: /report/v2f-stage5-findings.md.