Self-Supervised Learning for Video→Force

What we tried, what worked, and where the ceiling is

Ivan Domrachev

2026-06-01

The goal

video2force (v2f): predict contact force / contact-gate from RGB video of robot manipulation.

  • Pipeline: frozen V-JEPA 2.1 ViT-L → cached per-frame features → trainable head.
  • Three datasets: FMB (Franka peg-insertion), REASSEMBLE (Franka connector assembly), RH20T (4 robots, cross-embodiment).
  • Contact-gate F1 is the north-star metric; force direction + magnitude secondary.

Note

The frozen video encoder is a deliberate choice — it makes everything cheap. The question of this work: can self-supervised learning squeeze more out of it?

Background — how the pieces work

Legend

Colour code used in every diagram:

  • ❄️ blue = frozen weights · 🔥 green = trainable · grey = data/tensor · amber = loss.

How we use V-JEPA 2.1

  • Encoder stays frozen; we pay its forward pass once and cache the features.
  • A sliding 16-frame window is pooled to one 2048-d vector per frame (2 cameras × 1024).
  • Everything downstream trains on these cached vectors → minutes per run, not days.

Spatial mean-pool

  • The patch grid (H×W) is averaged per channel → spatial dims collapse, the 1024 channels are kept.
  • Output = one 1024-d vector per frame — what the TCN consumes.

JEPA vs JEA vs Generative

Three ways to self-supervise (after the I-JEPA taxonomy). We tried JEA and JEPA.

(a) Joint-Embedding — score two views’ embeddings. Our Class 3 (alignment).

(b) Generative — decode pixels ŷ. Not tried.

(c) Predictive (JEPA) — predict target latent ŝy given context + z. Our Class 1.

  • JEPA/JEA score in latent space, not pixels; collapse avoided by the EMA target (y-)encoder. V-JEPA is family (c).

What is a TCN?

Temporal Convolutional Network — stacked 1-D dilated convolutions over the frame sequence.

  • Dilation doubles per layer (1·2·4·8) → receptive field grows geometrically (here 16 steps from 4 layers).
  • We use it acausally (v2f is offline post-processing) → taps can also reach future frames.

The v2f baseline (what all SSL is compared against)

  • This frozen-V-JEPA + TCN is the baseline. Every SSL idea must beat this.

The SSL question

The central question

Frozen V-JEPA features cap performance (gate F1 ≈ 0.87 REASSEMBLE, 0.78 FMB, 0.71 RH20T). Can an SSL objective beat that ceiling — ideally without more labels?

We tried four classes of SSL:

  1. Cross-modal JEPA (predict force latent from video)
  2. Force-only SSL (learn a force embedding)
  3. Cross-modal JEA alignment (DINO-style video↔︎force)
  4. Inverse-dynamics / motion-constraint SSL

Spoiler: one narrow win (cross-robot), three negatives, and a clean law for when SSL helps.

Class 1 — Cross-modal JEPA ❌

Predict the force latent from video context (V-JEPA-2 style, EMA target + VICReg).

direction cos
supervised TCN 0.57
cross-modal JEPA 0.55
  • NEGATIVE — bounded by the same frozen-video information; never beats plain supervised.
  • First confirmation of the frozen-feature ceiling.

Class 2 — Force-only SSL ⚠️ (narrow win)

Learn a robot-agnostic force embedding from 125 Hz force alone (no video) — masked-latent JEPA.

contact-AUC SSL raw
in-distribution 0.99 0.94
cross-robot (LORO) 0.94 0.80
  • Real but narrow: helps cross-robot transfer only.
  • No in-distribution gain, no action-classification edge.

Class 3 — Cross-modal JEA alignment

DINO-style no-negative alignment: pull matched (video-window, force-window) embeddings together.

  • Scene-blind force teacher → forces video to drop appearance, keep contact structure (kills scene-shortcut).

Ran it on all three datasets with different OOD splits. The results define a clean law →

Class 3 — RH20T (cross-robot) ✅

Hold out an entire robot; eval on it.

RAW V-JEPA DINO-aligned
contact AUC, holdout flexiv 0.48 0.81
contact AUC, holdout kuka 0.58 0.81
contact AUC, holdout ur5 0.60 0.79
per-frame v2f dir, holdout kuka 0.13 0.30
  • Raw V-JEPA ≈ chance cross-robot → aligned robot-agnostic rep wins big, even with less capacity.
  • The one genuine SSL win.

Class 3 — REASSEMBLE + FMB ❌ (the catch)

Milder gaps: held-out objects (REASSEMBLE), single→multi-peg (FMB).

FMB multi-peg OOD raw+linear aligned+linear raw+TCN aligned+TCN
contact F1 0.53 0.54 0.59 0.54
direction cos 0.81 0.83 0.87 0.86
  • At a weak (linear) readout alignment looks like it wins (+0.03 dir)…
  • …but under a capacity-matched TCN readout the win reverses — raw beats aligned on every metric.
  • REASSEMBLE: same story (tie / raw wins). No real win where the gap is mild.

Class 3 — Two checks that flipped “wins” to ties

Fair readout

  • Linear-probe SSL gains were illusory.
  • Equal-capacity TCN readout reversed them on both REASSEMBLE and FMB.

Contact-only metrics

  • All-frame magnitude “−26% MAE win” was a no-contact-frame artifact.
  • On contact frames: correlation ≈ 0 for both.

Capacity sweep (FMB, hidden 256→1024): both raw and aligned saturate at 256; gap constant. → the deficit is representational (aligned 384-d loses info), not readout capacity.

The unified law (Class 3)

JEA cross-modal alignment helps only when the domain gap is so severe that raw frozen features fail to transfer.

dataset gap raw transfers? JEA
RH20T cross-robot no (≈chance) WIN
FMB single→multi-peg yes (0.59–0.87) no
REASSEMBLE cross-object yes (0.81) no

Mechanism: alignment trades information (2048→384-d compression) for invariance. Only pays off when the invariance is needed.

Class 4 — Inverse-dynamics SSL ❌ (shelved cheaply)

Idea: contact = motion-constraint residual (commanded action ≠ achieved motion). Bonus: pretext could LoRA-adapt the encoder label-free.

Grounded in prior work (V-JEPA-2-AC, VPT/DynaMo, generalized momentum observer, ICM, Lee et al.).

Step-0 go/no-go (proprioceptive, no video, no training — the cheapest possible test):

signal (FMB) AUC vs contact
commanded-vs-achieved residual 0.43 ❌
low achieved speed (proxy) 0.77
raw V-JEPA supervised contact 0.92
  • Residual is dead: human teleop slows at contact, so commanded motion drops with achieved → the gap never opens.
  • Best proxy (0.77) < supervised (0.92). Label-free detector underperforms what we already have.
  • Killed with one analysis script before building any pipeline.

Scoreboard

SSL class result
Cross-modal JEPA ❌ negative (frozen ceiling)
Force-only SSL ⚠️ narrow — cross-robot transfer only
JEA alignment ✅ RH20T cross-robot · ❌ FMB/REASSEMBLE
Inverse-dynamics ❌ shelved at go/no-go (teleop confound)

Frozen-video information is the binding ceiling — confirmed 5 independent ways. No frozen-feature SSL trick beats it.

Conclusion

  • SSL on frozen V-JEPA for v2f is exhausted.
  • It helps only under a severe embodiment gap (RH20T cross-robot) — a real but narrow niche.
  • There is no label-free SSL shortcut to break the ceiling.
  • The one proven lever left: supervised E2E encoder unfreeze (LoRA V-JEPA-L) — needs contact labels, not SSL.

Method lessons (keep these)

  1. Verify SSL / linear-probe gains against an equal-capacity supervised readout. Linear wins reverse — repeatedly.
  2. Evaluate force direction/magnitude on contact frames only. All-frame metrics are swamped by trivial no-contact frames.
  3. Run the cheapest mechanism go/no-go first (proprioceptive, no video, no training). It saved a multi-day build on Idea A.

Forward path

  • Primary: supervised E2E LoRA V-JEPA-L (the proven ceiling-breaker) — sinew-193 / 208.
  • Keep (narrow win): force-only SSL embedding + JEA alignment as a cross-robot transfer tool / zero-shot contact detector on unseen robots.
  • Parked: label-free contact via dynamics — would need non-teleop (autonomous/scripted) data where the residual mechanism survives.

Refs: docs/ssl_v2f_conclusion_2026-06-01.md, xmodal_align_*, fjepa_force_ssl_*, idm_contact_ssl_research.md.

Update 2026-06-02 — cross-modal force augment beats the baseline OOD

The reframe: don’t replace — augment

Earlier law: cross-modal alignment helps only severe gaps and costs in-distribution (it compresses 2048→384-d, trading information for invariance).

New idea: keep the raw features. Concatenate the force-aligned embedding onto them → SSL can only add information, never lose it. The invariance tax is paid on the added branch only.

Note

End goal restated: predict the 3-D contact force (direction + magnitude) + gate from video — not just a binary gate.

Setup — 4-camera REASSEMBLE, held-out-camera OOD

  • Re-extracted REASSEMBLE with 4 cameras (hand · hama1 · hama2 · capnode) → frozen V-JEPA (T,4,1024).
  • Train random-camera; OOD = hold one camera out entirely, test on it.
  • LOCO finding: random-cam interpolates seen-adjacent views, but hama1 extrapolation collapses (held-out F1 0.16) — a genuine severe OOD gap, the right place to test SSL.

Method — force as a view-invariant anchor

flowchart LR
  V[("video feats<br/>any of 4 cams")] --> G["force-align enc g ❄️<br/>(SSL-trained, frozen)"]
  V --> R[("raw 2048-d")]
  F[("force<br/>world frame")] -.->|InfoNCE / JEPA-regress| G
  G --> C[concat]
  R --> C
  C --> T["TCN force head 🔥"] --> O[gate / dir / mag]
  classDef frozen fill:#e8f0fe,stroke:#4285f4,color:#174ea6;
  classDef train fill:#e6f4ea,stroke:#34a853,color:#0d652d;
  classDef data fill:#f1f3f4,stroke:#9aa0a6,color:#3c4043;
  class G frozen; class T train; class V,R,F,O data;

  • Force is physical / camera-independent. Aligning video(any cam) → a shared latent ← force, over all 4 cams, forces the video encoder view-invariant (ImageBind-style binding).
  • Freeze g; augment: feature = [ raw 2048-d ‖ g(video) ] → same TCN force head.

Result — OOD held-out camera (hama1)

force model OOD F1 OOD dir in-dist F1 in-dist dir in-dist MAE
trivial (all-pos gate / const dir / mean mag) 0.461 0.473 0.461 0.473 2.17 N
V-JEPA + TCN (baseline) 0.280 ❌ 0.409 ❌ 0.857 0.802 2.11 N
+ cross-modal augment 0.626 ✅ 0.777 ✅ 0.864 0.860 1.82 N
cross-modal replace (256-d) 0.701 ✅ 0.772 ✅ 0.853 0.826 1.94 N
  • Baseline OOD is sub-trivial (F1 0.28 < 0.46, dir 0.41 < 0.47) — the held-out camera makes V-JEPA+TCN worse than constant guessing. SSL augment clears trivial on both (F1 0.70, dir 0.78).
  • In-distribution preserved & ≫ trivial (augment dir 0.80→0.86, MAE 2.11→1.82 N).
  • Not capacity: replace uses 256-d < baseline 2048-d yet wins OOD → it is the force-aligned embedding, not extra parameters.

How this fits — and refines — the law

  • Consistent: SSL wins the severe gap (unfamiliar camera where raw collapses to 0.28).
  • Refinement: augment removes the old in-distribution cost — raw info kept, invariance added. The first SSL config here with OOD gain and zero in-dist loss.
  • Caveat kept: direction cos is autocorrelation-inflated (persistence ≈ 0.99); the real signal is the relative OOD jump (baseline 0.41 ≪ persistence → genuine failure; SSL 0.78 → genuine fix).

Cross-dataset transfer — REASS ↔︎ FMB (2026-06-02)

Setup — fair, frame-correct, head-to-head

  • Force → world frame (REASS base; FMB R(tcp_quat wxyz)·tool_wrench), both +Z-dominant ⇒ direction comparable across datasets. (Frame-matching is necessary — prior cross-dataset direction was ≈0 / negative.)
  • Equal-param head-to-head: feature = [ raw 2048 ‖ g(cam0) ‖ g(cam1) ], dim 2560 for both arms; only g differs — random (baseline) vs force-aligned (SSL). Same MixStyle + feature-aug on both.
  • Direction is the transferable physical target; gate / magnitude are label-τ (2 N vs 8 N) and force-scale bound across datasets.

Result — direction cosine on contact frames

transfer (dir cos, contact frames) REASS→FMB FMB→REASS
trivial — constant mean direction 0.687 0.473
V-JEPA + TCN (baseline) 0.413 ❌ 0.396 ❌
SSL augment — strict zero-shot (g sees source only) 0.311 ❌ 0.095 ❌
SSL augment — UDA (g sees unlabeled target video) 0.865 ✅ 0.829 ✅
joint training (both datasets labeled) 0.96 0.87
(in-distribution, for reference) ≈0.95 ≈0.95
  • The baseline is below trivial — V-JEPA+TCN transfers an actively wrong learned direction (worse than predicting the constant mean). Strict-zero-shot SSL too.
  • Only SSL-UDA clears the trivial bar (+0.18 / +0.36) → cross-dataset direction goes below-trivial → above-trivial, the real win.
  • Needs unlabeled target video (UDA). Real, not autocorrelation: baseline ≪ persistence; UDA ≈ persistence.

Mechanism + the law, extended

  • Force = domain-invariant physical anchor. Aligning video→force over both domains’ (unlabeled) video forces the added branch to encode contact direction, not scene → it transfers.
  • Law (extended): cross-modal force-augment is a UDA lever — it pays off exactly when (a) the gap is severe and (b) unlabeled target data exists. No target access → no gain; full target labels → just train jointly (SSL then ties).
  • Gate / magnitude stay label-/scale-bound across datasets → joint training is their fix (reaches native: REASS F1 0.876 · FMB F1 0.769; dir 0.87 / 0.96).

Honest scorecard — where SSL wins

regime winner metric
OOD held-out camera SSL augment F1 0.28→0.70 · dir 0.41→0.78
cross-dataset direction, unlabeled target SSL augment (UDA) dir 0.40→0.85
cross-dataset, both labeled joint training native ceilings (SSL ties)
gate / magnitude across datasets joint training label/scale-bound

Takeaway: the augment + force-anchor reframe turns the old “SSL costs in-distribution, only narrow cross-robot win” into a clean win on both severe gaps we tested — viewpoint and cross-dataset direction — at zero in-distribution cost, whenever unlabeled target data is on hand.

What does the SSL latent space look like? (2026-06-02)

Why analyze the embedding, not just the task metric

We earned two OOD wins — but why? Probe the latent space of three per-frame encoders (each maps a V-JEPA cam-feature → unit vector):

  • RAW — frozen V-JEPA (1024-d), the baseline.
  • DINO-JEA — cross-view self-distillation over REASS’s 4 cameras (256-d).
  • FORCE-JEPA — video frame → force-aligned latent (256-d).

Three questions (refs): (Q1) diversity / collapse, (Q2) cross-view closeness, (Q3) grouping by camera / task / robot.

Important

Alignment without diversity is a lie. A collapsed encoder fakes perfect cross-view similarity. So Q2 is only trustworthy read jointly with Q1 — the alignment–uniformity plane is the headline.

The headline figure — alignment–uniformity plane

  • RAW (top-left): spread out, but view-variant (high cross-view misalignment) — it encodes which camera.
  • DINO-JEA (bottom-right): cross-view-perfect but collapsed (uniformity ≈ 0, singular values crash) → its invariance is the collapse-lie.
  • FORCE-JEPA (centre): low misalignment and good uniformity — view-invariant and diverse. The only encoder in the good region.

Numbers — diversity (Q1) + cross-view (Q2)

encoder dim eff-rank (RankMe) uniformity per-dim std cross-view cos
RAW V-JEPA 1024 396 −0.38 0.0085 0.895
DINO-JEA 256 33 ⚠ ≈0.00 ⚠ 0.0004 ⚠ 1.000*
FORCE-JEPA 256 61 −0.26 0.0154 0.962
  • DINO-JEA’s eff-rank 33 + per-dim-std 0.0004 + uniformity≈0 = partial collapse → its cross-view 1.000 (*) is not real invariance.
  • FORCE-JEPA: cross-view 0.962 backed by healthy diversity (eff-rank 61) → genuine view-invariance.

Numbers — grouping (Q3): kmeans-NMI / kNN-purity

encoder camera (want low) task (want high) robot (descriptive)
RAW V-JEPA 1.00 / 1.00 0.00 / 0.63 0.53 / 1.00
DINO-JEA 0.79 / 1.00 0.00 / 0.56 0.22 / 0.96
FORCE-JEPA 0.009 / 0.46 0.13 / 0.71 0.10 / 0.86
  • RAW is a pure camera encoder — camera NMI 1.0, task NMI 0.0. It clusters by viewpoint, not by what’s happening → exactly why it fails to transfer.
  • FORCE-JEPA erases camera identity (NMI 1.0 → 0.009) and is the only one with task structure (purity 0.71). CKA(raw, force)=0.014 → geometrically orthogonal to raw.

Visual — camera (t-SNE, REASS 4 cams)

RAW: four clean camera blobs. DINO-JEA: still camera-separated (collapsed). FORCE-JEPA: all four cameras fully intermixed — the latent forgot the viewpoint.

Visual — task (REASS) & robot (RH20T)

  • Task: FORCE-JEPA shows the cleanest insert/remove structure (purity 0.71 vs raw 0.63).
  • Robot: scored descriptively — robot-agnostic is preferred for v2f, so FORCE-JEPA’s weak robot clustering (NMI 0.10) is fine, not a defect.

Conclusion — the latent space explains the wins

  • RAW V-JEPA = camera encoder (camera NMI 1.0, task 0.0): the latent is dominated by viewpoint/scene → negative transfer.
  • DINO-JEA collapses (eff-rank 33): “view-invariant” only by losing information → why it was lossy in-distribution.
  • FORCE-JEPA = view-invariant + task-structured + uncollapsed (camera NMI 0.009, task 0.71, eff-rank 61, orthogonal to raw). The force anchor strips viewpoint while keeping contact structure — a direct mechanistic explanation of the OOD-camera (F1 0.28→0.70) and cross-dataset-direction (0.40→0.85) wins.