The missing modality

• Data factories collect enormous visuomotor datasets.
• Almost all of it is vision-only — no haptic / force.
• Idea: generate the missing force modality from video — augment vision data for free.

vision → generate missing modality → augmented data → policy training

The idea — climb a hierarchy

• Contact — where is the contact? (gate)
• Direction — which way is the interaction?
• Force — how strong is it? (magnitude)

Across embodiments: manipulator → dexterous hand → gloves.

Goal of the feasibility check

Self-supervised learning — three families

The shared recipe

• X — context signal (for us: video)
• Y — target signal (for us: force)
• Z — latent variable that conditions prediction
• D(·,·) — energy: low for compatible pairs

(a) Joint-Embedding (b) Generative (c) JEPA

(a) Joint-Embedding — CLIP & DINO

match s_x, s_y

CLIP — image↔text contrastive

DINO — teacher↔student self-distillation

Strong transferable backbones — but representation collapse & limited prediction.

(b) Generative — MAE

decode ŷ from s_x + z

MAE — reconstruct masked patches in pixel space

Pixel-level objective → reproduces noise, weaker inductive bias for downstream tasks.

(c) JEPA — predict the latent

predict s_y from s_x + z, in latent space

V-JEPA 2 — masked latent prediction, EMA target

Best of both: latent prediction → versatile features, no pixel noise. We build on V-JEPA.

Sparsh — SSL for touch (closest to our work)

Sparsh-MAE / DINO / JEPA on 462k tactile images (unpaired, mixed sensors)

• Goal: a sensor-invariant tactile backbone.
• Eval: force, slip, pose, grasp stability, texture, maze.
• Sparsh-DINO best for physics (force, pose); IJEPA best for touch semantics.
• Gap for us: ignores F/T data; tactile understanding, not generation.

per-task radar: E2E vs Sparsh variants

Goal, refined

Plan: find datasets → train supervised baselines → derive SSL approaches.

FMB — Functional Manipulation Benchmark

REASSEMBLE — NIST task board

RH20T — diverse, multi-robot

real RH20T rollout — Panda (our held-out OOD robot)

7 configs · 4 robots · mixed F/T sensors

Features

• 147 tasks · 4 robots · ~25M frames
• 100 Hz · ~30 s median episode · substages

Issues

• Too big — hard to load & work with
• Mediocre quality — F/T bias, no wrist camera

Comparison

What we predict & how we measure

cos-lift matters — a high cosine can be trivial when force barely changes direction.

What is a TCN?

Temporal Convolutional Network — stacked 1-D dilated convolutions over the frame sequence.

• Dilation doubles per layer (1·2·4·8) → receptive field grows geometrically.
• Used acausally (offline post-processing) → taps reach future frames too.

Frozen V-JEPA → TCN

FMB — contact & magnitude work

true vs predicted ‖F‖ (top) and direction cosine vs trivial baseline (bottom)

FMB	F1	dir	lift	MAE
in-dist	0.779	0.965	+0.015	1.06
OOD (9th peg)	0.767	0.953	+0.042	1.06

• Contact feasible; errors ≈ data noise.
• Magnitude surprisingly accurate.
• Direction high cosine but near-trivial lift.

REASSEMBLE — clean force, real direction

direction supremacy: model cos 0.928 vs trivial 0.572 (lift +0.356)

	F1	MAE	dir	lift
REASSEMBLE	0.874	0.74	0.906	+0.34
FMB	0.779	1.06	0.965	+0.015

• Better F/T data → better contact (+9.5% F1).
• Direction lift +0.34 — genuinely non-trivial.

RH20T — generalization is hard

split	F1	MAE	dir	lift
in-dist	0.71	1.86	0.39	0.27
OOD task	0.68	2.23	0.60	0.33
OOD robot	0.60	1.96	0.26	0.17
OOD both	0.57	1.55	0.25	0.08

• OOD task moderate; OOD robot poor.
• Multi-robot generalization possible but messy.

OOD-task: dir ok, mag under-scaled

OOD-robot (Panda): misses spikes

What encoder & head?

Hypotheses, answered

SSL of a force representation

masked-prediction SSL (JEPA-style)

X visible force tokens · Y masked force latents · Z mask tokens (target positions, no video)

• Use force as the self-supervisory anchor — physical, camera-independent.
• Two routes: force-as-image vs force time-series.
• Force-only SSL — no video in this stage; video enters at alignment (next slide).

Cross-modal DINO alignment

Augment & predict

What else we tried

DINO-style camera↔camera was the natural baseline — it collapsed. Force as the anchor is what worked.

Unseen camera: 0.28 → 0.70

Cross-dataset direction — lift restored

The latent forgets the camera

…but keeps the task

…and becomes robot-agnostic

View-invariant and diverse — not collapsed

The trap: a collapsed encoder fakes invariance. So read cross-view closeness with diversity.

	cross-view	eff-rank
Raw	view-variant	396
DINO	invariant*	33*
Force-JEPA	invariant	61

*DINO collapses (uniformity≈0) — fake invariance. Force-JEPA sits in the good corner; CKA vs raw ≈ 0.01 (orthogonal).

What SSL bought us

• A force-anchored embedding is view-invariant, task-structured, uncollapsed.
• Wins the severe gaps — unseen camera & cross-dataset direction — at zero in-dist cost.
• Pays off when the gap is severe and unlabeled target video exists.

Where each tool wins

Where we go from here

Transfer on our terms · scale to joint training

These are the directions the next chapter's funding & startup plan has to resource.

Funding & startup tracks

Does the “N-consecutive” contact rule help F1?

No contact-F1 gain from the debounce on any dataset → keep N=1. Each model peaks on its own eval-N, but the matched diagonal is flat — the rule only changes what counts as a contact, not how well it’s predicted.