The missing modality

• Data factories collect enormous visuomotor datasets.
• Almost all of it is vision-only — no haptic / force signal.
• Idea: generate the missing force modality from video — augment existing vision data for free.

vision → generate missing modality → augmented data → policy training

The idea — climb a hierarchy

• Contact — where is the contact? (gate)
• Direction — which way is the interaction?
• Force — how strong is it? (magnitude)

Across embodiments: manipulator → dexterous hand → gloves.

Goal of the feasibility check

Using existing datasets, try to predict contact · direction · magnitude — and ask:

• Are videos enough to extract interaction force?
• Which contact info is feasible to extract?
• Is there any room for generalization?

Self-supervised learning — three families

JEA — CLIP & DINO

CLIP — image–text contrastive alignment

DINO — teacher–student self-distillation

• Strong, transferable backbones.
• Limitations: representation collapse; limited prediction.

Generative & JEPA

MAE — reconstruct masked patches in pixel space

V-JEPA 2 — predict masked latent (EMA target)

• JEPA = best of both — latent prediction, versatile features. We build on V-JEPA.

Sparsh — SSL for touch

Sparsh: MAE / DINO / JEPA on 462k tactile images

DINO best for force & pose; JEPA for touch semantics

• Gap: ignores F/T data; tactile understanding, not generation.

Goal, refined

Three datasets with force

FMB boards & pegs

RH20T — 7 configs, 4 robots

Comparison

• 16 772 rollouts · 178 hours.
• Train each separately, then combine.

What we predict & how we measure

Frozen V-JEPA → TCN

• Encoder frozen; cache features once → small TCN head.

FMB — contact & magnitude work

FMB	F1	dir	lift	MAE
in-dist	0.779	0.965	+0.015	1.06
OOD	0.767	0.953	+0.042	1.06

• Contact feasible; magnitude surprisingly accurate.

REASSEMBLE — clean force, real direction

	F1	MAE	dir	lift
REASSEMBLE	0.874	0.74	0.906	+0.34
FMB	0.779	1.06	0.965	+0.015

• Better F/T data → better contact (+9.5% F1).

RH20T — generalization is hard

split	F1	dir	lift
in-dist	0.71	0.39	0.27
OOD task	0.68	0.60	0.33
OOD robot	0.60	0.26	0.17
OOD both	0.57	0.25	0.08

• OOD task ok; OOD robot poor.

What encoder & head?

Hypotheses, answered

• H1 — contact: feasible ✓.
• Magnitude: surprisingly good.
• H2 — direction: partial.
• H3 — generalization: partial → motivates SSL.

SSL of a force representation

• X masked force · Y true force · Z encoded video.
• Force as the self-supervisory anchor — physical, camera-independent.

Cross-modal force-augment

• Augment, don't replace: concat force-aligned embedding onto raw features.

Unseen camera: 0.28 → 0.70

• Baseline below trivial; SSL clears it, in-dist preserved.

Cross-dataset direction

• Needs world-frame force + unlabeled target video.

Why? The latent forgets the camera

	cam NMI	eff-rank
Raw	1.00	396
DINO	0.79	33
Force-JEPA	0.009	61

Force-JEPA kills camera identity without collapse.

What SSL bought us

• View-invariant, task-structured, uncollapsed force embedding.
• Wins severe gaps at zero in-dist cost.

Where each tool wins