Video → Force

Recovering the missing contact modality from vision · feasibility synthesis

① Motivation ② Goals ③ Datasets ④ Approaches ⑤ Supervised ⑥ Self-supervised

agenda

Six questions, one thread

① MotivationVision data scales; force is missing.

② GoalsPredict contact · direction · magnitude from video.

③ DatasetsFMB · REASSEMBLE · RH20T + our own.

④ ApproachesSupervised vs self-supervised.

⑤ SupervisedIn-dist ceiling, all datasets.

⑥ SSLWhat generalizes OOD — force embeddings + v2f.

Thread: supervised wins the task in-domain; self-supervision wins cross-domain transfer.

① motivation

The missing modality

Data factories collect enormous visuomotor datasets — and vision will stay the dominant way manipulation data is gathered.
Almost all of it is vision-only — no haptics, no force/torque.
Yet contact force is what makes contact-rich tasks work: assembly, insertion, handling.
Idea: generate the missing force modality from the video itself.

Why touch matters for contact-rich manipulation.

② goals

What we set out to predict

Can a model read contact off a video?

Contact event — when something touches (gate).
Direction — which way the force pushes (unit vector).
Magnitude — how hard (Newtons).

Constraints

Inference is video-only (no proprioception at test time). Force is post-processing — acausal allowed. Frozen V-JEPA encoder throughout.

Three prediction targets, easy → hard.

③ datasets

The data is already there

dataset	task	cams	force
FMB	peg / board insertion	2–4 ext	wrist F/T, biased
REASSEMBLE	NIST board insert/remove	4	clean <1 N
RH20T	147 tasks · 4 robots	8–11	base-frame, mixed
ours (crisp_ws)	3 contact tasks	front + wrist	clean, sensor-frame

≈16.8k rollouts · ~178 h · ~6.4M frames — ≈ 14× Meta's haptic corpus. Public, abundant, mostly unused.

RH20T 89% · FMB · REASSEMBLE of the training corpus.

③ datasets · public

Three public sources

FMB — 3 boards · 54 pegs. Wrist F/T biased (no gravity-comp): contact clean, magnitude noisy. τ≈10 N.

REASSEMBLE — NIST board. Cleanest force (sub-Newton). Real direction signal → where dir-cosine shines.

RH20T — 89% of corpus · 4 robots · only multi-task source. No wrist-force frame; per-robot τ. The generalization stress-test.

③ datasets · ours

Our own dataset — 3 contact tasks, ~90–100 demos each

peg-insert — round peg-in-hole, axial insertion.

ethernet-insert — rectangular connector, fine alignment.

box-flip — lateral contact, per-frame 6-D force.

In-lab, clean bias-removed sensor-frame force (NetFT, server-published). Held-out validation set for OOD / cross-lab transfer.

③ datasets · F/T rate

Force/torque sampling — harmonized to 10 Hz

dataset	sensor rate	stored	frame	contact τ
FMB	10 Hz (observer est.)	10 Hz	EE	~10 N
REASSEMBLE	~500 Hz	10 Hz (LP-filt)	base→EE	2 N
RH20T	100 Hz	10 Hz (unfilt)	base / world	per-robot ~1.5–3 N
ours	250 Hz	10 Hz	sensor→TCP→world	3.65 N

Why 10 Hz

All forces resampled to 10 Hz to match the V-JEPA RGB frame rate. After downsampling, the residual walls are label quality (FMB bias) and per-robot τ heterogeneity (RH20T) — not sample rate.

④ approaches

Two ways to learn it

Supervised — force labels → TCN

Best in-domain: directly fits contact / direction / magnitude.
Scales poorly OOD: new lab / camera / robot → drops hard, direction can invert.
Needs force labels for every target domain.

Self-supervised — align video ↔ force latent

No task labels; learns a shared video–force representation.
Generalizes OOD much better: robot-/camera-agnostic, restores cross-dataset direction.
Ties — not beats — supervised in-domain.

Verdict preview: complementary — supervised owns the in-domain ceiling, SSL owns the transfer gap.

④ approaches · SSL families

Self-supervision — three families

Joint-Embedding — contrastive (CLIP/DINO).

Generative — reconstruct (MAE).

JEPA — predict the latent.

X = video (context), Y = force (target), Z = latent. We keep the frozen V-JEPA video encoder and learn a force / alignment head on top.

⑤ supervised

Frozen V-JEPA → dilated TCN → 3 heads

Encoder: V-JEPA 2.1 ViT-L/16@384, frozen, mean-pool → 1024-d/cam.
Trunk: dilated TCN, 7 blocks (dil 1→64), hid 384 — causal, cheap.
Heads: gate (BCE) · direction (cos) · magnitude (smooth-L1).
Multi-cam view-attention fuser.
Dilation 1·2·4·…·64 → a window sees 2⁶ past steps.

⑤ supervised · in-dist

In-distribution ceiling — contact & direction work

dataset	gate-F1	raw cos-dir (+lift)	mag MAE (N)
REASSEMBLE	0.86	0.91 (+0.34)	1.6
FMB	0.78	0.97 (+0.02)	2.8
RH20T	0.63	0.39 (+0.27)	5.4
ours · peg-insert	0.90	0.83 (+0.02)	2.4
ours · ethernet-insert	0.91	0.93 (+0.00)	2.4
ours · box-flip	0.91	0.97 (+0.08)	2.4

H1 (contact) ✓ across all. H2 (direction) ✓ where force is clean (REASSEMBLE, box-flip). FMB + peg/ethernet direction is near-trivial (axial / biased +Z); magnitude is the residual wall.

Predicted vs true direction — REASSEMBLE.

Direction-cos vs magnitude — FMB (per-shape).

⑤ supervised · FMB contact

Contact prediction on FMB

Predicted contact gate overlaid on an FMB insertion rollout — the model fires exactly when the peg meets the board.

Contact is the cleanest signal even when FMB force magnitude is biased.
Gate-F1 ≈ 0.78 at τ≈10 N (force-histogram valley).
Video-only inference — no proprioception.

⑤ supervised · OOD

Out-of-distribution — zero-shot transfer breaks

Direction inverts (raw cos, zero-shot → after K-shot FT): box-flip −0.74 → 0.97; peg −0.33 → 0.83; ethernet +0.39 → 0.93.
Front-cam contact collapses: 0.86 → 0.39–0.43 (viewpoint extrapolation).
Magnitude scale mismatch: MAE 1.6–2.8 → 4.8–7.6 N (per-dataset τ).
Adaptation recovers: K≈20–60 target demos + multi-cam → pretrained-OOD ≥ from-scratch in-domain.

OOD after few-shot target fine-tuning — contact & magnitude recovered.

⑤ supervised · ceiling

The encoder is not the lever

Frozen V-JEPA-L + dilated TCN + 3-head + mean-pool = best frozen recipe.
Bigger / different frozen backbones cap at the same ceiling (F1 ≈ 0.78 FMB).
Ceiling is label- / sensor-bound, not capacity-bound.
Direction is frame- & viewpoint-locked → multi-cam + frame-unification matter most.

Multiple encoders / encodings → same OOD ceiling.

⑥ self-supervised

What we trained — and is it collapsed?

S1cross-modal DINO align video↔force.

S2anti-collapse + intermix (VICReg-cov).

S3augment & predict (raw ‖ aligned).

S4high-quality trunk + force embed.

S5close video↔force gap.

Train: REASSEMBLE + FMB + RH20T (jointly). Eval: in-dist val + our held-out crisp_ws.

embedding	RankMe (eff. rank)	verdict
raw V-JEPA (video)	396	full
naïve DINO align	33	collapsed
Stage-1 joint	9.8	near 1-D
Stage-2 (ours)	139	fixed
force tower (ours)	51	active

RankMe — collapsed (before) → restored (after). Every alignment claim is paired with a collapse check.

⑥ ssl · t-SNE · camera & robot

The latent forgets the camera and the robot

REASSEMBLE by camera — raw V-JEPA = 4 clean view clusters; force-aligned = intermixed.

RH20T by robot — aligned embed is robot-agnostic (4 embodiments overlap).

Viewpoint and embodiment — the two things that break supervised OOD — are exactly what the SSL embed discards.

⑥ ssl · force embeddings

Force embeddings — binding video to force

Old — video & force on separate parallel cones (modality gap).

Recent · by modality — clouds interpenetrate (video↔force overlap).

Recent · by dataset — FMB/REASS/RH20T intermix in the shared space.

A trainable force tower (RankMe 51, active not collapsed) is pulled onto the frozen video latent. Naïve cosine is offset-blind; centering the common mode + covariance-matching converts a fake alignment into a real one. The force embedding is non-trivial — it carries contact structure beyond hand-crafted force features.

⑥ ssl · video2force results

Video→force with SSL — where it pays off

setting	raw V-JEPA	+ force-aligned embed
in-distribution contact / direction	best	ties (no lift)
held-out camera — contact F1	0.28	0.70
cross-dataset direction · REASS→FMB	0.41	0.87
cross-dataset direction · FMB→REASS	0.40	0.83

Augment, don't replace: concat[ raw V-JEPA ‖ force-aligned embed ] → TCN. Zero in-distribution cost; large OOD / cross-domain gains — unseen camera (F1 0.28→0.70) and cross-dataset direction (UDA, 0.40→0.87 both ways).

verdict

Supervised vs self-supervised — complementary

question	supervised (trunk + FT)	SSL embedding
in-dist contact / direction	best	ties raw V-JEPA
OOD task (our data, K-shot FT)	0.87–0.91 F1	no extra lift
unseen camera (zero target)	0.28	0.70
cross-dataset direction (UDA)	negative	restored 0.40→0.85
robot-agnostic intermix	—	yes

Supervised wins the task; SSL wins cross-domain transfer. The frozen-feature ceiling is the next lever — encoder unfreeze.

Takeaways

Force is recoverable from video — contact ✓ everywhere, direction ✓ where force is clean.
The data exists at scale (≈14× Meta's haptic corpus) and is mostly unused.
Supervised sets the in-domain ceiling; self-supervision closes the OOD gap — camera-/robot-agnostic, restores cross-lab direction.
Quality, honestly measured: RankMe (no collapse) + UMAP/t-SNE (true intermix).
Next: unfreeze the encoder — the frozen ceiling is the last wall.