Video2Force
Feasibility check — predicting contact force from video
Ivan Domrachev · sinew · 2026 · → chapters · ↓ detail
The missing modality
- Data factories collect enormous visuomotor datasets.
- Almost all of it is vision-only — no haptic / force.
- Idea: generate the missing force modality from video — augment vision data for free.
The idea — climb a hierarchy
- Contact — where is the contact? (gate)
- Direction — which way is the interaction?
- Force — how strong is it? (magnitude)
Across embodiments: manipulator → dexterous hand → gloves.
Goal of the feasibility check
Using existing datasets, predict
- contact · direction · magnitude
…and ask
- Are videos enough to extract force?
- Which info is feasible to extract?
- Is there room for generalization?
Hypotheses
- H1 — video detects contact / no-contact.
- H2 — video gives an approximate direction.
- H3 — large-scale data → generalization to unseen robots / scenes.
Self-supervised learning — three families
The shared recipe
- X — context signal (for us: video)
- Y — target signal (for us: force)
- Z — latent variable that conditions prediction
- D(·,·) — energy: low for compatible pairs
(a) Joint-Embedding (b) Generative (c) JEPA
(a) Joint-Embedding — CLIP & DINO
Strong transferable backbones — but representation collapse & limited prediction.
(b) Generative — MAE
Pixel-level objective → reproduces noise, weaker inductive bias for downstream tasks.
(c) JEPA — predict the latent
Best of both: latent prediction → versatile features, no pixel noise. We build on V-JEPA.
Sparsh — SSL for touch (closest to our work)
- Goal: a sensor-invariant tactile backbone.
- Eval: force, slip, pose, grasp stability, texture, maze.
- Sparsh-DINO best for physics (force, pose); IJEPA best for touch semantics.
- Gap for us: ignores F/T data; tactile understanding, not generation.
Goal, refined
Step 1 — Supervised
Pre-trained visual encoders + paired video–force data → predict force.Step 2 — Self-supervised
Learn a force representation given vision as context (no force labels at test).Plan: find datasets → train supervised baselines → derive SSL approaches.
FMB — Functional Manipulation Benchmark
Features
- 4 cameras (2 external + 2 wrist)
- Panda F/T, 10 Hz · ~385k frames, ~11 h
- Substage division provided
Issues
- Biased force — poorly compensated
- Low variety — one task, one robot, one env
REASSEMBLE — NIST task board
Features
- Same board, 17 object insertions
- Separate F/T sensor, 100 Hz, <1 N error — very clean
- Event cameras, microphones, RGB; substages
Issues
- Pick & place show zero force → dropped
- One robot, one env
RH20T — diverse, multi-robot
Features
- 147 tasks · 4 robots · ~25M frames
- 100 Hz · ~30 s median episode · substages
Issues
- Too big — hard to load & work with
- Mediocre quality — F/T bias, no wrist camera
Comparison
| Dataset | Task | Samples | F/T quality | Variety |
|---|---|---|---|---|
| FMB | Peg insertion | 1 844 | Poor | Poor |
| REASSEMBLE | Object insert / remove | 2 262 | Perfect | Mediocre |
| RH20T | Diverse manipulation | 12 666 | Mediocre | Huge |
16 772 rollouts · 178 hours of high-freq F/T (after heavy preprocessing).
Plan: train on each separately, then combine for joint training.
What we predict & how we measure
Predict — 3 heads
- Contact event (bool)
- Force unit direction (3-D)
- Force magnitude (scalar)
Metrics
- Contact — F1
- Direction — cosine + cos-lift (gain over trivial mean dir)
- Magnitude — MAE (N)
cos-lift matters — a high cosine can be trivial when force barely changes direction.
Supervised feasibility
method → results → ablations → conclusions
What is a TCN?
Temporal Convolutional Network — stacked 1-D dilated convolutions over the frame sequence.
- Dilation doubles per layer (1·2·4·8) → receptive field grows geometrically.
- Used acausally (offline post-processing) → taps reach future frames too.
Frozen V-JEPA → TCN
Encoder frozen → cache features once → train a small TCN in minutes.
Three heads predict contact · direction · magnitude together.
FMB — contact & magnitude work
| FMB | F1 | dir | lift | MAE |
|---|---|---|---|---|
| in-dist | 0.779 | 0.965 | +0.015 | 1.06 |
| OOD (9th peg) | 0.767 | 0.953 | +0.042 | 1.06 |
- Contact feasible; errors ≈ data noise.
- Magnitude surprisingly accurate.
- Direction high cosine but near-trivial lift.
REASSEMBLE — clean force, real direction
| F1 | MAE | dir | lift | |
|---|---|---|---|---|
| REASSEMBLE | 0.874 | 0.74 | 0.906 | +0.34 |
| FMB | 0.779 | 1.06 | 0.965 | +0.015 |
- Better F/T data → better contact (+9.5% F1).
- Direction lift +0.34 — genuinely non-trivial.
RH20T — generalization is hard
| split | F1 | MAE | dir | lift |
|---|---|---|---|---|
| in-dist | 0.71 | 1.86 | 0.39 | 0.27 |
| OOD task | 0.68 | 2.23 | 0.60 | 0.33 |
| OOD robot | 0.60 | 1.96 | 0.26 | 0.17 |
| OOD both | 0.57 | 1.55 | 0.25 | 0.08 |
- OOD task moderate; OOD robot poor.
- Multi-robot generalization possible but messy.
What encoder & head?
Encoder
| model | F1 | dir |
|---|---|---|
| DINOv2-B (image) | 0.52 | — |
| V-JEPA ViT-B (video) | 0.64 | — |
| V-JEPA ViT-L | 0.76 | 0.969 |
| V-JEPA ViT-g (1B) | 0.764 | 0.974 |
| ViT-L E2E (LoRA) | 0.779 | 0.965 |
Head
| head | F1 |
|---|---|
| TCN | 0.768 |
| BiGRU | 0.723 |
| Transformer | 0.689 |
Predict Mag+Dir+Contact together (0.51→0.78). Mean pooling ≈ attentive → keep mean.
Hypotheses, answered
- H1 — contact: feasible ✓ (REASSEMBLE F1 0.874).
- Magnitude: surprisingly good — catches trends.
- H2 — direction: partial — real lift on clean data, trivial on FMB.
- H3 — generalization: partial — OOD-task ok, OOD-robot poor.
Open problem: cross-domain transfer (camera, robot, dataset). → motivates SSL.
Self-supervised force learning
idea → method → results → conclusion
SSL of a force representation
- Use force as the self-supervisory anchor — physical, camera-independent.
- Two routes: force-as-image vs force time-series.
Cross-modal force-augment
Align video → force (view-invariant anchor) → encoder forgets the camera.
Augment, don't replace: concat onto raw features → can only add.
What else we tried
| approach | idea | outcome |
|---|---|---|
| Cross-view DINO | align camera ↔ camera (video only) | collapses (eff-rank 33) — fake invariance |
| InfoNCE force-align | contrastive video ↔ force | loss floors (force autocorrelation) |
| Replace features | force-aligned embedding alone | OOD win, but lossy in-distribution |
| Augment (ours) | concat force-aligned onto raw | OOD win · zero in-dist cost |
| Joint training | train on both datasets (labeled) | reaches native ceilings; SSL ties |
DINO-style camera↔camera was the natural baseline — it collapsed. Force as the anchor is what worked.
Unseen camera: 0.28 → 0.70
| held-out camera | contact F1 | direction |
|---|---|---|
| trivial (constant) | 0.46 | 0.47 |
| V-JEPA + TCN | 0.28 | 0.41 |
| + force-augment | 0.70 | 0.78 |
- Baseline is below trivial — negative transfer.
- SSL clears trivial; in-distribution preserved.
Cross-dataset direction — lift restored
| direction cosine | REASS→FMB | FMB→REASS |
|---|---|---|
| trivial (constant) | 0.69 | 0.47 |
| V-JEPA + TCN | 0.41 | 0.40 |
| + force-augment (UDA) | 0.87 | 0.83 |
- Plain transfer is below trivial — negative cos-lift (Cycle-1's metric).
- World-frame force + unlabeled target video → wall falls.
The latent forgets the camera
t-SNE by camera (REASS, 4 views): Raw = 4 clean clusters → encodes which camera. Force-JEPA = fully intermixed → viewpoint forgotten.
| cam NMI (want low) | |
|---|---|
| Raw | 1.00 |
| Force-JEPA | 0.009 |
…but keeps the task
t-SNE by task (insert / remove): Force-JEPA shows the cleanest task separation — it strips the camera but retains what is happening.
| task purity (want high) | |
|---|---|
| Raw | 0.63 |
| Force-JEPA | 0.71 |
…and becomes robot-agnostic
t-SNE by robot (RH20T, 4 robots): Raw separates robots by scene/embodiment. Force-JEPA intermixes them — useful, since v2f wants a robot-agnostic force signal.
| robot NMI (descriptive) | |
|---|---|
| Raw | 0.53 |
| Force-JEPA | 0.10 |
View-invariant and diverse — not collapsed
The trap: a collapsed encoder fakes invariance. So read cross-view closeness with diversity.
| cross-view | eff-rank | |
|---|---|---|
| Raw | view-variant | 396 |
| DINO | invariant* | 33* |
| Force-JEPA | invariant | 61 |
*DINO collapses (uniformity≈0) — fake invariance. Force-JEPA sits in the good corner; CKA vs raw ≈ 0.01 (orthogonal).
What SSL bought us
- A force-anchored embedding is view-invariant, task-structured, uncollapsed.
- Wins the severe gaps — unseen camera & cross-dataset direction — at zero in-dist cost.
- Pays off when the gap is severe and unlabeled target video exists.
Where each tool wins
| regime | winner | signal |
|---|---|---|
| contact event (H1) | feasible ✓ | clean force → F1 0.87 |
| in-distribution force | frozen V-JEPA + TCN | REASSEMBLE lift +0.34 |
| unseen camera | force-augment SSL | F1 0.28 → 0.70 |
| cross-dataset direction | force-augment (UDA) | cos 0.40 → 0.85 |
| cross-robot (H3) | open | major room |
Next
Video is enough for contact & magnitude — and SSL unlocks the transfer.