Sinew
academic talk · 2026
[ video → force ]

Recovering contact
force from video.

Per-frame contact force from RGB manipulation video — with no force sensor at inference. A feasibility check: problem formulation, datasets, and the approaches we tested.
Ivan Domrachev · Igor Alentev · Lev Kozlov
KAIST IRiS · adv. Prof. Jee-Hwan Ryu
the gap

Manipulation data is vision + actions — never the force.

The large corpora — Open X-Embodiment, DROID, BridgeData — are vast camera + action datasets; the force is never recorded and can't be added back.

Goal — a feasibility check: is it possible in principle to recover per-frame contact force (localization, direction, magnitude) from RGB video alone, with no force sensor at inference?

Ego4D — typical vision-only manipulation data: rich video, zero force.
What's recorded
RGB video + actions — cheap, standardized, at scale.
What's missing
Contact force & torque — the result of every touch. Almost never captured.
Why it matters
Contact-rich tasks are won at the instant of contact — when cameras occlude the fingertips.
motivation
02 / 14
the two axes

Two ladders: what we touch with, & what we predict.

What we predict — abstraction
Localization (where) → direction (which way) → magnitude (how strong).
What we touch with — embodiment
Fixed tool → gripper (object · tool · env) → dexterous hand (many contacts).
setting
03 / 14
this stage

This stage: fixed tool, three force values.

This stage = the fixed-tool rung (single rigid contact). We predict three force values, each with its own metric, evaluated only on contact frames.

① Localization

Is / where it is touching — the contact gate.

F1 vs trivial floor

② Direction

Which way the force acts (EE frame).

cos − costrivial (lift)

③ Magnitude

How strong, in newtons.

MAE (N)

Report the direction lift over the best-constant prior (raw cosine is inflated by the −Z insertion prior). End goal: climb to the dexterous-hand rung.

focus
04 / 14
datasets

Four corpora — train in-dist, test cross-lab.

datasetrobot(s)force sensorτrole
FMBFranka Pandaobserver (σ≈2 N)10 Ntrain
REASSEMBLEFranka FR36-axis wrist (≈0.4 N)2 Ntrain
RH20Tflexiv·ur5·kuka·frankaexternal ATIper-robottrain
crisp_wsFranka (held-out lab)server-clean TCPper-taskOOD eval

16,772 rollouts · 178 h · ~6.4 M frames · 7 embodiments · 4 force-frame conventions harmonized.

Two scenarios

  • In-distribution — held-out episodes of the training labs.
  • OOD / cross-lab — zero-shot to crisp_ws, a lab never seen in training.
REASSEMBLE — train (in-distribution)
crisp_ws — held-out lab (OOD target)
datasets
05 / 14
supervised · pipeline

Supervised pipeline — three steps.

V-JEPA 2 ViT-L video features → dilated-TCN 3-head.
① Frame unification
Express every dataset's force / direction in a common end-effector (EE) frame — the cross-lab direction lever.
② Dataset distillation
Keep only force-rich, direction-coherent episodes (per-episode SNR & direction-change screen) — drop diluting clips.
③ Train frozen → unfrozen
Probe heads on frozen features, then fine-tune the last-8 ViT blocks (LP-FT) — breaks the frozen ceiling.
supervised
06 / 14
supervised · in-distribution

In-distribution results.

in-dist metricfrozen feats+ encoder FT
contact F1 (best)0.78 (ceiling)0.97
REASSEMBLE F10.870.92
REASSEMBLE dir cos (lift)0.880.93 (+0.37)
RH20T kuka F10.710.92
group (unfrozen)F1dir cos (lift)
REASSEMBLE0.920.93 (+0.37)
RH20T kuka0.920.80 (+0.21)
FMB0.720.92 (≈triv)
RH20T flexiv*0.47→0.790.38
RH20T ur50.640.24

*after distillation. Clean-force datasets clear both bars; observer / weak-force sets are data-bound.

  • Encoder fine-tuning is the lever — frozen ceiling 0.78 → 0.97 in-dist contact.
  • Direction is real on clean F/T (REASSEMBLE lift +0.37); near-trivial on observer-noise force (FMB).
supervised
07 / 14
supervised · in-distribution

In-distribution — contact-prediction overlays.

REASSEMBLE — contact F1 0.975
RH20T KUKA — F1 0.976
FMB — F1 0.736

Contact-prediction overlays: on-frame contact-probability bar (0.5 threshold), red border = GT contact, live GT/pred ‖F‖.

supervised
08 / 14
self-supervised · architecture

Using force as a self-supervised teacher.

Idea: reshape the visual stack without force labels at the target — use force as a self-supervised teacher, then attach the same heads.
Cross-modal objective: align / predict force latents from video.
Cross-modal alignment
Pull the video embedding toward a force embedding (cosine / CORAL / InfoNCE) so features become force-predictive, not just appearance.
Force-as-anchor augmentation
Concat a force-aligned embedding onto frozen features (augment, never replace) → can only add.
Then: the same supervised heads
The TCN gate / direction / magnitude heads train on top of the SSL-adapted features.
self-supervised
09 / 14
self-supervised · results

Self-supervised results.

scenariosupervised+ SSL
In-distribution (contact)0.920.92 (tie)
OOD crisp lab (contact)0.850.85 (tie)
OOD held-out camera (contact)0.280.70
OOD cross-dataset (direction)0.400.85
  • SSL ties supervised in-distribution and on the standard OOD lab.
  • It helps only under a severe domain gap (unfamiliar camera, unlabeled new dataset) — and there it needs unlabeled target video.
  • The in-distribution ceiling is raised by encoder fine-tuning, not SSL.
Overlay — REASSEMBLE, F1 0.975
Best contact — FMB, F1 1.00
self-supervised
10 / 14
self-supervised · embeddings

SSL embeddings — per-task structure.

The learned latent space (UMAP), coloured by task: force-anchored SSL pulls same-task episodes together across cameras and datasets — features organize by physics, not appearance.

self-supervised
11 / 14
results · OOD generalization

Zero-shot to a held-out lab.

metricOOD peg / ethtrivial
Contact F10.82 / 0.860.76 / 0.68
Direction cos0.76 / 0.740.81 / 0.88
Magnitude MAE~3.5 / 4.1 N—
  • Contact transfers cross-lab — clears trivial, ≈ in-domain.
  • Zero-shot direction reaches but doesn't beat the insertion prior (extrapolation-hard).
  • Few-shot (~30–60 target eps) breaks the ceiling: F1 0.42→0.90.
Zero-shot overlay — crisp peg (held-out lab), F1 0.946
results
12 / 14
summary

What we achieved & showed.

  • Recovered contact, direction, and magnitude from RGB video — sensor-free at inference.
  • In-distribution: fine-tuning the visual encoder breaks the frozen ceiling — contact F1 0.78 → 0.97; REASSEMBLE direction lift +0.37.
  • Built & harmonized a 16,772-rollout corpus (7 embodiments, 4 force-frame conventions) + a force-frame ground truth & a force-richness distillation metric.
  • Cross-lab: contact transfers zero-shot (clears trivial, ≈ in-domain); few-shot (~30–60 eps) breaks the OOD ceiling.
  • Studied self-supervision: it ties supervised, helping only under a severe domain gap.

Feasibility check: done

Recovering contact force from video works in principle — the idea is validated. What remains is scaling generalization and climbing the embodiment ladder.

summary
13 / 14
what's next

Future work.

Cross-lab generalization
Close zero-shot direction (viewpoint coverage); one model that transfers across labs without target data.
Climb the modality ladder
From the fixed tool up to the gripper and the dexterous (multi-finger) hand — many simultaneous contacts.
Augmented data → better policies
Feed the recovered force back into vision-only datasets and train policies on the augmented data → improved downstream success.
Toward egocentric human data
Recover per-finger force on human-hand video at scale — the end goal.
Ego4D — the dexterous, egocentric human-hand video we ultimately target.

Thank you — questions welcome.   video2force · KAIST IRiS

future
14 / 14