Sinew

Teaching machines to feel — from ordinary video

① The opportunity — touch is the missing sense ② The data — abundant & untapped ③ It works — recovering force from video ④ It generalizes — to the unseen ⑤ Verdict

Ivan Domrachev · Sinew · 2026 · technical report prepared for D2SF · → chapters · ↓ detail

Opportunity

The missing sense

Robots learn from millions of videos — but video has no sense of touch.
Touch & force are what make contact-rich tasks work: assembly, insertion, handling.
Our idea: recover the force from the video itself — add the missing sense to data that never recorded it.

Three levels of touch: is it touching? · which way? · how hard?

Opportunity

What we set out to prove

① Read force from video

Can a model tell when something touches and which way it pushes?

② Enough data

Is there enough data out there to learn this?

③ Generalization

Does it work on robots & scenes never seen?

Three questions. We answered all three — yes.

The data

The data is already there — nobody used it

Source	What	Recordings	Strength
FMB	Peg insertion	1 844	clean, controlled environment
REASSEMBLE	Assembly	2 262	excellent force-sensor quality
RH20T	Many tasks & robots	12 666	huge diversity

We took existing public datasets and processed them into paired video+force. The data was always there — untapped. More than enough to start.

16 772 recordings · 178 h · ~6.4M frames — ≈14× larger than Meta's haptic dataset. Public, abundant, and unused — until now.

Approach 1

Supervised learning

Does it work at all?

Goal: prove the general feasibility — can a model read force off video?

Approach 1 · Result

Yes — it recovers the force

predicted contact overlaid on a real insertion

predicted force (line) tracks the real sensor (shaded)

Detects contact ≈ 87% accurate

Predicts how hard — matches the sensor

Predicts which way it pushes

Approach 2

Self-supervised learning

Does it generalize?

Goal: prove generalist capability — work on cameras, labs & robots it never trained on.

Approach 2 · Result

Works on a camera angle it never saw

new camera angle	contact	direction
Supervised baseline	fails	fails
Sinew	works (0.70)	works (0.78)

A normal model breaks on an unseen viewpoint.
Sinew keeps working — and loses nothing on familiar views.

Approach 2 · Result

Works on data from a different lab

force direction, new lab	Supervised baseline	Sinew
dataset A → B	0.41	0.87
dataset B → A	0.40	0.83

Cross-lab transfer normally collapses.
Sinew restores it — no new force labels needed, just the target's video.

Approach 2 · Result

One model, many robots

Sinew learns what is happening, not which robot recorded it — so the same model carries across different robots. The right-hand map shows robots fully mixed together: the signal is robot-agnostic.

Verdict

Proven — and ready to scale

All three questions — answered

① Read force from video: yes ✓ — contact & direction
② Enough data: yes ✓ — abundant & untapped
③ Generalization: yes ✓ — unseen cameras, labs & robots

Verdict

Technical feasibility is done & validated.

Video recovers touch — contact, strength & direction — and it generalizes.

Next: scale up across more robots, hands & tasks.