Video2Force — Stage-3 report

TL;DR

Goal: make a brand-new lab (crisp_ws: peg/ethernet insertion, box-flip) reach the in-distribution ceiling on contact (gate-F1) and magnitude (MAE) — video-only at inference, multi-camera fusion allowed. Achieved & proven apples-to-apples.

A cross-dataset-pretrained model, adapted with a short multi-cam video fine-tune (K=60 demos/task), reaches gate-F1 ≥ 0.90 on every set and MAE at the in-domain level — and on the identical split matches or exceeds a from-scratch in-domain model trained on the same K. The "OOD" model is, if anything, a better init than training on the target itself.

crisp_ws held-out (K=60, multi-cam fused, video-only)	peg gate / MAE	eth gate / MAE	box gate / MAE
pretrained-OOD (transfer)	0.908 / 2.85	0.935 / 3.87	0.899 / 2.39
from-scratch in-domain (same-K bar)	0.887 / 3.10	0.930 / 4.13	0.892 / 2.35

Pretrained-OOD ≥ from-scratch in-domain on gate (all sets) and MAE (peg/eth; box a +0.04 N tie) → OOD = in-dist on contact and magnitude, video-only.

Update — direction matched too. Extending the same recipe with a per-frame cosine head, OOD direction cosine reaches 0.825 / 0.933 / 0.973 (peg/eth/box) — ≥ from-scratch in-domain on all three and beating the best-constant trivial. On box_flip (the only set with a real, non-vertical force direction), it goes from −0.74 zero-shot to 0.97. See Direction.

The gap (zero-shot → ceiling)

Before adaptation, the model trained on REASS/FMB/RH20T sits far below the crisp in-domain ceiling (~0.90 gate / 1.9–3.8 N):

crisp set	zero-shot gate	zero-shot MAE	in-domain ceiling
peg (front / wrist)	0.43 / 0.76	4.8 / 3.3	0.90 / 2.4
ethernet (front / wrist)	0.39 / 0.76	7.6 / 4.5	0.91 / 3.6
box_flip (wrist)	0.62	4.8	0.91 / 1.94

Front-cam contact collapses (0.39–0.43): crisp's front viewpoint is outside the training-viewpoint span.
Magnitude scale mismatch: crisp contact threshold τ≈19 N ≫ training τ≈1–8 N → the model under-reads force.
Plain zero-shot late-fusion of the two views caps at ~0.74 (the good wrist view dominates, but doesn't reach the ceiling).

Recipe (frozen V-JEPA; video-only at inference)

Base: Stage-1 cross-dataset pretrain (s1_iter2_vp) — TCN video tower + gate/mag heads on frozen V-JEPA, multi-view aug.
Multi-cam fuser: a per-frame view-attention module (Set-Transformer PMA, k=1 seed) over the camera views → one fused token → the heads. view-dropout p=0.3 so it degrades to one view (box_flip is wrist-only) and never over-trusts a stream.
Adapt: multi-cam video fine-tune on K=60 crisp demos/set (force/proprio used only as labels; input is video). Balanced gate (pos_weight 9→4) to fix recall-over-firing. Magnitude head learns crisp's force scale.

Champion ~/out_stage3/s3_champion_multicam.pt · script train_crisp_multicam_ft.py. Direction champion (adds a per-frame cosine head): s3_champion_dir_w5.pt · train_crisp_multicam_dir.py (--w-dir 5.0 --epochs 90).

The adaptation ladder

**Contact + magnitude climb to the ceiling.** Gate-F1 jumps from ~0.61 (zero-shot) to ~0.87 by **K=5** and meets/exceeds the in-domain ceiling by K=40–60 (eth crosses to 0.935). MAE falls from 3.9–6.8 N onto the ceiling lines by K≈20–40. Target adaptation — not fusion — is the lever; the front-cam collapse is fixed at K=5 because FT makes crisp's viewpoint in-domain.

Airtight A/B — OOD = in-dist, apples-to-apples

**Pretrained-OOD ≥ from-scratch in-domain at the same K=60.** The "ceiling" trained on full crisp data is an unfair bar (full-data vs 60-shot). At the same adaptation budget, the cross-dataset-pretrained model **matches or beats training from scratch on crisp itself** — gate on all three sets, MAE on peg/eth (box a +0.04 N tie). So the residual vs the full-data ceiling is a data-quantity gap, not a transfer deficit.

Mechanism — the fuser down-weights the bad view

**Per-frame view-attention (front vs wrist).** During contact, mean front-cam attention is **0.010 (peg) / 0.009 (eth)** vs 0.059 / 0.075 off-contact — the fuser leans ~0.99 on the transferable **wrist** view exactly when force matters. That is the intended fix for the front-cam collapse, learned automatically.

Champion predictions (held-out crisp, video-only)

**Fused pred-vs-GT timelines** on held-out episodes (2/set). Gate probability snaps on at contact onset (GT-contact shaded); magnitude tracks the GT force trace. Contact + magnitude both recovered from video alone, on a lab the model never trained on.

Prediction rollouts — best / worst held-out episodes

Champion, multi-cam fused, video-only, on held-out crisp episodes (K=60 val split, 106 eps), ranked by per-episode contact gate-F1. Left = gate probability (GT-contact shaded); right = magnitude pred vs GT (N).

**5 best.** Gate-F1 0.99–1.00 — a single sharp gate step-up exactly at contact onset, held through the span; predicted |F| tracks the GT profile (MAE 1.6–3.7 N). Covers all three sets.

**5 worst.** Gate-F1 0.69–0.76 — failure mode is **gate timing / false positives** (low precision P≈0.53–0.73, recall stays high): the gate chatters and fires extra intervals in no-contact frames, concentrated in sparse-contact box_flip. Magnitude stays well-tracked (MAE 1.5–3.5 N) — the residual is contact timing, not force scale.

Direction — OOD = in-dist on cosine too contact-frame cosine

The model also predicts a per-frame unit force direction. We score mean cosine on GT-contact frames against two trivial baselines: +Z const and best-const (cosine vs the mean contact direction — the bar a single constant vector hits). The honest win is model > best-const AND pretrained-OOD ≥ scratch in-domain.

crisp set (K=60, fused, video-only)	zero-shot	pretrained-FT	scratch in-domain	best-const	+Z const
peg-insert (+Z-degenerate)	0.33	0.825	0.801	0.806	0.802
ethernet-insert (+Z-degenerate)	0.39	0.933	0.913	0.931	0.929
box_flip (real lateral dir)	−0.74	0.973	0.971	0.892	0.528

**Left:** direction cosine per set. Pretrained-FT (green) ≥ scratch in-domain (blue) on every set and clears the best-const trivial (grey). peg/eth are **+Z-degenerate** (insertion force is near-pure vertical, so best-const ≈ 0.81/0.93 is already the ceiling — nothing to beat). **box_flip** is the only non-degenerate test: zero-shot is **anti-correlated (−0.74)** from cross-lab azimuth mismatch, recovered to **0.97**, far above the 0.89 constant. **Right:** a box_flip episode — predicted Fy/Fz (dashed) track the GT (solid) unit-force components through the flip (contact-cos ≈ 1.00), i.e. genuine per-frame rotation, not a constant.

Frame. crisp's raw sensor is TCP/EE, but the label is rotated to world via ee_quat (direction = unit F_world, verified), exactly as the REASS/FMB training labels — so the comparison is world-vs-world, consistent. The zero-shot gap is lab azimuth ambiguity: the gravity-Z sign is shared across labs (so peg/eth +Z partly transfers), but yaw-about-gravity differs (so box_flip's lateral force collapses until fine-tuned). A yaw-invariant EE-frame label is the lever for zero-shot direction (ee_quat is label-side; inference stays video-only).

Key findings

Target adaptation is the lever, not fusion. Front-cam contact collapse (0.43) is fixed by K=5 video-FT (→0.875); zero-shot multi-cam fusion alone caps ~0.74.
Multi-cam view-attention + view-dropout down-weights the OOD front-cam (attention 0.01 during contact), gives one clean video-only fused prediction, and handles variable view counts (box_flip = 1 view).
Balanced gate (pos_weight 9→4) cleared the last contact gap (peg 0.874→0.90+).
Magnitude reaches the same-regime in-domain level; it floors ~0.5 N above the full-data ceiling on peg/box (vision/few-shot limit — affine + w_mag/epoch all neutral), i.e. a data-quantity gap.
Direction matches too. Per-frame cosine reaches 0.825/0.933/0.973 (peg/eth/box) — ≥ from-scratch in-domain on all three and above the best-const trivial. box_flip (real lateral direction): −0.74 → 0.97. peg/eth direction is +Z-degenerate (constant ≈ ceiling). Frame: world-vs-world via ee_quat; zero-shot gap is lab azimuth ambiguity (EE-frame label is the lever for zero-shot dir).
Proven OOD = in-dist on contact + magnitude + direction, video-only, K=60 — pretrained-OOD ≥ from-scratch in-domain on every cell.

Full trace: docs/v2f_stage3_iteration_log.md · research docs/v2f_stage3_research.md. box_flip is wrist-only at source (domrachev03/box_flip_fb, no front cam, no high-rate force).