Stage-6 — E2E V-JEPA2 encoder-unfreeze

Fine-tune the video encoder end-to-end on streamed data — break the frozen-feature ceiling, then the cross-lab OOD ceiling. 2026-06-20

in-dist 0.26→0.97OOD few-shot F1 0.91 / dir 0.88branch stage6-e2e-stream

TL;DR. The recurring "encoder-unfreeze is the next lever" (named across Stages 1–5) is validated. Unfreezing V-JEPA2 ViT-L end-to-end (trained on raw video streamed from HuggingFace, no pre-extraction) lifts in-distribution contact-F1 from 0.26 → 0.97 — the frozen encoder capped ~0.78 with direction collapse. Cross-lab zero-shot (held-out lab "crisp") needs more: direction is fixed by EE-frame label unification, contact by augmentation, but pure zero-shot plateaus ~0.47. A small dose of target data — few-shot adaptation on crisp episodes — breaks the OOD ceiling: 30 episodes → held-out F1 0.79, and 60 episodes → F1 0.91 / dir 0.88, matching the frozen-adapted 0.90 and far exceeding it on direction (which the frozen pipeline could never recover).

1. In-distribution — encoder-unfreeze breaks the frozen ceiling

checkpoint	contact-F1	dir cos	MAE
frozen V-JEPA + TCN (Stages 1–5)	~0.78	collapse	—
probe (frozen, head only)	0.26	0.71	1.59
E2E LP-FT ep5	0.72	0.89	0.51
E2E LP-FT final (run2)	0.97	0.96	0.35

LP-FT: 2 probe epochs (frozen) → unfreeze last 8 blocks, LLRD 0.70, EMA 0.999, bf16. 4×A6000, streamed FMB+fmb_multi+REASSEMBLE.

2. Zero-shot cross-lab — the lever stack, then a plateau

Crisp is a held-out lab, never trained on. run1 (world-frame dir) → direction collapses (cos −0.14/−0.21). run2 expresses force in the end-effector frame (R(ee_quat)ᵀ·F_world) → direction recovers (+0.20/+0.28). run3 adds photometric+crop augmentation → contact + direction both rise (F1 0.45/0.47, dir 0.33/0.46). But zero-shot contact-F1 plateaus ~0.47 — below the trivial all-positive baseline (0.76). The cross-lab gap is too wide for pure zero-shot.

3. Few-shot target-adaptation — breaks the OOD ceiling

crisp episodes	peg F1 / dir	eth F1 / dir
0 (zero-shot run3)	0.39 / 0.38	0.44 / 0.49
15	0.53 / 0.52	0.54 / 0.57
30	0.79 / 0.75	0.78 / 0.72
60	0.91 / 0.88	0.89 / 0.84

Fine-tune the run3 E2E checkpoint (encoder last-8 + TCN head, enc-lr 1e-5, EMA) on a few crisp train episodes; evaluate on held-out crisp episodes. ~30 episodes clears the trivial baseline; ~60 episodes reaches F1 0.91 / dir 0.88 — matching (and on direction, far exceeding) the frozen pipeline's 0.90, which was likewise crisp-adapted but with collapsed direction.

Engineering notes

Stream tap: lerobot 0.5.2 + lvjonok fsspec-streaming addon (fixes a chunked-video frame-index bug on HF Hub). Gotchas solved: 1-parquet-shard datasets → DDP shard by disjoint episodes; buffer-size 48 (256 OOMs multi-rank); batched encode (GPUs 100%); bf16 not fp16; delta_timestamps smooths force → crisp uses cached canonical labels joined to streamed video by frame index; 2-cam decode via a meta.video_keys property patch. All scripts + checkpoints on git branch stage6-e2e-stream.

Next: RH20T joint phase (video-tactile co-training — M3L / VTAM balanced cross-modal attention / ForceMapping) when RH20T encoding lands.