Sinew — Knowledge Base

• Frozen video features + TCN 3-head — the baseline; caps contact F1 ≈ 0.78.
• Encoder unfreeze / E2E (LP-FT) — unfreeze last-8 V-JEPA blocks, probe-then-finetune, LLRD 0.7, EMA 0.999, bf16 (fp16 silently flat-lines loss). THE in-dist lever: 0.78 → 0.92–0.97.
• Encoder ablation — DINOv2-B 0.52 < ViT-B 0.64 < ViT-L 0.76 ≈ ViT-g(1B) 0.764; video ≫ image; size saturates past ViT-B.
• Head ablation — TCN 0.768 > BiGRU 0.723 > Transformer 0.689 (receptive field is the head lever).
• Multi-task 3-head — contact+dir+mag (0.779) beats mag+dir (0.768) / contact-only (0.68); raw 3D-force regression collapses (0.51).
• Contact-gate rebalance ("c9": neg-keep 0.5, gate-pos-weight 5, w-contact 30) for sparse-positive OOD; in-dist uses neg-keep 1.0 instead.
• Frame-unification (EE frame) — decisive cross-lab direction enabler (a supervised lever, not SSL): REASS flip (x,−y,−z), FMB stays base, RH20T sign-inverted.
• Force-richness curation — label-free pre-train screen: contact = SNR (peak/free-baseline); direction = dchg (temporal dir change), dchg<0.2 = coherent.
• Per-source τ + force-norm, robot-AGNOSTIC (robot-ID was harmful).
• Two-head split — contact from UNFROZEN encoder, direction from FROZEN encoder (frozen ≫ unfrozen for OOD direction).

Frozen V-JEPA-L ceilings (in-dist):

Encoder-unfreeze (E2E) ceilings (in-dist):

Headline: frozen ceiling ~0.78 → E2E unfreeze 0.92 on clean force; dedicated in-dist contact reached ~0.97. The win: force-richness curation lifts flexiv 0.47→0.785 (+0.31) ⇒ dilution, not encoder capacity, was dominant; residual ~0.78 ceiling on observer-class force is a sensor wall. Caveat: contact-F1 is NOT cross-dataset comparable (base rates differ); MAE is per-source scale.

Contact-prediction overlays (in-dist)

Side video + on-frame contact-probability bar (0.5 threshold); red border = GT contact; live GT/pred |F|.

Best validation episodes (in-dist) — 3-panel: contact · magnitude · direction

Zero-shot to crisp_ws (held-out lab). Trivial contact F1 peg 0.76 / eth 0.68; in-domain ceiling ≈ 0.90.

Few-shot target adaptation breaks the OOD ceiling: 0 eps F1 0.42 → 15 eps 0.53 → 30 eps 0.79 (dir 0.74) → 60 eps 0.90 (peg 0.91/dir 0.88, eth 0.89/dir 0.84).

Walls (honest): cross-lab direction beyond the prior is unsolved (prior-bound; insertion is +Z-degenerate even in-domain, lift +0.01–0.02); cross-lab direction = camera-viewpoint extrapolation (lever is viewpoint coverage, not encoder); RH20T hurts the direction corpus (raw −0.72 sign-inverted; sign-fixed +0.2 off-target) → keep RH20T for contact only.

Contact-prediction overlays (OOD — crisp held-out lab)

Best validation episodes (OOD crisp) — 3-panel: contact · magnitude · direction

• Cross-modal video↔force JEPA — predict masked force latent from frozen-video context (EMA target, latent L1, VICReg anti-collapse).
• Force-JEPA (force-only SSL, "ForceTok") — masked-latent SSL on 125 Hz force alone → robot-agnostic force embedding.
• DINO-JEA alignment — trainable video tower distilled to a frozen, scene-blind Force-JEPA teacher (no-negative self-distill).
• cosine / CORAL / InfoNCE alignment — close modality gap on raw V-JEPA (InfoNCE floors at ln(batch): force autocorrelation false-negatives).
• x-modal force AUGMENT — concat[ raw frozen V-JEPA ‖ g(video) ] → same TCN head; augment-NOT-replace, so SSL can only add.
• Sinkhorn-OT binding to a frozen teacher — force student bound to the frozen video encoder via entropic-OT contrastive + MoCo queue + contact-weight.
• VICReg / covariance collapse fix (decorrelation off-diagonal = master fix); DARE-GRAM/RSD for force regression DA (classification-DA DANN/CORAL HURTS regression).

Where SSL WON (severe-gap regime):

Caveat: camera/cross-dataset wins are UDA, not strict zero-shot — needs unlabeled target video (strict zero-shot is worse than baseline).

Where SSL was NEUTRAL / NEGATIVE:

The unified law: SSL/JEA helps only when the gap is severe enough that raw frozen V-JEPA fails to transfer (unfamiliar camera / unlabeled new dataset / cross-robot). Otherwise neutral.

• RH20T cross-robot: raw ≈ chance (0.13) → aligned wins even with less capacity.
• FMB-multipeg + REASS cross-object: raw already transfers → raw+TCN beats aligned+TCN (linear-probe "wins" reverse under capacity-matched readout — an artifact).
• When both datasets are present → just train jointly (reaches native ceilings).
• In-dist cost: REPLACE is lossy in-domain (collapse); concat-AUGMENT is zero-cost → clean OOD win.
• Geometry ≠ transfer: minimizing modality gap (CKA 0.07→0.51) didn't fix swap; the best-swap arm has the largest gap.
• Binding ceiling (confirmed 5×): no frozen-feature SSL trick beats frozen-video information; only lever = supervised E2E unfreeze.

• Raw V-JEPA = pure camera encoder: camera kmeans-NMI 1.00, task 0.00, eff-rank 396 → clusters by viewpoint → negative transfer.
• Force-JEPA (winner) = view-invariant WITHOUT collapse: camera NMI 1.00→0.009, cross-view 0.962, eff-rank 61, task purity 0.71, orthogonal to raw (CKA 0.014).
• DINO-JEA (multi-view) = COLLAPSED: eff-rank 33, per-dim std 0.0004; cross-view 1.0 is the "collapse-lie" (still leaks camera NMI 0.79).
• Collapse fix: RankMe 9.8 → 138.9; dataset silhouette 0.66 → 0.06. Intermix: MMD 1.25 → 0.21.
• The wall: local dataset-kNN floors ~0.93 (target 0.41) on frozen feats — high-rank embed must retain dataset-specific appearance until encoder unfreeze.
• Metric canon: RankMe/eff-rank + uniformity (collapse guards), cross-view cosine, silhouette/NMI/kNN-purity, CKA; align-uniformity plane = headline; UMAP not t-SNE.

• Force-JEPA / "ForceTok": masked-latent SSL on 100–125 Hz force (corpus rh20t_hifreq), trains clean (no collapse). Teacher fssl_v1/best.pt reused as the force tower in alignment.
• Force as a view/domain-invariant physical anchor (ImageBind-style): scene-blind → forces the aligned video tower to drop appearance, killing the camera shortcut.
• Cross-robot WIN + limits: raw force carries robot SCALE (~4× span) → fails cross-robot; SSL embedding is robot-agnostic → contact AUC 0.94 vs 0.80, weak-robot direction ~2× (ur5 0.10→0.21). Frozen-bound, no in-dist edge, no edge on non-circular task classification (≈ chance).
• Force embeddings are non-trivial: per-dim R² mean .45, 0% dims > .9, 52% residual variance beyond 181 hand-crafted features.

# Technical Progress — Summary & Critique (video→force, "v2f")

Internal honest assessment. Not marketing. Numbers are exact; where a claim is single-seed, best-epoch, or otherwise fragile, it is flagged. "Lift" = margin over a trivial baseline (best-constant direction, or all-positive contact), which is the only honest framing for our metrics.

(1) TL;DR — where v2f stands

We have a real, working in-distribution video→force model: encoder-unfreeze (E2E fine-tune of V-JEPA2 ViT-L, last-8 blocks, LP-FT) broke the long-standing frozen-feature contact ceiling from ~0.78 to ~0.97 in-dist (clean before/after on the same model: contact-F1 0.26→0.935 run1, dir cos 0.71→0.96), which validated the "encoder-unfreeze is the next lever" theme that recurred unbroken through Stages 1–5. Cross-lab transfer is the unsolved core: pure zero-shot contact plateaus at ~0.47 (below the trivial all-positive 0.76) and is only rescued to ~0.82–0.86 by a combination of frame-unification + frozen feats (encoder-unfreeze adds only ~+0.05) and/or few-shot target adaptation (~30–60 crisp episodes); zero-shot direction never beats the trivial best-constant on the two insertion tasks (peg/eth) — frozen-encoder + dchg-curation reaches cos-lift +0.76/+0.74 but that is still below best-const 0.82/0.90, is a still-rising cherry-picked epoch, and comes from a different (RH20T-dropped) corpus than the contact champion. SSL is exhausted as a value-add: across ~four SSL classes and multiple stages it is NEUTRAL (ties supervised) except in severe-gap UDA regimes, and even there it needs unlabeled target video (not zero-shot). The honest one-liner: in-dist is solved, contact transfer is mostly solved via frame-unify + few-shot, direction transfer and a single shippable cross-lab model remain open, and almost every headline number is single-seed.

(2) Stage-by-stage arc

• Stage-1 — feasibility / frozen SL ceilings. Asked: establish per-dataset frozen V-JEPA-L→TCN ceilings and whether any model-side lever moves them; later, close the crisp_ws OOD gap. Result: contact-F1 FMB 0.779 / REASSEMBLE 0.874 / RH20T ~0.71; direction real only on REASSEMBLE (cos-lift +0.34), near-trivial on FMB (+0.015); encoder size/head/pooling all tie (mean-pool 0.873 ≈ attn 0.869); the sinew-250 in-domain control (front-cam 0.000→0.912) proved the OOD collapse is a transfer gap, not a feature ceiling. Caveat: single-seed throughout; memory says "E2E→0.9" but ablations say 0.90 was never reached; the "OOD=in-dist" Stage-1 epic win is few-shot (K=60 labels), not zero-shot.

• Stage-2 — latent health / intermix. Asked: heal the joint latent (collapse-free, datasets+modalities intermixed) on a frozen encoder without regressing downstream. Result: collapse solved (RankMe_v 9.79→130, root-caused to a missing VICReg covariance term — a clean A/B, the strongest claim in the program); global dataset silhouette 0.66→0.085. Caveat: the goal metric (local dataset kNN-5) only moved 1.0→0.886, missing both the 0.50 target and the 0.70 hard floor; "intermix WON" rests on the global metric the recipe optimizes by construction; log and memory disagree on every champion number (130/40.6/0.085/0.886 vs 138.9/49.4/0.062/0.93); no random-feature baseline.

• Stage-3 — OOD=in-dist via target FT. Asked: match the crisp in-domain ceiling on held-out crisp with video-only inference. Result: contact reached ceiling (K=60 fused: peg 0.900 / eth 0.939 / box 0.906, all ≥0.90); airtight A/B shows pretrained-OOD ≥ scratch-in-domain on every cell; box_flip direction -0.740→0.973 (a genuine per-frame win, with target FT). Caveat: magnitude never reached the true full-crisp ceiling (peg 2.94 vs 2.40 N); A/B margins are ~0.01 F1 / ~0.1–0.3 N on n=31–42, single-seed — the log itself calls comparable deltas "eval-noise"; peg/eth direction is +Z-degenerate so the "win" is a tie dressed as a pass; zero-shot direction explicitly prior-bound (ITER-10: nothing beats best-const).

• Stage-4 — SSL embeddings. Asked: one frozen-trunk SSL embedding scored on four intrinsic axes (multi-cam, FMB intermix, per-cam closeness, force meaningfulness). Result: global intermix win (fmb_mmd 1.254→0.206); force tower activated (contact-AUC 0.885→0.977); anti-collapse verified (rankme_v 11.7→115). Caveat: the local FMB-vs-rest kNN-5 stayed 1.0 across all 5 recipes (a real frozen wall, but a failure on that sub-target); vp_minus_ds never crossed zero (its actual target); xview_cos was near-saturated at baseline (0.986→0.998); the "+0.07 multi-cam" is a linear-probe proxy — the spec-defined gate-F1 metric moved only +0.026; scoring harness absent from disk; FMB 2-cam (spec prerequisite unmet); intended DARE-GRAM/RSD never run.

• Stage-5 / 5.5 / 5.6 — cross-modal video↔force alignment. Asked: close the latent geometry gap and make it transfer task semantics. Result: geometry gap decisively closed (CKA 0.07→0.51; exposed the Stage-4 0.998 "alignment" as a common-mode cosine collapse — paired_cos_centered 0.014); A1→A6 self-correction proved the SSL embed reproduces Stage-3 OOD under a fair recipe; A5b showed every SSL embed is a deterministic function of frozen V-JEPA. Caveat: geometry buys nothing downstream — swap-at-chance under the symmetric recipe; the headline Stage-5.6 "contact swap solved (0.831)" is unverifiable on disk (local doc stops at a 1-epoch smoke of 0.72); OOD dirCos peg 0.765 / eth 0.828 sit below their trivial best-const (0.82 / 0.88); single-seed; no prevalence baseline.

• Stage-6 — E2E encoder-unfreeze + few-shot. Asked: finally unfreeze the encoder (long-deferred). Result: the decisive in-dist win — contact-F1 0.26→0.935 (run1) / ~0.97 (run2/3), dir 0.71→0.96; few-shot crisp scales 15ep 0.53 → 30ep 0.79 → 60ep 0.90 (peg F1 0.91/dir 0.88, eth 0.89/0.84). Caveat: "0.97" has no per-task/per-source breakdown and is single-run; pure zero-shot caps at 0.47 (< trivial 0.76); few-shot only matches (not beats) the prior frozen-adapted 0.90 contact ceiling — the real new win is direction; direction recovery is raw-cos with no trivial baseline and is retroactively tainted by a later-found FMB frame double-rotation bug; box_flip excluded; no MAE/AUC reported; numbers live only in 8-day-old memory.

• Stage-7 / 7-redo — frame-unify + force-richness + direction. Asked: per-dataset SL ceilings + visual-only/cross-modal SSL until OOD crisp matches SL in-dist on contact+direction. Result: first zero-shot crisp contact clearing trivial (peg 0.826 / eth 0.870 vs trivial 0.76/0.68); frame-unification is the decisive direction lever (REASS flip [1,-1,-1]; FMB must stay base-frame — tool-frame flips all arms to ~-0.6; RH20T base-frame is antipodal, dir_rh20t -0.66/-0.72 ≈ mirror of champion +0.76/+0.74); SL E2E REASS F1 0.92/dir 0.93 passes both bars; force-richness metric (SNR→contact, dchg→direction) predicts learnability; SSL declared NEUTRAL (kill-rule 2/2). Caveat: the contact "win" is ~90% frame-unify + frozen feats (ep0 frozen probe already clears trivial at 0.815/0.760; unfreeze adds ~+0.05); direction +0.76/+0.74 is a still-rising cherry-picked ep17, from a different corpus, and still below best-const 0.82/0.90; the "two-head production split" (unfrozen-contact + frozen-direction) was never trained/evaluated as one model — the headline is a best-of-each-arm composite; epoch selected on the OOD eval itself.

• Stage-8 — RH20T supervised-vs-SSL. Asked: does multi-robot RH20T + SSL match/beat supervised on OOD crisp? Result: SSL matches supervised on OOD contact (0.845 vs 0.848, all arms within 0.01, all beat trivial) — user goal met (tie, not beat); RH20T did not lift contact but lifted frozen ep0-probe direction (0.71–0.73 vs control 0.63). Caveat: iter-1 used only 2 of 7 RH20T configs (~2275 of ~10000+ eps, flexiv absent); all verdicts rest on sub-0.01, single-seed, best-epoch gaps; the direction "win" is a frozen ep0-probe raw cosine with no direction baseline and no deployed checkpoint (fine-tuned ep11 gives only 0.47/0.37); iter-2 (full RH20T) is paused on a loader bug.

• Cross-cutting laws. SSL-exhausted (frozen-SSL can't beat supervised; only escape is supervised E2E unfreeze, which still needs labels); gap-floor (joint training hits each domain's native ceiling — REASS 0.876≈ceiling 0.874, FMB 0.768≈0.779 — but the absolute 9.5% gap was called "irreducible," a word the later Stage-6 0.97 unfreeze contradicts, since 0.779 was a frozen ceiling); camera-direction (single-view direction collapses, random-cam interpolates [+0.13/+0.24] but does not extrapolate [hama1 -0.53], lever = viewpoint coverage); force-frame (EE/tool frame is canonical, contact+magnitude are frame-invariant ‖F‖, frame/quat/sign poison direction only — the #1 silent bug).

(3) Honest scorecard

(4) Genuinely solved vs still open / shaky

Genuinely solved

• In-distribution contact via encoder-unfreeze. 0.26→0.935 (run1), ~0.97 (run2/3), dir 0.71→0.96 — the frozen ceiling is broken, doc-corroborated, the central success of the program.
• Latent collapse (Stage-2). Root-caused to the missing VICReg covariance term; clean A/B; RankMe_v 9.79→130; anti-sphere-artifact control (PR persists 1.45 on L2-norm) actually run.
• Frame-unification / force-frame ground truth. EE-frame canonical; contact+magnitude frame-invariant; per-dataset sign conventions empirically pinned and replicated; the FMB base-vs-tool fix is a real, falsifiable bug with a clean A/B.
• The transfer-gap diagnosis (sinew-250). In-domain crisp hits ~0.90 on every cell (incl. front-cam 0.000→0.912) — proves OOD collapse is transfer/calibration, not a feature-information ceiling.
• SSL-is-neutral, stated honestly. Pre-registered kill-rules, no-SSL control arms, numerically-identical results — disciplined negative results, not spin.
• Few-shot target adaptation as a reliable lever. Scales monotonically with target data (0.42→0.53→0.79→0.90) and breaks the OOD ceiling — when ~30–60 labeled crisp episodes are available.

Still open / shaky

• A single shippable zero-shot cross-lab model. Contact and direction champions come from different encoders (unfrozen) and different corpora (RH20T-dropped for dir). The "two-head production split" was never trained or evaluated end-to-end — current cross-lab headlines are a best-of-each-arm composite.
• Zero-shot OOD direction. Prior-bound; +0.76/+0.74 still below best-const 0.82/0.90; box_flip only solved with target FT. This is the hardest remaining wall.
• Magnitude at the true ceiling. Persistent ~0.5 N gap to full-crisp; "data-quantity not transfer" asserted but never proven with a learning-curve control.
• Statistical reliability. Single-seed everywhere; "within noise" claims rest on an unmeasured noise floor; multiple headline orderings hang on 0.003–0.01 deltas.
• RH20T's contribution. Provisional — 2 of 7 configs, flexiv absent, direction "win" is a frozen-probe artifact with no baseline. iter-2 paused.
• Numeric provenance. Stage-5.6 "contact swap solved (0.831)" and the gap-floor "irreducible" framing are unverifiable / contradicted on disk; log-vs-memory champion numbers disagree (Stage-2). Several stages live only in stale memory with no dedicated doc.
• Sensor-bound datasets. flexiv/ur5/FMB fail the SL bars regardless of recipe (sigma~2N observer noise, uncalibrated robot_ft); these are data-quality walls, not model walls.

(5) Top methodological lessons (recurring critique themes)

1. Report cos-LIFT over best-constant, never raw cosine. Half the "direction wins" evaporate against the trivial baseline: FMB dir 0.965 is +0.015 lift; peg/eth are +Z-degenerate so a high cosine is structural; the +0.76/+0.74 frozen-dir headline still sits below best-const 0.82/0.90. Any raw dir-cos quoted bare is a red flag. Also prefer margin-over-persistence (direction cosine is autocorrelation-inflated and frame/viewpoint-locked).

2. Contact-masked, base-rate-aware evaluation. Cross-dataset contact-F1 is not comparable — crisp pos ~0.61 vs FMB/REASS ~0.10 (~6×). Read gate-AUC not gate-F1 for OOD (F1 zeros on peg/eth are a compressed-sigmoid threshold artifact). Always carry the trivial all-positive floor next to the number. Direction must be evaluated on contact frames only, ideally on direction-CHANGE / high-|dF/dt| frames.

3. Equal-capacity, same-data, same-recipe comparisons. "Encoder not the lever," "TCN > Transformer," "intermix WON," and the in-dist DINO bars all suffer from cross-dataset/cross-arm or capacity confounds (head-arch run on different datasets; tau 8-vs-10 mixed within the encoder ladder; champion bundles raw+anchor+bs+cov changes at once). Isolate one variable; provide a random-feature / equal-capacity supervised readout floor.

4. Non-comparable scales (MAE, base rate). Per-source force scales span an order of magnitude (RH20T 0.08–0.33 N … kuka ~11.5 N); raw MAE is not aggregable or rankable across sources — normalize or report per-source. F1 is prevalence-dependent; never read "OOD ≈ in-dist F1" without prevalence-matching.

5. ep0-probe and frozen-probe artifacts. Stage-8's "RH20T lifts direction" lives entirely on the ep0 frozen probe (same trunk for all arms; only the readout differs) with no deployed checkpoint behind it. A frozen-probe result is not a trained-model result; attribute it to the readout, not the data, until isolated with a matched-readout control.

6. Single-seed discipline ⇒ humility, not false precision. Project policy is single-seed (and that's fine), but that makes every "within noise," "every cell," and "robust floor" claim an assertion, not a measurement. Lever deltas of +0.05–0.06 are within observed epoch-to-epoch swings (champ_fmbbase swung -0.11 across epochs). Best-epoch selection on the OOD eval target is selection-on-test — needs a held-out crisp split. When the delta is <~0.02, do not interpret the ordering.

7. Provenance hygiene. Pin a single source of truth and reconcile log-vs-memory disagreements before declaring "converged" (Stage-2 numbers disagree on every digit; Stage-5.6 champion exists only in memory). "Irreducible" / "solved" should never be written for a result that is single-seed, frozen-bound, or unverifiable on disk.