sinew

R2S2R — real-to-sim-to-real for contact-rich manipulation

★ Knowledge Base — the whole startup, one page →

Single consolidated wiki: idea & thesis · technical progress (ordinary manipulation — datasets, metrics, SL & SSL approaches + results + generalization + embedding viz + force-embedding pretraining; dexterous/OmniReset) · honest progress summary & critique · business plan (market · model · roadmap) · venues · team. Hyperlinked to every deck & stage report.

★ Video + Joints → Force — does proprioception help? (sinew-338) →

Controlled A/B (vision / joints / vision+joints) + training-only SSL: joints help in-distribution & same-lab but shortcut cross-lab (joint-dropout fixes it); joints-alone don't transfer force. Lit review (FACTR2) + report. Wiki §2.1.6.

★ Video → Force — main deck (start here) →

Consolidated feasibility + Stage 1–5: motivation · goals · datasets (incl. our 3-task set) · supervised vs SSL · in-dist ceilings · SSL generalization (RankMe + UMAP/t-SNE)

▶ Sinew — 3-minute pitch (video → force) →

Tight narrative: manipulation is impressive but data-bound · collected data is vision+actions only · why force (with clip) · we recover force for robot learning · foundation model ≈14× Meta Sparsh · market · moat · GTM · team · who we've worked with

Sinew — business plan (for NAVER D2SF) →

10-slide business-plan: market · customers · model · pricing · competition/moat · GTM · roadmap · ask — naver-d2sf palette

Sinew — technical report (for D2SF) →

Condensed sell-deck: the opportunity · data moat · supervised feasibility · self-supervised generalization · verdict

Video2Force — stage reports ▾

Stage-6 E2E encoder-unfreeze (in-dist 0.97 · few-shot OOD 0.79) · Stage-5.6 force↔frozen-V-JEPA alignment · Stage-5.5 do SSL embeds earn their keep · Stage-5 video↔force gap closing · Stage-4 SSL embedding quality · Stage-3 OOD=in-dist · Stage-2 latent health · Stage-1 feasibility

★ Stage-7-redo — richness curation, visual↔force SSL, the direction fix (latest) →

Zero-shot to a held-out lab now matches in-dist on both metrics — but not via SSL (video↔force align is neutral; the unified law holds). Wins from supervised levers it surfaced: a force-frame bug fix, a label-free force-richness data metric (SNR→contact, dchg→direction), and frozen-encoder direction. OOD contact-F1 0.86 ≈ in-dist; OOD direction +0.76 (2× the Stage-7 champion). Two-head split: contact-unfrozen / direction-frozen · 2026-06-25

Stage-6 — E2E V-JEPA2 encoder-unfreeze →

Fine-tune the encoder end-to-end on streamed HF video: in-dist contact-F1 0.26→0.97 (broke the frozen 0.78 ceiling); cross-lab zero-shot via EE-frame direction-unification + augmentation plateaus 0.47; few-shot crisp adaptation (~30 eps) breaks the OOD ceiling → held-out F1 0.79 / dir 0.75 · 2026-06-20

Stage-5.6 — aligning force to a frozen visual encoder →

Bind a force student to frozen V-JEPA: contact cross-modal swap SOLVED via new Sinkhorn-OT (champion 0.831 ≈ teacher ceiling) — refutes Stage-5.5's swap-fail; direction stays frozen-bound (within-vis 0.575 → encoder-unfreeze); latent distance ≠ transfer (best arm has the largest modality gap) · 4 iters · 2026-06-19

Stage-5.5 — do the SSL embeddings earn their keep? →

Validation ablation on the Stage-5 champion: SSL embed never beats raw V-JEPA for v2f or BC, cross-modal swap fails (alignment geometric not task-semantic) — but force embed is non-trivial & can drive a policy; lever = encoder-unfreeze · 2026-06-12

Stage-5 — closing the video↔force modality gap →

Frozen-trunk SSL: genuine cross-modal alignment (CKA 0.07→0.51, centered-cos 0.014→0.51, MMD 1.30→0.11) via cosine+CORAL on raw V-JEPA; goal-2/3 frozen-feature ceilings · 2026-06-12

Stage-4 — high-quality SSL embeddings →

Frozen-trunk SSL scorecard + 7 latent figures: global FMB intermix (mmd 1.25→0.21) + meaningful force embeds (probe AUC 0.98) + multi-cam law (+0.07 informative views) + same-view cohesion; local-kNN appearance & v↔f retrieval are frozen/physics walls · 2026-06-10