sinew · progress

force-from-vision on FMB · R2S2R pipeline · 2026-05-22 snapshot

research 33/33 wave 1 sim · 11/11 wave 2 RL · ⅖ phases wave 4 v2f scaffold wave 4 v2f training wave 3 data-gen wave 4b real fine-tune

Companion to the v2 exec deck at /slides/. Live task board at /tracker/.

TL;DR — where we are, where we're stuck

end goal

video2force (v2f) — predict 6-D EE wrench from RGB(+depth) for downstream contact-rich policies. End deliverable: a model + benchmark on FMB.

how we get there

RL = data factory (not deployed policy). Sim F/T = direct label via simulated wrench. Visual sim2real gap: χ² = 1.88 pre-Tier 1 (FoAR-style threshold = 1.0).

active blockers

DL_A6000 reboot pending → PPO Phase 2-5 (sinew-35) + v2f stage-2 training (sinew-55) both queued behind it
χ² re-measurement post-Tier 1 (sinew-39) — open, validates that the visual fixes closed the gap

Pipeline at a glance

Epic ledger (bd-as-truth)

50 bd issues · 45 closed / 3 in_progress / 2 open

Research v1 + v2 sinew-1 + sinew-5 epics

33/33

Wave 1 · Sim foundation sinew-6 Epic A

11/11

Wave 2 · RL data factory sinew-28 Epic B

PPO P1 ✓ · P2-5 ⏳

Wave 3 · Data generation Epic C — placeholder

not started

Wave 4 · v2f training sinew-40,42,44,45,47,52,55

scaffold ✓ · train ⏳

Wave 4b · Real fine-tune Epic E — placeholder

deferred

Live state: /tracker/ (kanban + list, polls every 10s)

Research — DONE · 33/33

Two waves of closed sprints

v1 · sinew-1.1..1.5 + sinew-5.1..5.14 (14 issues)

FMB benchmark deep-dive (obs / act / control / dataset schema)
isaac_twins audit + RL-side wrapper design
force-from-vision landscape: 7 papers, FoAR/Force-from-Motion as closest priors
EE-frame pinned as canonical wrench convention

v2 reframe · sinew-5.16..5.23 (8 issues + synth)

BC warmstart DROPPED after χ²=1.88 visual-gap measurement (sim/real not interchangeable enough for behaviour cloning)
RL repositioned as data factory, not deployed policy
Sim F/T → direct label (no force prediction during RL)
SimDist action-corruption recipe locked: dropout / jitter / dead-zone
v2 reward Φ_insert specified with clean-wrench dot products
N=5 rliable evaluation protocol (IQM + 95% bootstrap CI)

All 33 closed. Output: docs/r2s2r_research_report.html §12 = binding implementation plan.

Wave 1 · Sim foundation — DONE · 11/11

Epic A (sinew-6, feature). SimWorker primary · RLWorker support on reward.

Sim plumbing (SimWorker)

A.1 SubstageDetector class (5 predicates: grasp / lift / place / regrasp / insert)
A.2 Threshold-envelope startup assertion
A.3 Inverted-physics sanity probes (5/primitive)
A.4 Temporal-smoothness check on detector predicates
A.5 peg_tip_local_offset baked on peg USDs
A.6 read_eef_wrench_ee w/ clean + noisy paths
A.7 Batched grab_franka_view(num_envs)
A.8 Per-env reset_episode(env_ids)
A.9 Observation packager isaac_twins.fmb.obs.get_obs

Reward + verification (RLWorker + SimWorker)

A.10 Φ_insert v2 reward function (RLWorker)
A.11 Reward-decomposition logging hook (RLWorker)
sinew-50 validation: SubstageDetector predicates against sinew-5.20 replay traces → GREEN

SubstageDetector independently validated by replaying FMB ground-truth trajectories in sim — predicates fire at correct primitive boundaries.

Wave 2 · RL data factory — PARTIAL · PPO P1 ✓

Epic B (sinew-28). RLWorker primary.

Done (Phase 1 + supporting infra)

Issue	Title	Result
`B.0`	Epic A integration smoke	FmbInsertionEnv + reward end-to-end green
`B.1`	FmbInsertionEnv → `DirectRLEnv` subclass	landed
`B.2`	EEDeltaCorruptedActionMapper	SimDist recipe wired
`B.3`	Sub-expert ckpt preservation	50-iter cadence
`B.4`	Curriculum scaffolding	phase advance on IQM > 0.7
`B.5`	PPO Phase 1 (grasp-only)	IQM 0.83 / N=5 seeds (0.81–0.85)
`sinew-51`	wandb per-stage + per-peg success	live on `sinew-r2s2r/phase-1`
`sinew-56`	EE-delta DIK init bug	fixed (was crashing every env reset, falling back to joint-delta)

Phase 1 IQM = 0.83 (mean 0.83, range 0.81–0.85 across N=5 seeds). Comfortably above the 0.70 curriculum-advance gate.

Wave 2 · PPO Phase 2–5 — IN PROGRESS

⏳ IN PROGRESS — content TBD when training run completes

sinew-35 · PPO training run, phase 2-5 (full chain)

Assignee: RLWorker. Currently blocked on DL_A6000 reboot pending. Sub-stages 2 (grasp + lift), 3 (+ place), 4 (+ regrasp), 5 (+ insert) train sequentially via the curriculum gate from Phase 1.

What this slide will hold once landed

Per-phase IQM + 95% CI table (N=5 seeds each)
Curriculum advance timeline (steps until IQM ≥ 0.7)
Per-peg success rate at phase 5 (rect / big_long / pegs from FMB taxonomy)
Action-corruption ablation if SimDist regions resolve before write-up
wandb run URLs under sinew-r2s2r/phase-{2,3,4,5}

Live status: /tracker/#sinew-35

Wave 4 · v2f scaffold — DONE

VisionWorker. Pre-data scaffolds for the force-from-vision predictor.

Scaffold issues (closed)

Issue	What
`sinew-40`	v2f model arch (28 M params per sinew-43 revision)
`sinew-42`	train loop
`sinew-44`	dataloader
`sinew-45`	eval harness
`sinew-47`	wandb wrapper + branch consolidation
`sinew-52`	FmbEpisodeSource adapter (commit adae620)
`sinew-54`	FMB-real subset: 193 eps / 17 GB / 50 shards filtered

Verification

58/58 tests green across model / train / dataloader / eval
FmbEpisodeSource remaps FMB-real keys → v2f schema (tcp_force/torque, RGB(+depth) per cam, action delta-pose, gripper bit)
EE-frame wrench convention enforced at the adapter boundary (per sinew-1.5 lockdown)

Pre-data work fully landed. Training is the next step — gated on compute, not on code.

Wave 4 · v2f training — IN PROGRESS

⏳ IN PROGRESS — content TBD when stage-2 run completes

sinew-55 · v2f stage-2-from-scratch experiment on FMB-real

Assignee: VisionWorker. Parallel to PPO Phase 2-5, also blocked on DL_A6000 reboot. Goal: validate the scaffolded train loop on the 193-episode FMB-real subset before scaling to full sim pretrain.

What this slide will hold once landed

train/val loss curves (MSE on 6-D wrench, EE-frame)
per-axis error breakdown (Fx,Fy,Fz / Tx,Ty,Tz)
error vs contact-state stratification (free-space / contact / insertion)
direction-only baseline comparison (per Direction-Matters prior art)
FoAR-threshold comparison (does ‖F̂‖ > 1 N correlate with ground-truth contact?)
wandb run URL

Live status: /tracker/#sinew-55

Wave 3 · Sim data generation — NOT STARTED

📋 PENDING — Epic C placeholder, runs after PPO Phase 5 lands

Goal

Collect a large sim rollout dataset (target ≥10 k episodes) using the converged Wave 2 PPO policy as the actor, with full ground-truth labels per sinew-5.19 spec: RGB(+depth) per camera, tcp_pose / tcp_vel / clean wrench / per-axis joint vel / contact map / segmentation / substage labels.

What this slide will hold once Wave 3 starts

Recording cadence (Hz capture vs Hz step)
Storage layout + sharding strategy
Episode-level stats: success rate, length distribution, peg/board diversity
Train/val/test split protocol
Capacity numbers (GB on disk vs episodes)

Spec: docs/research/sim_recording_spec.md. Dependency: Wave 2 Phase 5.

Wave 4b · Real fine-tune — DEFERRED

📋 DEFERRED — Epic E placeholder, awaits sim-pretrain convergence

Plan

Once Wave 4 sim-pretrain v2f converges, fine-tune on the FMB-real subset (sinew-54: 193 eps / 17 GB / 50 shards) to close the visual sim2real gap on real-world frames.

Why deferred (not skipped)

FMB-real subset is already pulled + filtered (sinew-54 closed)
Adapter (FmbEpisodeSource) already mapped to v2f schema
But sim-pretrain hasn't started, so fine-tune target weights don't exist yet
Tier 1 scene fixes (sinew-36/37/38) closed the χ² gap → fine-tune burden should be smaller; sinew-39 will re-measure χ² to confirm

What this slide will hold once Wave 4b starts

Sim-pretrain → real fine-tune transfer error
Real-only baseline comparison
Per-axis error on held-out real episodes

Visual sim2real gap — pre/post Tier 1

Pre-Tier 1 (research-phase measurement)

per-cam χ² range: 1.35–2.10
worst at wrist (closest to insertion contact, biggest texture mismatch)
mean χ² = 1.88 vs FoAR "visibly distinct" threshold = 1.0
concrete enough to motivate dropping BC warmstart

Tier 1 fixes (Wave 1, all CLOSED)

sinew-36 procedural cable mesh in wrist field-of-view
sinew-37 lab-clutter distractor objects
sinew-38 background workshop texture

sinew-39 OPEN: re-measure χ² post-Tier 1 to confirm gap closed (Researcher). Until then, the "Tier 1 worked" claim is untested.

Figure: docs/research/figures/sim2real_visual_gap_grid.png (4×2 cam grid, sim/real side-by-side). Memo: docs/research/sim2real_visual_gap.md.

Infrastructure — live

Public hosting

https://sinew.tail23daca.ts.net/ — landing
/slides/ — v2 exec deck (30 slides)
/report/ — long-form research report
/tracker/ — live bd kanban + list (10s poll)
/progress/ — this deck

Stack: python3 stdlib HTTP server (subclass) + tailscale funnel. Dynamic /tracker/data.json shells out to bd list --json with 3 s in-memory cache.

5 active GH repos

sinew-robotics/isaac_twins — sim scene + USDs (SimWorker)
sinew-robotics/isaaclab_sinew — RL env + PPO (RLWorker)
sinew-robotics/sinew_video2force — v2f predictor (VisionWorker)
sinew-robotics/sinew_r2s2r_web — this site + docs (WebWorker)
bd issues (local sqlite, JSONL committed)

Per-agent git identity, no user-co-author lines. Branch pattern: <agent>/<bd-id>-<slug> → --no-ff merge to main.

Open work · bugs · next-up

45closed

3in_progress

2open

50total bd

currently in_progress

sinew-35 RLWorker — PPO Phase 2-5 · blocked on A6000 reboot
sinew-55 VisionWorker — v2f stage-2 training · blocked on A6000 reboot
sinew-58 WebWorker — this deck (closing on push)

open (not yet claimed)

sinew-28 Epic B feature — closes when Phase 2-5 lands
sinew-39 Researcher — re-measure χ² post-Tier 1

known bugs (backlog)

sinew-57 P3 — disk-gate calibration in deployment runbook

Next steps

⏳ Gated on DL_A6000 reboot

Two parallel tracks once compute returns

Track	What	Owner	bd
RL	PPO Phase 2-5 (lift → place → regrasp → insert)	RLWorker	`sinew-35`
v2f	stage-2-from-scratch training on FMB-real subset	VisionWorker	`sinew-55`

After both land

Wave 3 starts (sim data-gen with converged PPO actor)
sinew-39 re-measures χ² post-Tier 1; informs Wave 4b fine-tune burden
This deck updates: WIP placeholders → real charts

Refresh cadence: this deck regenerates on demand (no auto-build yet — see sinew-22 backlog for critical-path overlay on the tracker). Source: sinew_r2s2r_web/src/progress_slides.html.

summary

research locked · sim stable · RL on Phase 1 · v2f scaffold green · two training runs queued behind compute

33 / 33 11 / 11 ⅖ scaffold train ⏳ datagen fine-tune

snapshot 2026-05-22 · live tracker · v2 exec deck · research report