sinew R2S2R

The deliverable

What ships

• predictor_real_finetune.pt - DINOv2-S backbone, RGBD patch embed, 4 heads (force-direction, 6D wrench, contact-point, in-contact gate)
• Inference at 10 Hz from 2 D405 cameras (side_left + wrist_left) on real FMB testbed
• Trained on a sim corpus generated entirely inside the sinew stack, fine-tuned on the FMB-real insert subset

What the predictor does NOT consume

• No F/T input - RGBD in, wrench out. Single-direction flow.
• No proprio. No commanded action. No history beyond a per-frame RGBD tensor.

Why force-from-vision

Contact-rich manipulation - peg insertion, assembly, tool-use - depends on force feedback. Real F/T sensors are noisy, slow, and absent on most low-cost hardware.

FMB upstream is a benchmark for contact-rich insertion but ships no force-prediction baseline. sinew fills that gap with a predictor trained on sim-generated (image, force) pairs and anchored on the FMB real subset.

Three building blocks, one ship artifact

Two gaps, separately handled

Visual and F/T sim-to-real gaps need different fixes. Conflating them is the easy mistake.

Predictor sees clean sim images, learns to match noisy real-Franka F/T. The visual gap requires a real fine-tune; the F/T gap is handled entirely inside the sim label pipeline.

Sim environment foundation

Goal 2 - isaac_twins + isaaclab_sinew

cfg = FmbInsertionEnvCfg()  # defaults: fmb_big_demo + big_long/rect peg
                            # + side+wrist cams
env = FmbInsertionEnv(cfg)
obs, info = env.reset()

# obs keys: side_left, side_right, wrist_left, wrist_right (RGBA 720x1280),
#           q (7,), dq (7,), tcp_pose (7,),
#           tcp_force (3,), tcp_torque (3,),
#           gripper_pose (1,)

for _ in range(N):
    action = policy(obs)          # 7-vec in [-1, 1]
    obs, rew, term, trunc, info = env.step(action)

Key API surface

• read_eef_wrench_ee(art, sensor, *, noisy, state, rng) - the sim F/T pipeline (clean + noisy paths)
• SubstageDetector at isaac_twins/src/isaac_twins/fmb/substage.py - 5 primitive predicates, single source of truth for reward + labels
• grab_franka_view(num_envs), SceneConfigurator.reset_episode(env_ids) - multi-env handles

How we detect substages

Goal 2 - sim_substage_detection.md + addenda Q1/Q2

Three ContactSensors cover all five primitives

• finger_contact - antagonistic finger-force pattern → grasp closed
• peg_contact - per-partner zero/non-zero distinguishes floor / fixture / hole. The substage-defining signal.
• board_contact - sanity heartbeat (board should stay seated)

Predicate accuracy bounds (full-state detector)

Detector is single-source-of-truth for reward gates AND recorded v2f labels. Threshold-envelope assertion + inverted-physics probes + temporal-smoothness check land pre-RL (~2 days, addenda Q1).

Sim F/T sensor pipeline

Goal 2 - sim_ft_sensor.md - replicates Panda K_F_ext_hat_K

Two callsites, one API

• noisy=False → reward, SubstageDetector, v2f direction-head label (geometry-derived)
• noisy=True → policy obs, v2f wrench-head label (matches real Franka distribution)

Force sigma=0.025 N/axis, torque sigma=0.01 Nm/axis (Franka resolution /2). Per-episode bias drift sigma=0.05 N / 0.02 Nm. LP lag tau_lag = U(20, 80) ms.

RL as a data factory

Goal 1 - rl_revised_plan.md + sim_dist_review.md

Yield gates (data-gen pass)

• ≥ 1.6M (image, force) pairs total
• ≥ 5% contact-transient frames
• F/T magnitude coverage [0, 30] N
• Per-cam diversity chi2 ≥ 0.3

Sim dataset shape

Goal 2 - sim_recording_spec.md + rl_revised_plan.md 4

Per-episode trajectory labels (4-way)

v2f architecture (~43M params)

Goal 3 - v2f_arch.md + v2f_pipeline_revised.md - 22M trainable + 21M frozen

Four heads

"Force direction transfers sim→real; magnitude does not" (Direction Matters) - dictates head weights, freeze list, and DR-vs-no-DR split.

v2f training schedule

Goal 3 - v2f_pipeline_revised.md 3

Why this partition

• Direction transfers - geometric constraint normals are sim/real-identical
• Magnitude doesn't - real F/T has payload model error, gravity-comp residual, thermal drift
• Contact-point doesn't have a real label - no per-pixel "contact happened here" ground truth on FMB-real
• Backbone frozen at stage 2 - protects visual-DR features from catastrophic forgetting

Stage 1: 300 epochs, AdamW lr=3e-4 cosine + 2k warmup, bs=128, bf16. Stage 2: 30-50 epochs, lr=3e-5, no warmup. Output: predictor_real_finetune.pt - the ship artifact.

Visual sim-to-real strategy

Goal 3 - v2f_dr_spec.md + sim2real_visual_gap.md - heavy visual DR, NO dynamics DR

Critical knob: depth dropout on low-texture pixels. D405 is passive stereo - textureless 3D-printed pegs give sparse depth in real but clean in sim. Without this, any RGBD head will be sim-tuned.

Five hard-required categories: lighting + color jitter + material BRDF + cam intrinsic jitter + background clutter. Tier-1 scene fixes (cable mesh, lab clutter, background plane) land alongside DR to drive chi2 from 1.88 to ~1.4.

The sim-to-real visual gap

Goal 3 - sim2real_visual_gap.md - chi2=1.88, FoAR threshold for "visibly distinct" is 1.0

Per-camera χ² (worst at wrist) — FoAR "visibly distinct" threshold = 1.0

Headline: wrist cams have the worst gap because they see hand, fingers, board screws, peg layer-lines - all of which the sim does not model. This is what makes a real fine-tune non-optional and what the contact-point head freezes against.

Sim vs real - 4-camera grid

Scene fix plan

Goal 3 - substage_and_scene_addenda.md Q3 - leverage x inverse-cost ranking

Tier 1 (~2.5 days, biggest leverage)

Expected χ² trajectory — before → after

Out of scope: PathTracing (3-5x render cost), hand simulation (stage-2 FT carries this), photographed bin texture (not on critical path).

Six epics, 37 tasks, ~26 wall-days floor

impl_epic_plan.md - one wave per epic; Wave 4 splits into 4a sim pretrain + 4b real fine-tune

Aggregate: ~7 GPU-days across Epics B + D + E; ~26 wall-days floor if teammates fully available + GPU not contested; ~45 days realistic with contention.

Dependency graph

Epic A - Wave 1 sim surface foundation

~4 wall-days - 0 GPU-days - SimWorker (+ RLWorker)

Deliverables (11 tasks)

1. SubstageDetector class at isaac_twins/src/isaac_twins/fmb/substage.py per spec - offline unit tests + Kit runtime smokes
2. Threshold-envelope startup assertion + inverted-physics probes (5 per primitive) + temporal-smoothness check
3. Bake peg_tip_local_offset attribute on each peg USD via the author script
4. read_eef_wrench_ee(art, sensor, *, noisy, state, rng) - stateful when noisy=True, stateless when noisy=False; smoke: push peg into board, wrench grows monotonically with depth
5. Multi-env handles: batched grab_franka_view(num_envs), SceneConfigurator.reset_episode(env_ids), observation packager
6. Phi_insert reward function (clean wrench, PBRS form) - offline unit tests against synthetic histories
7. Reward-decomposition logging hook - every 10K steps writes all 8 reward components + per-primitive share dict

Pass criterion: Integration smoke runs FmbInsertionEnv -> reset -> 100 steps -> reward returns correct decomposition AND read_eef_wrench_ee returns non-zero on contact in the same loop. Test suite green.

Epic B - Wave 2 RL data factory

~6 wall-days + ~4 GPU-days - RLWorker (+ SimWorker, Researcher)

Deliverables (10 tasks)

1. Promote FmbInsertionEnv to DirectRLEnv subclass; multi-env smoke at N=4
2. EEDeltaCorruptedActionMapper subclass with per-DOF Gaussian burst; gripper bit excluded; per-env sigma fixed at run start
3. Sub-expert checkpoint preservation every 50 PPO iters -> 11 .pt files (iters 0, 50, ..., 2000)
4. Curriculum scaffolding - phase advance on IQM > 0.7, eval cadence every 100K steps
5. PPO training run - phase 1 (grasp-only), 1.5 GPU-day, N=5 seeds
6. PPO training run - phase 2-5 (full chain), 2-3 GPU-days, full curriculum
7. Scene Tier 1 fixes #1-3 in parallel (SimWorker): procedural cable mesh + lab clutter + workshop background
8. Re-measure chi2 post-Tier 1 (Researcher) - target drop from 1.88 toward ~1.4

Pass criterion: 11 sub-expert + 1 expert PPO checkpoint on disk for full grasp->insert curriculum; post-Tier-1 chi2 re-measured and documented; scene fixes committed.

SimWorker scene fixes run alongside RLWorker PPO training (different repos, no contention).

Epic C - Wave 3 data-gen pass

~3 wall-days + <1 GPU-day - SimWorker (+ RLWorker)

Deliverables (7 tasks)

1. FmbDataRecorder outer loop - burst-noise applied here (NOT during PPO training); 10-episode test corpus validated
2. HDF5/zarr schema additions: 4 obs/sim_* keys (sim_action_noised, sim_policy_iter, sim_never_noised, sim_noise_std) + 4 episode-meta labels (successful, disturbed, failed, clean_expert)
3. episode_uploader.py - nohup-resumable, plain FTP via curl --ftp-method nocwd, idempotent on NAS
4. Data-gen yield eval script - reports total pairs, contact-transient fraction, F/T distribution, per-cam diversity chi2
5. Small pilot - 100 episodes; label distribution matches expectation
6. Full sweep - 5k-10k episodes, ~150-300 GB on NAS at 224 2mixed_rgbd
7. Build TFDS sinew_fmb_v2f shards on A6000 - tfds.load succeeds locally

Storage budget

• ~6 MB per 50-step episode at 224 2mixed_rgbd; 22k FMB-equivalent corpus ~243 GB (45% of FMB upstream's 545 GB zip)
• NAS: ftp://143.248.121.169:7002/IntelligentManipulationTeam/DomrachevIvan/sinew/recordings/

Pass criterion: Sim corpus on NAS (label distribution per spec; yield eval green); TFDS shards verified loadable on A6000.

Epic D - Wave 4a v2f sim pretrain

~3 wall-days + ~1-2 GPU-days - VisionWorker

Deliverables (4 tasks)

1. V2FPredictor model class: DINOv2-S frozen + 4-ch RGBD patch embed + 2-layer cross-cam fusion + 4 heads (~43M params, 22M trainable)
2. Training script - AdamW lr=3e-4 cosine + 2k warmup, bs=128, 300 epochs, bf16; checkpoint save every 50 epochs; one-epoch wall-time matches budget (~7 min for 5k episodes)
3. Stage 1 sim pretrain run on A6000 - all 4 heads trained; gate-head loss upweighted 1.5x on noised steps
4. Per-shape stratified eval on sim test split - 9-row per-shape table, identifies worst shape, per-shape gate F1

GPU budget

• Single A6000, bf16, batch 128
• Stage 1: ~1-2 A6000-days for 300 epochs over 5-10k episodes
• Held-out validation: clean_expert subset (~1.25% of corpus)

Pass criterion: predictor_sim_pretrain.pt on A6000; sim-test direction-acc > 0.85 (loose sanity bar; if not hit, pipeline is broken).

Epic E - Wave 4b v2f real fine-tune

~3 wall-days + ~1 GPU-day - VisionWorker (+ Researcher)

Deliverables (5 tasks)

1. fmb_parse.py - filter chain + zero-bias subtract + 8N gate threshold + (qx,qy,qz,qw) + BGR->RGB + cam renames; parses 100 episodes without warning
2. Pull FMB insert-only filtered subset - ~20-35 GB via GCS HTTPS-range; 15-30 shards; ~3000-4000 episodes; per-shape distribution recorded
3. Stage 2 real fine-tune run on A6000 - direction + gate heads only; backbone, wrench, contact-point heads frozen; 30-50 epochs, lr=3e-5, no warmup
4. Per-shape stratified eval on FMB-real test - direction-acc + gate F1 table; worst-shape direction-acc ≥ 0.45 (acceptable tier)
5. Branch decision per outcome matrix A-E (team-lead + VisionWorker) - written addendum, next-epic plan adjusted

GPU budget

• ~1 A6000-day for stage-2 fine-tune (30-50 epochs at 1/10 stage-1 lr)
• Output: predictor_real_finetune.pt - the ship artifact

Pass criterion: Real-test direction cos-sim ≥ 0.60 global, ≥ 0.45 worst-shape (acceptable tier); aspirational ≥ 0.70 / ≥ 0.60. Outcome decision recorded.

Epic F - Wave 5 real-robot eval (hardware-gated)

~7 wall-days - 0 GPU-days - VisionWorker (+ User) - DEFERRED until hardware access confirmed

Deliverables (5 tasks)

1. Hardware setup confirmation - Franka + 4x D405 + FMB workspace match sim layout within +-5 mm
2. Deploy v2f predictor inference on real D405 streams at 10 Hz; publish to ROS topic / shared mem
3. Collect 50 real rollouts with predictor running alongside real F/T ground truth
4. Real eval bench - predicted vs real direction-acc + gate F1 + per-shape stratified report card
5. Final results memo + decision on next steps - docs/research/final_real_eval.md

Open chunks

• Real-robot data-collection pipeline
• Safety envelope review (presumed via FMB testbed; safety task added if not)
• Impedance gains tuning

Pass criterion: 50-trajectory real eval done; per-shape direction-acc table on real Franka; final memo committed.

RL eval protocol (IQM + bootstrap CIs) carries over for real-eval reporting.

Open questions and unknowns

Each open item has a "when" trigger - they enter the plan on a specific event, not on a schedule. The chi2 re-measurement is the most important; it gates how aggressively we interpret v2f real-test numbers.

Compute envelope summary

Cross-cutting

• Critical path: A -> B -> C -> D -> E -> F. Each wave gates the next.
• Intra-wave parallelism: Epic B (SimWorker scene fixes alongside RLWorker PPO), Epic E (Researcher parser alongside VisionWorker training).
• Single A6000 covers the entire training budget.
• Realistic with GPU contention + integration churn: ~45 days end-to-end.