sinew R2S2R — Force-from-Vision on the FMB Testbed

4.1 Evaluation protocol

Owner: RLWorker · sinew-5.2 · docs/research/rl_eval_protocol.md · Refs: references/eval/rliable, Andrychowicz 2020.

The R2S2R compute budget forces a permanent few-run regime (3–10 seeds per config). Per rliable §4, point estimates at this seed count carry only 50–70% probability of being real improvements. The protocol stays defensible by reporting interquartile mean (IQM) with stratified bootstrap CIs.

Logging cadence

PPO defaults (Plan A, Andrychowicz "competitive base")

SAC is the fallback only if PPO plateaus below 50% success after the 5-seed pass.

Two-stream evaluation under the reframe

Per sinew-5.22 §5 the eval surface forks into two streams:

• PPO training eval — unchanged. The sinew-5.2 protocol gates curriculum advancement.
• Data-gen pass eval — new. The product is (image, force) pairs, not return. Yield gates: ≥ 1.6M total pairs, ≥ 5% contact-transient frames, F/T magnitude coverage [0, 30] N, per-cam diversity χ² ≥ 0.3.

4.2 Substage detection and gate verification

Owner: SimWorker · sinew-5.3 · sinew-5.19 · addenda Q1+Q2 (2026-05-21) · sim_substage_detection.md + substage_verification.md

FMB upstream has no automated substage detection — the operator hits Enter to advance primitives at rollout time (sequential_rollout.py:250). Sim has full ground truth, so deterministic detectors are tighter than real; deltas are documented intentional.

Sensor surface

Three Isaac Lab ContactSensor instances using filter_prim_paths_expr cover all five FMB primitives plus transitions:

• finger_contact — sense = panda_(left|right)finger, filter = peg. Antagonistic finger-force pattern → grasp closed.
• peg_contact — sense = peg, filter = [board, fixture, both fingers, Bin]. This is the substage-defining signal: per-partner zero/non-zero distinguishes "peg on floor" / "peg on fixture" / "peg in hole".
• board_contact — sense = board, filter = Bin. Sanity heartbeat — board shouldn't tip during insertion.

Per-primitive predicates

The SubstageDetector class lives at isaac_twins/src/isaac_twins/fmb/substage.py. It exposes {p}_success(), {p}_failure(), transition_ready(p) (success AND TCP at z_safe ≥ 0.20 m), and diagnostics(). Reward authors never recompute distances — the detector is single source of truth, eliminating reward-vs-eval drift by construction.

Q1 (addenda): are the gates meaningful before execution?

Beyond the three core tests already specced in sinew-5.3 §6 (offline unit, runtime smoke, negative case), four additional checks are ranked by ROI. The recommendation lands three before RL kickoff (~2 days total):

Landing the threshold-envelope + inverted-physics + temporal-smoothness checks lifts predicate confidence high enough to trust the recorded obs/sim_substage_predicate as a v2f gate-head training label.

Q2 (addenda): FP/FN rates and how to combine state-only with full-state

The gap is big — full-state has 1–15% error, state-only has 5–50%, with rotate and insert worst because peg orientation isn't observable from state alone.

Recommendation: option (d) for the training loop, (b) for offline audit. Full-state detector is canonical for reward + recorded labels. State-only adapter is offline calibration only — never enters training. Mixing them as a confidence-weighted auxiliary loss (option a) would inject the state-only 5–50% error into the reward gradient, correlated with the actual physics on the weak primitives (rotate, insert) — exactly the wrong shape for reward shaping. State-only's only role is flagging audit-worthy disagreements (option b) and measuring sim-bias when state-only is the proxy for FMB-real labels at stage 2 (option c).

4.3 Reward design v2 (Φ_insert with clean wrench)

Owner: RLWorker · sinew-5.1 superseded in part by sinew-5.22 §1 · rl_reward_design.md + rl_revised_plan.md

Composition: dense potential-based shaping + sparse substage bonus + small action regularizer. Per-primitive activation — only the current primitive's reward contributes per tick.

R_p(s, a, s') = λ_success · 1[p_success(s')]       # sparse terminal
              - λ_failure · 1[p_failure(s')]       # sparse anti-terminal
              + γ · Φ_p(s') - Φ_p(s)               # dense potential-based (Ng 1999)
              - λ_action  · ||a[:6]||²             # small action regularizer
              - λ_clip    · Δsafety_clip_count    # safety-box pressure

PBRS form preserves the optimal policy (Ng 1999): cumulative dense return ≈ Φ(final) − Φ(initial), so the policy cannot bank arbitrary shaping reward.

Φ_insert v2 — the load-bearing change post-reframe

Under the reframe sim F/T is a real, reliable signal via read_eef_wrench_ee(art, noisy=False). The v1 "degenerate-when-F/T-zero" branch is dropped — clean wrench is non-zero only in real contact, so the force term naturally degenerates.

d_xy      = ||peg_tip_xy - hole_xy||
d_z       = max(0, peg_tip_z - hole_z_bottom)
align     = 1 - cos²(peg_long_axis, hole_axis)
f_clean   = read_eef_wrench_ee(art, sensor, noisy=False)[:3]   # (3,) clean N, EE frame
f_mag     = ||f_clean||

Φ_insert(s) = -d_xy
            - 1.5 · d_z · 1[d_xy < r_align_xy]
            - 1.0 · align · 1[d_z < z_align_thresh]
            - 0.2 · f_mag · 1[d_xy < r_align_xy AND d_z < z_align_thresh]

Coefficient lifted 0.1 → 0.2 because the signal is now reliable. The force-term gate ensures we only penalize contact force when we should be making controlled contact (inside the alignment and seat-depth window); outside that window force is exploration cost and is not penalized.

Constants (starting values, all tunable)

Curriculum

1. Phase 1: grasp-only. Pass: N=5 seeds, IQM > 0.7, 95% CIs not crossing 0.5.
2. Phase 2: + place_on_fixture.
3. Phase 3+: + rotate, + regrasp, + insert one at a time.

4.4 BC dropped — historical record

Owner: RLWorker · superseded by sinew-5.22 §2 · originals: il_bc_warmstart.md + rl_revised_plan.md §2.

The original plan (sinew-5.12) recommended Option C: BC warmstart → PPO/SAC fine-tune using the 22,550 FMB demos. The χ²=1.88 visual gap (sinew-5.16) invalidated the precondition that FMB-real images are drop-in compatible with sim-camera obs. Three options were considered post-reframe:

Curriculum + sub-expert mixture compensate for losing BC. The 22,550 FMB demos are kept on the project shelf for future real-robot work but do not enter the sinew RL training loop. The four data-side issues from the original sinew-5.12 spec (frame convention, quat order, F/T mismatch, BGR-on-disk) remain documented as a reference for any future BC revival.

4.5 SimDist disturbance recipe

Owner: Researcher · sinew-5.18, adopted by sinew-5.22 §3 · sim_dist_review.md + cloned code at isaac_twins/references/sim_dist/ (CLeARoboticsLab/simdist).

The only piece of SimDist sinew adopts is the action-only burst noise recipe. The latent world model + MPC planning machinery is out of v2f scope.

Adoption cost is ~2 days: an EEDeltaCorruptedActionMapper subclass that wraps the existing mapper (~0.5 d), the burst-state machine, and the HDF5 schema additions (sim_action_noised, sim_policy_iter, sim_never_noised).

4.6 Trajectory labels for v2f filtering

Owner: RLWorker · sinew-5.22 §4

The recorder writes four orthogonal per-episode labels so the v2f trainer can stratify by data quality at HDF5 load time. Predicted fractions per a 1.6M-pair data-gen pass:

Cross-tabulated: ~30% successful + clean, ~30% successful + disturbed, ~25% failed + disturbed, ~14% failed + clean. The trainer's default behaviour is to ignore the flags and train on the full pool (the distribution is already mixed); gate-head loss optionally upweights noised steps by 1.5× because contact-transient frames carry the cleanest gate signal.

5.1 Camera subset bench

Owner: SimWorker · sinew-5.5 · camera_subset_benchmark.md · 48-row JSON at isaac_twins/docs/research/camera_subset_benchmark.json

48-cell sweep: 8 camera subsets × {RGB, RGBD} × N ∈ {1, 4, 8}. Mixed-render steps/s:

Winner by consumer

RGBD costs +5–15% over RGB at fixed cam count — cheaper than the cliff above. The bench appendix documents the nohup + resumable-driver pattern that the NAS recorder inherits.

5.2 GT recording spec + sim F/T sensor (unified data surface)

Owner: SimWorker · sinew-5.6 + sinew-5.21 · sim_recording_spec.md + sim_ft_sensor.md

Sim F/T sensor pipeline (sinew-5.21)

Replicates Panda's K_F_ext_hat_K in four stages. New public API: read_eef_wrench_ee(art, contact_sensor, *, noisy, state, rng) → dict.

contact sensor             coord transform         DR noise model            output
    │                          │                       │                       │
    │ world-frame net force    │ rotate by R_world_EE  │ + bias + Gauss + lag  │
┌───▼───────────┐    EE-frame  ┌▼─────────────────┐    ┌▼──────────────────┐   │
│ clean F_w (3,)│ ── world→EE ─│ F_clean_ee (3,)  │ ──▶│ noisy_lagged_ee   │──▶│ obs/eef_force
│ clean τ_w (3,)│   adjoint    │ τ_clean_ee (3,)  │    │ (3,) + (3,)       │   │ obs/eef_torque
└──────────────-┘              └──────────────────┘    └───────────────────┘   │
                                       │                                       │
                                       ├──▶ obs/sim_eef_force_clean (3,)       │
                                       ├──▶ obs/sim_eef_torque_clean (3,)      │
                                       │                                       │
                                       ▼                                       │
                               ‖F_clean‖ > 0.1 N ───▶ obs/sim_in_contact (bool)

Integration discipline

Noise model parameters

Recording schema (FMB-parity + sim extras)

The recorder writes FMB-RLDS-parity keys (images, joint_pos/vel, eef_pose/vel/force/torque, action, primitive, language) plus namespaced obs/sim_* extras for v2f training:

• obs/sim_eef_force_clean, obs/sim_eef_torque_clean — pre-noise labels paired with the noised obs/eef_force/obs/eef_torque.
• obs/sim_in_contact — gate label from contact reporter.
• obs/sim_contact_point_local, obs/sim_peg_local_axis — contact-point head targets in peg-local frame.
• obs/sim_force_dir_ee — direction-head target, unit-vec EE frame.
• obs/sim_substage_predicate — per-primitive boolean vector from the detector.
• obs/sim_dr_profile_blob — JSON of per-episode DR knobs (replay reproducibility).
• obs/sim_action_noised, obs/sim_policy_iter, obs/sim_never_noised — SimDist columns.
• episode_metadata/{successful, disturbed, clean_expert} — trajectory labels.

Per-frame loop ordering (validator invariants)

1. F/T capture, clean-then-noised: read sim wrench → write obs/sim_eef_force_clean; apply DR noise (per-frame Gaussian + per-episode bias + per-episode lag) → write obs/eef_force.
2. Depth pipeline order: dropout (low-texture mask) BEFORE Gaussian noise BEFORE range-clip. Different order changes the noise distribution.
3. F/T bias and lag-τ sampled per-episode (constant within episode); frame noise per-frame.

Format and storage

224² matches DINOv2-S patches (16×14 = 224), saves train-time resize and ~23% disk. Even the v1 default (~243 GB) is ~45% of FMB upstream's 545 GB single-object zip.

5.3 FMB ↔ sim data matching

Owner: Researcher · sinew-5.7 · fmb_sim_data_match.md + validator isaac_twins/scripts/validate_fmb_schema.py

Three FMB schemas exist (raw .npy, RLDS, live gym env). The schema work locks one canonical sim record that's drop-in compatible with FMB's RLDS while adding sinew ground truth via the obs/sim_* prefix.

libfranka channels FMB drops (sinew picks them up)

libfranka's franka::RobotState exposes ~30 channels FMB ignores. High-value ones recorded under obs/sim_*:

• tau_J — direct link-side torque, better SNR than the external wrench estimator
• tau_ext_hat_filtered — low-pass filtered external torque
• cartesian_contact — per-Cartesian-dim contact bit (the load-bearing FoAR gate label)
• O_T_EE_d — last commanded EE pose (reveals controller tracking lag)
• time — strictly monotonic libfranka clock

Camera name mapping (sinew canonical)

Caveat: the wrist L/R mapping is a sinew choice — FMB upstream binds arbitrary serials. If real-Franka eval shows mirrored-wrist artifacts (policy reaches the wrong way), the fix is to flip the wrist mapping and retrain, not look for a bug elsewhere.

5.4 FMB trajectory replay in sim — honest negative

Owner: SimWorker · sinew-5.20 (honest negative) · fmb_replay_feasibility.md + isaac_twins/scripts/tests/test_replay_mechanism.py

Verdict: cannot reliably prove end-to-end FMB grasp+insert replay in current sim. This is the load-bearing reason substage verification falls back to a state-only adapter (§4.2 Q2), and the reason BC option (b) "BC on sim-rendered FMB-replay images" was rejected (§4.4).

Four structural blockers, none individually trivial:

1. No IK layer in the replay path — joint-replay drifts in EE space because the Plan A DifferentialIK that sinew-3 wired into the env isn't reused at replay.
2. Peg spawn is randomized; it doesn't match the FMB-recorded peg pose at the grasp moment, so the contact geometry doesn't line up.
3. STEP→USD tessellation deflection ~0.5 mm ≈ medium/small board clearances — tight-tolerance insertions are geometrically infeasible in sim.
4. FMB raw .npy (545 GB) not downloaded; only 5-frame cached smokes exist locally.

Mechanism smoke confirmed sim infra works end-to-end: scene builds, physics settles, gripper actuates 0.08 → 0.0002 m closed. Predicted yield shape-by-shape: ~30–50% best case, < 10% for asymmetric shapes. There's a ~4-day path to reliable replay if needed (proper IK in the replay loop, peg-pose forcing, board re-mesh, FMB raw pull), but it's not required for the v2f-end-goal pipeline because substage verification falls back to the state-only adapter for offline calibration.

5.5 NAS + DL_A6000 pipeline

Owner: SimWorker · sinew-5.8 · data_pipeline.md + draft isaac_twins/scripts/episode_uploader.py

local PC (4070 Ti SUPER, 16 GB)        NAS (143.248.121.169:7002, ftp)          DL_A6000 (24 GB+)
┌─────────────────────┐                 ┌──────────────────────────┐            ┌────────────────────┐
│ FmbRecorder         │                 │ /IntelligentManipulation │            │  pixi env          │
│   → zarr/episode_*  │  FTP (curl)     │  Team/DomrachevIvan/     │   FTP      │  → TFDS reader     │
│   → episode.zarr.   │ ─push────────►  │  sinew/recordings/       │ ◄─pull──── │  → train_v2f.py    │
│       tar           │                 │    2026-05-21/seed_07/   │            │                    │
│ episode_uploader.py │                 │    sinew/tfds/           │            │                    │
└─────────────────────┘                 └──────────────────────────┘            └────────────────────┘

Protocol locks (per user global CLAUDE.md)

Two-process model: FmbRecorder writes zarr atomically; episode_uploader.py watches and pushes per-episode opportunistically. Idempotent — mid-upload crash → next pass overwrites. Recording rate ~60 MB/min = 1.0 MB/s at 224², vs ~12 MB/s home-link ceiling — network is never the bottleneck. Even four parallel collectors stay well under.

5.6 Scene visual-gap mitigation (Q3 answer from the addenda)

Owner: SimWorker (addenda) over Researcher (sinew-5.16) · substage_and_scene_addenda.md §3 + sim2real_visual_gap.md

Researcher's measured baseline (sinew-5.16): χ² = 1.88 mean across 4 cams, wrist cams worst at χ² ~ 2.0 with real-edge-density 51–67% denser. Wrist cams see hand, cables, fingers, peg-print lines — none of which the current sim USD models. The addenda enumerates 10 candidate fixes and ranks them by leverage × inverse-cost. Two tiers land before the v2f stage-1 pretrain.

Tier 1 — do first (~2.5 days total)

Tier 2 — next ~2 days if budget

Defer or reject

Expected χ² trajectory (SimWorker estimate, not measured)

Honesty note: these Δχ² estimates are SimWorker fix-table predictions per the leverage analysis in sim2real_visual_gap.md §3. The original measurement is N=1 (one real episode, one timestep, one sim env, 4 cams). Re-measuring χ² post-DR + post-Tier-1 is the highest-priority sinew-5.16 follow-up. What scene authoring cannot fix — unmodeled lab clutter, lighting hardware noise, sensor-level rolling-shutter / chromatic-aberration artifacts — is absorbed by the stage-2 real fine-tune.

5.7 Parallel-development architecture

Owner: SimWorker · sinew-5.4 · parallel_dev_architecture.md

Two-repo separation

num_envs scaling roadmap

Multi-env RL gated on three SimWorker follow-ups (carried into Wave 1):

1. Batched articulation handle — grab_franka_view(num_envs) → Articulation wrapping /World/envs/env_.*/Scene/Robot regex.
2. Per-env reset — SceneConfigurator.reset_episode(env_ids).
3. Observation packager — isaac_twins.fmb.obs.get_obs(cfgr, art_view, sub_detector).

Until those land, single-env is the only contract. DirectRLEnv migration is then a rename-only ~200-line PR.

The hot rules (workspace CLAUDE.md proposed)

1. One author per docs/research/*.md.
2. No USD re-bake in a PR that also changes Python code.
3. isaaclab_sinew imports from isaac_twins only via the 8 published symbols.
4. Paired tickets for cross-repo work; never one bundled.
5. Topic branches per ticket; no long-lived feature branches.
6. Asset additions are additive — never rename or remove existing.
7. DR variants don't need new assets.
8. nohup + resumable driver for any Kit-loop sweep > 5 min.
9. bd ready is the conflict-avoidance gate — claim before editing.

6.1 Visual sim2real gap quantified

Owner: Researcher · sinew-5.16 · sim2real_visual_gap.md

Per-camera breakdown

Wrist cams have the largest gap (real-edge-density 51–67% denser) because wrist cams see hand, fingers, board screws, peg layer-lines — sim doesn't model the foreground. Side cams cover the larger workspace that the sim USD captures more faithfully.

Implications (load-bearing for the rest of Goal 3)

• Stage-1 sim pretrain cannot ship without the full DR knob set. A predictor trained on sim-only without DR deploys onto inputs that are 1.5× brighter, 0.7× edge-dense, 0.7× tonal-variance vs what it saw at training.
• Stage-2 real fine-tune is non-optional. The hand-in-wrist-cam slice cannot be simulated away.
• The "70% direction-acc bar" needs caveats. Until χ² is re-measured post-DR + post-Tier-1, the primary v2f metric is "monotonic improvement post-fine-tune," not absolute ≥ 0.70.
• The N=1 study is sufficient for the design decision — a 20-episode × 5-timestep study would tighten the mean by maybe 0.3–0.5, not 10×. Re-measurement is deferred to a sinew-5.13 follow-up.

6.2 Force-from-vision lit review v2

Owner: VisionWorker · sinew-5.9 · v2f_lit_v2.md · cloned code at references/v2f_lit_v2/code/{FoAR, reactive_diffusion_policy, forcesight}

Backbone consensus (2025)

• Frozen DINOv2 ViT-S per cam, 4-channel RGBD patch embed (ForceSight pattern at references/v2f_lit_v2/code/forcesight/prediction/models.py:RGBDDinov2). Depth channel trainable; init from RGB conv1 mean.
• ResNet still works for purely-RGB or PCD inputs (Force Map ResNet50; FoAR MinkowskiEngine ResNet14 for PCD).
• No paper uses a video-native backbone (VideoMAE, TimeSformer) for force prediction yet.

Strongest impl-detail finding

Training defaults to crib

6.3 v2f FMB-only feasibility

Owner: VisionWorker · sinew-5.17 · v2f_fmb_only_feasibility.md + isaaclab_sinew/scripts/train_v2f_fmb_only.py (480 lines, AST-valid)

The feasibility check answers a single load-bearing question before committing to the staged-pretrain plan: can v2f be learned from FMB-real labels alone? The spec is closed; the A6000 run is a separate impl ticket per the user CLAUDE.md "training runs on DL_A6000 not local PC" rule.

6.4 Data leverage analysis

Owner: Researcher (coord w/ VisionWorker) · sinew-5.10 · v2f_data_leverage.md

Why this partition

• Direction transfers — geometric constraint normals are sim/real-identical (Direction Matters confirmed on Franka peg-in-hole).
• Magnitude doesn't transfer — real F/T includes payload-model error, gravity-comp residual, thermal drift; sim contact stiffness is calibrated arbitrarily. Cross-distribution training corrupts both.
• Contact-point doesn't transfer — no real-world per-pixel "contact happened here" label exists. Sim derives from peg/board poses + cam extrinsics.
• Gate transfers if masked correctly — threshold at 8 N (FoAR default, 3–5× DR σ) gives a clean binary signal.
• Backbone frozen at stage 2 — catastrophic-forgetting risk on visual-DR features.

Hard exclusions

• Multi-object assemblies (7,200 episodes) — wrong physics.
• Non-insert primitives (grasp/place/rotate/regrasp) — gripper-peg contact direction is uninformative for peg-board contact direction.
• FMB real depth (D405 passive stereo on textureless 3D-printed pegs is the worst real distribution shift). RGB-only stage 2.
• FMB language_embedding — no language conditioning.
• FMB action labels — were for BC (now dropped); not for v2f.

Filter chain (parser-side)

1. Primitive filter: keep only primitive == 'insert'.
2. Gripper filter: keep only state_gripper_pose == 1 (peg held).
3. Episode validity: drop episodes with < 10 post-filter steps.

Steps 1+2 collapse 22,550 episodes → ~3,000–4,000 insert-only-with-peg episodes, ~20–40 GB RGB-only.

F/T zero-bias subtraction (per-episode)

Franka external wrench carries a per-episode baseline drift (payload model + thermal). The first 5 pre-grip-close timesteps (state_gripper_pose == 0) provide the bias estimate. If fewer than 5 pre-grip-close steps exist (peg already grasped), skip subtraction — falling back to in-contact bias estimation would bake contact force into the "bias" and contaminate all downstream direction labels.

6.5 3-head architecture

Owner: VisionWorker · sinew-5.11 · v2f_arch.md (~43M params, 22M trainable + 21M frozen)

Four heads

Key locks across memos

• F/T noise on labels not inputs — predictor sees clean sim vision and learns to match noisy real F/T.
• NaN-mask direction loss at samples where ||F|| < 8 N — direction at the noise floor is uninformative.
• Gate-gated wrench loss — per-sample MSE × gate label suppresses magnitude gradient on no-contact frames.
• No F/T input to predictor. RGBD in, wrench out. Single-direction flow. Eliminates train-test divergence at deploy.

Training budget

6.6 Domain randomization spec

Owner: VisionWorker · sinew-5.13 · v2f_dr_spec.md

Heavy visual DR, NO dynamics DR. F/T noise applied to labels, not inputs. 46-row master knob table organized in three schedule buckets:

Critical knob: depth dropout on low-texture pixels (knob #20). D405 is passive stereo — textureless 3D-printed pegs give sparse / noisy depth in real but the sim depth is clean. Approximate by masking texture_grad < τ pixels and setting them to inf. Without this knob, any RGBD-using head will be sim-tuned.

Five categories elevated to hard-required (post sinew-5.16)

Per the visual-gap quantification: lighting + color jitter + material BRDF + camera intrinsic jitter + background clutter must all ship in the stage-1 DR set. None of these is "optional ablation."

6.7 Revised pipeline — branches A–E

Owner: VisionWorker · sinew-5.23 · supersedes sinew-5.10 + 5.11 + 5.13 · v2f_pipeline_revised.md

One pipeline, five outcome branches set by the sinew-5.17 FMB-only A6000 run result.

                     ┌─ A. Strong validation (real-only ≥0.85) ── drop sim stage 1; ship real-only
                     │
                     ├─ B. Validated, expected (≥0.70, <0.85)── staged pretrain-sim + fine-tune-real (canonical recipe)
[sinew-5.17 result]──┤
                     ├─ C. Marginal (≥0.70 global, <0.60 worst)─ per-shape ensembles OR more real data
                     │
                     ├─ D. Below-bar (0.55-0.70)─────────────── triage: unfreeze backbone, swap to ViT-B, more real data
                     │
                     └─ E. Failed (<0.55)────────────────────── HALT — frame audit, zero-bias check, linear probe

Outcome B (the canonical, most-likely branch)

• Stage 1 sim pretrain. 1.6M (image, force) pairs from the RL data-gen pass; DR per sinew-5.13 calibrated to sinew-5.16 deltas; all 4 heads trained; 300 epochs, AdamW lr=3e-4 cosine + 2k warmup, bs=128, bf16. SimDist disturbance integration via gate-head loss upweight ×1.5 on noised steps. Output: predictor_sim_pretrain.pt.
• Stage 2 real fine-tune. Filtered FMB insert subset (~3–4k eps × ~100 steps × 2cam RGB). Direction + gate heads only; backbone, wrench, contact-point heads frozen. 30–50 epochs, AdamW lr=3e-5, no warmup, bs=128. Direction (L1 NaN-masked) + gate (BCE) weighted 1.0 : 0.1. Output: predictor_real_finetune.pt — the ship artifact.
• Eval. Stratified per-shape direction cos-sim, magnitude-stratified direction cos-sim, gate F1 on FMB-real test.

Why no F/T input or disturbance-conditioning input

Two related decisions that both prevent train-test divergence at deployment:

• No F/T input to predictor. Real Franka reports K_F_ext_hat_K — that's a separate channel the policy may read; the predictor's job is to produce wrench from RGBD alone.
• No sim_action_noised conditioning input. The flag doesn't exist at deployment time; training a conditioning input the trainer can't reproduce at test time is exactly the kind of subtle divergence we should refuse to introduce. The disturbance shows up implicitly in RGBD (object position, contact geometry); the model can learn the regime from images.

The 70% bar caveat

Direction Matters's 70% bar was derived under their own sim2real distribution. Our χ² = 1.88 makes it likely the absolute bar is closer to 0.55–0.65 in the worst case. Therefore:

Primary metric becomes monotonic improvement post-fine-tune until χ² is re-measured post-DR. If real fine-tune doesn't lift over stage-1 sim-pretrain, the visual gap absorption isn't working.