EgoWM Lane B — Iter 209 Direction (post-v10)

Date: 2026-07-10 · Author: Haoran Geng (@geng-haoran) with Claude Opus 4.7 (1M context) · Branch: h200_1_dev

What we learned from v10, and where v11+ should go. Written after v10 step 3000 converged at 159 mm on ARCTIC-lite (from v9b's 222 mm baseline; oracle is 54 mm), and while v11 (pixel-focused finetune of v10) is training in background.

Quick links: What v10 achieved · What's still on the table · v11 recipe · Beyond v11 · Metrics gap

What v10 achieved

v10 was the first Wan-DiT + full 7-head recipe with per-head GT masking. It trained 3000 steps on the 33k all-7-dataset pool and landed at:

Metric	v9b step 200	v10 step 3000	Δ	Iter 206 oracle	Iter 204 oracle
abs MPJPE	222 mm	159 mm	−28 %	54 mm	17 mm
rr MPJPE	92 mm	89 mm	−3 %	—	13 mm
PA MPJPE	13 mm	15 mm	~ (floor)	—	13 mm
wrist trans	130 mm	121 mm	−7 %	0	0
pixel err	327 px	105 px	−68 %	0	—
depth log-res	—	0.24	new	—	—

The two biggest wins are pixel error (327 → 105 px) and absolute MPJPE (222 → 159 mm). The Iter 208 blueprint predicted this: dense root_head + soft-argmax breaks the pooled-features cap, and the per-head GT masking makes training on the 33k pool safe.

What's still on the table

Even at 159 mm, we're 3× the Iter 206 oracle and 9× the Iter 204 oracle. Where is the gap?

Gap 1 · Pixel-error floor 105 px vs oracle σ=5 target

The overfit sanity test (Iter 208) showed the dense_root_head can reach ~11 px Euclidean error on memorized clips. On unseen clips at v10 step 3000, it hits 105 px. Where does 100 px go? - Signal-to-capacity: the Wan trunk features aren't finely enough resolved for sub-token pixel prediction. The token grid is 30×52 covering a 480×832 frame → each token is ~16 px wide. Soft-argmax over 60×104 bilinear-upsampled heatmap gives sub-token accuracy in principle, but the underlying features may not disambiguate at sub-token scale. - Distribution shift: overfit was on 2 clips; val is 3307 clips with many poses/objects/lighting not in the training distribution. - v11 test: doubles pixel loss weight (5.0 → 10.0). If pixel drops to 60-80 px, we've been under-supervising; if it stays at ~100, the trunk is the wall.

Gap 2 · Wrist trans 121 mm vs oracle 0

Given pixel error 105 px, and the lift formula wrist_z = exp(s_log) * pmap.z, the wrist 3D error decomposes into: - XY error = pixel error × depth × pixel-size. At 105 px × 0.5 m × (1/416) ≈ 125 mm — matches the observed 121 mm wrist_MPJPE almost exactly.

So wrist_trans is bounded by pixel_err, not by the depth head. If pixel drops to 20 px, wrist_trans should drop to ~25 mm. That's what to watch for in v11.

Gap 3 · rr MPJPE 89 mm — global orient

Root-relative MPJPE (wrist-subtracted) is 89 mm at v10 step 3000, vs PA floor 13 mm. The 76 mm gap is global hand orient error. Iter 204 diagnosed this as a rodrigues+MSE mean-collapse issue: the model was outputting a near-constant orient across frames.

v10 hasn't fixed this — the hand_head is still 61-d MANO with rodrigues global_orient, trained with plain MSE. The Iter 208 blueprint recommended switching to 6D rotation with geodesic loss for global_orient, but this hasn't been implemented yet.

Gap 4 · Depth log-res 0.24 — SSI floor?

Depth SSI residual descended smoothly from 0.60 to 0.24 across training and was still decreasing at step 3000. The floor is not obvious — could be SSI's intrinsic per-frame median-normalization noise, or the depth head just needs more capacity.

v11 keeps depth loss weight at 0.5 (unchanged from v10). If the trajectory continues, expect ~0.15-0.20 at step 3000.

v11 recipe

Currently training in background:

python data_engine/train_egowm_v10_scale.py \
  --out _ckpt/lane_b/v11_pushpix \
  --init-from _ckpt/lane_b/v10_scale/ckpt_step003000.pt \
  --steps 3000 --lr-heads 1e-4 --lr-trunk 5e-6 \
  --w-pix 10.0 --w-s 2.0                             # doubled
  --w-hand 1.0 --w-cam 1.0 --w-present 0.5 --w-vis 0.5 --w-depth 0.5

Changes vs v10: - w_pix: 5 → 10 — push pixel accuracy harder - w_s: 1 → 2 — s_log matters more when pixel is more accurate - lr_heads: 3e-4 → 1e-4 — lower LR for finetuning from v10 (was training from v9b) - lr_trunk: 1e-5 → 5e-6 — half the trunk LR - Same manifest, same split, same trunk-freeze recipe

Predicted outcome: pixel 105 → 60-80 px, abs MPJPE 159 → 130-145 mm. If achieved, next step is to also tackle Gap 3 (6D-rot global_orient).

Beyond v11

Ordered by expected impact vs. engineering cost:

v12 · Switch global_orient to 6D + geodesic (Gap 3)

Cost: ~200 LoC. Change MANO head output shape (add 3 dims), replace rodrigues with 6D-rot decoder, replace MSE with geodesic loss on rotation matrix.

Impact: Iter 204 estimated Δ 49 mm on absolute MPJPE. If v11 gets to 130 mm, v12 should reach ~80 mm.

v13 · Add VGGT point_head via adapter (Iter 208 blueprint)

Cost: ~500 LoC (adapter + DPT wrap + weight-load logic). VGGT's point_head is 32.65 M frozen parameters + ~10 M adapter that trains.

Impact: Depth quality should improve substantially (from 0.24 SSI to ~0.10 or better). Also unlocks confidence-weighted depth for PnP.

v14 · Multi-resolution root_head (higher spatial fidelity)

Cost: ~300 LoC. Add a second dense head at higher resolution (120×208 or 240×416) fed from earlier DiT layers via skip connections.

Impact: Should push pixel error below 30 px, matching SomantisModel v14d territory (13.4 px is their champion).

v15 · Full-precision temporal transformer on pooled features

Cost: ~400 LoC. Add a small temporal transformer over the T=13 pooled features before feeding to heads. SomantisModel v14d uses this.

Impact: Smooth temporal consistency, better vis prediction, slight MPJPE improvement.

Metrics gap: v10 → SomantisModel v14d

SomantisModel v14d (external champion): 13.4 px 2D MPJPE. Our v10 step 3000: 105 px 2D MPJPE.

We are 8× off in 2D pixel accuracy. Ways to close: - v11 (this run) should get us to 5× (60-80 px) - v12 (6D-rot global_orient) — mainly helps 3D, not 2D - v13 (VGGT depth) — helps 3D lifting, not 2D pixel - v14 (multi-resolution root_head) — this is the pixel-focused move - v15 (temporal smoothing) — SomantisModel uses this heavily

Realistic path: after v11-v13, tackle v14+v15 explicitly for pixel accuracy. Or, alternatively, distill SomantisModel v14d directly into our root_head as a pseudo-labeler.

What lives on this page

v10 → v11 rationale (why we picked pixel/s doubling and lower LRs)
Concrete v12-v15 backlog with expected impact and cost
The Gap analysis — where the remaining 105 mm of absolute MPJPE goes and which gap each future recipe would attack

The 12h autonomous exploration loop kicks off v11 training now, and while it runs, produces per-source benchmarks + best/worst gallery + depth qualitative comparisons. Milestones committed as they land.