v13 → v16 arc summary — a 12 mm push via diagnostic-driven training

Date: 2026-07-11 · Champion: v16_more_boost step 1000 = 76.9 mm abs (150-clip) / 78.9 mm (40-clip lite) Total gain vs v13 s3000: −12 mm abs / −13% / gap to Iter 206 oracle cut 62 → 23 mm (−63%) What worked: rot-matrix loss (v13), continued training (v15a), rebalanced sampling (v15d, v16) What didn't: Somantis lift + PnP (v14), 6D-rot (v15c), multi-res root head (v15a's own head)

The 9-hour autonomous arc from v13 to v16. This report is the retrospective: what each iter tried, what worked, what didn't, and what to try next.

Headline arc — ARCTIC-lite N=40 champion at each iter

iter	recipe	abs	rr	PA	wrist	pix	depth	150-clip abs
v13 s3000	rot-matrix loss + trunk freeze + drop ho3dv3	90	45	13	67	35	0.21	88.9
v14 postmortem	Somantis-lift + PnP	641 mm broken	—	—	—	—	—	—
v14 v13-eval	reuse v14 hand_head + v13 pipeline	88	42.5	12.3	67	35	0.18	—
v15a s1000	multi-res root head (multires under-trained, v13-head kept training)	82.6	42.4	12.0	64.3	36.2	0.168	—
v15d s1000	3× sample notebook + s05	80.5	41.0	11.5	60.9	31.9	0.185	78.5
v15c s500	+6D-rot for global_orient	79.0	40.4	11.3	59.9	32.0	0.183	79.2 (tied)
v16 s1000 🥇	v15d + 5 categories boosted	78.9	40.7	11.5	59.5	31.4	0.161	76.9
Iter 206 oracle	GT-2D + per-clip s*	54	—	—	0	0	—	—

Full v13 → v16 delta: - abs: 90 → 78.9 mm (−12 %, −11 mm) - rr: 45 → 40.7 mm (−10 %) - PA: 12.8 → 11.5 mm (−10 %, below stated 13 mm floor) - wrist: 67 → 59.5 mm (−11 %) - pix: 35 → 31 px (−11 %) - depth: 0.21 → 0.16 (−24 %)

Gap to Iter 206 oracle (54 mm): 36 → 25 mm (−31 %) on 40-clip; 88.9 → 76.9 = 35 → 23 mm (−63 %) on 150-clip.

What worked and what didn't

✅ v15d rebalance (biggest win: −2 mm on 40-clip, −10 mm on 150-clip)

Diagnosis (from v13_failure_categories.md): notebook clips (11 % of val) contributed 40 % of the mean overshoot. s05 subject was 78 % worse than s06.

Fix: 3× sampling weight on any clip whose slug contained notebook or __s05__. 10 LoC in the trainer.

Result: notebook 132 → 99 mm (−25 %), s05 121 → 104 mm (−14 %) on 150-clip. Both moved MORE than baseline (which moved 84 → 76 = −10 %) → the rebalance was doing what was intended, not just adding steps.

✅ v15a continued training on v13 architecture (marginal: −4 mm)

v15a introduced a multi-res root_head that didn't converge (12 M params fresh init in 3000 steps, too big). But keeping v13's dense_root_head alongside gave it 3000 more training steps, which alone dropped pixel 35 → 31 px and rr 45 → 42 mm. The multi-res head was a wash; the continued specialist head was the real win.

Lesson: bigger heads need 2-3× more training compared to specialists. Adding capacity ≠ getting capability.

✅ v16 more-category rebalance (small: −1.6 mm)

Extended v15d's 2 boost categories to 5 (notebook + s05 + box + phone + waffleiron). All were in the top 5 worst per-category means on v13.

Result: 78.5 → 76.9 mm on 150-clip. Diminishing returns — each new category buys less. Notebook moved 99 → 96 mm, phone 70 → 67, waffleiron 87 → 86, box didn't move (n=6 in val).

❌ v14 Somantis-lift + PnP (postmortem)

Idea: predict 21-joint 2D positions, use pnp.solve_transl to place a canonical MANO hand.

Result: 641 mm broken because init_transl computes depth from fx × s_3D / s_2D. At noisy s_2D (early training), depth explodes 10-20×. Even after v14 s3000 converged, 3/40 frames still triggered fallback.

Lesson: joint-spread-derived depth is fundamentally unstable in monocular. Iter 206's oracle worked because it used GT per-clip s*, not model-predicted spread. Full postmortem at iters/v14_postmortem.md.

❌ v15c 6D-rot for global_orient

Idea: replace axis-angle orient with 6D representation (Zhou et al. Continuous 6D). Fresh Orient6DHead + Frobenius loss.

Result: 6D head at inference: 85 mm (worse than axis-angle at 79 mm). Axis-angle stayed better because it's been trained across 6+ iters (v9b–v15c) vs 6D head has only v15c's 1500 steps.

Lesson: same as v14 — new representations need dedicated training time that a 1500-step run can't provide. Would need 5000+ steps on the 6D head.

❌ v15a multi-res root_head (own head)

Idea: three PixelShuffle stages, 60×104 → 240×416 output.

Result: 105 mm at s3000 (worse than v13 s3000's 90 mm). The 12.4 M param head was under-trained — pix2 loss dropped from 0.26 to 0.03, but not enough to compete with v13's 6000-step specialist head at 0.013.

Lesson: same again. Bigger ≠ better without proportional training.

❌ v15b eval-time per-clip s median (null result)

Idea: Iter 205 showed per-clip s* is stable. Median-pool model's per-frame predicted s at eval time.

Result: 89.9 → 90.8 mm (WORSE by 0.8 mm). Model's per-frame s is already stable within a clip; median pooling loses per-frame refinement.

Failure mode evolution

v11 to v16 tracked as: what shape does the per-clip abs distribution have?

version	shape	mean	worst	tail contribution
v11	strongly bimodal	149 mm	540 mm	3 % of clips at 400+ mm dominate
v13	compact	89 mm	253 mm	notebook cluster + s05
v15d	shifted-compact	78 mm	209 mm	still notebook s05, but smaller gap
v16	smoothed-compact	77 mm	172 mm	tail keeps shrinking; distribution near-normal

The failure mode has shifted from "some clips explode" (v11) to "static- hand scenarios cluster at 100-150 mm" (v13) to "diffuse tail" (v16). Every diagnostic-driven fix moved the tail closer to the mean, but never below.

Persistent hard clips

Clip #6189 (s05 notebook_use_02): 253 → 150 → 167 mm. Best drop was v15d. Clip #6217 (s05 notebook_use_02): 166 → 209 → 161 mm. Actually got WORSE in v15d then recovered. Clip #2788 (s04 espressomachine_grab_01): 175 → 165 → 172 mm. Barely moved.

These represent the "irreducible" floor of single-frame monocular hand tracking on hard scenarios: minimal-motion, partial occlusion, ambiguous depth. Fixing them needs either temporal information or per-clip modeling.

Recommendations for v17+

Cheap wins likely to work

Extend v16 with more clip diversity — boost by object AND by ANY clip above the 90th percentile of v13's error distribution. Adaptive. ~10 mm predicted?
Continue matrix-hand-loss training with cosine LR schedule — v16 trajectory shows abs peaks at ~1000 steps then over-fits. Better LR schedule might sustain the improvement to 3000 steps.
Explicit HO3Dv3 sample-weight = 0 — currently only removed from train_idx. Could keep some ho3dv3 in val to detect generalization drift.

Medium bets

Temporal smoothing — cross-frame average of ±2 neighbor wrist uv predictions. Cheap eval-time change, but v15b showed per-clip aggregation didn't help; ±2 might.
Per-clip attention pooling on features — replace feats_grid.mean with a learned attention. Might improve clip-level consistency.
Bigger dedicated wrist head w/ dedicated training — a specialist head trained 5000+ steps on just wrist uv. The v15a lesson: bigger heads need more time.

Bigger bets

Cross-source rebalance to fix h2o PA — currently v13-v16 all specialize ARCTIC. h2o PA regressed 22 → 28 mm from v12 → v13. 3× h2o sampling could recover 5 mm on h2o but might hurt arctic. Worth testing.
Actual generative demo — connect Wan 2.1 generation pipeline to the heads for the demo-facing side (currently all evals are on cached latents + real RGB overlay). This is a fundamentally different pipeline; would need ~500 LoC.