v17 → v21 arc summary — the diagnostic-driven push from 78.9 to 75.0 mm

Date: 2026-07-12 · Champion: v18_adaptive step 2500 = 75.0 mm arctic-lite / 72.0 mm 150-clip v13 → v21: 90 → 75 mm (−17%), 88.9 → 72.0 mm on 150-clip (−19%) What worked: v17 cosine LR (small), v18 layered adaptive weights (BIG), v21 h2o boost (cross-source) What didn't: temporal smoothing, v19 re-scoring, v20 low-LR polish

This report covers the second exploration wave after the v13 → v16 arc. It documents which levers still had juice and which hit the diffuse floor.

Headline arc — ARCTIC-lite N=40 across iters

iter	recipe	abs	rr	pix	wrist	depth	150-clip
v16 s1000 (prev champ)	keyword-boost 5 categories	78.9	41	31	60	0.161	76.9
v17 s1500	+cosine LR	78.3	40	28	59	0.159	—
v18 s2500 🥇	+adaptive top-25% error boost (layered)	75.0	40	28	55	0.174	72.0
v19 s2500	re-score on v18	75.7	40	27	55	0.167	—
v20 s1000	low-LR polish (3e-5)	76.0	40	28	56	0.174	—
v21 s1000	+h2o boost	75.6	40	28	55	0.176	—

Full v13 → v18 delta on 150-clip: - ALL: 88.9 → 72.0 mm (−17 mm, −19%) - notebook: 132 → 89 mm (−33%) - s05: 121 → 92 mm (−24%) - Gap to Iter 206 oracle (54 mm): 35 → 18 mm on 150-clip

Cross-source (h2o) trajectory: - v13: 289 mm (broken by ho3dv3 pollution) - v18: 145 mm - v21 s1500: 133.7 mm (−11 mm from v18 via cross-source boost)

What worked and what didn't

✅ v17 cosine LR — small but real

Warmup + CosineAnnealingLR on top of v16 recipe sustained the abs peak beyond v16's step-1000 over-fit point. v16 s1000: 78.9 → v17 s1500: 78.3 (−0.6 mm). Pixel improved 31 → 28 and depth 0.161 → 0.159 simultaneously.

Cheap change (5 LoC), consistent gain. Kept as recipe for all later iters.

✅ v18 layered adaptive weights — the biggest win

The key insight: v13-v16 had all been boosting by CATEGORY (keyword-based). But even within a category, some clips are hard and some are easy. A category-wide 3× boost dilutes signal across easy clips too.

v18 replaced the flat keyword boost with layered weights: 1. Run inference on 500 random arctic train clips using v17 s1500 2. Identify top-25% highest-error clips 3. Boost those PER-CLIP × 3 4. MULTIPLIED with existing keyword boost (v16 recipe)

Total: 7406/28948 clips above baseline, with ~75 clips at 9× (both hit).

Result: v17 s1500 78.3 → v18 s2500 75.0 (−3.3 mm) and v16 76.9 → v18 72.0 on 150-clip (−4.9 mm). Biggest single-step gain since v13→v15d.

Lesson: layered signals (broad keyword + specific per-clip) capture more failure modes than either alone.

✅ v21 h2o cross-source boost — small arctic drift, big h2o gain

Added h2o to the boost keyword list (979 clips × 3× weight). Total boost pool jumped 7406 → 8385 clips.

Result on same eval: - Arctic (target): 75.0 → 75.6 (−0.6 mm within N=40 noise) - H2O: 145 → 137 (v21 s1000) → 133.7 (v21 s1500) — −11 mm total

Cross-source generalization improved without hurting the primary metric. Recipe worth keeping when h2o performance matters.

❌ Temporal smoothing (eval-time) — hands are moving

Tested ±1 neighbor mean and full-clip median of wrist uv predictions at inference. Both catastrophic: - baseline: 78.9 mm - mean-3: 126.9 mm (+48 mm) - median-all: 170.7 mm (+92 mm)

Root cause: hands aren't static within our 5-9 latent frame span (~12 RGB frames stride 4). Averaging blurs positions across genuine motion.

❌ v19 re-scoring — diminishing returns

After v18 shifted the failure distribution, re-scored 800 train clips on v18 s2500 to identify NEW hard clips. Found 200 clips (top-25% of 800).

Result: v19 s2500 = 75.7 mm — 0.7 mm ABOVE v18 champion. Sub-metrics improved (pix 27.3, rr 39.9, PA 11.21 all new bests) but abs didn't.

Interpretation: the remaining errors on v18's hard clips are largely intrinsic (occlusion, ambiguous depth). Boosting them lets the model memorize specific instances (dropping sub-metrics) but doesn't yield better generalization to unseen frames.

❌ v20 low-LR polish — disturbs champion

Fine-tune v18 s2500 with LR 3e-5 (3× lower), 1500 steps, same weights.

Result: v20 s500 = 77.0, v20 s1000 = 76.0. Both above 75.0 baseline. Killed early.

Interpretation: at 3e-5 the model is still moving parameters enough to shift away from the local optimum but not enough to converge to a better one. Would need much lower LR (1e-6) or fewer steps.

Persistent hard clips (unchanged since v13)

Same 3 clips keep showing up in the worst-5: - clip #6217 s05·notebook_use_02: v13 166 → v18 165 mm (barely moved) - clip #2788 s04·espressomachine_grab_01: v13 175 → v18 146 mm (−29) - clip #4769 s05·box_use_02: v13 was ~130 → v18 169 mm (GOT WORSE)

These represent the diffuse floor. Notebook clips are static-hand-on-flat- surface (minimal motion cue). s05 has distinct hand shape. Espressomachine has aggressive occlusion. These are edge cases of monocular pose estimation, not learnable from more of the same data.

Recommendations for v22+

Given multiple recent nulls (v19, v20, v21 arctic), the champion appears stable at ~75 mm on ARCTIC-lite. Further gains likely need different levers:

High leverage (untested)

Higher-quality scoring signal — score 2000-3000 train clips instead of 500-800. More statistical power for adaptive weights.
Multi-eval scoring — score with 3 different ckpts (e.g., v18 s1500, s2000, s2500), take median. Reduces noise in per-clip hardness estimate.
Curriculum scheduling — start with all-clip equal weight, then gradually shift boost weights over training. Might avoid v19's over-fit-then-recover pattern.

Architecture / representation (bigger bets)

Deeper hand_head — currently just Linear from pooled features. MLP with 2-3 layers might give another 1-2 mm.
Per-joint scale prediction — currently single s scalar for the whole hand. Per-joint scale would handle fingertip depth ambiguity better.

Fundamentally different pipelines

Cross-frame attention for temporal-aware features. Would need to modify DiT features aggregation.
Generative demo path — actually use Wan 2.1 generation with the heads as conditioning, not just extraction.