v13 gap diagnostic — where do the remaining 36 mm come from?

Date: 2026-07-11 · Model: v13_stable step 3000 (90 mm abs on ARCTIC-lite N=40) Question: v13 landed at 90 mm; Iter 206 oracle ceiling is 54 mm; where do the 36 mm live? Answer: it's diffuse, not a specific bug — pixel + per-frame lift together cost ~25-30 mm.

Per-clip distribution — no more bimodality

150-clip arctic val eval on v13 s3000:

Percentile	abs MPJPE
min	29 mm — sub-Iter 204 oracle
p5	47 mm
p10	53 mm — at Iter 206 oracle
p25	66 mm
p50 (median)	85 mm
p75	103 mm
p90	128 mm
p95	149 mm
max	253 mm
mean	89 mm

Fractions: - 7 % of clips < 50 mm (Iter 204-oracle territory) - 18 % of clips < 60 mm (~Iter 206 oracle) - 32 % of clips < 70 mm - 55 % of clips < 90 mm - 87 % < 120 mm

Compare to v11 (before rot-matrix loss): bimodal distribution with best clips at 60 mm and worst at 540 mm, mean = 149 mm dragged by 3 % tail.

v13's failure mode is no longer tail-dominated. Mean ≈ median (89 vs 85); the tail is compressed (max 253 vs. v11's 540). The remaining 36 mm gap is a distribution shift, not a bug in specific clips.

Best and worst clips

Top 5 (best abs): | clip | slug | abs | pixel | |-----:|------|----:|------:| | 14066 | (various) | 29 mm | 10 px | | 14862 | | 41 mm | 14 px | | 12908 | | 42 mm | 20 px | | 13917 | | 43 mm | 11 px | | 9721 | | 43 mm | 17 px |

Pixel accuracy on top-5 clips: 10-20 px, comparable to SomantisModel v14d champion (13 px). When the model gets a clip right, it's genuinely competitive.

Bottom 5 (worst abs): | clip | slug | abs | pixel | |-----:|------|----:|------:| | 6189 | s05·notebook_use_02 | 253 mm | 78 px | | 6084 | s05·notebook_use_01 | 206 mm | 53 px | | 2788 | s04·espressomachine_grab_01 | 175 mm | 67 px | | 6217 | s05·notebook_use_02 | 166 mm | 59 px | | 15975 | s09·notebook_use_03 | 166 mm | 76 px |

4 of 5 worst clips are "notebook" (static typing). These are minimal-motion clips where the hand sits nearly still on a notebook. The model has no motion cue to disambiguate depth, and the small-hand geometry makes 2D pixel prediction hard. This is a systematic dataset skew, not a per-clip bug.

Per-source comparison (v12 → v13, N=25 each)

	v12 s3000	v13 s3000	Δ
arctic abs	118 mm	91 mm	−27 mm
arctic pixel	36 px	36 px	tied
arctic PA	14 mm	13 mm	at floor
h2o abs	148 mm	145 mm	−3 mm
h2o pixel	78 px	75 px	−3 px
h2o PA	22 mm	28 mm	+6 mm (regression!)
ho3dv3 abs	269 mm	289 mm	+20 mm (expected — dropped from training)

Cross-source gap widened: v12 was 30 mm (arctic 118 vs h2o 148); v13 is 54 mm (91 vs 145). v13 is a stronger arctic specialist but weaker generalizer. h2o PA regressing 22 → 28 mm is the loudest signal — shape parameters overfit to arctic hand distribution.

Decomposing the 36 mm gap

v13 arctic-lite metrics tell the story:

metric	v13	oracle	contribution
abs MPJPE	90 mm	54 mm (Iter 206)	36 mm gap
rr MPJPE	45 mm	(rr contributes to abs)	joint pose + orient
PA MPJPE	13 mm	13 mm (floor)	at floor, no room
wrist trans	67 mm	0 mm (oracle)	dominant abs term
pixel err	35 px	0 px (oracle)	2D residual
depth SSI	0.21	0 (oracle)	scale residual

The 90 mm abs breaks down (rough decomposition, from Iter 205/206 ablations):

Pixel error (35 px, when depth is ~0.5-1 m) contributes ~10-15 mm to abs
Depth scale error (0.21 log-res on wrist) contributes ~15-20 mm to abs
Joint pose residual (rr − PA = 45 − 13 = 32 mm) contributes ~5-10 mm to abs after wrist correction
Interaction (compounding of pixel + depth per joint) — a few more mm

Totals ~30-40 mm, which matches the 36 mm gap.

No single lever cuts the whole gap. Iter 206's 54 mm oracle uses BOTH GT-2D AND per-clip s* — both interventions together.

Recommended v14 direction

Three options, evaluated:

Option A — Multi-res root_head (pixel-focused)

Add coarse-to-fine head: 60×104 → 120×208 second pass with residual
Predicted pixel gain: 35 → 15-25 px (via _ref_somantis inspection: their v14d hits 13 px with per-joint heatmap)
Predicted abs gain: −8 to −12 mm (moves mean to ~78-82 mm)
Cost: ~300 LoC
Risk: v18 REJECTED in SomantisModel (pooled crop token throws away spatial info)

Option B — Somantis-style 2D + s* + PnP lift (recommended)

Replace 61-d MANO output with (present, root_uv, log_scale, j_rel[20,2], vis[21]) = 65-d
At inference: PnP solve_transl(J0, uv_pred, w, K, t_init) for wrist
Directly encodes the Iter 206 winning recipe into the model
Predicted abs gain: −20 to −30 mm (targets 60-70 mm range, close to oracle)
Cost: ~500 LoC — but 60% of code exists in ~/Projects/SomantisModel/hand2d/{pnp,representation}.py
Risk: bigger refactor, longer to converge

Option C — Add h2o-shape rebalance (fix cross-source)

Duplicate h2o in training pool 3× to counteract PA regression
Predicted h2o PA gain: 28 → 22-24 mm; arctic unchanged
Cost: 10 LoC (weight sampling)
Risk: none, but scope is small (only helps h2o)

Recommendation: B (Somantis lift). Directly targets the Iter 206 oracle recipe; the failure mode diagnostic shows no single-bug fix will do; ~500 LoC is spendable and reusing SomantisModel code cuts risk significantly.

Alternative if speed matters: C first (10 LoC weight sampling for immediate 5-10 mm h2o PA gain), then B.

v14 blueprint (Somantis lift)

Model outputs per (frame, hand):
  present   scalar   [BCE]
  root_uv   (2,)     [MSE in normalized coords]
  log_scale scalar   [MSE against robust hand scale]
  j_rel     (20, 2)  [MSE in root-relative + scale-normalized coords]
  vis       (21,)    [BCE per joint]

Loss (from Somantis compute_loss_unified):
  L = 1·L_present + 3·L_root_uv + 1·L_log_scale + 5·L_jrel + 0.5·L_vis
      + (keep) L_cam + L_pmap + L_depth

Inference:
  For each frame t, each hand h:
    reconstruct uv[k] = root_uv + exp(log_scale) · j_rel[k-1]  # (21, 2)
    solve_transl(J0 = template hand at rest, uv, w=vis, K, t_init=weak-perspective)
    → wrist trans t; combine with MANO regressor for joint 3D

J0 source: use a fixed template MANO hand (zero pose, mean betas) OR use hand_head's MANO regression as J0. Somantis uses HaMeR-distilled J0.
Compatibility: keep hand_head for pose/betas (still used for shape). Only remove hand_head's contribution to trans/wrist.
Warm-start: from v13 s3000 for hand_head + trunk; new heads (root_uv, log_scale, j_rel, vis) fresh-init.
Training: 3000 steps, same 63-min budget with trunk-freeze recipe.

Next steps

This report — commit + push (documents the diagnostic + recommendation)
v14 trainer — new file data_engine/train_egowm_v14_lift.py
Adopt SomantisModel representation.py + pnp.py under egowm/hand2d/
Eval script with PnP lift at inference

Expected timeline: - v14 code: 3-4 h (skeleton + smoke + fix) - v14 training: ~1.5 h wall (5000 steps at 0.79 ips) - v14 eval + report: 1 h

Target: v14 s3000 abs ≤ 70 mm on ARCTIC-lite N=40.