v13 diagnostic — why 116 mm is unacceptable, and what v13 does about it

Date: 2026-07-10 · Trigger: user pushed back on v12 champion (116 mm) — "hand tracking is too bad, this isn't even generated video, overfit was fine, why does scale collapse?" Verdict: it is bad, and it's not a capacity problem. Four concrete failures in the v10/v11/v12 training recipe. v13 fixes all four.

The overfit vs. scale gap

The Iter 208 overfit proved the architecture, losses, and data pipeline all work: 2 clips, trunk fully frozen, 500 steps → loss_hand 20.56 → 0.31 (−98.5 %), loss_pix 0.128 → 0.00035 (~11 px, at Iter 206 σ=10 target). Every head converged.

At scale (v10/v11/v12 on 33 k clips), the training loss on hand does not decrease:

The eval MPJPE (116 mm) is a combination of untrained hand head + still-improving pixel + s-scale + partly-broken depth. It's not a proof that the model learned; it's a proof the peripherals kept working.

Four concrete failures in v10–v12

1. Axis-angle MSE causes mean-collapse at scale

Current loss:

loss_hand = masked_mse(hand_pred, hand_gt, hv_l)
# hand_pred, hand_gt: (B, T, H, 61) axis-angle MANO

The 61-d MANO layout is [orient(3) + body(45) + betas(10) + trans(3)]. The 3-d orient and 45-d body are axis-angle (rodrigues) — the angle magnitude is ‖v‖ and the axis is v/‖v‖. This has:

• π discontinuity: v = (π, 0, 0) and v = (−π, 0, 0) represent the same rotation but MSE sees them as distance 2π apart. Gradients flip sign across this boundary.
• Non-Riemannian metric: MSE on (θx, θy, θz) doesn't correspond to any sensible rotation distance. The optimum for a diverse GT distribution isn't the mean rotation — it's the mean axis-angle vector, which usually lies near the origin. This is why Iter 203/204 observed pred_spread only 16.5° on ARCTIC (near-collapsed to identity).

At scale (33 k varied poses), the collapse pins loss_hand at 25–42. On overfit (2 clips, near-identical poses), collapse doesn't happen because the target is essentially a single point.

Fix: convert both pred and GT orient + body axis-angle to rotation matrices via Rodrigues formula (differentiable everywhere), then use Frobenius MSE ‖R_pred − R_gt‖²_F. This is what the MANO paper recommends and what most modern hand-pose regressors use. See train_egowm_v13.py matrix_hand_loss.

2. Trunk drift outpaces heads

Current recipe unfreezes patch_embed + last-4 DiT blocks at LR 5e-6 for 3000 steps. Heads are trained at LR 1e-4 (20 × higher). Effect: the DiT features slowly shift each step, so heads are chasing a moving target. Overfit worked because the trunk was fully frozen — heads had a static feature to fit.

Symptoms: - val_pix NaN at 4 of the last 6 eval steps (soft-argmax numerical instability when features drift into unusual regions) - val_hand increasing in the last third of training (32 → 40) - Overfit-scale gap 20× on loss_hand

Fix: freeze the entire trunk. Only heads train. LR 3e-4 on heads only. Bonus: forward-only trunk in no_grad() context → 2× training speedup (smoke test: 0.8 ips vs. v12's 0.42 ips), so 3000 steps takes ~1 h instead of ~4 h.

3. HO3Dv3 pollutes both train and val

Three independent diagnostics now converge on the same conclusion:

Training on ho3dv3 pulls the model toward noise. Val'ing on ho3dv3 makes the eval metric penalize good predictions. The evidence is overwhelming.

Fix: drop ho3dv3 from both train_idx and val_idx. Effect: train 29 770 → 28 948 clips (−2.8 %), val 3307 → 3219 (−2.6 %). Small pool loss, big signal gain.

4. NaN val batches poison the running mean

v12 val log shows val_pix=NaN and val_s=NaN at steps 1750, 2000, 2500. The current eval aggregation is np.mean([d[k] for d in vp]), which propagates NaN into the reported mean. Any downstream analysis that compares val losses across steps then sees garbage for those steps.

Fix: skip batches where any part is non-finite. Report n_valid/16 so we can see if this is happening.

What v13 does concretely

Diff from v10-scale to v13:

Training cost: ~1 h @ 0.8 ips (measured on smoke) vs. v12's ~4 h @ 0.42 ips.

Expected outcome

We already know the ceilings:

• Iter 206 oracle (GT-2D + per-clip s*): 54 mm on ARCTIC
• Iter 204 oracle (GT-orient + GT-trans): 17 mm
• PA floor: 13 mm

Current: 116 mm. Gap to Iter 206 oracle: 62 mm.

v13 targets the two biggest levers directly: - Rotation-matrix loss fixes the ~48 mm orientation contribution quantified in Iter 204. Not all 48 mm — mean-collapse recovery gives some fraction, plausibly 15–30 mm. - Trunk freeze + no-drift eliminates the val-instability that was costing 5–10 mm at the end of v12 training.

Rough forecast: 60–80 mm abs MPJPE on ARCTIC-lite. If we hit that, we're within 2× of the Iter 206 oracle and the remaining gap is depth ambiguity + fingertip lift (which is what Iter 206 lift's per-joint analysis flagged as the residual bottleneck: fingertips have irreducible ~60–80 mm depth ambiguity from a single view).

If v13 lands at 60–80 mm, the next step is v14 with SomantisModel's lift representation (predict 2D + per-clip scalar → PnP → wrist) for the 54 mm oracle floor. If v13 lands at 90+ mm, we know rotation representation was not the dominant issue and depth ambiguity is the true floor.

What "generation" would look like — noted, deferred

The user is correct that our demo videos are RGB-input + prediction overlay, not generated video. Wan 2.1 is currently used as a cached-latent feature encoder for tracking heads. The generative capability (prompt → video) is neither exercised nor demonstrated.

If we go to a true grounded-generation demo (option C): - Sample noise → run Wan 2.1 diffusion → decode → new video - Each generated frame has a hand pose in it (either injected via conditioning or predicted post-hoc) - The heads become a conditioning source, not a prediction target: "generate a video where the hand does X at frame Y"

That's a fundamentally different pipeline. It's on the backlog but option A (fix tracking) is what user picked. If v13 gets us to 60-80 mm, generation-style demos become much more sellable because the underlying model actually knows where the hand is.

Reproducibility

# v13 training (~1 h on H200)
/home/ubuntu/miniforge3/envs/ml/bin/python data_engine/train_egowm_v13.py \
    --steps 3000 --val-every 250 --save-every 500 \
    --out _ckpt/lane_b/v13_stable

# eval same protocol as v12 for fair comparison
/home/ubuntu/miniforge3/envs/ml/bin/python data_engine/eval_v10_scale.py \
    --ckpt _ckpt/lane_b/v13_stable/ckpt_step003000.pt \
    --n-clips 40 --source-filter arctic \
    --out benchmark/results/v13_stable_step3000.arctic-lite.json

Meta

• Author: Haoran Geng (@geng-haoran) with Claude Opus 4.7 (1M context)
• Precursors: Iter 208 overfit · v12 final · gtfilter finding
• Trigger: user's blunt "这个hand tracking也太差太差了" after seeing 116 mm champion