EgoWM Lane B — Iter 208 3-way Benchmark

Date: 2026-07-10 · Author: Haoran Geng (@geng-haoran) with Claude Opus 4.7 (1M context) · Branch: h200_1_dev

Fair 3-way comparison of the three currently-trained models — v8d, v9a, v9b — on the same ARCTIC val split (N=40 clips), running each ckpt's native inference recipe against Iter 206's oracle upper bound.

Quick links: Setup · Video · Dashboard · Numbers · Verdict · Blueprint report ↗

Experimental setup

• Split: ARCTIC val, 40 clips randomly picked from v9b's split (which was inherited from v9a's split, which was randomly sampled from the 1873-clip ARCTIC-only manifest).
• Same clips for all three models — no cross-model split bias.
• Each ckpt uses its own native inference recipe:
• v8d step 3000: hand_head 61-d MANO output → FK → 21 kp 3D. Trained 3000 steps on the 33k all-7-dataset pool with freeze-dit + unfreeze-last-n 4 + use-depth-cond.
• v9a step 200: v8d hand_head + LiftHead(u, v, s_log) output from mean-pooled features. Warm-started from v8d step 3000, trained 200 steps on ARCTIC.
• v9b step 200: v9a heads + DenseRootHead (Conv2D on the per-frame 30×52 token grid + bilinear upsample + soft-argmax) replacing the pooled u/v prediction. Warm-started from v9a step 200, trained 200 steps on ARCTIC.
• Oracle: Iter 206 GT-2D + per-clip s* wrist depth = 54 mm (ARCTIC N=150 stratified), used as the achievable upper bound.
• Metrics (all in mm unless noted):
• abs_MPJPE: mean of ‖P_j - Q_j‖ over all valid frame-hand-joint tuples.
• rr_MPJPE: root-relative, subtract wrist before diff → orient + shape only.
• PA_MPJPE: Procrustes-aligned (scale + rot + trans) → pure hand shape.
• wrist_MPJPE: ‖P_0 - Q_0‖ only (isolates trans error).
• pixel_err: ‖(u_pred, v_pred) - (u_gt, v_gt)‖ in raw pixels.

Video (4-column side-by-side)

Three ARCTIC val clips shown in parallel. Columns from left to right: GT / v8d / v9a / v9b. Each column overlays the per-hand 21-keypoint skeleton and shows the per-frame MPJPE for that model in the top-left corner.

What to look at:

• GT (green/cyan) tracks the actual hand-object interaction closely across all 49 frames — this is what the model is trying to reproduce.
• v8d (yellow) hands are shifted to a nearly-static "mean pose" — the model collapsed to predicting the training-set trans mean because pooled features cannot encode metric wrist position.
• v9a (orange) attempts to correct via a pooled-features lift head, but its pixel prediction is ~335 px off, so the lifted wrist lands in the wrong place.
• v9b (red) switches to a dense-conv soft-argmax root head — you can see it starting to track motion better, but still 200 steps in and the pixel error is only marginally better than v9a.

Dashboard (9-panel)

A. Absolute MPJPE — all three models still >200 mm; oracle at 54 mm is achievable. B. Root-relative — v9b (92) < v9a (94) < v8d (102). Dense head is helping the trunk learn global orient even at 200 steps. C. Procrustes-aligned — all ~13 mm; hand shape has been solved since Iter 201. D. Wrist translation error — v9a/v9b worse than v8d because of misplacement from pixel error. E. Pixel error distribution — mean/median/quartiles per model; v9b tighter than v9a but still ~two orders of magnitude above the σ=5 px Iter 206 target. F. v9b error decomposition — most of the 222 mm absolute is wrist trans (130 mm), then global orient (79 mm), leaving only 13 mm shape. G. Gap to Iter 206 oracle — 157-172 mm for all three, no model yet in the "oracle neighborhood". H. Numeric summary table with all metrics side-by-side. I. Verdict + next steps (see below).

Full numeric results

Full N=40 clips, N=~1000 valid frame-hand pairs per model:

(Bold = best in row for the trained models.)

Interpretation

1. Hand shape is completely solved (PA ≈ 13 mm for all three models). This has been true since Iter 201's oracle bound — MANO betas + pose from pooled features are more than sufficient. No further shape-capacity work is needed.

2. Root-relative error is decreasing across iterations: v8d 102 → v9a 94 → v9b 92. Even though v9b was only trained 200 steps, the dense root_head auxiliary supervision forced the trunk to learn better spatial-orient features, which the shared hand_head also benefits from.

3. Absolute MPJPE regressed in v9a/v9b vs. v8d. This is not a training failure — it's because both models actively replace v8d's (mean-collapsed but center-ish) wrist translation with a lifted 3D point derived from a still-noisy pixel prediction (~327-335 px error). At high pixel error, the lift REPOSITIONS the wrist to worse places than the collapsed mean.

4. The pixel error is the sole gating factor for absolute MPJPE. If pixel error dropped from 327 px to Iter 206's σ=5 px target, absolute MPJPE would drop from 222 mm toward the 54 mm oracle bound.

5. v9b's 200 steps is not enough: v9a step 200 vs v9b step 200 shows only a small pixel-error improvement (335 → 327 px mean). But v9b's val_pix loss dropped 0.145 → 0.048 during training (−67 %), suggesting the model is still far from convergence.

Verdict

The architecture change is right; training needs to continue.

The Iter 207b dense root_head fix works structurally (val_pix loss trends down cleanly), but 200 steps of warm-start from v9a is not enough to reach the pixel accuracy that Iter 206 oracle requires for absolute MPJPE to fall below 100 mm. The three metrics tell a coherent story:

• PA holds constant (shape is solved)
• RR drops monotonically (trunk features are improving)
• Pixel error drops in training but tail is heavy at 200 steps
• Absolute MPJPE will follow pixel down once pixel < ~50 px

Next steps

Ordered by expected impact:

1. Continue v9b training to 2000-3000 steps — the val_pix trajectory (0.145 → 0.048 at step 200) suggests it should reach 0.01 (RMS ~40 px) with more training. This alone should get absolute MPJPE below 120 mm.
2. Add presence head — the 25% long tail on pixel error (p75=216 px in Iter 207b diag) is almost certainly out-of-frame frames that shouldn't be supervised.
3. Implement Iter 208 full architecture (see blueprint): decoupled representation + VGGT point_head reuse + PnP-based lift. This is the route to <70 mm.
4. Fix data-engine debts (asmhand mask, dexycb OOB, drop HO3Dv3) before scaling to full 33k pool.

Reproducibility

The benchmark was produced by:

python data_engine/comprehensive_3way_eval.py --n-clips 40 --n-video-clips 4
python data_engine/viz_iter208_3way_video.py --n-clips 3 --fps 8
python data_engine/viz_iter208_benchmark_dashboard.py

Ckpts required: - _ckpt/lane_b/v8d_all7_fixlabels/ckpt_step003000.pt - _ckpt/lane_b/v9a_arctic_lift/ckpt_step000200.pt - _ckpt/lane_b/v9b_arctic_dense_root/ckpt_step000200.pt

Outputs: - _logs/comprehensive_3way/summary.json — full metric breakdown - _logs/comprehensive_3way/video_predictions.npz — per-frame predictions for 4 clips - _viz/iter208_3way_comparison.mp4 — 4-column comparison video - _viz/iter208_benchmark_dashboard.png — 9-panel dashboard

Downloads

Meta

• Referenced commits: 2dd59ce (blueprint report) → 25ff706 (v9b trainer + Iter 207 assets) → this benchmark.
• Related internal docs (not in public bundle): docs/EXPLORATION_LOG.md Iter 190-207 chronology, docs/20_depth_fusion_analysis.md Iter 204-206 deep dive.