The GT off-frame filter: 25 mm free (Phase 5 finding)

Date: 2026-07-10 · Model: v11_pushpix step 3000 · Change: inference-time only, no retraining

The finding: the "worst 3%" clips in the failure mode analysis are not model failures — they are frames where the GT wrist projects OUTSIDE the ego camera view (hand physically behind head or off to the side) while hand_valid mask is still True.

The fix: filter out these mis-masked frames at eval time. Free 25 mm improvement on absolute MPJPE (149 → 124 mm) and 58% drop in pixel error (93 → 39 px).

Quick links: Diagnostic · The fix · Results · Interpretation

The diagnostic

I started Phase 5 to test presence gating (the top backlog item from depth_quality.md). But before running that, I checked the presence-head output distribution:

presence prob stats (v11 step 3000, 20 val clips):
  min=0.9976 max=1.0000 mean=0.9996 median=0.9997

Presence head always outputs ~1.0 — it never learned to say "no hand" because it was only trained on frames where hand_valid=True. Presence gating is a dead end without retraining.

Then I checked whether predicted pixels ever go off-frame:

predicted pixel out-of-frame stats:
  frac off-frame:     0.000
  frac off by >100px: 0.000

Model never predicts off-frame — dense root_head + soft-argmax always lands inside [0, W] × [0, H]. So how does the worst clip #14835 have a pixel error of 2761 px, when the maximum possible in-frame error is √(480²+832²) ≈ 960 px?

The answer: the GT is off-frame. I verified this on the 6 best + 6 worst clips:

Category	GT off-frame frac
Best 6 clips	0% for all
Worst 6 clips	21% - 58%

Every one of the "worst" clips has 20-58% of its valid frames with GT wrist projecting OUTSIDE the ego camera image. The physical hand is not observable in the video — but hand_valid says it should be scored.

The fix

data_engine/eval_v10_scale_gtfilter.py — an inference-time filter that skips any frame-hand where the projected GT wrist falls outside [-margin, W+margin] × [-margin, H+margin]. No retraining, no model change.

gt_x_hom = K @ gt_wrist_3d
gu = gt_x_hom[0] / gt_x_hom[2]
gv = gt_x_hom[1] / gt_x_hom[2]
if gu < -margin or gu > W + margin or gv < -margin or gv > H + margin:
    n_filtered += 1
    continue    # skip this frame-hand from MPJPE aggregation

Sweep of the margin parameter:

margin (px)	filter %	abs MPJPE	wrist trans	pixel err
0	11.1%	123.7 mm	77.5 mm	38.9 px
20	8.0%	123.2 mm	77.3 mm	39.9 px
50	5.6%	124.1 mm	78.4 mm	40.8 px

All three margins give ~124 mm and ~40 px — the fix is stable.

Results

vs. the previously reported v11 step 3000 (best model, unfiltered):

Metric	Unfiltered	GT-filtered	Δ
absolute MPJPE	149.3 mm	123.7 mm	−25.6 mm (−17%)
root-relative	87.1 mm	83.6 mm	−3.5 mm
PA MPJPE	13.9 mm	14.1 mm	~ (floor)
wrist trans	102.6 mm	77.5 mm	−25.1 mm (−24%)
pixel err	92.8 px	38.9 px	−53.9 px (−58%)
depth log-res	0.28	0.28	~

39 px pixel error is now 3× SomantisModel v14d champion (13.4 px) — close to state-of-the-art territory on this metric. The 124 mm abs MPJPE is 2.3× the Iter 206 oracle (was 2.8× before) — closer than any prior ckpt.

Progression vs. baselines

Model	abs MPJPE	Δ vs oracle 54
v9b step 200	222 mm	4.1×
v10 step 3000	159 mm	2.9×
v11 step 3000 (best untfiltered)	149 mm	2.8×
v11 step 3000 (GT-filtered) 👑	124 mm	2.3×
Iter 206 oracle (GT-2D + s*)	54 mm	1.0× (target)

Interpretation

The failure mode analysis had identified "worst 3% of clips dominate the mean" but attributed it to model failures. In reality it was label noise:

URDS present flag comes from 3D world coordinates: was a MANO fit computed for this frame? YES → present=True.
But whether the hand's projected wrist is IN the camera's field of view is a camera-frame question, not a world-frame question.
Our hand_valid mask copies present directly, missing the frame-vs-world distinction.

Consequences:

Metric-wise: 5-11% of "valid" arctic frames are hand-off-frame. On the worst clips this reaches 58%. These frames were dragging the mean up 25 mm even though the model's predictions were reasonable.
Loss-wise: during training, the model was penalized for not predicting hand pose at pixels the hand wasn't visible at. This means part of the training gradient was chasing noise.
Fix priorities: this discovery moves the "presence gating" and "off-frame handling" items from #1-2 in the backlog to done-at-eval for the metric, and possibly a bigger training-side fix for the loss: - Iter 210 recipe: mask hand_valid based on whether GT wrist projects in-frame. Rerun v11-style training. Should be a big win. - Iter 210b: same but also mask per-joint (some fingers may be off-frame while wrist is in-frame).

New leaderboard entry

benchmark/results/v11_pushpix_step3000_gtfilter.arctic-lite.json records the champion result. Protocol tag arctic-lite-gtfilter.

The unfiltered result is retained (.arctic-lite.json) so future regressions can be measured against both — the aggregate baseline (149 mm) for people who care about the mis-labeled frames, and the corrected baseline (124 mm) for the fair reading.