# Held-out v3 results (18 October 2026)

## What this set is

- 21 plans, 70 faults, 23 decoys, and 11 out-of-scope faults (listed in `out-of-scope.md`).
- Two agents wrote and labelled it blind. Neither saw the readiness rules, the fixes, v1 or v2. `README.md` has the details.
- `tears-of-steel-tower-watch` rebuilds one dialogue scene of a professionally edited film (Blender Foundation, CC-BY 3.0; credit: (CC) Blender Foundation | mango.blender.org). It is a **clean control**: no faults were planted in it.

## How it was used

It was the **test set** for E10 (#57). It was measured twice with `dpl bench`, scoring each mutant against its own plan's open findings:
- on `main` 0686a6ff, which has E9 and the format gaps G1/G2 (#56);
- on the E10 branch.

The builder never saw this set.

**Status now: a development set.** The lead has read its failures, so the next honest figure needs a fresh blind set (v4).

## Summary

| | Before E10 | After E10 |
|---|---|---|
| `dpl bench` | **Caught** 86 % (60/70) · **caught or other** 87 % (61/70) · **masked** 0 · **missed** 9 · **decoy alarms** 6/23 · **safe fix offered** 48/70, **clears the fault** 48/60 caught · **faults with an extra new finding** 9 · **findings on clean plans** 24 on 10/21 plans (RDY-BEAT-PRESENTER 2, RDY-EYELINE 4, RDY-GEOGRAPHY 1, RDY-HOLD 1, RDY-JOURNEY-CARRIED 4, RDY-STATE 11, RDY-TRAVEL 1). | **Caught** 89 % (62/70) · **caught or other** 90 % (63/70) · **masked** 0 · **missed** 7 · **decoy alarms** 5/23 · **safe fix offered** 52/70, **clears the fault** 52/62 caught · **faults with an extra new finding** 10 · **findings on clean plans** 22 on 9/21 plans (RDY-BEAT-PRESENTER 2, RDY-EYELINE 4, RDY-GEOGRAPHY 1, RDY-HOLD 1, RDY-JOURNEY-CARRIED 2, RDY-NEWCOMER 1, RDY-STATE 11). |

**The professional-cut control raised no findings, before or after E10.**

## Per label (caught under the expected code / planted)

| Label | Before E10 | After E10 |
|---|---|---|
| `RDY-AXIS` | 5/6 | 5/6 |
| `RDY-EYELINE` | 6/6 | 6/6 |
| `RDY-GEOGRAPHY` | 5/5 | 5/5 |
| `RDY-HOLD` | 2/5 | 2/5 |
| `RDY-JOURNEY-JUMP` | 4/4 | 4/4 |
| `RDY-JOURNEY-PROGRESS` | 3/3 | 3/3 |
| `RDY-JOURNEY-UNANCHORED` | 4/4 | 4/4 |
| `RDY-JUMP` | 5/6 | 5/6 |
| `RDY-REACTION` | 7/7 | 7/7 |
| `RDY-SPEAKER` | 1/6 | 3/6 |
| `RDY-STATE` | 7/7 | 7/7 |
| `RDY-TRAVEL` | 6/6 | 6/6 |
| `RDY-UNSEEN` | 5/5 | 5/5 |

## Decoy alarms

E10 added no new decoy alarms and fixed one.

- **Before:** fd-decoy-menu-either-side-of-the-change, dc-decoy-neutral-wide-from-other-side, mv-decoy-second-chorus-still-at-the-cafe, st-decoy-long-insert-under-narration, ca-decoy-marisol-turns-her-back, st-decoy-narration-over-screen
- **After:** fd-decoy-menu-either-side-of-the-change, dc-decoy-neutral-wide-from-other-side, st-decoy-long-insert-under-narration, ca-decoy-marisol-turns-her-back, st-decoy-narration-over-screen

## Missed after E10

tr-axis-take-two-arc-not-marked, hm-speaker-solomon-turn-over-okafor, dt-speaker-sam-question-over-mel, mv-jump-ina-watches-bus-same-frame, np-hold-welder-cutaway-eleven-seconds, np-hold-council-report-insert-eleven-seconds, ca-hold-empty-tin-insert-six-seconds

## Reading

- The gain on this untouched set (86% → 89% caught, alarms 6 → 5) is smaller than on the development sets. Part of E10's effect there is fitting.
- What remains weakest:
  - holds (2/5);
  - speaker faults (3/6);
  - narration over pictures (two decoys);
  - a state that may show on either side of a change.