# TT/TO P6 confirmatory campaign — 2026-09-24

**Verdict: `same_quality_for_less`** under the frozen plan `tt_to_p6_confirmatory_v1`
(sha256 `399b6ec626e174fde8ea692114ec2d3b4967b3de0bcff6f39071f6cd14e330c7`).

Authority: `docs/SPECs/SPEC_TT_TO_Campaign_P0_Contract.md` A15 (pilot, plan freeze) and A16 (this run).

## Setup

| Item | Value |
|---|---|
| Stack | isolated `g3-20260924g`: TokenOven `b5754f8`, TokenTumbler `d433676`, Verdict gateway `4c261a3b42`, runner `147b7a85ad` |
| Model | `verdict-4-35b-toolterse-v5-mlx`, local LM Studio, $0 |
| Arms | baseline `features_off`, treatment `features_all_supported` |
| Sample | held-out instances 1–3 × 13 task families × 1 rep = 39 pairs, 78 attempts |
| Contract | temperature 0, `max_tokens` 4096, one-sided α 0.05, NI margin 0.05, 10,000 family-cluster bootstrap resamples, seed 20260924 |
| Evidence | receipts sha256 `71e7fce7045c5ced54a6d4e05b53adc7b9009510f77ed3065b9163f312a89a56`; report sha256 `70e331d5cf94330bfcb0d2e98677b432bc5bdbf16802d27fc78b08e292c4eea0` |

## Results

| Metric | features_off | features_all_supported |
|---|---|---|
| Pass | 35 / 39 | 35 / 39 |
| Wrong answer | 1 | 1 |
| Reasoning exhausted (both hops at cap) | 3 | 3 |
| Upstream tokens, incl. retries | 346,283 | 280,857 |
| Prompt / completion | 304,106 / 42,177 | 239,759 / 41,098 |
| Tokens per success | 9,894 | 8,024 |
| Client wall | 1,131 s | 1,007 s |
| Energy (estimated) | 67.3 Wh | 54.6 Wh |

- Paired quality: 0 discordant pairs; difference 0.0; one-sided lower bound 0.0 (margin −0.05) → noninferior.
- Cost: token ratio 0.811; one-sided 95% upper bound 0.968 → significant. Point reduction 18.9%; the bound guarantees about 3%.
- Integrity: 0 issues, 0 unpaired, 0 unscorable; stores reconciled.

## Where the saving comes from

| Family | off tokens | on tokens | ratio |
|---|---|---|---|
| lc_multi_needle | 24,309 | 731 | 0.03 |
| log_triage | 37,276 | 5,853 | 0.157 |
| doc_needle | 23,160 | 12,735 | 0.55 |
| other ten families | — | — | 1.0 (identical) |

`log_error_count` is identical in both arms because `log_pruning` still reports `bypassed` there.

## Limits

One local model, a synthetic corpus, temperature 0, one rep per held-out instance. The finding is that the
features cut tokens on long-context retrieval and log triage without changing any answer here, not that
they help every workload.

## Disclosure

One held-out request (`cache_retention@i2:features_off`) was sent once under pilot-phase keys before the
`p6c` phase existed. It wrote no receipt and was stopped; nothing was tuned on it. See A16.

## Correction (2026-09-24, after independent review)

The token result stands. Two statements above overstate the quality result:

- **Noninferiority is not demonstrated by an exact bound.** Every paired difference was zero, so the family-cluster bootstrap resampled only zeros and returned a lower bound of 0.0. That interval is degenerate and says nothing about unseen losses. With 0 treatment losses in 39 pairs, the exact (Clopper-Pearson) one-sided 95% upper bound on the loss rate is 7.4%, wider than the plan's 5-point margin. An exact test would need at least 59 loss-free pairs to clear it. The frozen plan's procedure did return `same_quality_for_less`; read the quality half as "both arms passed the same 35 of 39 tasks," not as demonstrated noninferiority.
- **"Without changing any answer" (Limits) is too strong.** Six pairs gave differently worded answers that scored the same (e.g. "Saturdays at 21:20 UTC" vs "21:20 UTC on Saturdays"). The analysis compares pass/fail outcomes, not answer text.

Also:
- "The bound guarantees about 3%" should read: the one-sided 95% upper bound on the ratio is 0.968. It describes the procedure, not a guaranteed saving.
- Six attempts used a counted retry, three per arm. The plan allowed this, and their tokens are included.

The receipts and the analysis report JSON (hashes above) are unchanged; this section is the only edit to this file.