# Interval-construction defect in the EDH recovery test (r2-edh-b1, Block B)

**Status:** confirmed, analytically and in simulation. Dated 7 August 2026.
**Effect:** every bootstrap-derived 90% interval in the Block B ledger is narrower than
it should be by a factor of approximately √2.
**Consequence:** the published diagnosis of the 9/16 coverage result is wrong in its
principal claim. The scorecard stands; its explanation does not.

---

## 1. The defect

Block B's intervals are built by resampling provinces **with replacement from the
visible half only**, computing the record-weighted aggregate on each resample, and
taking the 5th and 95th percentiles of the resulting distribution.

Two call sites, and they are the whole method:

- `apps/scriptome/management/commands/derive_edh_recovery_predictions.py:314`
  — `bootstrap_ratio_target()`, used for the four conditional stratum targets
  (B3 Latin-West, B3 Greek-East, B7 Mediterranean, B7 Continental).
- `apps/scriptome/management/commands/derive_edh_recovery_predictions.py:403`
  — `rng.choices(pool33, k=n32)`, used for the whole-holdout aggregates.

That distribution describes the sampling variability of **the visible-half estimate**.
The quantity the interval must cover is **the hidden half's value** — a different set
of 33 provinces, carrying its own independent error.

Writing θ̂_vis for the point prediction and θ_hid for the truth:

    Var(θ_hid − θ̂_vis) = Var(θ_hid) + Var(θ̂_vis) ≈ σ²/33 + σ²/33 = 2σ²/33

The implemented bootstrap delivers approximately σ²/33 — one of the two components.
The interval is therefore too narrow by √2 ≈ 1.414.

The same factor arises under the finite-population framing, which is the more accurate
one here: EDH's 66 provinces are the entire population, split without replacement into
halves. Because ȳ_vis + ȳ_hid = 2μ exactly, ȳ_hid − ȳ_vis = 2(μ − ȳ_vis), so
Var(ȳ_hid − ȳ_vis) = 4·Var(ȳ_vis) ≈ 2σ²/33 after the finite-population correction.
Same answer, different route.

A related and separate point: a with-replacement bootstrap is not the appropriate
scheme for a finite population divided without replacement. Design-consistent
finite-population procedures exist for exactly this case.

## 2. Confirmation in simulation

4,000 simulated populations of 66 provinces, random 33/33 splits, 800 resamples each,
province record counts lognormal to reproduce EDH's size skew. Nominal coverage 0.90.

| province size skew | current method | width | two-sample | width | width ratio |
|---|---|---|---|---|---|
| equal (σ=0.01) | 0.745 | 0.0572 | 0.881 | 0.0808 | **1.413** |
| moderate (σ=0.6) | 0.719 | 0.0645 | 0.861 | 0.0912 | **1.414** |
| heavy (σ=1.2) | 0.664 | 0.0814 | 0.801 | 0.1164 | **1.430** |

The width ratio is √2 to three decimal places. The defect is exactly as derived.

Expressed as targets covered out of 16:

| | current method | corrected two-sample |
|---|---|---|
| equal sizes | 11.7 | 14.1 |
| moderate skew | 11.6 | 13.9 |
| heavy skew | 10.8 | 12.9 |

Nominal is 14.4/16. **Observed was 9/16.**

## 3. What this re-attributes

The published account said the misses shared one diagnosable cause: an alphabetical
split is not a random split, so ranges built assuming the halves resembled each other
came out too narrow.

The correct account is that the ranges came out too narrow *by construction*, on any
split whatever, including a random one. The missing variance component alone moves the
expectation from 14.4 to roughly 11.5 before a single structural effect exists. Only
the remaining ~11.5 → 9 is available to be explained by the alphabet.

So the ordering in the published text is inverted. The dominant cause is a method
defect that has nothing to do with the partition; the partition effect is real but
secondary.

**The structural observations remain true and remain instructive.** Noricum does sit on
Alpine quarries and does run 36% marble against visible-side analogues below 3%;
fifteen holdout provinces do hold at least 1,000 inscriptions where the prediction
expected six to twelve. Those facts are not withdrawn. What is withdrawn is their
status as *the* explanation of the coverage failure.

**The scorecard is unaffected.** The predictions were registered before the holdout
existed on our systems, the holdout reconciled 33/33 exactly against EDH's published
counts, and scoring was mechanical. Nine of sixteen registered intervals covered the
landed value. What was mislabelled is the intervals' confidence level: they were not
90% intervals.

## 3a. Measured on the real province data

The simulation above uses synthetic populations. Repeating it on EDH's actual 66
provinces — all 81,855 records, the holdout included, since it has long since
landed — replaces the estimate with a measurement. 3,000 random 33/33 splits,
1,000 resamples each, seed 20260807, over the eight whole-population ratio
constructions the ledger uses:

| target | current | two-sample | width ratio |
|---|---|---|---|
| non-Latin share | 0.693 | 0.846 | 1.449 |
| epitaph share | 0.653 | 0.786 | 1.423 |
| votive share | 0.717 | 0.847 | 1.415 |
| dated share | 0.729 | 0.864 | 1.422 |
| range-precision share | 0.685 | 0.876 | 1.440 |
| bracket density | 0.663 | 0.835 | 1.421 |
| no-image rate | 0.694 | 0.812 | 1.420 |
| marble share | 0.702 | 0.844 | 1.460 |
| **mean** | **0.692** | **0.839** | **1.431** |

Expected covered of 16 targets: **11.1** under the current method, 13.4 under the
naive two-sample fix, 14.4 nominal. Observed was 9.

Two results here matter more than the headline. First, the width ratio on real
data averages 1.431, *above* √2 — the finite-population structure and the size
skew add to the variance defect rather than being absorbed by it. Second, **the
naive two-sample correction reaches only 0.839, not 0.90.** Supplying the missing
variance component is necessary and not sufficient; a residual defect of roughly
six coverage points survives it, and the repair specified in §5 must address that
rather than stopping at the √2.

This measurement is diagnosis, not certification. It uses EDH, which is why EDH
cannot also certify the repair.

## 4. Precision of the decomposition

The 14.4 → 11.1 → 9 decomposition now rests on the measurement above rather than
on synthetic populations, but it remains **partial**. It covers the eight
whole-population ratio constructions and not the four stratum-conditional ratios,
and three of the sixteen targets are not simple ratios at all (two province-count statistics
rescaled to a declared holdout total, and the B2 century-share vector, which is scored
against a one-sided total-variation ceiling). The B2 ceiling carries the same defect in
a different form: it bounds how far a *resample of the visible half* drifts from the
visible anchor, when what it must bound is how far the *hidden half* drifts from it.

An exact decomposition for EDH's real targets requires the redesigned split audit and
is not claimed here. What is established here is the existence, direction and magnitude
of the width defect, which do not depend on those details.

## 5. The repair

Not "widen the intervals until EDH reaches 90%." The repair is specified by the defect:

1. **Two-sample / predictive construction.** The interval must carry both estimation
   uncertainty and future-set uncertainty. Resampling an independent pseudo-holdout and
   taking percentiles of the *difference* is the minimal correct form.
2. **No separate finite-population correction.** *(Corrected 7 August 2026; an earlier
   version of this section listed one.)* For complementary halves the identity is
   Var(z̄_H − z̄_S) = (1/n + 1/m)·S², which at n = m = 33 is exactly 2S²/33 — the
   two-sample variance. The complement's negative covariance with the visible sample
   cancels the separate finite-population corrections in the difference. Multiplying by a
   further √(1 − n/N) would count the finite-population effect twice and make the
   intervals too wide. The √2 *is* the finite-population answer, not an approximation to
   be corrected afterwards.
3. **Leverage, which is the residual.** Measured on real provinces, the two-sample
   construction reaches 0.839 against a nominal 0.90 (§3a). That gap is a
   leverage and non-regularity problem, not a missing correction: a few provinces
   dominate every record-weighted aggregate, and no non-parametric bootstrap can learn a
   giant province absent from the visible half. It requires studentization, predeclared
   leverage-eligibility gates and a bounded fallback rather than a wider interval.
   *m*-out-of-*n* subsampling is not an automatic cure and may not be the primary repair.
4. **Report interval width or a proper interval score** alongside coverage, so that a
   repair cannot succeed merely by making every interval enormous.
5. **Version it.** The corrected construction is a new protocol with its own identifier.
   Block B's original numbers are not retrofitted; they are marked miscalibrated.

## 6. Consequences for the programme

- **EDH becomes development data.** The defect was found on it, so it cannot also
  certify the repair. The corrected protocol must pass the synthetic calibration suite
  and then a genuinely untouched holdout. The census walk supplies those continuously.
- **Amendment 12** is entered: *prediction intervals must carry both estimation and
  future-set uncertainty; an interval derived by resampling only the observed side
  states a confidence level it does not have.*
- **Amendment 8's derivation is corrected.** The rule — no exchangeability assumed
  across a declared partition — stands on its own merits and is retained. But it was
  described as "the 9/16 interval failure turned into a rule," and that attribution is
  now known to be mistaken. The 9/16 failure was principally an interval-construction
  defect. Amendment 8 keeps its content and loses its origin story.
- Nothing has been promoted, so no downstream result requires retraction. The defect
  was caught at the least damaging moment available.

## 7. Provenance

The missing-variance hypothesis was raised by GPT-5.6-sol (OpenAI) during an
adversarial review of a proposed permutation study, as a check to run *before* spending
computation on that study. It was then confirmed here against the code and by the
simulation above. The proposed study's two kill-conditions were withdrawn as a result:
both explanations being partly true is precisely the case that would have made the
alphabetical result non-extreme among random splits, and the study as designed would
have read that as refuting a diagnosis that is partly correct.
