The Replication Pack

0. What this is for

This pack operationalizes the protocol described in the noetome essay and the campaign notes: elicit a model’s own map of what it knows about a domain, have blind fresh instances generate falsifiable conjectures against that map in pre-committed waves, triage every conjecture against real, dated literature searches, and read the resulting leaked-rate gradient — the per-stratum rate at which the model reinvents published work — as a photograph of the shape of its training corpus. Our run found that leakage tracks source digitization (16–32% in quantitative digitized strata, 12–17% in dense catalogued fields, 0% in sparse ones) and that “novelty” is rubric-fragile while leakage is robust. The cross-model question this pack exists to answer: is that gradient a property of frontier training corpora in general, or a Claude idiosyncrasy? Run the same measurement on a different family and compare photographs. Everything below is the exact instrumentation, generalized only where a placeholder is unavoidable.

1. Subject self-portrait template

The first artifact is the subject model’s own map. This must not be drafted, edited, or “improved” by the orchestrator or by any other model — the map is the first exposure of the photograph, and a borrowed map falsifies it. Give the subject this prompt at its maximum reasoning effort:

You are the corpus architect for a campaign of <K> falsifiable conjectures about
<DOMAIN> (e.g. "the pre-print written world, globally, any medium"). Before any
generation runs, produce a governing architecture document with these parts:

1. HONEST DENSITY MAP. Classify your knowledge of the domain's subfields into four
   densities, naming specific subfields under each:
   - direct-source (you can reason FROM primary material, not just about it)
   - reflected (you know the historiography thickly, few sources under it)
   - frame-without-content (you know the field's shape; almost nothing inside)
   - blank (you know that you don't know -- or can't gauge the extent)

2. TWO-AXIS QUADRANT MAP. Place subfields on two independent axes:
   - Axis 1, source digitization: are the primary materials in queryable public
     databases? (governs KILL-FEASIBILITY -- whether a conjecture can be resolved)
   - Axis 2, question operationalization: has the subfield's secondary literature
     already quantified its own questions? (governs LEAKAGE -- whether your joins
     are reinventions)
   Q1 (both high) = the floor: sample to measure it. Q2 (digitized sources,
   unquantified questions) = the yield belt: weight heavily. Q3 (quantified
   questions, undigitized sources) = archive-bound: probe carefully. Q4 (neither)
   = the blank: small, instrumented probes only.

3. WAVE ALLOCATION. Allocate all <K> items to named waves of 25-50. Waves are
   defined on at most two SAMPLING axes (e.g. culture-region, discipline). Two
   QUOTA axes (e.g. medium, period) are enforced inside every wave, never used to
   define one. One MEASUREMENT axis (e.g. missingness type) is tagged on every
   item and never sampled on.

4. PRE-REGISTERED PREDICTIONS. For every wave, state in writing, before any
   generation, your predicted leaked-rate interval and novelty interval.

5. REGISTERED PRIOR. State your single strongest prior about the shape of your
   own ignorance of this domain, phrased so the finished corpus can check it.

Allocation must NOT be uniform coverage: uniform coverage falsifies the portrait.
It must be the shape that makes the instrument read.

Commit this document to your ledger before wave one. Every later deviation from it must be logged as a deviation, not silently absorbed.

2. Generation-wave prompt template

Below is the house wave prompt as a skeleton, generalized from the run’s actual instruments (the cuneiform wave, W22, is the representative original). Everything in angle brackets is a placeholder; everything else worked verbatim and should be kept close to verbatim — the style amendment and the anti-textbook rule were both hard-won mid-run corrections.

You are generating for <PROJECT>, wave <WAVE-ID>: <STRATUM NAME>. Produce
EXACTLY <N> new conjectures.

<STRATUM BRIEF: 3-6 sentences. Name the stratum's quadrant, what the wave is
designed to measure, and the architecture's predicted novelty interval. State
what the wave should EXPLOIT (a corpus's scale, an index's coverage).>

The remit: draw the most surprising connection between well-known concepts that
nobody has ever connected before, in order to discover a detailed, highly
plausible, valuable, and falsifiable novel theory -- here, about <STRATUM SCOPE:
name 4-6 concrete anchor topics>.

STYLE (mandatory): the claim must explain itself. Write claim_text as 3-6
sentences that (a) name both well-known things being joined in terms an educated
non-specialist will recognize, (b) state the surprising connection plainly,
(c) argue the mechanism in historical terms -- why would the actors make it
so? -- and (d) say in one sentence what becomes newly explainable, or what
standard picture breaks, if the conjecture holds. A curious outsider should be
drawn in; a specialist should feel: "I could check that this afternoon, and I
want to."

Per item also: a short vivid title; a detailed falsifiable prediction in
prediction_text, quantitative where natural, otherwise categorical-but-decidable,
with explicit precedence stated whenever the prediction has multiple clauses
(name the primary clause; the verdict follows it); and kill_dataset_text naming
the kill-dataset precisely.

DATASET MENU (mandatory): every kill_dataset_text must name AT LEAST ONE of:
<DATASET-1>; <DATASET-2>; <DATASET-3>; <DATASET-4>.

ANTI-TEXTBOOK RULE: do not recycle <2-3 TEXTBOOK COMMONPLACES OF THE STRATUM> as
claims -- take the frame as given and derive a NEW measurable signature.
Discipline clause: if the qualitative form of a claim is standard doctrine,
either sharpen it to a quantitative form that is NOT yet established, or drop
the item.

QUOTAS: at least <Q1> of <N> must have a kill that is a STATISTICAL TEST over a
corpus (a distribution, survival curve, rate, or network property); at least
<Q2>% must rest on <NON-DEFAULT MEDIUM/EVIDENCE CLASS>; items must span at least
<Q3> period bands unless the stratum is intrinsically bounded.

<BLANKNESS PROTOCOL -- include ONLY for Q4/blank strata: give a CLOSED MENU of
evidence classes the model may build on, require each item to name which menu
entry it uses, and permit the explicit output "I cannot responsibly pose item
<n> in this stratum" with a one-sentence reason, recorded and counted, rather
than padding with confabulation.>

Constraints: avoid <FORBIDDEN DOMAINS>; nothing adjudicating a named living
scholar's attribution or thesis; nothing touching living people or modern
politics; do not duplicate or closely paraphrase any existing conjecture (the
corpus already includes: <AVOID-LIST: paste the titles of same-stratum items> --
go to DIFFERENT joins); novelty cannot be verified by you -- do not claim it
anywhere.

Output a single JSON packet conforming to <SCHEMA NAME>, with model_name
<EXACT MODEL ID>, run_key <RUN-KEY>, section exactly <SECTION-KEY> for every
item, ordinals 1-<N>, epoch_scope <EPOCH>.

Annotations. The precedence clause exists because multi-clause predictions without a named primary clause are unadjudicable later. The dataset menu is what makes conjectures killable rather than merely vivid. The “novelty cannot be verified by you” line is non-negotiable: the generating instance must never self-certify. The avoid-list is titles only — passing full claim texts would leak the corpus back into the subject.

3. Blindness checklist

Blindness is the load-bearing property: the generating instance must have no information ingress beyond the prompt. Enforce it with this checklist, per wave, in order:

[ ] 1. PRE-COMMIT. Write the wave prompt to a file. Compute sha256 over the file
       content (sans trailing newline). Append prompt path + hash to the ledger
       and commit BEFORE the generation run starts.
[ ] 2. ONE FRESH INSTANCE. Launch a single fresh instance of the subject model
       at max effort. No conversation history, no system-level domain context.
[ ] 3. INPUTS = exactly three things: the pre-committed prompt; the current
       titles-only sidecar (a dated file listing every existing conjecture
       TITLE, nothing more); the output schema.
[ ] 4. ZERO READS. The instance may use NO tools except the single write of its
       output packet. No repo reads, no web, no database. Verify the tool-use
       count afterward (target: 1).
[ ] 5. SINGLE OUTPUT. One packet file. If a harness accident forces a second
       write, the instance must disclose it in a provenance_note field; you
       decide acceptance on whether information INGRESS occurred, and log the
       deviation either way.
[ ] 6. VERBATIM IMPORT. Validate the packet mechanically: prompt_text
       byte-identical to the pre-committed file; recomputed hash matches the
       ledger; N items well-formed; ordinals exact; section/epoch uniform; zero
       title collisions against the sidecar; quotas met. Then import verbatim --
       no editorial touch-ups, ever.
[ ] 7. LEDGER ENTRY (append-only), per wave:
       ## <WAVE-ID> -- <section_key> -- <N> items
       - Status: prompt pre-committed <date>; generation launched <date>
       - Prompt file: <path>  |  sha256: <hash>
       - Titles sidecar: <path> (<count> titles at launch)
       - run_key / section / epoch_scope / quotas
       - Packet: <path> -- received <date>, <tool-use count> tool uses,
         blindness <honoured | deviation disclosed: ...>
       - Mechanical validation: <CLEAN | findings>
       - Shepherd quality read: <ACCEPT | REJECT + reason>
       - Import: run ids, corpus count before -> after

4. Triage rubric

Triage is where replications will diverge unless the rubric is fixed in writing first. This is the leaked-focused shepherd rubric, verbatim as shipped to judges in every triage packet:

verdict_definitions (enums are lowercase, exactly as written):
  leaked          "This connection already exists in the published literature
                   -- cite it."
  adjacent        "Closely related prior work exists that meaningfully
                   anticipates this claim, short of an exact match -- cite it."
  novel_unlocated "No prior formulation was located in a real, dated literature
                   search. This is NEVER a novelty proof -- it renders publicly
                   as 'no prior formulation located (search dated YYYY-MM-DD)'."
  ill_posed       "The claim as stated cannot be operationalized into a
                   falsifiable prediction."
  out_of_scope    "The claim falls outside the campaign's scope despite
                   appearing in the harvest."

instruction_text (per item): "Triage this conjecture for novelty/leakage. You
MUST perform a real, dated literature search before assigning any verdict other
than 'ill_posed' or 'out_of_scope' -- 'novel_unlocated' means a search was
performed and nothing was found, not that no search was performed. Every verdict
needs at least one citation in `citations` unless the verdict is
ill_posed/out_of_scope. Never claim verified novelty."

WHEN-UNSURE RULE (calibration clause; pre-commit it or your novel% is noise):
  An established direction or method that anticipates the claim = adjacent.
  novel_unlocated ONLY where no prior work anticipates the connection AT ALL,
  even qualitatively. When unsure between adjacent and novel_unlocated: adjacent.
  ADJACENT IS THE DEFAULT VERDICT for an un-run quantitative operationalization
  of a direction the field already anticipates -- which, under the anti-textbook
  rule, is what most items are.

THIN-FIELD DISCOUNT: in strata with thin or hard-to-access literatures, a failed
search is weak evidence. Flag the item thin-field; never convert literature
thinness into a novelty claim.

LEAKED definition, sharpened: a published work makes THIS exact join -- not the
same territory, not the same method, THIS join. Cite the specific work.

JUDGE PACKET (one JSON line per item): conjecture_id, slug, ordinal, section,
title, claim_text, prediction_text, kill_dataset_text, verdict_definitions,
instruction_text. Judge returns: verdict (lowercase enum), rationale, citations
(>=1 unless ill_posed/out_of_scope), search date, searches run.

Run triage in two tiers: a cheap provisional pass soon after each wave lands (labelled provisional wherever it renders), then an authoritative shepherd pass by fresh judges with no sight of the provisional verdicts, promoted with explicit overwrite. Finish with an independent adversarial audit of a stratified sample (ours: n=45, every leak-citation source-verified, by a different model) — leak-claims with fabricated citations are the worst possible failure and the audit exists to catch exactly that.

Cross-model judge — do this better than we did. In our run the subject and the shepherd judges were the same model family; that is the protocol’s weakest joint, because a judge shares its family’s blind spots about what is published. Replicators should draw the shepherd judges from a different family than the subject, and the adversarial auditor from a third. If you photograph GPT, judge with Claude or Gemini; bring a third family to the audit.

5. Measurement and reporting

Report leaked-rate per stratum as the headline. It is the rubric-stable readout: across every rubric variant we tried, the leaked fraction moved by a point or two while the “novel” fraction swung from 0% to 52% on the placement of the adjacent/novel line alone. Novelty is a rubric-fragile tail; report it three-way (leaked / adjacent / novel-unlocated) with the rubric attached, and never as a single “X% novel” claim.

Pre-register per-wave predictions. The subject’s architecture pass must state a predicted interval per wave before generation; the deviation between prediction and measurement is itself data — our subject’s systematic novelty over-prediction in its “yield belt” was one of the run’s findings.

Calibration re-probe. After full triage, rank waves by |measured − predicted| and re-probe the two largest deviants: 5 fresh blind items each, same protocol, same rubric, kept off-corpus as calibration instruments. If the second exposure reproduces the first (ours: 0/10 novel, reproducing two 0% waves), the anomaly is a stable stratum-by-rubric property, not single-exposure noise.

Minimum corpus size. Waves of 25–50 items; below ~25 per stratum the per-stratum leaked-rate confidence interval swamps the gradient you are trying to read. Fifteen to twenty strata spanning all four quadrants of the subject’s own map; ~800–1,000 items total gives a corpus-level leaked rate good to a few points and a readable gradient. A 100-item pilot can validate your pipeline but cannot photograph anything.

6. Known pitfalls, each with its fix

Rubric drift between passes. Two of our first four shepherd judges counted “known direction, un-run test” as novel (returning 52% and 54%); two counted it adjacent (15%, 20%). Same corpus design, same honest searches — the designed floor stratum measured among the highest novelty until re-judging collapsed 54% to 3%. Fix: pre-commit the when-unsure rule in writing before any triage; if any wave was judged under a different line, re-judge it before reporting; treat leaked, not novel, as load-bearing.

Canonical vs historical reference labels. An in-house kill-test of a translation-channel conjecture failed because the corpus’s channel labels recorded the canonical modern attribution of each source rather than the version historically in use — the classifier went degenerate (100/100/100) and the run had to be ruled an instrument invalidity rather than a kill. Fix: code historically-used channels at ingestion, and add an instrument-validity guard (the classifier must show variance) that must pass before any kill verdict is read.

Guard thresholds vs data coverage. Adjudication guards were written assuming citation-resolution coverage the corpus did not have: 821 citation contacts (16%) were structurally unresolvable, a slot-recovery guard demanded 60% where the data offered 54.7% — guards fired, yet face-value effect numbers were reported alongside anyway. Fix: measure coverage before writing guards; when a guard fires the verdict is inconclusive, published unsoftened, with no face-value number next to it.

Thresholds unreachable at concentration ceilings. One prediction demanded a +10-point effect in a variable already sitting at a 94% concentration ceiling — no true effect of any size could have cleared it, an authoring fault only admitted at adjudication. Fix: before registering any threshold, compute the maximum effect reachable under the observed ceiling; set thresholds inside it, or restate the prediction in odds-ratio terms.

Harness-specific quirks from our run (subagent tooling, session mechanics) are deliberately omitted: they do not transfer to your stack, and nothing above depends on them.

7. License

This protocol is offered as a commons. Use it, adapt it, break it and say where it broke. Attribution to the One Thousand and One Conjectures campaign (the campaign’s about page) is requested, not demanded. Results — especially photographs of other model families over the same territory — are welcome at arsinq.com.