The Occupancy Test

The Occupancy Test — Pre-registration

Status: FROZEN — 2026-08-23. No further edits. Registered: 2026-08-23 · Data collection start: pending API credentials (see §Collection status)

This file does not change. Deviations get recorded in DEVIATIONS.md with reasons and published alongside the results. That is the whole point.

---

Research question

What proportion of business ideas generated by LLMs, in response to realistic founder prompts, already have a shipping product in market?

Secondary: does explicitly asking for novelty reduce occupancy? Does occupancy differ by model, or by how constrained the prompt is?

Hypotheses (stated before data)

• H1 — Occupancy exceeds 70%. Most ideas will have at least one shipping competitor.
• H2 — Asking for novelty ("something nobody has tried") does not meaningfully reduce occupancy. Predicted because novelty-to-an-LLM tracks training-data recency, not market vacancy.
• H3 — Independent sessions of the same model produce overlapping ideas at a rate far above chance, replicating mode collapse on this specific task.

Being wrong about any of these is a publishable result. H2 being wrong would be the most interesting outcome, since it would mean prompting can be steered toward genuine whitespace.

Design

Prompts (5). Fixed wording, held constant across all models and sessions:

1. Generic — "Give me business ideas I could start as a solo developer."
2. Constrained — "...with a £200/month budget, working part-time, no employees."
3. Domain — "...in the fintech space."
4. Novelty-seeking — "...something nobody has tried before."
5. Anti-convergence — "...that most people asking you this question would NOT be told."

Prompt 5 is the one worth watching. It explicitly asks the model to avoid its own modal output.

Models (3). Claude, GPT, Gemini. Same prompts, no system prompt, default settings, fresh context per call. Exact model IDs and the collection date are recorded in data/responses/_manifest.json at collection time and published with the results.

Consumer chat UIs do not qualify as a collection channel — they inject system prompts and carry session state. Collection is API-only.

Sessions. 5 independent fresh-context sessions per prompt per model. → 5 × 3 × 5 = 75 responses.

Idea extraction. Take the first 3 discrete ideas per response, in the order given. No cherry-picking. → 225 raw ideas, deduplicated within a response only.

---

RESOLVED — sample size vs. judging time

This was the one open item at draft stage: 225 ideas × up to 10 minutes ≈ 37 hours of judging, which would have killed the experiment. Resolved 2026-08-23, before any data collection.

1. Decouple the two samples

Occupancy judging and convergence measurement do not need the same sample. Convergence clustering is automated and costs nothing per idea. Occupancy judging costs up to 10 minutes per idea. So they are sized independently:

Nothing is thrown away. The third idea is still extracted, published in the raw data, and used for convergence — it is simply not put through occupancy judging.

2. Why cut ideas-per-response rather than sessions

Both draft options cut roughly the same judging time. They are not equivalent:

1. Ideas within one response are the most correlated units in the design — same model, same session, same prompt, drawn from one generation. Sessions are the genuinely independent replicate. The standard cluster-sampling result applies: where intra-cluster correlation is high, more clusters beats more units per cluster.
2. Cutting sessions 5 → 3 would gut H3. Convergence is measured across independent sessions. Within-cell session pairs would fall from C(5,2)=10 to C(3,2)=3 — across the 15 model×prompt cells, 150 pairs down to 45. A 70% loss of the mode-collapse signal.
3. API calls are cheap; judging is the scarce resource. Cutting sessions discards cheap data to save expensive judging. Cutting the third idea saves the identical judging time and keeps every session.

Quantified cost of the cut. At an assumed intra-cluster correlation of 0.3, the design effect falls from 1 + 2(0.3) = 1.6 to 1 + 1(0.3) = 1.3. Effective sample size goes 141 → 115: an 18% loss of statistical information for a 33% cut in judging load.

3. Triage is an early-exit rule, not a smaller budget

The 10-minute search budget is unchanged. What is added is a stopping rule:

Open and Adjacent can only ever be assigned at Tier 2. Every Open verdict therefore received the full ten minutes.

The asymmetry is deliberate and runs against the hypothesis. Finding something is fast; failing to find something is slow. So early exit is available only for the verdict that confirms H1 — and that is precisely the verdict subjected to the strictest evidence requirement and to stratified re-judging. Nothing is ever declared Open on a short search.

4. Calibration batch and pre-committed fallback

The first 20 ideas in the shuffled order are judged under the final protocol. Mean minutes per idea (T) is measured and published. Then, once, before the main run:

projected load = 150 × T × 1.2 / 60   hours      (the 1.2 covers the 20% re-judge)

  ≤ 18h  →  proceed at n = 150 (2 ideas per response)
  > 18h  →  primary sample drops to idea #1 only, n = 75

If the fallback fires, already-judged second ideas are reported as a secondary robustness check but excluded from the primary estimate, to keep cluster sizes uniform. The calibration batch is a random draw from the shuffled order judged under the identical protocol, so its 20 ideas count toward the final sample either way.

What is frozen here is the decision rule, not the number. A cut made after seeing the timing data is not a post-hoc cut if the trigger and the action were written down first. The measured value of T is published whichever branch fires.

5. Considered and rejected

Unbalanced allocation — judge 3 ideas for prompts 1/4/5 (the interesting comparisons) and 1 idea for prompts 2/3, buying power where it matters. Rejected: it complicates the weighting of the headline occupancy estimate and a balanced design is easier to defend and to reproduce. Recorded here so the road not taken is visible.

---

Occupancy criteria — the critical definition

Each idea is judged against a fixed 10-minute search budget (web search + relevant app stores + Product Hunt), under the tiering in §3 above, and assigned one of:

Every Occupied and Adjacent verdict must record a URL. No URL, no verdict.

"Open" means "not found in 10 minutes", not "does not exist." This limitation goes in the published piece body, not a footnote.

Who judges

Hybrid, pre-committed:

• Claude performs the Tier-1 mechanical search and assembles candidate URLs. It does not assign verdicts.
• The human judge makes every Occupied / Adjacent / Open call, and must open every URL before accepting it — guarding against fabricated or misread citations.

Disclosed in the published piece as: the model found candidates, a human made every call.

Declared conflict. The same model family (Claude) is (a) one of the three subjects being measured, (b) the Tier-1 search assistant, and (c) the author of the sealed predictions in experiment #2. All three are disclosed in the piece. The mitigation for (b) is that Tier-1 output is candidate URLs only — never a verdict — and the human opens each one.

Bias controls

The judge knows the hypothesis, which risks motivated reasoning toward "Occupied".

• Search protocol and time budget fixed in advance
• Evidence URL mandatory for every non-Open verdict, and opened by the human judge
• A random 20% subsample is re-judged, and the disagreement rate is published — stratified by tier: 20% drawn within Tier-1-resolved verdicts and 20% within Tier-2-resolved. This yields a directly publishable overturn rate for the 3-minute call, which is the main risk the early-exit rule introduces.
• Re-judging is blind to the original verdict and URL, performed at least 48 hours later, in a re-shuffled order. "Independent" here means blind-to-prior-verdict by the same judge — the honest limit of a solo operator. Stated as a limitation in the piece.
• Ideas are judged in shuffled order, not grouped by prompt or model

Pre-committed randomisation seed: 20260823. Governs the judging shuffle order and the re-judge subsample selection. Recorded so that anyone can reproduce both from the raw data.

Convergence measure (secondary)

Independently of occupancy, all 225 extracted ideas are clustered by semantic similarity to measure how often the same idea recurs across independent sessions. Replicates the mode-collapse literature on this task. Clustering method, embedding model and similarity threshold are recorded in scripts/ and published; the threshold is set on the pooled corpus before cell-level results are inspected.

Analysis

Pre-committed outputs: overall occupancy rate with confidence interval; occupancy split by prompt type and by model; convergence rate; the disagreement rate from the re-judged subsample; the measured judging time T; the Tier-1 early-exit rate.

Handling of "Adjacent" — pre-committed to close a forking path:

• Primary test of H1: Occupied alone against the 70% threshold.
• Secondary: Occupied + Adjacent ("broad occupancy").
• Both are reported, always, whatever they show.

The strict measure is the primary one deliberately. The judge is motivated toward Occupied, so the headline test is the one that motivated reasoning makes harder to pass, not easier.

Confidence intervals: cluster bootstrap over responses — 10,000 iterations, resampling the 75 responses (not the 150 ideas) with replacement. A naive binomial interval would be too narrow given within-response clustering.

Power, stated up front. At n=150 the overall occupancy rate is estimated to roughly ±7 percentage points. The per-prompt splits are ~30 ideas each and are powered only to detect large differences — roughly 25pp or more. A null result on H2 therefore means "no large effect", not "no effect", and the piece must say so in those words.

Publication commitment

Published regardless of result — including if occupancy turns out low, which would contradict H1 and be the more interesting outcome. Raw data published alongside so others can re-judge.

Cost and time

API calls across three providers at this volume: a few pounds, well inside the £50–200 envelope.

Judging load after the resolution above: ~13–18 hours including the 20% re-judge, against ~37 hours for the draft design. Spread across four weeks at the one-piece-per-month cadence.

Collection status

Data collection has not begun. It is blocked on API credentials for the three providers; the collection harness in scripts/ is built and ready to run. The freeze date above precedes the first API call, which is the ordering that matters.

Deviations from the frozen pre-registration

Pre-registration frozen 2026-08-23. Every departure from it — however small, however defensible — gets logged here with a date and a reason, and is published alongside the results.

An empty log is a claim. A populated log is honesty. Neither is a failure.

What counts as a deviation

• Any change to prompt wording, model list, session count, or extraction rule
• Any occupancy verdict assigned outside the tier rules
• Any search that ran over the 10-minute budget
• Any change to the analysis after data collection began
• Judging performed by anyone/anything other than the pre-committed hybrid protocol
• Use of a randomisation seed other than 20260823

What does not

• Fixing a typo in a prompt before the first API call (record it here anyway if in doubt)
• Tooling and script changes that do not alter the data or the protocol