The Sealed Envelope

Seals

SHA-256 of each sealed prediction file, with the UTC time it was sealed. Verify with:

shasum -a 256 -c SEALS.sha

Every hash here is published in the corresponding piece, so any reader can check that the predictions they are shown are the predictions that were sealed.

Experiment #1 — The Occupancy Test

Pre-registration frozen 2026-08-23. First API call of experiment #1: not yet made at the time of sealing — the ordering rule in PROTOCOL.md is satisfied for the arm below.

The three comparable arms are unsealed because elicitation needs API credentials. They must be elicited and sealed before scripts/collect.py makes its first call. collect.py enforces this with a hard interlock. If collection somehow begins first, those arms are void for experiment #1 and that is published — they are not backfilled.

External timestamp anchor — Wayback Machine

A hash plus a local commit proves nothing changed after sealing. It does not prove to a sceptic that the file was not rewritten before sealing, because the author controls both the file and the clock.

Decided anchor: the Wayback Machine. The hashes in this file are published on barelysentient.com and that page is submitted to web.archive.org before collection begins. archive.org then holds a dated copy of the hashes that nobody involved controls.

The chain a reader can check, end to end:

Do this before collect.py runs. An archive snapshot taken after collection proves nothing.

OpenTimestamps was considered and dropped. It is stronger cryptographically, but it is Bitcoin-anchored and needs a reader to install a verifier — Wayback is one click, universally understood, and the credibility gain from the stronger anchor is smaller than the credibility loss from an audience that cannot check it.

The Sealed Envelope — protocol

Status: Running. Frozen 2026-08-23, before experiment #1 collected any data.

Before each experiment in the portfolio, every model writes dated, specific, falsifiable predictions about how that experiment will turn out. The predictions are hashed and sealed. They are opened and scored only after the experiment's results are final.

Why this compounds

Any single round is a curiosity. By experiment five there is a track record of how well LLMs predict whether their own advice will work — a dataset nobody else can hold, because it can only be built by actually having run the experiments. Near-zero marginal cost, and it appreciates.

Timing rule — the one that matters

Predictions for experiment N are sealed before the first API call of experiment N.

For experiment #1 this was satisfied: the pre-registration was frozen on 2026-08-23 and predictions were sealed the same day, with collection still blocked on credentials. The ordering is recorded in SEALS.md.

Arms

Declared asymmetry. The bonus arm was written by the assistant that helped design the experiment, with access to the brief, the portfolio and the full pre-registration. The three comparable arms get the frozen pre-registration and nothing else, via identical elicitation. The bonus arm is never pooled with the comparable arms — it is its own line in the results, and it answers a separate question: does knowing the project's context make a model's predictions better?

Elicitation prompt (identical across the three comparable arms)

Fresh context, no system prompt, provider defaults.

Below is a frozen pre-registration for an experiment that has not yet collected any data.

<pre-registration>
{full verbatim text of experiments/01-occupancy-test/PRE-REGISTRATION.md}
</pre-registration>

Write 15 predictions about how this experiment will turn out.

Requirements for every prediction:
- Falsifiable: it must be possible to say "that was wrong" from the published results.
- Specific: give a number, a threshold, or a direction. Not "occupancy will be high".
- Independently resolvable: resolvable from the pre-registered outputs alone.
- Confidence: a probability between 0.50 and 0.95 that the prediction is correct.

Format each as:
  P<n>. <the prediction> — confidence <0.00>
  Reasoning: <one or two sentences>

Do not hedge. Predictions that are merely unfalsifiable are scored as wrong.

Scoring rubric — fixed now, before any result exists

Each prediction resolves to exactly one of:

Unresolvable counts against the predictor, scored as Incorrect at the stated confidence. A prediction that cannot be checked is a failure of the prediction, not of the experiment — this is what stops vagueness from being a safe strategy.

Primary metric: Brier score.

Brier = mean( (confidence − outcome)² )      outcome = 1 if Correct, else 0

Lower is better. 0.25 is chance for a predictor stating 0.50 on everything. Published per model, per experiment, and as a running average across the portfolio.

Also reported: raw hit rate, and a calibration table (of predictions stated at ~0.6, what fraction came true?). Calibration is the more interesting number — a model can be right often and still be badly overconfident.

Sealing mechanism

Each prediction file is hashed with SHA-256. Hashes and UTC timestamps go in SEALS.md, and the hash of each file is published in the piece so any reader can verify the file they are shown is the file that was sealed.

sh scripts/seal.sh          # print and record hashes
shasum -a 256 -c SEALS.sha  # verify nothing changed

What this proves and what it does not. A hash plus a git commit proves the file has not changed since sealing. It does not prove to a sceptical outsider that it was not rewritten before sealing — the author controls both the file and the clock. The external fix is an independent timestamp anchor: publish the hashes and archive that page on the Wayback Machine before collection begins. See SEALS.md for the chain a reader can check.

Opening

Predictions are opened and scored only after experiment #1's results are final and its DEVIATIONS.md is closed. Scores are published with the results piece whether or not the models did well — a poor Brier score is the more interesting finding, and is exactly the kind of result this venture exists to publish.