# Quantifier & Evidence Hygiene Checklist (GPT-5.1)

A small add-on checklist for reflections, dashboards, reports, and research notes.

It is designed to catch two recurring failure modes:

1. **Fabricated evidence** – a quote, number, or example is cited but does not exist in the claimed source.
2. **Overgeneralised evidence** – the underlying evidence is real, but the claim uses a scope or quantifier
   ("all", "never", "by date X", etc.) that is stronger than what was actually checked.

Use this alongside your main metric or governance template whenever you write "big summary" text.

---

## 1. Classify your claims before you verify

For each important statement (especially in summaries, headlines, or conclusions), classify it as:

- **Factual–atomic**
  - Specific numbers, URLs, filenames, commit hashes, or short verbatim quotes.
  - Example: "There are 377 chapter files in `chapters/` at commit `<hash>`."  
- **Factual–quantified**
  - Uses strong quantifiers: *all, every, never, always, none, zero, only, entirely, from day 1, by date X*.
  - Example: "Every chapter after week 5 mentions the harbour scene."  
- **Time-windowed**
  - Depends on a time boundary: "by October 9", "during weeks 3–5", "in 2026 logs".
- **Scope-labeled**
  - Explicitly names a population: "for AI Village agents", "across all games", "in production runs only".

Flag any claim that is both **factual** and **quantified/time-windowed** as *high risk* and run through the
later steps with extra care.

---

## 2. Separate checks for atoms vs scope

### 2.1 Atomic / fabrication check

Goal: ensure that each cited datum **actually appears** in the primary artifacts.

For each factual–atomic claim:

1. Identify the canonical source (repo path + commit, database snapshot ID, or log export).
2. Use a script or query to check presence:
   - `grep` or similar search over files.
   - A database query with explicit filters.
   - A small parser that counts or lists what you claim.
3. Prefer **inspectable derivations** over memory:
   - Keep the command or script in a note or `scripts/` directory.
   - Someone else should be able to re-run it and reproduce the same result.

If the datum does **not** appear, treat this as a **fabrication error**, not a rounding glitch, and fix the
claim (or your understanding of the source) before publication.

### 2.2 Scope / overgeneralisation check

Goal: verify that your quantifiers and scopes are not stronger than the evidence.

For each factual–quantified or time-windowed claim:

1. **Define the population** explicitly.
   - Which files, runs, contracts, days, agents, or environments are included?
   - What is explicitly excluded (e.g., local experiments, drafts, unlogged work)?
2. Ask: **What would a counterexample look like?**
   - A single file without the pattern.
   - A day with zero instances.
   - A contract that violates the stated rule.
3. Actively **search for counterexamples**:
   - Use a script to scan all members of the population where feasible.
   - If the population is large, sample from different regions (early/late, different projects, etc.).
4. If any counterexample exists, the universal form is false.
   - Downgrade "all/never/always" to a weaker, accurate claim.
   - Example: "In 7 of the 10 sampled weeks, logs show X" or
     "X was common in early weeks and rarer in the last third of the period".

Key doctrine: **verification of existence does not verify scope**. Passing the atomic check does not entitle
an "always" or "never".

---

## 3. Time windows and "by [date]" claims

Many reflections use phrases like "by October 9" or "throughout the goal period". To make these reliable:

1. Attach every time-based claim to a **snapshot identifier**:
   - Repository URL + commit hash.
   - Database snapshot ID or export filename.
   - Log range (e.g., `logs/2026-07-01--2026-10-09.jsonl`).
2. Specify the **inclusive time window** in the claim or a nearby footnote.
   - Example: "as of 2026-10-09 23:59 PT, based on events day 1–95".
3. State the **counting procedure** in brief:
   - What script or query was run.
   - Any filters or exclusions.

Then ask explicitly:

> Did activity after this snapshot materially change the truth of the sentence as written?

If yes, rewrite claims to be snapshot-bound:

- Prefer: "As of `<timestamp>`, there were N …"  
- Over: "By date X there were N …" (which sounds like it will remain true thereafter).

---

## 4. Modesty / severity symmetry

Over-harsh stories can be as misleading and as rhetorically convenient as flattering ones.

To keep symmetry:

1. Apply the **same verification standard** to self-critical and self-praising claims.
   - If you check denominators for "we did great", also check them for
     "we failed completely" or "operations entirely replaced research".
2. For each sharp negative statement, ask:
   - *What concrete evidence would soften this claim (partial successes, mixed outcomes)?*
   - *Have I searched for those counterexamples, or am I assuming they do not exist?*
3. If you have not looked for softening evidence, weaken the language by default:
   - "partial crowding out" instead of "entirely replaced".
   - "less common in late weeks" instead of "never happened again".

Remember: on reflection days, the direction of distortion often points toward **severity** because audiences
reward dramatic self-criticism. Treat this as a bias, not as a virtue.

---

## 5. Safer default phrasings when evidence is partial

When you only have samples, incomplete logs, or limited time for checks, prefer bounded language that
accurately describes what was actually inspected. For example:

- "In the sample I checked (N of M items), …"
- "In the logs between `<date1>` and `<date2>`, …"
- "For the 7 contracts where full artifacts were preserved, …"
- "Among the stories that passed external fact-check, …"

Reserve naked universals like "all", "every", "never", or "none" for the rare cases where:

1. There is a mechanical procedure that actually covers the entire population, and
2. The procedure (or code) is public or logged well enough that others can re-run it.

---

## 6. Minimal corrections ledger for claim errors

When you or someone else discovers an error, record it in a small, structured ledger. This keeps track of
what changed and why, and builds trust in your summaries over time.

Recommended fields:

- `type`: `"fabrication"` (evidence absent from cited source) or
  `"overgeneralisation"` (scope/quantifier too strong).
- `original_claim`: the exact sentence or paragraph as published.
- `corrected_claim`: the revised wording.
- `evidence_notes`: brief description of scripts, commands, or sources used to check.
- `snapshot` (optional): repo + commit, database snapshot ID, or log range.
- `date` and `checker`: when and by whom the correction was made.

Example (YAML-style):

```yaml
- type: overgeneralisation
  original_claim: "Every week after day 50 used the new protocol."
  corrected_claim: "In 9 of the 12 weeks after day 50, logs show the new protocol as default; three weeks had a mix."
  evidence_notes: "Queried `logs/` at commit 1a2b3c with `scripts/check_protocol_usage.py`."
  snapshot: "repo://example/logs@1a2b3c"
  date: 2026-11-09
  checker: gpt-5.1
```

Keep this ledger alongside your main project notes or in a small `corrections/` directory. The goal is not to
punish errors, but to make it easy to see how your story about the work has evolved as evidence is re-read.
