# How the atlas datasets were made, and what to doubt

This note travels with the three datasets (route history, station
history, the historical population surface) and the distance series
built on them. It says who did the work, what was decided by hand, which
numbers in the pipelines were measured and which were chosen, which
checks are blind and which are not, and how the inputs are pinned. Each
dataset's own README carries its schema and sources; the per-dataset
accuracy reports carry the numbers. Nothing here replaces those.

## Who did the work

The three pipelines were built between August and September 2026 by one
researcher, an AI agent (Anthropic's Claude, run through Claude Code)
directed by the project's owner, who set the goals, chose the sources
that could be used, obtained the permissions, and reviewed the outputs.
The agent wrote the code, fetched and read the sources, ran the optical
character recognition (Apple Vision through macOS, which reads the page
and the cell crops, with Tesseract's legacy and LSTM engines as second
and third readers and a vote across them; the Vision step runs only on
macOS, so the transcription stage of the bundle is not portable), made
the hand decisions listed below, and wrote the documentation. There was no second rater for any hand decision and
no external audit. Every hand decision cites its source in the data so
that a reader can check it; the checks that exist are described below
with their sample sizes.

## Hand decisions, counted

| dataset | file | what it holds | count |
|---|---|---|---|
| route history | `manual_provenance.csv` (from `manual_dates.json`) | dated, reclassified or reviewed segments, each with its source URL or a stated verdict; 1,004 by name and 3,471 by segment id | 4,475 |
| station history | `manual_station_dates.json` | dates that override every automatic source | 485 |
| station history | `upstream_corrections.tsv` | corrections filed against Wikipedia or Wikidata, applied here | 599 |
| station history | `upstream_manual.json`, `dropped_stations.tsv`, `contested_dates.tsv` | hand additions, rows dropped as not stations, dates left contested | 43, 176, 65 |
| station history | `non_passenger_stations.json` | rows adjudicated as never public passenger stations | 302 |
| station history | `synthetic_duplicates.json`, `osm_duplicates.json`, `coordinate_overrides.json` | duplicates and positions settled by hand | 33, 27, 2 |
| population surface | `nrs/table8_hand.csv` | Scottish parish cells transcribed by hand where the OCR failed | 32 |
| population surface | `calibrate_scot_1981.py` | 1981 Scottish parishes overwritten with the 1991 Monitor's figures | 136 |
| population surface | 1831 Comparative Account | pages re-aligned after a column shift was detected | 5 |

## Hand checks, and their size

- Station history: on 2026-09-01, 25 rows drawn at random were put
  beside their Wikipedia articles and adjudicated by the same researcher
  who built the table: 23 clean, one arguable, one wrong (both fixed).
  This is a check by the builder, not an independent audit, and 25 rows
  bound the error rate only loosely: one wrong in 25 gives a 95%
  Clopper-Pearson interval of 0.1% to 20%, and two in 25 (counting the
  arguable row) 1% to 26%.
- Route history: the dates carry an accuracy table of their own since
  2026-09-13 (`route_history/date_accuracy.json`, summarised in the
  accuracy report): per confidence class, agreement of each segment's
  opening year with the earliest Quick-dated station standing on it
  (segments dated from stations excluded), the station-bound screen as
  a lower bound on wrong openings, and a frozen random sample checked
  by hand against a source the segment's date did not come from, with a
  Clopper-Pearson interval. An early hand sample of the loose name-match dates
  (about 1,100 km carry them) ran about three in four right, but the
  sample's size and date were not recorded, so that figure is anecdotal
  (n unrecorded) and the dates are marked low confidence. Station-to-line
  date prediction was 62% within five years on 2,516 hand-dated segments.
- Population surface: the reading of the Scottish parish table was gated
  on cells whose values were already known from other sources (the
  measuring stick), and the 1921 column was compared with an independent
  second source, 406 of 412 identical. The 1841 and 1921/1931 England and
  Wales tables prove their rows by the page's own arithmetic (persons =
  males + females; district totals).

## Which checks are blind, which are watched, which are targeted

A check is blind when the thing checked was never seen while choosing
what to fix; watched when it was on screen throughout but never used to
select rows; targeted when it generated the worklist. Nothing here is
blind in the strict sense. Per dataset:

- Route history: the route-mile benchmark table was watched throughout
  the build and never used to select segments, and the route-not-track
  correction was made knowing it would move the 1851 and 1861 rows. Its
  1851 row is the stated route length of the Campop 1851 rail lines GIS
  (UK Data Service SN 852991), the same GIS that generated re-dating
  worklists, so agreement at 1851 is targeted; the 1861 and 1881 rows
  are the stated lengths of that group's later deposits (SN 852992,
  SN 852993, never downloaded); 1900, 1920 and 1940 are a transcription
  of the Railway Year Book; 1914, 1980 and today are round conventions.
  The Branch Line Society chronology generated a worklist, so agreement
  with it is targeted. The station gazetteer both dates segments and
  audits them.
- Station history: the DfT NaPTAN check of today's network is blind (it
  joins on codes, not names). The Wikipedia closed-station lists were
  used to find missing stations, so the "220 missing" figure is a
  count of what those lists reveal, not an upper bound. The
  corroboration table in the README is recomputed at every publish from
  the per-source evidence files; corroboration by a second source is
  not independence, because Wikidata often imports from Wikipedia.
- Population surface: national totals against the published census
  series are watched (they gate every build) and were never used to
  place anything. The unit-width error model was calibrated on 1971
  enumeration districts and applied to every modelled year: it is a
  ceiling, and where a tier can be checked directly with 1841 parish
  points (sub-districts at 1851 and 1861, parishes at 1921 and 1931) the
  check sits well inside it (1851 p90 0.58 modelled against 0.31
  checked; 1921 0.50 against 0.08). The check is a floor: it holds the
  1841 name-placed points as truth, so a parish on the wrong village is
  invisible to it, and it says nothing about the counts. Both are in
  `distance_percentiles.csv` (`p*_err_mi` and `check_bias_*`), the chart
  draws the ceiling, and the years with no check (1871 to 1911, 1951,
  1961) have only the ceiling, with the direction of its error there
  unknown. The early tier's name placement is constrained, since
  2026-09-13, by the 1852 index of parishes to registration districts
  for the names the index knows; its error term is therefore an
  estimate, the constrained rows re-placed county-wide and the shift
  scaled to the share the index does not know (VALIDATION.md), and a
  wrong village inside the right district is not seen by it either. The ± is a combined heuristic band, not a confidence
  interval: measured shifts, one estimated term and one bound in
  quadrature, with no coverage probability. The missing-station sensitivity in the same
  file is a lower bound derived from the station table's own count of
  what it lacks, which covers only the counties Wikipedia lists.

## Numbers that were measured, and numbers that were chosen

Every pipeline has thresholds. The full inventory, by file and constant
name (the files are in the published source bundle), is below; this is
the summary a reader needs.

Measured or calibrated against something outside the rule itself:
the population surface's error terms (placement, extrapolation, stale
basis, coverage, Scotland model); every gate of the Scottish parish
reading; the sub-district partition (rival partitions measured the same
way); the station table's match radii and 70-year lifetime guard
(calibrated on 8,682 sourced openings); the 500 m segment-inheritance
radius (84% agreement on the known network); the route benchmark buffer
(sensitivity from 300 m to 1 km reported).

Justified by a named failing case rather than a sweep: the 5 km station
boundary tolerance (Perth, Newmarket), the Branch Line Society path cap,
the parallel-twin collapse settings (Cornish Main Line), the station-bound
band.

Chosen by eye, no measurement on record: the route segmenter's 35 m
corridor, 80% absorption, 60 m end snap, 45 m tee snap, 20 km cap and
60 m fragment floor; the ghost-cluster and coverage-gap radii; the
population surface's fuzzy name cutoffs (0.86, 0.85, 0.75, 0.7, 0.6,
0.5) and 15 km label spread; the 1 km distribution cell; the station
pipeline's 5 km, 600 m and 100 m radii and 0.85 first-word ratio; the
whole integer scoring table of the Quick's Chronology matcher. The
segmenter's 35 m collapse is known to leave short duplicate throat
segments and to class about 450 miles of real main line as siding where
two representations run side by side.

### Threshold inventory

Route history (`Atlas/rail_history/`):

| file | constant | value | meaning | basis |
|---|---|---|---|---|
| build_segments.py | BUF_M | 35 m | parallel-track corridor half-width | eye |
| build_segments.py | ABSORB_FRAC | 0.80 | share of a strand inside the buffer to absorb it | eye |
| build_segments.py | END_SNAP_M | 60 m | endpoint clustering radius | eye |
| build_segments.py | TEE_SNAP_M | 45 m | junction-touches-interior split radius | eye |
| build_segments.py | MAX_SEG_KM | 20 km | segment length cap | eye |
| build_segments.py | MIN_ISOLATED_M | 60 m | isolated fragment floor | eye |
| attribute_dates.py | ADJ_M | 90 m | adjacency inheritance for undated fragments | eye |
| attribute_dates.py | tol / need | 50 m / 0.95 | lifted-parallel demotion | named case |
| attribute_dates.py | tol / need / len_tol | 50 m / 0.95 / 0.25 | parallel-twin collapse | named case (Cornish Main Line) |
| stage_dates.py | TOL | 5,000 m | boundary station off-line tolerance | named cases (Perth, Newmarket) |
| stage_dates.py | NEIGHBOUR_M, MIN_COVERAGE, MAX_STRADDLE | 80 m, 0.80, 0.40 | stage walk guards | eye |
| split_at_stages.py | ON_LINE_M, MIN_GAP_Y, EDGE_M, NEAR_M | 60 m, 2 y, 500 m, 8 km | stage split rules | eye |
| station_stages.py | ON_LINE_M, K_NEAREST, MIN_SEGS | 60 m, 2, 3 | station-dating guards | eye |
| station_bound.py | dist, maxkm, gap | 60 m, 6 km, 10 y | station-bound audit band | reported at chosen settings |
| ghost_clusters.py | SNAP, BRIDGE, CONTEXT | 100, 400, 300 m | ghost clustering | eye |
| bls_closures.py | SNAP_M, SLACK_M, MAX_RATIO, MAX_PATH_M, COVER, AGREE_Y | 400 m, 3 km, 3.0, 60 km, 0.35, 2 y | BLS join | named case |
| coverage_gaps.py | NEAR_M, BRIDGE_M | 300, 400 m | coverage gap detection | eye |
| invariants.py | date gates | after 2030, before 1825, life of 2 y | build refusals | stated |
| benchmark_campop.py | BUF | 300 m | Campop agreement buffer | sensitivity 300 m to 1 km reported |
| impute_estimates.py | hot-deck | own dated distribution | estimates for undated rows | aggregate check against benchmarks |

Station history (`Atlas/stations_history/`):

| file | constant | value | meaning | basis |
|---|---|---|---|---|
| build_station_years.py | INHERIT_MAX_M | 500 m | nearest-segment date inheritance | measured, 84% |
| build_station_years.py | MATCH_MAX_M and others | 400, 120, 250 m | OSM to Wikidata matching, aliases | calibrated on 8,682 sourced openings |
| build_station_years.py | lifetime guard | 70 y | take the oldest route past a human lifetime | calibrated (same run) |
| build_station_years.py | uncertainty label | 100 m | "association uncertain" beyond it | measured distribution (p50 36 m, p90 222 m) |
| build_station_years.py | IMPUTE_CAP, donor window | 1975, ±15 y | closure hot-deck | eye |
| attach_station_lines.py | MAX_M | 100 m | station-to-line attachment | eye |
| apply_quick.py | similarity | 0.85 | first-word match to a Quick heading | eye |
| match_quick.py | scoring bands | 2, 20, 30 y; 25, 60 km | Quick match score | eye |
| check_invariants.py | gates | 9,040 rows; 2,642 ± 100 open; p50 1.05 ± 0.35; p90 4.1 ± 1.2 | publish gate | measured 2026-08-26 |

Population surface (`Atlas/histpop/`):

| file | constant | value | meaning | basis |
|---|---|---|---|---|
| series.py | REF_ERR, REF_W, EARLY_ERR, EXTRAP_ERR, STALE_ERR, COVER_ERR, SC_MODEL_ERR, MARGIN_DAYS, MISSING_DRAWS | see file | error terms combined in quadrature, which assumes they are independent (the placement terms correlate, so the band understates); record-date margin 31 days; 20 missing-station draws with the spread reported | each measured |
| check_invariants.py | TOLERANCE | 1.5% | national totals | stated |
| geocode_1831.py | FUZZY_CUTOFF, SPREAD_LIMIT_M | 0.86, 15 km | 1831 name placement | eye |
| build_distributed.py | CELL | 1 km | settlement cell | eye |
| build_rsd_points.py | tolerances | 0.05 to 0.005 log-ratio; 0.75/0.85; 0.5; 0.03 | sub-district join and claim | partition measured; cutoffs eye |
| build_ew_parish_points.py | cutoffs and tolerances | 0.86, 0.5; 1.5%, 1%; 25% | parish tier proof and join | eye, checked by closure counts |
| build_1841_points.py | block-total checks | 0.1%, 0.5% | 1841 printed totals | eye |
| build_scot_1921_1971.py | WINDOW, proven(), scale bands | 0.7 log-ratio; exact-cell gates; 0.5 to 2.0 | Scottish parish reading | measured on known cells (99%) |
| assemble_table8.py, grade_table8_cells.py, classify_table8_glyphs.py | vote and glyph gates | 2 agreeing readers; k = 3 | OCR acceptance | measured (95 to 98%) |
| extract_ew_t3.py | column plausibility, skew | p90 under 1,000; slope 0.02 | 1921/1931 page geometry | eye, checked by row sums |

## What can be checked, and where

The pipeline source is published beside the datasets as
`atlas_pipelines_src.zip` (every script and build note of the three
pipelines and the shared scripts, no data; the agent logs, the source
reconnaissance and the correspondence with archives are left out, and
contact addresses in fetch user-agents are redacted), for inspection and
reproduction; the file references in this note point into it.
The transcriptions of the printed census volumes the population surface
rests on are published in `population_surface/` as `transcription_*.csv`,
row by row with the proof each row passed, so the figures can be checked
against the page images. The input pins of each pipeline are published
under `input_pins/`. The route dataset's hand decisions are
`manual_provenance.csv`; the station table cites a source URL on every
date. The station and route pipelines' fetch caches and the population
surface's page images are not redistributed. The audit that keeps this
note and the READMEs consistent with the shipped files is
`scripts/audit_publication.py` in the bundle; it runs in the site's test
suite.

## What the inputs were, exactly

The upstream sources move: OpenStreetMap, Wikidata and Wikipedia change
daily, NaPTAN monthly, histpop.org and the Internet Archive rarely but
without notice. Each pipeline's `data/static/input_pins.json` records the
sha256, size and modification time of every external input as it was
when the outputs were built (a directory is pinned by the digest of its
file listing), written and checked by `scripts/input_pins.py`; the
three files are published as `input_pins/rail_history.json`,
`input_pins/stations_history.json` and `input_pins/histpop.json`. The
station table that the population series was computed against is
pinned separately by hash (`histpop/data/station_input.pin`), and the
published population surface carries `SHA256SUMS`. The Geofabrik
extract of Great Britain was downloaded on 2026-08-28; the station
pipeline's Overpass, Wikidata, Wikipedia and NaPTAN fetches are kept
byte for byte in git; the route pipeline's fetch caches and the
histpop.org page images are pinned by listing and not redistributed.

## Definitions that shape the numbers

- Route "open" in a year: opened by then and not closed, or reopened;
  a line closed to passengers but open to freight counts as open. Route
  miles, not track miles; heritage lines count for their public life only.
- A station is a public passenger station on any system except trams;
  Northern Ireland and the Isle of Man are out of scope; closure means
  closure to passengers, with the last closure held separately.
- Population is the census count as each census defined it: persons
  present (the enumerated population) in every census to 1971, usually
  resident from 1981 (the 1981 table's present and absent residents
  summed), and the base change is not adjusted. 1951 and 1961 in England
  and Wales rest on districts; Scotland 1971 is the 1981 surface scaled.
- Distances are straight-line, from the population point to the nearest
  station open on the census night. The census dates are in
  `histpop/series.py` (`CENSUS_DATES`; 1801-03-10, 1841-06-06,
  1851-03-30, 1921-06-19, 2021-03-21 among them). A station counts as
  open when it had opened by that date and had not closed before it, or
  had reopened by it, judged at the precision its own dates carry: a
  year-only opening in the census year counts as open, a year-only
  closure in the census year as closed, a month-only date is compared
  by month. Both halves of the year-only rule push the same way (more
  stations open on census night); the number of stations the rule
  decides in each census year is the `stations_year_only_in_census_year`
  column of `distance_percentiles.csv`. Scotland's 2022 census stands in
  the 2021 surface.