Case study
Data acquisition at scale
Eighteen upstream sources become 33 collector modules and 54 databases; a gate turns a named ingestion hole into a refusal at the call site.
Context
Research writing has one failure mode that care in the writing cannot catch: the sentence the company did not disclose this, written over a hole in the writer's own ingestion. Classifying 19 such findings settled the argument for everything on this page: 3 were genuine non-disclosure. Five were present at the source and absent from the platform. Eleven were sitting in the platform and had never been retrieved.
So the failure the estate is built around is not a dead feed. It is absence of data mistaken for knowledge about the world. Eighteen upstream sources, 33 collector modules and 54 stores are the easy part; the part that took the design work is a nightly contract that measures what the estate actually holds, names its holes in machine-readable form, and a gate that turns a named hole into a refusal at the call site.
What I built
A source is admitted on paper before it is coded. The admission card answers five things: which data type the module owns, which endpoints were exercised live, not read from documentation, what storage shape the rows land in, what cadence the module runs at, and — the column that does the most work — what I am not taking from this source but could. A chosen absence and an unnoticed absence are different objects, and only the first can be governed.
Modules are one per data type, not one per vendor. SEC EDGAR alone decomposes into twelve: spine, XBRL facts, dimensioned facts and notes, insider transactions, institutional holdings, stakes, proxy compensation, material events, subsidiary trees, fails-to-deliver, full text, request logs. The contract is the same for all of them — parsers that fail loud, collect / verify / status verbs with exit codes 0, 1, 2, 3, and a suite that runs offline against recorded upstream responses; 30 of the 33 carry that suite today.
Identity needed the same treatment. Company identity hung on one primary-ticker column, so a registrant with two share classes silently lost one, and calls filed under the second class anchored to nothing. An append-only identifier registry — 10,429 rows over 3,558 entities, 17 of them carrying more than one ticker — moved coverage of speaking companies from 3,479 of 3,495 to all 3,495. The residual 16 were never missing companies. They were corporate actions, and they resolve with date windows.
The coverage gate. The only way past it is not to call it.
"filing_spine": {
"rows": 18697338,
"date_span": ["1980-01-02", "2026-08-21"],
"holes": []
},
"material_events": {
"rows": 342275,
"text_states": {"fetched": 108006, "none": 220850, "skipped": 13419},
"holes": [
{ "hole": "8-K bodies: 108,006/342,275 fetched, 220,850 pending",
"status": "backfill_running",
"consumer_rule": "An event with text_state != 'fetched' has METADATA ONLY. Never claim its content is unavailable — it was not fetched yet." }
]
},
"dimensioned_facts": {
"num_rows": 103444701,
"implausible_ddate": 78,
"holes": [
{ "hole": "fiscal 2015-2019 not loaded",
"status": "awaiting_disk",
"consumer_rule": "Pre-2020 dimensioned facts are absent BY STATE, not by disclosure." }
]
}Decisions
Chose one collector module per data type over one module per vendor, because a schema change or a block then lands on one module and one store instead of on everything that source supplies — SEC EDGAR alone decomposes into twelve modules, each with its own resume state and its own circuit breaker.
Chose calendar-anchored discovery over provider-listing discovery, because a what-is-new surface reports what the provider is showing, not what happened — so a collector reconciles against an independent expectation source (an earnings calendar, the filing spine, a trading-day calendar) and chases the difference rather than trusting the index.
Chose a refusal at write time with no override parameter over a warning in the report template, because a warning is dismissed by the same person under the same deadline that produced the gap — and a contract older than 36 hours is treated as no contract at all, so a stale measurement cannot quietly authorise the claim.
Evidence
Research artifact · evidence ledger
| Claim | How it was checked | Status |
|---|---|---|
| 18 upstream sources decompose into 33 collector modules — 32 live and 1 retired but retained; 30 of the 33 carry an offline test suite that runs against recorded upstream responses. | ls -1 <raw-store-root> | wc -l → 18 ; find <providers-root> -mindepth 2 -maxdepth 2 -type d | wc -l → 33 ; the same listing filtered to directories containing a tests/ subdirectory → 30. | published |
| The estate is 54 database files holding 105.4 GB (98.2 GiB); the whole platform tree, including raw blobs, code, logs and backups, is 114.4 GB (106.5 GiB). | find <estate-root> -name '*.db' -type f -printf '%s\n' | awk '{s+=$1}END{print s}' → 105,441,099,776 B across 54 files ; du -sb <platform-root> → 114,355,413,279 B | published |
| 397,679,393 rows under governance across 28 named surfaces. | read-only SELECT COUNT(*) on each of the 28 row-bearing surfaces, summed. The sum mixes grains — filings, facts, positions, speaker turns — so it is a governed-row count, not a count of facts. | published |
| SEC EDGAR spine: 18,697,338 filings spanning 1980-01-02 to 2026-08-21, 27,155,455 filer-to-filing links, 982,170 companies. XBRL: 125,307,255 company facts and 103,444,701 dimensioned fact rows, of which 78 carry implausible dates that are stored and named rather than dropped. | SELECT COUNT(*) and MIN/MAX per table with each store opened mode=ro; dimensioned counts read from the nightly coverage contract fields num_rows and implausible_ddate. | published |
| The 8-K narrative hole is named rather than silent: 342,275 events, 108,006 bodies fetched, 220,850 pending, 13,419 skipped. | coverage contract, text_states block. The hole carries status backfill_running and a consumer rule that makes an absence-of-disclosure claim from event text illegal for that surface. | published |
| Identifier registry: 10,429 identifier rows over 3,558 entities, 17 entities carrying more than one ticker. At the migration that built it, coverage of speaking companies moved from 3,479 of 3,495 to 3,495 of 3,495, with none unresolved at that point. | SELECT COUNT(*) on the identifier table; COUNT(DISTINCT entity_id); GROUP BY entity_id HAVING COUNT(DISTINCT ticker) > 1 — all three re-run today. The coverage pair is not re-run: it is a migration-time record in the engineering log, and every identifier row carries a build timestamp from that single pass, so the pair is dated to the build and today's coverage is not measured here. | published |
| Cross-provider overlap was measured before it was fixed: 166 calls and 1,429 questions net double-countable across the two transcript stores. | join on (ticker, fiscal quarter, fiscal year) across both stores, excluding intra-store duplicates that an earlier document had folded into a larger figure. Measured once, before the fix, and recorded in the engineering log — it is not reproducible today because the fix now suppresses the superseded copy in any read spanning both stores. The fix keeps an include-duplicates escape hatch and deletes nothing. | published |
| Deduplicated transcript corpus: 12,171 calls over 3,652 tickers, 2021-03-10 to 2026-08-21, from 12,712 raw captures — 541 rows (4.3%) removed by a cross-source pass and a within-source re-post pass. | SELECT COUNT(*) FROM calls on the derived corpus (mode=ro) → 12,171 ; SELECT COUNT(*) FROM documents on each upstream store → 9,764 and 2,948. | published |
| One holdings surface changed value-scale regime mid-history: roughly 76,898 filings on one convention through 2022 and 116,077 on another from 2023, with a mislabel rate around 7%. | SELECT CASE WHEN filed_date < '2023-01-01' THEN 'through 2022' ELSE 'from 2023' END AS era, value_scale, COUNT(*) FROM filings GROUP BY era, value_scale; the ~7% is a separate measurement — the share of filings whose stored scale value disagrees with a market-anchored normalisation of the same position, not a product of the grouping. The consumer rule written into the coverage contract forbids trusting the stored value and requires the normalisation instead. | published |
| The scheduled surface is 54 jobs: 43 timers loaded and scheduled, plus 11 active cron lines. | systemctl --user list-timers --all --no-legend | wc -l → 43 ; crontab -l | grep -vcE '^\s*(#|$)' → 11. This counts jobs, not runs per day. | published |
12 claims tested · 10 published · 2 withdrawn before publication. Full evidence record available on request — king.hippopotamus@yahoo.com.
Limitations
Not a real-time system. Price grain is daily, the options surface is three snapshot windows per session, and there is no tick data and no streaming feed anywhere in the estate.
Not full history everywhere, and the gaps are stated rather than smoothed: dimensioned XBRL for fiscal 2015-2019 is not loaded and waits on storage, 8-K narrative bodies are 108,006 of 342,275, stake coverage has a 2023-01-01 floor by decision and fails-to-deliver a 2009-07 floor. The 1984 price floor is deep but sparse — it is not dense daily coverage of a wide universe, and any backtest must state its own measured start date rather than inherit the table's minimum.
No uptime, no SLA and no data-quality percentage. None was measured; the estate publishes named holes instead of a score. The 7,048 test functions across the platform were counted, not run, so no pass rate is claimed either. This is a private research estate that extracts and normalises data for its own use — not a licensed or redistributable market-data product, and nothing here carries redistribution rights.
Live transcript coverage begins 2026-07-23. Everything earlier exists only in a frozen archive that is no longer collected, and the corpus is not a complete record of the market's earnings calls.
The gate is a library, not a wall around the estate. It refuses at write time, it has no override parameter, and the rule that every consumer must check a surface before claiming a company did not disclose something is written into the contract's own preamble — but a pipeline enforces that rule by calling it, and no report pipeline here imports it today. It also exempts one class of hole: a gap recorded as a deliberate decision is named in the contract and does not refuse, and one of the three holes the contract carries today is that kind.
Links
No demo — The estate is a private operating environment: there is no public endpoint and no repository behind this page. The commands that produced every figure are printed on the record page instead.
Authorship
I am the architect and author of record; drafts were produced by agent tooling under my protocol and every sentence was approved by me before publication.
Author approval — 2026-08-23, hash 2377b38c