Data platform architecture & governance

Context

What I built

{
  "name": "returns.beyond_bound_unflagged",
  "db":   "<the derived returns panel>",
  "sql":  "SELECT COUNT(*) FROM daily_returns
           WHERE (implausible IS NULL OR implausible = 0)
             AND (ret > 47.238328008997684 OR ret < -0.97926958)",
  "low":  0,
  "high": 0,
  "note": "The dead-safety-net detector: a row beyond the
           physical bound without a flag means the bound was not applied
           at the call site. Must be 0 forever."
}

Decisions

• Chose 54 single-writer SQLite files on one node over a distributed store with an orchestrator, because one host and one operator is the whole operating budget — a cluster and a scheduler would spend on themselves what the data needs, and one writer per file turns "who wrote this row" into a layout question instead of an investigation.

• Chose a refusal at write time over a review step after publication, because a reviewer who catches a wrong "the company did not disclose X" catches it once it is already written — the gate raises GapRefused at the call site, carries no override parameter, and treats a contract older than 36 hours as no contract at all.

• Chose assertions that return a number over checks that return a verdict, because a verdict hides its own drift — 30 SQL claims judged against inclusive bands produced 776 evaluations and 4 deviations across 31 runs, and every deviation named the surface and the value that had moved.

Evidence

Published claims and the verification policy →

Limitations

• Integrity was checked once, completely, on a single supervised date — 40 databases, all 40 that were in scope that day and 40 of the 54 the estate holds today. There is no recurring readability check across the large files. The absence of that check is a recorded open item, not something this page discovered.

• One derived product is proven reconstructible: the returns panel. Raw capture is not reproducible by definition — it is a capture of a source at a moment. The claim is that the derived layer rebuilds from sources, not that the platform is bit-reproducible.

• The schema-generated probe emitted 1,970 numbers over 4 databases and 19 tables. The estate has 423 tables. The method generalises; the coverage does not.

• Identity is not solved. About 10.3% of panel rows resolve to no security today — 9.71% at the cutover that fixed it, drifting as new rows land — 28 ticker rows across 14 tickers remain genuinely ambiguous, and about two-thirds of the entities in the archives nobody consumes yet have no bridge into the identity master at all — join yields of 39.8%, 36.4%, 30.0% and 27.6% against 99.4% and 93.7% for the surfaces already in production. What changed is that all of those are numbers on a timer rather than unknowns.

• Backups are host-local, one generation deep, perishable-only: 26 compressed database copies at 1.9 GB, one generation kept per store, because a source that stays downloadable forever is its own second copy. They cover corruption and bad writes on this host, not loss of the host, and off-host mirroring is an open decision. Separately, one major price panel is survivorship-selected at the source — delisted names were never collected — which is a procurement problem and cannot be repaired in code.

Links

No demo — the estate is a private operating environment and is not exposed; the read-only commands behind every row are published instead, with placeholder roots.

Authorship