Case study
Automation & reliability engineering
Fifty-four scheduled jobs on one host, and a watcher engineered so that a defect in the pipeline cannot take down the thing watching it.
Context
The failure mode this estate is built against is not a crash. It is a job that quietly did not run, a source that was never fetched, or a counter that lied — found mid-project, after something has already been built on the hole. The written doctrine names the disease in its own words: absence of data mistaken for knowledge about the world.
So the practice is a ledger, not a README. It holds 370 incident records across 96 subsystem areas in a fixed shape — timestamp, area, fault, cause, fix, automation, status — and the last two fields are the argument. An incident is not closed by a fix; it is closed by the check that would have caught it. Eighteen rows have reached that state. 114 are fixed, 20 carry a resolution, and 218 are open and visibly so.
The second half of the practice is that nothing may be silent. Every check mails. A check that finds a gap exits differently from a check that broke. And the thing that watches the nightly build is not permitted to depend on the nightly build.
What I built
Everything scheduled is a declared unit, not a script someone remembers to run. Fifty timer files were written by hand; the scheduler has 43 timers loaded and running, alongside 11 cron lines for the jobs whose sub-hourly or market-close semantics cron expresses better — 54 jobs in all. The unit file is where the reliability semantics live: 44 of the 50 catch up a firing missed during downtime, 29 start on a randomised delay so nothing stampedes one host, and 8 are pinned to the US market clock, because daylight-saving drift is a proven bug class here. Seventeen of the 50 are guards or hygiene rather than collection.
Guards do not run on hope. A census exits with a distinct status when it finds a gap, and its unit declares that status a success, so "found a problem" is never read as "crashed" — 29 of 61 service units say so explicitly. Underneath, the host sensor makes 96 full ten-layer passes and 288 three-layer passes a day, plus an hourly janitor and an hourly check that the outbound-only posture has not been altered; the 06:00 digest folds the layers into one message.
I built a local CI runner this week, because hosted CI is not available to this estate: a per-repository test run, plus a nightly check that installs each published Python release into a clean environment, runs its CLI and imports it, and installs the two npm packages into a throwaway project and resolves their entry points. Test suites stand green at 245, 157 and 16, fresh install passes on all three Python releases, and the end-to-end run is 15 pass / 1 fail — the one failure is the harness's own outbound-link check, which flags every external link the site carries.
Watcher independence — the watcher reads artifacts, never the pipeline
ONE NIGHT, HOST CLOCK -- what fires, and what it leaves behind
every 5 min host sensor, three-layer pass 288 / day
every 15 min host sensor, full ten-layer pass 96 / day
hourly janitor + outbound-only posture check 24 + 24
23:10 Mon-Fri nightly build -> run-log row, dated ledger file,
one gate verdict row
23:30 Mon-Fri pipeline watcher -> reads today's run-log row and
today's ledger. Non-trading day:
deliberately silent. Alert log
has been 0 bytes since 2026-06-05
23:50 screener sweep, full universe -> nightly snapshot
23:55 Mon-Fri collection watcher -> did tonight's sweep land? is the
weekly price refresh starved? the
perishable panel gets three
sub-checks: freshness, calendar
continuity, shape
00:45 Tue-Sat derived chain + price compose, one heavy lane per source
behind a shared lock file rather than by convention
01:10 Tue-Sat trading-day calendar rebuilt
01:25 Tue-Sat returns recomputed against that calendar
01:40 Tue-Sat perishable ledger closed for the day
02:30 Tue-Sat catalyst delta
03:30 database backup, after all night collection has finished
05:40 enrichment pass + morning brief
06:00 daily digest -> ten layers scored, every event of
the trailing 24h, one message
06:45 coverage contract -> what is held, per surface, with
the holes listed by name
07:30 change guard, --fix -> 7 invariants. What it repairs, it
repairs in place. What it cannot,
it writes to a file the next
working session must clear first
09:30 Mon-Fri census -> exits 2 on WARN; the unit declares
2 a success, so "found a gap" is
never read as "crashed"
Decisions
Chose declarative timer units over cron one-liners because the unit file is the one place restart, catch-up and drift semantics can be stated and then counted: 44 of 50 catch up after downtime, 29 randomise their start, 8 pin to the US market clock. Cron kept only the 11 jobs it expresses better.
Chose a watcher that reads artifacts over a watcher that imports the pipeline because a defect in the pipeline must not be able to disable the thing that watches it: 205 lines, standard library only, no pipeline import, and scheduled twenty minutes later so it inspects a result rather than racing one.
Chose a distinct exit status for "found a gap" over a generic non-zero failure because a scheduler otherwise cannot tell a working check from a broken one — which is why two finance-scope units sit in a failed state by design, holding findings, instead of quietly succeeding. The convention has its own failure mode, on record here: an argument-parsing error that also exits 2 reads as success, which is how one weekly job failed quietly until the mismatch was found.
Evidence
Research artifact · evidence ledger
| Claim | How it was checked | Status |
|---|---|---|
| 54 scheduled jobs run on one host — 43 timer units loaded and scheduled, plus 11 active cron lines. 50 timer files were authored by hand; of those, 44 catch up a firing missed during downtime, 29 randomise their start, and 8 are pinned to the US market clock. | systemctl --user list-timers --all --no-legend | wc -l → 43 ; crontab -l | grep -vcE '^\\s*(#|$)' → 11 ; ls -1 <units>/*.timer | wc -l → 50 ; grep -l 'Persistent=true' <units>/*.timer | wc -l → 44 ; grep -l 'RandomizedDelaySec' → 29 ; grep -l 'America/New_York' → 8 ; of the 43 loaded, 42 are from the authored 50 and 1 is supplied by the operating system | published |
| 17 of the 50 authored timer files are guards or hygiene rather than collection — roughly a third of the authored surface exists only to check the rest. | the 50 authored .timer files classified by role from the unit directory listing → 17 guard/hygiene, 33 collection or build. The matching pattern is withheld here because it enumerates internal job names. | published |
| 29 of 61 authored service units declare a non-zero exit status as success, so a check that found a gap is not reported to the scheduler as a crash. | ls -1 <units>/*.service | wc -l → 61 ; grep -l 'SuccessExitStatus' <units>/*.service | wc -l → 29 | published |
| On an ordinary day the host sensor makes 288 three-layer passes, 96 full ten-layer passes, 24 janitor runs and 24 outbound-only posture checks, plus one digest. Four different units of work; they are not summed. | crontab -l → five dispatcher lines: every 5 min quick (three layers — process, network, storage), every 15 min full (all ten layers), hourly janitor, hourly posture check, 06:00 digest. Sensor source: 1,611 lines, standard library only, 10 layers in its layer table. | published |
| The nightly build's watcher is 205 lines of standard library, imports no pipeline module, runs on its own schedule twenty minutes after the build, and its alert log has been 0 bytes since it was armed on 2026-06-05. | wc -l <pipeline-root>/<watcher> → 205 ; its import block is standard library only and names no pipeline module ; crontab -l → build 23:10 weeknights, watcher 23:30 weeknights ; stat -c '%s %y' <pipeline-root>/<watcher-log> → 0 bytes, armed 2026-06-05 | published |
| 61 gated nightly runs are on record: 56 PASS, 3 WARN, 2 FAIL, between 2026-06-05 and 2026-08-21. A WARN or FAIL holds publication. | verdict tally over the gate ledger CSV → 61 rows, [('PASS', 56), ('WARN', 3), ('FAIL', 2)] ; first/last run_at → 2026-06-05T01:28:59 → 2026-08-21T23:19:29 | published |
| The incident ledger holds 370 records across 96 subsystem areas, 2026-07-29 to 2026-08-21: 218 open, 114 fixed, 20 carrying a resolution, 18 closed by an automation. | JSON aggregation over the ledger file → records 370, areas 96, status {open 218, fixed 114, resolution 20, automated 18}; 218+114+20+18 = 370 ; window 2026-07-29 → 2026-08-21 | published |
| The change guard is 484 lines enforcing 7 invariants, runs daily at 07:30 with --fix, and writes whatever it cannot repair to a file the next working session is required to clear first. | wc -l <guard-root>/<guard> → 484 ; grep -c '^ C[0-9]' → 7 invariants ; the guard timer's OnCalendar → daily 07:30 ; the pending-fixes file is present and non-empty | published |
| Failed units at the moment of audit: 0 in system scope, 5 in user scope — 2 finance-scope units exiting 1 because they are holding findings, 3 inert leftovers from software no longer in use. | systemctl list-units --state=failed --no-legend | wc -l → 0 ; systemctl --user list-units --state=failed --no-legend → 5 rows ; systemctl --user show <guard>.service -p Result -p ExecMainStatus → Result=exit-code, ExecMainStatus=1 (findings pending, not a crash) | published |
| The local CI runner runs each repository's test suite plus a nightly check that installs each published Python release into a clean environment, runs its CLI and imports it, and checks the two npm packages against the registry: test suites green at 245, 157 and 16 tests, fresh install passing on all three Python releases, and an end-to-end run of 15 pass / 1 fail. | local CI status record, latest run per repository, 2026-08-23 → working trees green at 245 / 157 / 16 tests ; the e2e harness, 2026-08-23 18:24 → 15 pass / 1 fail, including install-from-index then CLI --help exit 0 for each of the three Python releases, and an import of the installed release; the single failure is the harness's outbound-link check | published |
12 claims tested · 10 published · 2 withdrawn before publication. Full evidence record available on request — king.hippopotamus@yahoo.com.
Limitations
One host. No failover, no replica, no second scheduler. What is demonstrated here is detection and recovery discipline, not availability engineering — and not uptime.
The register is disciplined, not clean. 218 of 370 incident rows are open and only 18 are closed by an automation. At the moment of audit five user-scope units were in a failed state: two finance-scope units holding findings by design, exiting non-zero rather than crashing, and three inert leftovers.
Coverage of the watchers is itself incomplete, and known to be. The August mail audit turned up a service that had been failing silently because its working directory no longer existed, and a job that hung for two hours with no check to notice.
No aggregate mean-time-to-detect has been measured. Two latencies are defensible — a reboot race alerted within the same boot cycle, and a reconciliation warning fired every morning until it was root-caused — but there is no estate-wide figure.
An empty alert log is not proof the pipeline never missed a night. Weekends, market holidays and deliberately silent days are excluded by design, and the switch has never been observed to fire. Nor is the change guard self-healing: it repairs a narrow enumerated set of mechanical defects and escalates everything else.
Links
No demo — The estate is a private operating environment: there is no public endpoint and no repository behind this page. The commands that produced every figure are printed on the evidence page instead.
Authorship
I am the architect and author of record; drafts were produced by agent tooling under my protocol and every sentence was approved by me before publication.
Author approval — 2026-08-23, hash f0cf48cc