Case study
Products & developer tooling
Every documented claim on the three Python packages I publish runs as a command, with a recorded output and one of three verdicts.
Context
Publishing to a registry is a promise made to someone I will never meet, and it is kept or broken at install time, on a machine I never see.
Three of these packages exist because I needed them. The CAD automation server exposes a modelling application's geometry, scripting and visual-programming surface as MCP tools, so an assistant can drive the model instead of a person clicking through it; it is for anyone who already runs that application. The vault is a local-first credential store for someone who wants their passwords on their own disk, with no account, no sync and no server to trust. The XBRL server wraps a mature filing processor so SEC filings can be queried as tools; its first consumer is my own research pipeline, and a transport fault found from that consumer's seat was filed as an incident against my own package, fixed, and shipped in the released version — full record on request.
Then I audited all three the way a stranger installs them.
What I built
The rule is one line: clone, install, run — nothing taken from the README on trust. Every documented claim becomes a shell command with a recorded output and one of three verdicts: held, failed, or not testable. The vault audit runs 22 claims that way; the CAD server audit runs five headline claims plus a census.
CAD server, 0.2.2. Each request carries a correlation identifier that is compared against the reply, the timeout path closes the socket, and the client reconnects only where a connection has already succeeded, so a slow operation cannot hand its answer to the next call. 89 tools counted at runtime, from a registry the package's own test reads; 245 tests, of which the 16 integration tests skip for want of the application. Every runtime dependency carries an upper cap with the reason written beside it; the 3-operating-system × 4-Python matrix carries a weekly schedule. The container image starts and waits on stdin.
Vault, 3.2.2. The packaging summary names the cipher the code runs, and a test pins the key derivation at both sites so the two cannot drift. 157 tests, most of them a static scan asserting that no module names a networking symbol. The self-destruct reports failure when the keychain entry survives it.
XBRL server, 2.1.1. Logging is held off the transport by three guard layers — file descriptor, Python object, logger — so no library write can corrupt JSON-RPC framing. 46 tools, 16 tests.
A nightly check reinstalls every published package into a clean environment — the three Python releases get their command line run and their module imported, the two browser elements get installed from the registry and their entry point resolved; all five pass.
The method: a claim, the command that tests it, the recorded output, one of three verdicts
Decisions
Chose to vendor the wire-protocol module twice — once for modern Python, once for the CAD application's stdlib-only embedded interpreter — over one shared compatibility shim, because the embedded half cannot install anything and a shim would have had to be judged against modern lint and type rules on both sides; instead each copy pins the same wire contract — a 4-byte big-endian length prefix, UTF-8, a JSON object body, a 64 MB ceiling — and the lint config excludes the old-dialect half by path.
Chose a machine-checkable assertion over a corrected string wherever documentation and code can disagree, because a corrected README drifts again in a month: the XBRL server reads its tool count from the live registry and fails any README or instruction string that disagrees, and the vault carries a banned-claim grep that must exit 1.
Chose to cap every runtime dependency and add a weekly scheduled matrix run over trusting a lower bound, because dependency drift is invisible to a workflow that only fires on a push — but the cap is the half that holds today: hosted workflow runs are not available on this account, so the schedule is a file waiting for a runner rather than a control that is running, and the releases were built and uploaded from the same host that runs the tests.
Evidence
Research artifact · evidence ledger
| Claim | How it was checked | Status |
|---|---|---|
| The regression test that ships with the connection fix fails against the pre-fix module and passes against the released one. | Rebuilt the pre-fix module in a scratch tree and ran the new test file against it — 5 failed, 2 passed of 7, reproducing the symptom the test was written for. The same file against the released source: 7 passed in 4.05 s. | published |
| The vault makes no network calls. | Most of its 157 tests are a parametrised static scan over 51 modules and 20 forbidden import roots, with an allowlist that is empty and a test that fails if it stops being empty. A Python audit hook around the login path caught a hostname resolution that the claim did not survive; that call was removed and the two log statements consuming the deleted field were rewritten. | published |
| The vault's self-destruct reports honestly when it cannot finish. | Independent verification against the pre-fix code returned 10 of 10 checks PASS with 4 residual observations, one of them a case where the routine reported success while the operating-system keychain entry survived it. It now reports the failure. | published |
| The XBRL server keeps stdout clean across a filing load, so nothing it logs can corrupt the transport. | Under the stdio transport, file descriptor 1 is the wire: a byte on it that is not a framed JSON-RPC message is a framing error by specification. A load path with no log file configured routed a library logger there. Three independent layers now hold it off — the real stdout duplicated onto a private descriptor with fd 1 re-pointed at stderr, a replacement sys.stdout object, and the library logger pinned to stderr with propagation off — under 16 tests, one of which asserts stdout stays clean across a filing load. | published |
| All five published packages install from their registries into a clean environment; the three Python packages run their command line, the two npm elements resolve their entry point. | Three PyPI installs at rc=0 in a throwaway venv with their console entry points exercised — golem, irondome, irondome-cli, arelle-mcp. Two npm installs (md-spawn 1.0.0, sst-spawn 2.0.0) into a throwaway project; each package's declared entry point (./dist/index.js) resolves. No browser run was performed. Releases across the three Python packages: 22 before today's uploads (4 / 11 / 7). | published |
12 claims tested · 5 published · 7 withdrawn before publication. Full evidence record available on request — king.hippopotamus@yahoo.com.
Limitations
Shipped today, not soaked. The index carries 0.2.2, 3.2.2 and 2.1.1, uploaded within minutes of each other; the CAD server and the XBRL server are tagged on the public repository, the vault carries its version in the package with no tag. The nightly reinstall check passes on all three from a clean environment, but one clean run on release day is not a soak, and nothing here has been running long enough to drift. The vault figures on this page were taken at 3.2.1 and re-checked against 3.2.2 before publication.
The build and release path is local, not hosted. Hosted workflow runs are not available on this account, so these releases were built and uploaded from the same host that runs the tests, and the weekly dependency schedule is correct as a file that has never fired.
157 self-authored tests are not a security audit. No independent third-party review of the vault exists, and its own README says so. The self-destruct is a single in-place overwrite followed by unlink — wear-levelling, copy-on-write filesystems and journaling can all leave the original blocks, filenames and directory entries are not overwritten, and hardlinked copies are not handled. The biometric check is an advisory boolean from a subprocess exit code; the key is released by an unconditional keyring read.
No adoption is claimed or measured. No download figures were taken, no issues are open against these packages, and nothing here says anyone else runs this code.
The watchdog CLI, the financial-data CLI and the earnings scanner are repositories, not products. None is published to a registry, two of the three carry no CI workflow at all, and their README figures are self-reported and were not re-measured here.
Links
No demo — The packages are public on their registries under the names printed here; the audit records, fix branches and independent verification passes live in a private working environment, so the commands are printed instead of a console and no account page is linked.
Authorship
I am the architect and author of record; drafts were produced by agent tooling under my protocol and every sentence was approved by me before publication.
Author approval — 2026-08-23, hash f45b7c2a