Case study
Transcript intelligence
A searchable corpus of earnings calls, held to a retrieval gate that refuses a change unless two named metrics both improve.
Context
This is a domain deep-dive, not a step in the spine. The estate's method — a gate that counts rows rather than files, a written record of what failed, numbers produced by code instead of asserted by prose — was built against filings and prices. Those arrive structured, with their own identifiers and their own idea of what quarter they belong to.
Earnings-call transcripts are none of that. Free text from two providers, no shared key between them, dates the provider itself often does not know, and no correct answer to compare a search result against. So this study sits beside the acquisition work rather than after it, and asks one question: does the method transfer to unstructured text?
What it was applied to is 12,171 calls across 3,652 tickers, deduplicated in two passes. Two results carry the page. A retrieval gate written in advance refused the change it was built to judge. And a one-line date fallback produced not errors but confident wrong answers — the harder class to find, because nothing in the interface looks broken.
What I built
The promotion rule was fixed before any numbers existed: a challenger index must beat the incumbent on nDCG@10 and hit@5-strong. In round one the challenger lost the first metric — 0.575 against 0.607 — while winning the second outright. The gate refused the promotion: the verdict recorded was STOP, change nothing live, with a list of what was not touched.
The write-up then examined the measurement rather than the threshold. The challenger was surfacing 92 to 101 ungraded hits per 300 against the incumbent's 26 at its winning setting, and the harness scores an ungraded hit as zero — the challenger was being charged for results the incumbent had never retrieved. I graded the 111 missing items against each query's written intent, re-ran the identical gate — 0.606 to 0.701, zero unjudged on both sides — and published the one category that regressed.
The second problem is smaller and harder to see. One line resolved a call's date to whichever candidate was non-null. One provider leaves that field blank on 36.4% of its documents, so hundreds of calls inherited the day their transcript page was posted instead. The join to the earnings feed then did not fail — it succeeded against the wrong quarter, displaying a price reaction that belonged to an unrelated one. The repair is a four-tier resolution chain that records which tier resolved every row, under a rule that decides the hard cases: leave the date NULL rather than stamp a known-wrong one.
Six stages, and the gate that governs the fourth — both rounds, verdicts intact
# before -- one expression, three fallbacks, no provenance date = call_date or published_at[:10] or url_date # one provider leaves call_date blank on 3,500 of 9,603 documents # -> the transcript PAGE date became the CALL date # -> 179 calls joined to the wrong quarter's price reaction # after -- four tiers, and every row records which one resolved it # 1 explicit provider call date .............. 5,925 48.7% # 2 fiscal period matched to a report date ... 5,649 46.4% # 3 page date, only within a plausible lag ... 291 2.4% # 4 left NULL by design ...................... 306 2.5% # # tier 2 is the one that matters: matching by fiscal period rather # than by proximity means the repair never consults the value it # does not trust. And the rejected repair is written down too -- # widening the join window makes wrong joins MORE likely, not fewer.
Decisions
Chose to grade the 111 unjudged results over relaxing the two-metric promotion rule, because round one's loss was an artifact of the evaluation and not of the index: the challenger surfaced 92 to 101 ungraded hits per 300 against 26 at the setting the incumbent won with, and an ungraded hit scores zero. Loosening the threshold would have shipped the same change on a weaker measurement; closing the grading gap and re-running the unchanged gate produced 0.701 against 0.606.
Chose to leave a call's date NULL over stamping the transcript page's publication date, because a blank cell degrades visibly and a wrong date degrades invisibly — 179 calls had already joined to the wrong quarter's price reaction with no signal anywhere in the interface. The obvious repair, widening the plus-or-minus-three-day join window, was named and rejected in the same document: it makes wrong joins more likely, not fewer.
Chose deterministic code over model-written numbers because every figure in the product has to be re-derivable from SQL. Language models are used at the edge only, to read and to draft prose, and a quote-anchoring step normalises each claim against the sentence set of the actual source material and drops the claims that do not anchor. 43 memos exist under that rule, and the analyst role is given no tools.
Evidence
Research artifact · evidence ledger
| Claim | How it was checked | Status |
|---|---|---|
| The corpus is 12,171 deduplicated calls across 3,652 tickers, spanning 2021-03-10 to 2026-08-21 — 9,376 from the archive provider and 2,795 from the live provider. | sqlite3 'file:<derived>?mode=ro' "SELECT COUNT(*) FROM calls" -> 12,171 ; "SELECT COUNT(DISTINCT ticker) FROM calls" -> 3,652 ; "SELECT MIN(call_date), MAX(call_date) FROM calls" -> 2021-03-10 | 2026-08-21 ; "SELECT source_db, COUNT(*) FROM calls GROUP BY source_db" -> 9,376 | 2,795 | published |
| 12,712 raw captures became 12,171 calls: 541 rows (4.3%) removed by two deduplication passes. The live provider loses 153 of its own rows not to the archive but to a fuller sibling capture in its own store — the second pass keeps the capture with the most speaker turns, and it is source-blind. | COUNT(*) on the documents table of each upstream store -> 9,764 + 2,948 = 12,712 ; 12,712 - 12,171 = 541. Pass 1 drops an archive capture colliding with a live capture on (ticker, fiscal quarter, fiscal year, call date); pass 2 keeps the fullest capture within a (ticker, fiscal quarter, fiscal year) group. | published |
| The corpus is concentrated in 2024 to 2026, with a thin 2021 tail and no 2022 to 2023 coverage at all: 2 calls in 2021, 780 in 2024, 3,310 in 2025 and 7,773 in 2026. | sqlite3 'file:<derived>?mode=ro' "SELECT substr(call_date,1,4), COUNT(*) FROM calls WHERE call_date<>'' GROUP BY 1 ORDER BY 1" -> 2021:2 | 2024:780 | 2025:3,310 | 2026:7,773. The 2022 and 2023 buckets are absent, not zero-filled. | published |
| The derived read model holds 146,579 question-and-answer pairs, 128,976 speaker rows with 1,878 analyst firms parsed out of free-text role labels, 35,825 linguistic-vitals rows and 230,925 extracted guidance sentences. The measured non-answer rate across all pairs is 6.54%. | COUNT(*) per table on <derived> opened mode=ro -> qa_pairs 146,579 | speakers 128,976 | vitals 35,825 | guidance_sents 230,925 | memos 43 ; "SELECT COUNT(DISTINCT firm) FROM qa_pairs WHERE firm<>''" -> 1,878 ; "SELECT ROUND(100.0*SUM(non_answer)/COUNT(*),2) FROM qa_pairs" -> 6.54 | published |
| The round-one loss was diagnosed as a bias in the evaluation, not a defect in the index: the candidate surfaced 92 to 101 top-10 hits per 300 that had never been graded, against 26 at the setting the incumbent won with, and the harness scores an ungraded hit as zero. All 111 ungraded items across the four gated configurations were then graded against each query's written intent — distribution 20 zeros, 54 ones, 37 twos, 33% top grade, checked against the earlier passes for leniency drift. | the ungraded-hit ledger the runner writes on every evaluation ; judgment pool grew 695 -> 806 with the prior state kept — three retained snapshots of the pool read 300, 454 and 695 grades, and the pool stands at 897 today. | published |
| Re-running the identical gate with zero unjudged hits on both sides promoted the candidate: nDCG@10 0.606 to 0.701 (+15.7% relative), P@5 with at least one relevant result 0.813 to 0.900, hit@5-strong 0.867 to 0.900, MRR@10 0.629 to 0.773. Warm query latency measured 0.52 to 0.92 s at the development flip and 0.56 to 0.92 s after the container deploy, against a 2 s bar. | same runner, same 30 queries, same two-metric rule, judgment pool at 806 ; unjudged@10 = 0 for both arms. Latency from the two flip verifications: three mandated queries re-run at the development flip (0.52 to 0.92 s) and again after the container deploy (0.56 to 0.92 s), the post-deploy result sets byte-identical to the development ones. | published |
| One of the six query categories regressed under the promoted backend and was published as such rather than averaged into the headline: macro_chain fell 0.657 to 0.621. The five gaining categories moved as the earlier failure analysis had predicted, the largest being incident_impact at 0.682 to 0.892. | per-category nDCG@10 table from the round-two evaluation run. The write-up also records 0.701 as a floor rather than a ceiling, because the fusion parameters beyond one knob were tuned against the previous backend's score distribution and never retuned. | published |
| A one-line date fallback corrupted at least 330 calls, and the damage was split by failure mode rather than reported as a total: 179 joined to a real earnings surprise and price reaction from an unrelated quarter with no signal in the interface, 150 joined to nothing and rendered blank, 1 had no ticker in the outcome feed. The archive provider leaves the call date NULL on 3,500 of 9,603 documents (36.4%). | the integrity investigation of 2026-08-03, run read-only against production data (the archive store held 9,603 documents at that date): a fiscal quarter-and-year cross-check with a 150-day plausibility cutoff, quantified explicitly as a floor and not a total, with ten worked examples and every downstream consumer of the join traced (three call sites plus the feed sort and the company timeline). The 3,500-of-9,603 NULL rate is recorded in that investigation rather than produced by a standing command. | published |
| After the four-tier repair every call records which tier resolved its date: 5,925 (48.7%) from an explicit provider call date, 5,649 (46.4%) matched by fiscal period to a report date in the outcome feed, 291 (2.4%) from a page date within a plausible reporting lag, and 306 (2.5%) left NULL by design. | sqlite3 'file:<derived>?mode=ro' "SELECT date_src, COUNT(*) FROM calls GROUP BY date_src" -> 5,925 | 5,649 | 291 | 306. The 306 NULLs are the rule working as specified, not a gap. | published |
12 claims tested · 9 published · 3 withdrawn before publication. Full evidence record available on request — king.hippopotamus@yahoo.com.
Limitations
Nothing here predicts returns. The corpus is joined to a one-day price reaction so that a measure can be falsified later; the backtest and the lookahead-bias guard are described in the plan and are not demonstrated. No measured predictive performance exists yet.
The retrieval numbers are a relative comparison under a fixed rule, not a benchmark score. nDCG@10 = 0.701 is measured on 30 queries against the 806-grade pool as it stood at that run — 897 grades today — judged by a single grader on a stated bar; the write-up flags single-judge bias as an unresolved caveat, and the figure is recorded as a floor because the fusion parameters were never retuned against the new backend. It is not comparable to anyone else's published number.
The corpus is not complete and is not evenly deep. 12,171 calls over 3,652 tickers, of which 7,773 are 2026 and 2 are 2021, with no 2022 or 2023 coverage at all; an individual company may be one quarter deep. Vector and exact-match search are built from one provider's identifier space and do not yet see the live provider's calls — keyword and speaker lookups span both.
The date problem is contained, not closed. 330 was stated as a floor: rows with a missing fiscal period, and rows whose gap was smaller but still wrong, were never fully cleared, and no post-fix re-check of that residual population exists. The standing suite is 11 PASS / 7 WARN / 0 FAIL — zero failures is the correct claim, and the seven warnings, including 64 duplicate (ticker, call date) groups, are open documented items rather than clean ones.
Several wedges are one layer deep. Of 230,925 extracted guidance sentences only 9,889 carry a directional verdict, and the link from a guidance tuple to its eventual actual is unbuilt. Only 8,486 of 35,825 vitals rows carry a z-score, because that requires prior quarters for that specific company — so the change-beats-level signal is available on roughly a quarter of the measured scopes. Two of three designed model roles run; 43 memos is a working feature, not a body of research.
Links
No demo — The estate is a private operating environment: there is no public endpoint and no repository behind this page. The transcript text itself is licensed third-party material and is never published — only counts and derived aggregates. The read-only commands that produced every figure are printed on the linked record instead.
Authorship
I am the architect and author of record; drafts were produced by agent tooling under my protocol and every sentence was approved by me before publication.
Author approval — 2026-08-23, hash 926a2097