How a Record Is Built
Every record Librarian Labs publishes is built the same way and clears the same gates before it ships. A record is a set of claims, and every claim is traced to a verbatim quote, a timestamp, and a source. This page is the whole method: how a record is made, what verification means here, how uncertainty is written, how sources are preserved, and what we never do.
▪THE PIPELINE
- CAPTUREThe full source is archived at capture time, exactly as the public web serves it: the complete recording or the complete text, captured logged off. Public sources expire; the archive is what keeps the record checkable.
- TRANSCRIBEAudio sources are transcribed in full, locally, with word-level timing. Text sources are read from the published writing itself.
- EXTRACTEach claim is pulled from the transcript as a unit: a proposition (the claim restated so it stands alone), the verbatim quote that grounds it, the timestamp, and the source. Claims are typed (factual, evaluative, normative, causal, and so on) and tagged with polarity and tense.
- VERIFYDeterministic checks run on every row: quote grounding against the transcript, timestamp verification, speaker attribution, quality flags. Hard failures are held out; soft flags ship disclosed in the data. Nothing is fixed silently.
- PUBLISHA release ships only after a full-population audit passes. Every release carries SHA-256 checksums and a changelog.
▪WHAT VERIFICATION MEANS
Quote grounding. Every quote must be found in its source transcript. The audit counts two tiers: exact verbatim containment, and near-verbatim matches where the transcript itself stutters or repeats a word. A quote that grounds in neither tier fails the build.
Timestamps. Each row discloses how its timestamp was established: checked against the transcript at extraction, or re-derived from word-level audio timing. Text sources carry no timecode. Estimated timestamps do not ship.
Attribution. Each row discloses its verification tier: the quote sits in a speaker-labeled segment of the recording, the source itself name-tags the speaker, or the words are the subject's own writing. Rows that clear none of these are held out. Identification never comes from biometrics.
Quality flags. Automated checks flag rows with cosmetic defects, such as a quote that ends mid-sentence. These flags ship in the data, disclosed per row, so any consumer can filter on them. Rows with harder defects are held out of the confirmed file for review.
Repair. A flagged proposition may be rewritten into a standalone claim only by a disclosed repair pass; anything that cannot be repaired cleanly stays flagged rather than guessed. The verbatim quote is never edited by anything.
The audit gate. Before a dataset ships, an audit runs over the full population, not a sample: schema and field constraints, consistency across the claim, source, and entity files, agreement between the CSV and JSONL, reproducibility of every quality flag, and verbatim grounding of every quote against its source transcript. The release ships only on a pass.
▪THE UNCERTAINTY LADDER
Dossier prose grades its language. In ascending order: appears, suggests, indicates, strongly indicates. A hard assertion is reserved for what a document, a recording, or a timestamp compels. Numbers never dress up judgment: no invented confidence percentages in prose.
In the datasets, uncertainty is structural instead: a confidence field, disclosed quality flags, disclosed attribution tiers, and review rows held out of the confirmed file rather than shipped as certainty.
▪PRESERVATION
Public sources expire: recordings get deleted, feeds get pruned, accounts disappear. The full source is archived privately at capture time, exactly as served, so every claim stays checkable long after the public copy is gone. Archives are never republished: what ships is the claim, its verbatim excerpt, and a pointer into the source. Recut and re-posted copies are cited to the original appearance.
Datasets are versioned. Corrections land as new versions with the change disclosed in the changelog, never as silent overwrites, and every release's files carry published SHA-256 checksums.
▪WHAT WE NEVER DO
Never paraphrase inside a quote. The proposition is the labeled restatement; the quote field is verbatim from the source audio or text. A join between non-adjacent spans is always marked with an ellipsis.
Never splice. Quotes are the speaker's own words, never assembled across contexts and never lifted from an interviewer's framing.
Never ship full transcripts or recordings. The record is claims and short verbatim excerpts, cited back to the source.
Never use biometrics. No voiceprints, no face recognition. Attribution comes from the record itself, and its tier is disclosed on every claim.
Never cover private individuals. Records document public figures speaking in public, on the record.
Never circumvent. Capture is logged off, takes only what a source serves publicly, and bypasses no paywall, gate, or protection.
Never estimate a timestamp. Every shipped timestamp is verified or derived from word-level timing, and each row says which.
Never rewrite silently. Automated checks flag; a disclosed review or repair lane decides; the verbatim quote is untouchable.
▪CORRECTIONS
Every claim carries a stable ID. If you believe one is wrong, write to zac@librarianlabs.com with the ID. Substantive challenges get an on-the-record response, and corrections land in the record and its changelog.