Parsing the Paper Mountain: Criminal Intelligence Is a Document Problem Before It Is an AI Problem
The TL;DR is that most of what we call criminal-intelligence "big data" is not data at all. It is documents. Scanned exhibits, phone extraction reports, transcripts of intercepted calls, handwritten ledgers, and decades of paper briefs sitting in evidence rooms. Before any model can reason over a case, something has to read the page, and the parsing layer that does the reading quietly sets the ceiling on what every downstream stage can ever know.
This article argues that the parsing layer is the real problem, and that we are ignoring it because it is unglamorous. We want to talk about the reasoning model at the top of the stack. The case is actually won, lost, or thrown out at the bottom, in the boring work of turning a smudged photocopy into a structured, provenanced, court-defensible assertion.
Here is the question worth sitting with: if your system cannot faithfully read the worst page in the brief, what is the accuracy number on the vendor slide actually measuring?
Introduction
Picture the start of a serious investigation. A cache of material lands on the desk. Some of it is clean and digital: bank statements as CSVs, structured transaction records, well-formed reports. Most of it is not. There is a box of photocopied receipts, a stack of intercepted-call transcripts, a device extraction dump with call logs and photos of a paper diary, and a ledger in a hand that takes a human ten minutes a page to decipher. This is the reality of the corpus, and it is nothing like the tidy dataset the AI conversation assumes.
The industry pitch skips straight to the clever part. Feed it to the model, ask it questions, watch the network graph assemble itself. That is the easy part to demo. But you did not come here for the demo.
The hard truth sits one layer down, in a stage nobody wants to own. Everything the reasoning model will ever say about this case is bounded by what the parsing pipeline managed to extract from the paper. If the parser drops a ledger row, mangles a phone number, or silently discards a relationship it has no category for, no downstream model can recover it. The model cannot reason about a fact that never made it off the page.
It Is Documents, Not Data, and the Stack Has Four Floors
Let us peel back the layers and name the stack plainly, because vagueness here is where programs get sold a false picture.
Optical character recognition. The pixels become text, and, if the tool is any good, every region carries a confidence score. That per-region confidence is not a nice-to-have. It is the signal that tells you which parts of the page to trust and which to route to a human.
Layout analysis. Before the text means anything, the system has to understand the geography of the page: columns, tables, headers, stamps, marginalia, signatures. Get this wrong on a ledger and a row swaps its amounts, so the debit becomes the credit and the whole account inverts. The characters were read perfectly and the meaning is still wrong.
Structure and entity extraction. Now the text and its layout are mapped into the ontology: this token is a person, that one an account, this is a transaction between them on a date. This is where records become entities and relationships.
Chunking and indexing. Finally the extracted content is broken up and indexed so a retrieval system can find it later. Chunk across a table boundary and you split a fact from its context.
Four floors, and the reasoning model everyone is excited about sits above all of them, inheriting whatever they got right or wrong.
The Arithmetic of Compounding Error
Here is the sum nobody puts on the slide. Each stage in that pipeline has an accuracy, and the stages are in series, so the errors do not add, they multiply.
Say each of five stages runs at 95 percent. That sounds excellent in isolation. End to end, you get 0.95 to the fifth power, which is about 77 percent. Add two more stages, so seven at 95 percent, and you are down to roughly 70 percent. Nearly a third of the corpus has an error somewhere in it, and every stage was individually "good".
And that is the flattering version, because it assumes the errors are independent and spread evenly across the corpus. They are not. Parsing errors cluster. They land on exactly the material an investigation cares about most: the degraded photocopy, the handwritten ledger, the document photographed at an angle in bad light, the page in a language the model was barely trained on. The clean, typed, born-digital bank statement sails through at 99 percent. The one page that might break the case is the one the pipeline is worst at.
This is why the average accuracy number is worse than useless. It is an average over a corpus that is judged, in the end, on its worst pages. A brief of evidence does not get graded on how well you read the easy 900 pages. It gets tested, adversarially, on the hundred that matter, and those are precisely the pages the compounding error concentrates in.
[Ben: personal example here about a document-quality or OCR surprise that blew out a data-migration or ingestion program, without naming the client. Something concrete about how the worst 5 percent of a corpus ate most of the effort. The article flows without it.]
I will be blunt about the uncomfortable part. When a vendor shows you a single accuracy figure for a document-AI product, they are almost certainly showing you the mean on a clean benchmark. That number tells you nothing about the material that will actually decide your case, and quietly hides where the harm lives.
Provenance Is a Legal Artefact, and So Is Your Ontology
There are two failures in the parsing layer that are not really about accuracy at all. They are about what the system is even capable of representing.
The first is the ontology. An extraction schema is a set of decisions about what counts as an entity and what counts as a relationship. Whatever the schema cannot represent, the system cannot see. If your data model has no category for a particular kind of association between two people, that association is dropped at parse time, silently, and no amount of cleverness in the reasoning model above can bring it back. The nuance that might have exonerated someone, or the connection that might have cracked the case, is gone before anyone reads a chart. Treat the schema as neutral engineering and you have quietly encoded a worldview into the evidence.
The second is provenance, and this is where the sector is least ready. In a criminal-intelligence setting, provenance is not metadata hygiene. It is a legal artefact. Every machine-read assertion has to trace back to the page, the region, and ultimately the pixel that a court can be shown, with the confidence score and the model version captured at the moment of parsing. Captured then or never, because you cannot reconstruct it after the fact. Evidence derived from machine-parsed documents will be challenged, and the challenge will be exactly this: show me how the machine got from that smudge to this assertion, and prove it was this model on that date. The documents-and-records provisions of the Evidence Act 1995 (Cth) are the frame for how machine-produced material comes before a court, and the discipline has to be built into the parser, not bolted on at trial. [Verify: exact Evidence Act provisions with legal review before publishing.]
This is the seam where value leaks. A confidence score that never travels with the assertion becomes false certainty by the time it reaches an analyst. A model-read fact with no trace back to its page becomes an orphan the moment someone asks where it came from.
The So What?
So if criminal intelligence is a document problem before it is an AI problem, what does an architect actually do? None of it starts with picking the reasoning model.
Capture provenance at parse time or lose it forever. Every assertion carries its page, region, confidence, and model version as first-class attributes from the instant it is created. There is no later. If it is not captured in the pipeline, it does not exist for the court.
Route human review by document quality, worst first. Human review is not a fallback for when the machine gives up. It is a designed stage. Send your reviewers to the degraded, low-confidence pages first, because triaging by quality buys back several times the fidelity of random QA sampling. Random sampling spends most of its effort re-checking the easy pages that were already right.
Treat the schema as a decision, not a default. Someone with authority should sign off on what the ontology can and cannot represent, understanding that anything outside it is invisible downstream. That is a decision with moral weight, not a config file.
Measure the worst page, not the average. Report accuracy stratified by document quality, not as a single mean. The number that matters is how the pipeline performs on the hardest decile, because that is the material the case will turn on.
Assume adversarial scrutiny of the ingestion layer. Build as though a defence expert will one day pull the worst-parsed page in the brief and ask you to defend the machine's reading of it. Because one day, they will.
Note: this is not a comprehensive account of everything that can go wrong in ingestion, and a proper treatment of chunking strategy is a separate conversation for my architecture friends. But if you get provenance and worst-page review right, you have fixed most of what actually sinks these programs.
Conclusion
Everyone wants to talk about the reasoning layer. It is the part that demos well, the part that feels like the future, the part with the impressive graph that assembles itself on stage. Meanwhile the parsing layer, the OCR and the layout analysis and the extraction, sits underneath doing the unglamorous work that decides whether any of it stands up.
The uncomfortable reframe is this. Commercial document AI is optimised for average-case accuracy on clean corpora, treats provenance as overhead, and throws it away, because for most commercial use that is a perfectly rational trade. A brief of evidence is the opposite kind of object. It is judged adversarially, on its worst page, in a room where someone is paid to break it.
So here is the question I would put to anyone standing up a criminal-intelligence AI capability. You have chosen your reasoning model with great care. Who is defending the pixel?
Thanks for reading.
The views expressed in this article are my own and do not represent those of my employer, or any of my clients.
See related articles: The 80,000-Entity Wave: Why Tranche 2 Is an Entity Resolution Problem, Not a Compliance One [link] and Starved by Design [link].
References
APA 7. Confirm current versions, dates, and URLs before publishing. Where a direct quote would strengthen a claim, insert it at the marked slots and add the page number. Do not publish the Evidence Act framing without the legal review noted in the body.
Evidence Act 1995 (Cth). https://www.legislation.gov.au/C2004A04858
[Verify source: OCR and document-parsing accuracy on degraded and handwritten corpora. Prefer a peer-reviewed benchmark or standards source over a vendor figure.]
[Verify source: provenance standard for machine-derived assertions, e.g. W3C PROV, if cited in the provenance section.]
