This fall we ran an experiment that had nothing to do with business. We pointed an AI pipeline at the archive of the Dutch East India Company: 4.35 million handwritten pages from the 1600s and 1700s, plus centuries of digitized newspapers. Over a few days it surfaced an uncatalogued meteorite fall from 1812, three Javan rhinoceroses missing from the historical record, and volcanic eruptions absent from the Smithsonian’s global list.
The discoveries make a good story, and Jesse tells it in full here. The architecture is in our case study. This post is about something more useful to most readers: what the project taught us about putting AI to work on real documents.
A 17th-century colonial archive turns out to be a near-perfect stress test for business AI. It is enormous, the text is messy, nobody has an index of what is in it, and any claim you make has to stand up to experts. That describes a lot of law firm file rooms, insurance claim systems, and shared drives too.
1. Keyword search fails on real-world data
17th-century Dutch spells “rhinoceros” about fifteen different ways. Text recognition on old handwriting adds its own errors. And the best find of the project, a meteorite report from 1812, never uses the word “meteorite,” because the word barely existed yet. The officer who wrote it called it “a great curiosity,” a stone “precipitated from a Thunder-cloud.”
A keyword search would have missed it entirely. We found it because the pipeline searched by meaning. Every passage was converted into an embedding, a numerical fingerprint of what it is about, so a query describing a heavy stone falling from the sky with a bang lands near every passage that describes the same thing, whatever words were used.
Business documents have the same problem in a quieter form. Contracts call the same obligation five different things. Customers describe the same defect in a hundred ways. Scanned PDFs and faxes come out of text recognition with typos baked in. If your search depends on guessing the exact words, it is quietly missing things, and you have no way of knowing what.
2. Don’t send everything to the biggest model
A single search across millions of passages can return tens of thousands of hits. Reading all of them with a large AI model is slow and expensive. Reading them by hand is out of the question.
So we did what a well-run office does: we gave the first read to someone cheap and fast, and saved the expert for what mattered. The first pass went to Jev, a tiny decision model that answers only narrow yes-or-no questions. Is this a real animal? Is it wild? Where is it? Jev costs a few cents per million words. Screening 59,000 mentions of elephants cost about $3. Only the passages Jev passed went on to Claude Haiku, a more capable model, for translation and close reading.
This pattern is called a model cascade, and it is the single most practical idea in the project. Most of the cost of AI document work comes from using a powerful model on text that a much smaller model could have rejected in a fraction of a second. Put a cheap judge in front, and reading everything stops being a budget question.
3. Test your search on answers you already know
Here is the lesson we think matters most, and it is the one most AI projects skip.
When a search finds nothing, you have learned something only if the search could have found something. So before looking for anything new, we made the pipeline find events we knew were in the records: a famous 1615 dodo sighting, the Laki eruption of 1783, Tambora in 1815, and a dozen more. It is the same idea as testing a metal detector by burying your own wristwatch in the yard first.
That discipline paid off. Our first target, the great “unknown” eruption of 1808/09, came up empty across every source we tried. Because the same pipeline had found every known eruption we tested it on, we could treat that empty result as real evidence, not a shrug.
In business terms: before you trust an AI system to tell you there are no responsive documents, no problem clauses, or no prior incidents, plant a few you know about and make sure it finds them. Without that test, “nothing found” is not an answer.
4. Every answer should end at a source document
AI models can be confidently wrong. In this project, one model attributed an ash fall on Java to a volcano 1,500 kilometers away. Out of 769 eruption reports the pipeline found, only three survived once each was placed using the letter’s own geography and checked against the official record.
So we adopted a strict rule: no claim stands on a model’s word. Every finding ends at a scan of the original page with an archive reference anyone can look up. An AI agent compared each transcription to its source image, and a person checked each finding against the catalogues the specialists use.
This is the difference between an AI system your team experiments with and one it can rely on. When every answer links to the page it came from, checking a result takes seconds, mistakes get caught, and trust builds on evidence instead of hope.
5. Scale changes which questions are worth asking
Reading the Company archive by hand, at two minutes a page and a normal working week, would take about 70 years. Indexing it took one overnight run on a single graphics card.
That gap does more than speed up existing work. It makes new work possible. Nobody had searched those records for rhinoceroses or meteorites because nobody could justify the decades of reading. Once reading everything costs dollars and hours, the question shifts from “can we afford to look?” to “what should we look for?”
Most organizations have questions like that sitting in their own archives. What did we promise customers in contracts signed before 2015? Which claims look like this one? Has this issue come up before, anywhere? They go unasked because answering them used to mean weeks of someone’s time.
6. A person still steers
The pipeline read at a scale no person could. It did not decide what to look for, which leads were promising, or when a line of inquiry was going nowhere. Jesse made those calls throughout, and some of the best choices in the project, including the cheap screening model, were human ideas rather than AI suggestions.
The AI did the reading. The judgment stayed with a person. That is the division of labor we recommend to every client.
What this means for your documents
The same four ideas work on your documents too: search by meaning, screen with a cheap model, test against known answers, and cite every source. Together they are what make an AI document system both affordable and trustworthy. They apply to discovery in a lawsuit, an audit of years of financial records, a review of insurance claims, or a company archive nobody has opened in a decade.
The workflow from the archive project is open source as Antiquity if you want to see how it fits together. If you would rather have it built for you, that is what our AI document analysis service does. Tell us what you need to find, and we will tell you honestly whether AI can find it.