Searching 4.35 million archival pages with a multi-model AI pipeline.

The GLOBALISE project has transcribed the Dutch East India Company archive, 4.35 million handwritten pages written between the 1600s and the 1790s. We added the Dutch national library's digitized newspapers, two centuries of American newspapers, and a set of ship logbooks. Read by a person at two minutes a page, the Company pages alone would take about 70 years of full-time work.

Volume was only half the problem. Text recognition on old handwriting and print is full of errors, and 17th-century Dutch spells "rhinoceros" about fifteen different ways. Writers also lacked the vocabulary we search with today: in 1812 the word "meteorite" barely existed in English, so a report of one never uses it. Keyword search misses most of what matters in a corpus like this.

The question was whether a small, affordable AI system could read all of it, find the handful of passages worth a historian's attention, and prove each one against the original document.

Index. A single graphics card turned every passage into an embedding, a numerical fingerprint of its meaning, in one twelve-hour overnight run. That made the archive searchable by what a passage was about rather than which words it happened to use.

Search. Queries described the thing we were looking for ("a heavy stone fell from the sky with a loud report") instead of guessing at spellings. The meteorite report surfaced this way, even though its writer called it "a great curiosity" and "a stone precipitated from a Thunder-cloud."

Screen. A single search could return tens of thousands of hits, too many for a person or even a large model to read affordably. The first read went to Jev, a tiny, fast decision model that answers only narrow yes-or-no questions: is this a real animal, is it wild, where is it? Jev costs a few cents per million words. Screening 59,000 mentions of elephants cost about $3.

Read. Only the passages Jev passed went to Claude Haiku, which read them closely, translated the Dutch, and pulled out dates, places, and people as structured data.

Verify. A Claude Code agent opened the scan of each original handwritten page to check the transcription against the source. Before anything was called new, it was checked against the catalogues that specialists use, such as the Smithsonian's Global Volcanism Program and the Meteoritical Bulletin.

The first target was the great "unknown" eruption of 1808/09, one of the largest of the last 500 years, which ice cores record but nobody has located. The pipeline searched American, Dutch, South American, and Indian newspapers, ship logbooks, and all 633 articles of Batavia's first newspaper. It found nothing.

Because the same pipeline had found every known eruption it was tested on, that null result means something: the reports most likely are not in those sources. A system you can trust to say "it isn't here" is as valuable as one that finds things, and only the known-answer tests make that possible.

Swap the Dutch East India Company for your own records and the problem is the same: too many pages to read, inconsistent language, scanned files with recognition errors, and a need to show exactly where each answer came from.

The workflow behind this investigation is published as Antiquity, a small open-source toolkit that lets anyone with a research question and a coding agent run a similar archival investigation. For the lessons we took from the project, read what searching 4 million pages taught us about applied AI.