Cleaning and Structuring Historical Data | Jim Clifford | AI and History Conference 2026

Jim Clifford (University of Saskatchewan) on cleaning and structuring historical data and grounding it in Wikidata, at the AI and History Conference 2026. Clifford walks through the pipeline behind the Imperial Careers Atlas, which maps the careers of British colonial officials, instead of running a hands-on workshop. The pipeline starts with optical character recognition (OCR) of the biographies printed in the Colonial Office List and the Indian Office List. A Python script splits the text into individual biographies and an open Qwen model converts each one to JSON, after which Python rules fix dittos and merge duplicate people. He then grounds places and people in Wikidata using an MCP (Model Context Protocol) server with vector search and QLever for deterministic matching, reaching 86 percent grounding for places. The discussion covers cost, how clean a dataset needs to be, and learning these methods with Claude Code. Speaker: Jim Clifford, University of Saskatchewan. 0:00 Why a pipeline walkthrough instead of a workshop 1:39 The source: biographies in the Colonial Office List and Indian Office List 7:31 Why Wikidata changes historical place grounding 9:47 The pipeline: OCR with olmOCR and Chandra 2 13:36 Exercise: annotating a biography by hand 17:20 Python first, and where regular expressions break down 22:28 Extracting structured JSON with Qwen on a cluster or OpenRouter 28:50 Auditing the output: dittos and duplicate people from OCR errors 35:16 Grounding places and people in Wikidata with an MCP server and QLever 43:49 Cost, and special rules for ambiguous places such as Victoria 46:34 How clean the data needs to be depends on the question 48:08 Q&A: publishing to Wikidata and learning with Claude Code Talk page: https://proflouishyman.github.io/ai_c... Conference program: https://proflouishyman.github.io/ai_c... All talks:    • AI and History Conference 2026   Recorded at the AI and History Conference 2026, Johns Hopkins University (SNF Agora Institute), in partnership with the American Historical Association. Hosted by the Center for Economy and Society. #DigitalHistory #AIandHistory #HistoryAndAI #Wikidata #OCR #LinkedOpenData