End-to-end Information Extraction from Archival Records with Multimodal Large Language Models
Abstract
This talk was presented on November 13, 2025 at the 34th ACM International Conference on Information and Knowledge Management (CIKM 2025) in Seoul, Korea. In the paper associated with this presentation, we first introduce BZKOpen, a new annotated dataset designed for key information extraction from historical German index cards. Furthermore, we systematically assess the capabilities of several state-of-the-art MLLMs on the task of extracting key information from these archival documents. We further provide practical insights into prompt engineering and inference settings, offering guidance for applying MLLMs to real-world key information extraction tasks. Additionally, we highlight the need for more ground truth datasets that include a wider range of historical documents with varying quality and in multiple languages, in order to fully explore the potentials and limitations of MLLMs for key information extraction from historical records.
Full text
Mahsa Vafaie, Sven Hertling, Inger Banse, Kevin Dubout, Harald Sack End-to-end Information Extraction from Archival Records with Multimodal Large Language Models 1
Vafaie, Mahsa, et al., End-to-end Information Extraction from Archival Records with Multimodal Large Language Models, CIKM 2025, November 13. Agenda Project Overview Experiments & Results Research Contributions Takeaways Dataset & Methodology 2
Vafaie, Mahsa, et al., End-to-end Information Extraction from Archival Records with Multimodal Large Language Models, CIKM 2025, November 13. Project “Wiedergutmachung” An attempt to compensate for National Socialist injustice in Germany Digitalisation of more than 100 km of archival documents to create a knowledge graph (KG) Wieder + gut + machung again + good + making 3
Vafaie, Mahsa, et al., End-to-end Information Extraction from Archival Records with Multimodal Large Language Models, CIKM 2025, November 13. The “Wiedergutmachung” Knowledge Graph P1 Friedrich Löwenthal Esslingen 26.5.98 D1 ES/12224-1 Landesamt für die Wiedergutmachung Stuttgart givenName familyName birthPlace birthDate cardNumber createdAt Org1 legalName mentionedIn 4
Vafaie, Mahsa, et al., End-to-end Information Extraction from Archival Records with Multimodal Large Language Models, CIKM 2025, November 13. The Online Collection https://www.archivportal-d.de/themenportale/wiedergutmachung Family and biography research 47% Local History research 26% General research 27% Infographic based on a survey conducted by the State Archive of Baden-Württemberg 5
Vafaie, Mahsa, et al., End-to-end Information Extraction from Archival Records with Multimodal Large Language Models, CIKM 2025, November 13. Research Contributions 1. A real-life born-analogue dataset for Key Information Extraction 2. Evaluation of the latest Multimodal Large Language Models for Key Information Extraction 6
Vafaie, Mahsa, et al., End-to-end Information Extraction from Archival Records with Multimodal Large Language Models, CIKM 2025, November 13. ❏Central card file of most of applications for compensation of National Socialist injustices in the Federal Republic of Germany ❏about 1.9 million cards with basic information on applicants, persecuted persons and proceedings ❏Kept from 1950s until today ❏Basis for the person search in Online Collection Wiedergutmachung The Bundeszentralkartei (BZK) Collection Source: Landesarchiv NRW – Abteilung Rheinland – BR 3015 ZK-Nr. 64083, 190667, 15932, scho8, 67800/II/6095 7
Vafaie, Mahsa, et al., End-to-end Information Extraction from Archival Records with Multimodal Large Language Models, CIKM 2025, November 13. The BZKOpen Dataset https://huggingface.co/datasets/MahsaVafaie/BZKopen Keys for data extraction: CompensationOffice1, BZKNr, ApplicantFirstName, ApplicantLastName, ApplicantAltFirstName, ApplicantBirthName, ApplicantAltLastName, ApplicantBirthDate, ApplicantBirthPlace, ApplicantCurrentAddress, VictimFirstName, VictimLastName, VictimAltFirstName, VictimBirthName, VictimAltLastName, VictimBirthDate, VictimBirthPlace, VictimDeathDate, VictimDeathPlace 8
Vafaie, Mahsa, et al., End-to-end Information Extraction from Archival Records with Multimodal Large Language Models, CIKM 2025, November 13. End-to-end MLLM-based Information Extraction <image>\nPlease provide the following information as you can see on the image as a Python dictionary [schema]… Input Image and Prompt Output Sequence Multimodal LLM Evaluation Model {""full_response"": ""```python\n{\n \""CompensationOffice1\"" : \""Nordrhein-Westfalen\"",\n…} BZKNr: 1906 67 ApplicantFirstName: Pinkas ApplicantLastName: Riesel ApplicantBirthDate: 19.6.67 ApplicantBirthPlace: Kossov VictimtFirstName: Jakob/Jerzb VictimLastName: Riesel VictimDeathPlace: Warschau Converted CSV 9
Vafaie, Mahsa, et al., End-to-end Information Extraction from Archival Records with Multimodal Large Language Models, CIKM 2025, November 13. Zero-shot Performance gain with prompt tuning Better results for arbitrary values - The bigger the model (with the same architecture) doesn’t necessarily mean the better the result -LMDeploy implementation speeds up the processing by 20 times Some Takeaways Model Size Zero-shot vs. few-shot Deployment 16 Few-shot Negligible performance gain with prompt tuning Better suited for values with a pattern/repetition
Thank you very much! Now it’s your turn! mahsa.vafaie@fiz-karlsruhe.de sven.hertling@fiz-karlsruhe.de Supervised by: Prof. Dr. Harald Sack harald.sack@fiz-karlsruhe.de https://www.linkedin.com/in/mahsa-vafaie/ https://sigmoid.social/@MahsaVafaie 17 If I must die, you must live to tell my story - Refaat Alareer, Palestinian poet (1979-2023)