Full text
Brown University Library — NIRD, 2025-12-05 Hall-Hoag at Scale: Taming 800,000 Pages Birkin Diana & Justin Uhr
Brown University Library — NIRD, 2025-12-05 the BDR
Brown University Library — NIRD, 2025-12-05 hall-hoag (above the fold)
Brown University Library — NIRD, 2025-12-05 hall-hoag (below the fold) 16.1-K orgs 681.7-K items (so far)
Brown University Library — NIRD, 2025-12-05 in this talk... ● the preparation and ingestion pipeline ● lessons learned (scale) ● improvements ○ discovery ○ item-organization
Brown University Library — NIRD, 2025-12-05 Pipeline Overview Images of Scanned Docs from Vendor Metadata Files from Librarians Preparation Check inputs and create output file directories Create page-level metadata based on org-level metadata Main Pipeline Detect Orientation and rotate if needed Run Tesseract OCR and output ALTO xml Create PDF - add document image - write invisible text, using location and size information to align it with the matching text in the image Create JP2 Derivative Image Write organd page-level tracker files Ingestion Find new candidates for ingest based on tracker files Double-check using solr to avoid duplication Ingest metadata and content files into BDR. First creating org-level ‘parent’ objects and then adding pages as ‘child’ objects Add ingestion info to tracker files
Brown University Library — NIRD, 2025-12-05 lessons learned... ● the more validation that can be done up-front -- the better ○ confirming metadata, checking vendor images ● there are _going_ to be errors -- accept them and prepare for them as best you can ○ make sure they don’t break the process ○ track them to be handled later ● figure out workflows with colleagues to reduce bottlenecks
Brown University Library — NIRD, 2025-12-05 lessons learned... ● at this scale, small things add up ○ an error that affects only 0.02% of pages occurs every week ○ an extra second or two per item adds months to the total ● it's ok to implement a best-for-now solution ○ by the time you implement a solution a better one will be available
Brown University Library — NIRD, 2025-12-05 discovery challenge The question... How can we make these items discoverable, without overwhelming results with possibly-offensive extremist content?
Brown University Library — NIRD, 2025-12-05 grouping items via multi-modal model (gemini) [ { "document_id": "1", "proposed_document_title": "House Committee Report: American Nazi Party Investigation", "page_number": 1, "is_final_page": false, "is_standalone": false }, { "document_id": "2", "proposed_document_title": "House Committee Report: American Nazi Party Investigation", "page_number": 2, "is_final_page": false, "is_standalone": false }, (--snip--) { "document_id": "2", "proposed_document_title": "Committee on Un-American Activities Publication Request", (--snip--) } ]
Brown University Library — NIRD, 2025-12-05 closing... We're looking forward to 2026 work! ● Applying these technology improvements ● Working with the team on policy issues We'd also like to thank the other members of our team, and the Hay librarians, who helped implement this project.
