CLEF HIPE-2026 - Shared Task Participation Guidelines

CLEF HIPE-2026: Participation Guidelines Shared Task on Person-Place Relation Extraction from Historical Documents v.2025-12-04 HIPE-2026 Team: Juri Opitz, Maud Ehrmann, Simon Clematide, Corina Raclé, Emanuela Boros, Andrianos Michail, Matteo Romanello Contents 1 Person–Place Relation Extraction from Multilingual Historical Texts 2 2 Task Description 2 3 Data 5 4 Evaluation Campaign and System Responses 8 5 Submission Instructions 9 6 Evaluation 10 7 CLEF Workshop and Notebook Papers 13 A Data Preparation 15 Useful Links •HIPE-2026 website: https://hipe-eval.github.io/HIPE-2026/ •Google Group: https://groups.google.com/g/hipe-2026 •HIPE-2026 data: https://github.com/hipe-eval/hipe-2026-data •HIPE-2026 evaluation: https://github.com/hipe-eval/hipe-2026-eval 1 HIPE Team CLEF HIPE-2026: Participation Guidelines 1 Person–Place Relation Extraction from Multilingual Historical Texts HIPE-2026 is a CLEF Evaluation Lab dedicated to the extraction of person–place relations in multilingual historical documents. A person-place relation corresponds to a semantic link between an individual and a location as evidenced in a document. Such relations may indicate where a person is said to be at a given moment, where they lived or worked, or places connected to notable moments in their life (e.g., birthplaces, residences, visits, travel destinations). Together, these relations can support the reconstruction of individuals’ geographical and temporal trajectories. Detecting these implicit or explicit, spatio-temporal relations cannot be achieved through simple document co-occurrence of entity mentions. Rather, it requires temporal reasoning, geographical inference, and interpretation of noisy historical texts–often with sparse or indirect contextual cues–to detect and qualify person–place relations with appropriate degrees of certainty. The objective of HIPE-2026 is to advance the automatic detection of such relations, enabling the reconstruction of individuals’ movements in space and time and the tracing of life trajectories in support of digital humanities scholarship. The task is designed to be approachable by both generative AI systems (LLMs) and more traditional classification models. 2 Task Description In essence, participants are tasked with developing systems that determine the nature of the relationship implied by each person-place pair in a historical document. Each pair consists of two entities, one person and one location, each of which appears in the text through one or more mentions. Specifically, systems must establish whether the text indicates that the person is currently at that place within the document’s temporal horizon (a recency-oriented isAt relation), suggests that the person was at that place at some point in time (a more general at relation), or provides no meaningful evidence linking the person and the place. As illustrated in Figure 1, the isAt relation is bounded in time and close to the document publication date, while the at relation is only right-bounded by the publication date: the relation could hold for any time in the past, but naturally not in the future. The next sections provide a more detailed and formal definition of the task. 2.1 Terminology Before defining the sub-tasks, we introduce the terminology. Each term corresponds to fields in the data structure presented in Section 3.2. •context: The full text of a document or passage together with its metadata (for example, newspaper title, language, publication date). 2 HIPE Team CLEF HIPE-2026: Participation Guidelines Figure 1: Schematic representation of the two person-place relation types. •person: An entity, represented by a cluster of mentions in a document, referring to a specific person. •location: An entity, represented by a cluster of mentions in a document, referring to a specific location. We use the terms ‘location‘ and ‘place‘ interchangeably. •(person, location): A pair consisting of one person entity and one location entity drawn from the same document. Not all possible pairs are included in order to avoid excessively large candidate sets. •at: A relation type indicating that the text provides evidence that the person was at the location at some point in time. •isAt: A temporally narrower relation type indicating that the text suggests the person is at the location within the temporal horizon of the news document (e.g., the time of reporting). •system: A system developed by participants that, given a context, a (person, location) pair, and a candidate relation type, assigns a label to that candidate relation. 2.2 Sub-tasks The shared task comprises two sub-tasks, each defined by one of the two relation types that systems must classify. Sub-task definition: at relation Given a context and a (person −location)pair, the general at relation captures whether the context implies that the person was at that place at any point in time. Systems should perform the following classification: 3 HIPE Team CLEF HIPE-2026: Participation Guidelines systemcontext, (person, location),at=                                      true if the context provides explicit evidence that the person was at the place at some point in time. probable if this can be inferred from implicit cues in the context and is thus a likely assumption. false if no evidence is present or the context contradicts such relation. Sub-task definition: isAt relation Given a context and a (person, location)pair, the temporally narrower isAt relation captures whether the context implies that the person was at that place within the document’s temporal horizon. Systems will perform the following classification: systemcontext, (person, location),isAt=            true if the person is located at the place shortly before the document’s publication time. false if this is not the case. Temporal window for isAt.By definition, isAt =true means there is evidence the person was at the location up to about one month before the publication date. Outside this window, it should be set to false. Relation between at and isAt.The isAt relation is a temporal refinement of at: it captures whether the at relation holds shortly before or at the time of publication. The following constraints apply. If at is false, then assigning true to isAt would be inconsistent. If at is true or probable,isAt further specifies whether the person’s presence falls within the document’s temporal horizon. For example, if an article reports a current visit of a US President to Berlin, then the relation between the president and Berlin is labeled as at and isAt. By contrast, if the article refers to a past trip (“Last year, the president visited Berlin.”), only at applies, since the event lies outside the temporal horizon of the document. For simplicity, however, the shared task evaluates the two relations independently. 4 HIPE Team CLEF HIPE-2026: Participation Guidelines 3 Data 3.1 Data Overview HIPE-2026 data consists of two sets of data: historical newspaper articles in French, German, and English spanning roughly 200 years (19C-20C), and literary documents in French. The newspaper material–originating from European and US library collections–composes the core dataset of the shared task and is released as training, development, and test sets (‘Test A’). The literary documents are reserved as a surprise test set (‘Test B’) to evaluate systems’ robustness and generalization on out-of-domain data. The historical newspaper data is derived from the entity-annotated HIPE-2022 material and includes only those datasets that contain person and location annotations. Data has been prepared and newly annotated for the present shared task. The literary surprise set has likewise been newly curated. Appendix 7details the preparation of the historical newspaper data, while a description of the surprise set preparation will be released after the evaluation campaign. 3.2 Data Representation and Contents Data are represented according to a dedicated JSON schema1and stored as JSON Lines (.jsonl) files, where each line corresponds to one document. Each file contains a collection of documents belonging to a single language (German, French, or English). These documents correspond to individual historical newspaper articles or literary text chunks, represented independently. Every document consists of four main components presented below and illustrated in Figure 2: •Metadata keys provide contextual information about the document and includes: –document_id: a unique identifier for the document; –media, including: ∗publication_title; ∗time_period: the year interval spanned by the media sample in the dataset; ∗source_type: e.g. “newspaper”; –source: data path in the original material; –language: the language of the document (one of de, fr, or en); –date: the article’s publication date. •The text key contains the full text of the document. This is the context within which persons and locations are mentioned, and from which systems must infer whether person–place relations hold. •The sampled_pairs list contains person–location pairs selected for annotation. Each pair entry includes: 1https://github.com/hipe-eval/HIPE-2026-data/blob/main/schemas/hipe-2026-data.schema.json 5 HIPE Team CLEF HIPE-2026: Participation Guidelines {"document_id":"NZZ-1798-12-12-a-p0003", "media": { "publication_title":"Neue Zürcher Zeitung", "time_period":"1780-1950", "source_type":"Newspaper" }, "source":"v2.1/hipe2020/de/HIPE-2022-v2.1-hipe2020-dev-de.tsv", "language":"de", "date":"1798-12-12", "text":"(...) Der be\nkannte Irländer Theobald Wolfe Tone, den man auf der\nBompartschen Eskader gefangen nahm, ward von einem\nKriegsgericht zum Tode verurtheilt. Er hörte sein Ur\ntheil mit Standhaftigkeit an , und verlangte nur, daß\nman ihn, seiner franz. Uniform halber, erschießen soll\nte. Da ihm das versagt ward, verwundete er sich mit\neinem Messer so, daß er schwerlich aufkommen wird. —\nDie von Nelson bey Abukir Lissabon angekommen; nur das Schif (...)", ,→ ,→ ,→ ,→ ,→ "sampled_pairs": [ { "pers_entity_id":"NZZ-1798-12-12-a-p0003_Q83235", "pers_wikidata_QID":"Q83235", "pers_mentions_list": ["Nelsons","Nelson","Adm. Nelson","Baron Nelson vom\nNil und Burnharm Torpe",→ ], "loc_entity_id":"NZZ-1798-12-12-a-p0003_Q12783927", "loc_wikidata_QID":"Q12783927", "loc_mentions_list": ["Abukir"], "at":"TRUE", "at_explanation":"(...)", "isAt":"FALSE", "isAt_explanation":"(...)" }, { "pers_entity_id":"NZZ-1798-12-12-a-p0003_Q335205", "pers_wikidata_QID":"Q335205", (...) Figure 2: Example of a HIPE-2026 input document. Key values are shortened for readability. In the training and development sets, the at/isAt fields contain gold labels, whereas in the test set they are set to null. Systems must predict the values for at and isAt, and may optionally provide explanations via the *_explanation fields. 6 HIPE Team CLEF HIPE-2026: Participation Guidelines Data Source Lang. #Docs #Pairs hipe2020 de 146 1,974 fr 214 3,055 en 94 827 sonar de 13 180 newseye de 4 64 fr 77 1,097 letemps fr 256 3,470 Total – 804 10,667 Split Lang. #Docs #Pairs train de 97 1,345 fr 328 4,521 en 56 501 dev de 32 432 fr 109 1,557 en 18 148 test de 34 441 fr 110 1,544 en 20 178 Total 804 10,667 Table 1: HIPE-2026 historical newspaper dataset statistics per data source (left) and per language split (right). Values reflect pre–manual-annotation counts and represent upper bounds; final released data will contain less pairs. –pers_entity_id and loc_entity_id: unique identifiers of the entities; –pers_wikidata_QID and loc_wikidata_QID: Wikidata identifiers, when available; –pers_mentions_list and loc_mentions_list: lists of mentions for the entities; –at and isAt: relation labels (gold value in training/dev; null in test/surprise); –at_explanation and isAt_explanation: optional explanations for the assigned labels. 3.3 Data Splits and Statistics For the historical newspaper data, training, development, and test sets are provided. For the literary data, only the surprise test set is released. Table 1reports statistics for the prepared data (prior to full manual annotation at the time of publication), and therefore reflects the upper bound on the number of person–place pairs that will be available. This section will be updated in next version of the guidelines. 3.4 Data releases The HIPE-2026-data directory is organized per dataset, release version and language, as shown in Figure 3. •There is one UTF-8 encoded, .jsonl file per language and split. •All files, including system submissions, adhere to the same JSON schema. •Files are named according to this schema: HIPE-2026-<hipeversion>-impresso-<split>-<language>.jsonl, where the value of split can be sample,train,dev, or test. 7 HIPE Team CLEF HIPE-2026: Participation Guidelines •HIPE-2022 releases are versioned with a two-part version number (Major.Minor) present in 1) the data directory structure and 2) the filename of each file. •Each HIPE-2026 release has an equivalent git repository release, with release notes. Figure 3: HIPE-2026 release data directory structure. 4 Evaluation Campaign and System Responses 4.1 Evaluation Campaign General Rules •Registration is open until 23 April 2026. Please refer to the HIPE timeline for more information. •The evaluation period runs from Tue 5 May 2026, 10 a.m. (CEST) to Thu 7 May 2026 8 p.m (CEST). At the beginning of this period, input test files for all languages and the surprise test set will be released on the hipe-data-2026 repository. At the end of the evaluation period, participants must send their system responses via email to the submission email (see below). Gold-standard data will be released after the publication of the evaluation results. •Teams may submit up to three runs per language. •Teams must provide predictions for all input test files (Test A for all languages + surprise Test B). •Teams may use any external resources (e.g. additional language resources available elsewhere or homemade, and other annotated data). 8 HIPE Team CLEF HIPE-2026: Participation Guidelines 5 Submission Instructions Participants are required to submit system responses in JSONL format, conforming to the HIPE-2026 schema. Each response file corresponds to a system run and must include predictions for all documents of a given input test set. 5.1 Submission Task Overview Each test input file contains a list of documents, each with sampled person–location pairs. For each such pair, systems must: •Replace null values in the at field with one of: "TRUE","PROBABLE", or "FALSE". •Replace null values in the isAt field with either: "TRUE" or "FALSE". •Optionally, provide a textual explanation by filling the corresponding at_explanation and isAt_explanation fields. Important: Any missing documents, or null label left in the at or isAt fields at submission time will be interpreted as "FALSE" during evaluation. 5.2 Task Scope •Both sub-tasks (at,isAt) are evaluated on the historical newspaper Test A. •Only the at sub-task is evaluated on the surprise Test B (literary domain). •Systems must return predictions for all languages of Test A. The surprise test set B is evaluated separately. •Teams may submit up to three runs per test set (maximum of 6 runs in total for both test sets A and B). 5.3 File Format and Schema Compliance Each system run must be submitted as a single UTF-8 encoded .jsonl file that: •Mirrors the structure and document ordering of the input test file. •Preserves all document and entity metadata. •Adheres strictly to the HIPE-2026 data schema hipe-2026-data.schema.json 9 HIPE Team CLEF HIPE-2026: Participation Guidelines 5. Final dataset creation: Dataset splits are assembled and packaged for release using the final JSON schema. A.2 Literary data Information will be published after the evaluation campaign. 16