Assisting Forensic Identification through Unsupervised Information Extraction of Free Text Autopsy Reports: The Disappearances Cases during the Brazilian Military Dictatorship

information Article Assisting Forensic Identification through Unsupervised Information Extraction of Free Text Autopsy Reports: The Disappearances Cases during the Brazilian Military Dictatorship Patricia Martin-Rodilla 1,* , Marcia L. Hattori 2and Cesar Gonzalez-Perez 2 1Research Center on Information Technologies (CiTIUS), University of Santiago de Compostela, Jenaro de la Fuente Domínguez, s/n, 15782 Santiago de Compostela, Spain 2Institute of Heritage Sciences (Incipit), Spanish National Research Council (CSIC), Avda. Vigo, s/n, 15705 Santiago de Compostela, Spain *Correspondence: patricia.martin.r[email protected] Received: 31 May 2019; Accepted: 3 July 2019; Published: 5 July 2019   Abstract: Anthropological, archaeological, and forensic studies situate enforced disappearance as a strategy associated with the Brazilian military dictatorship (1964–1985), leaving hundreds of persons without identity or cause of death identified. Their forensic reports are the only existing clue for people identification and detection of possible crimes associated with them. The exchange of information among institutions about the identities of disappeared people was not a common practice. Thus, their analysis requires unsupervised techniques, mainly due to the fact that their contextual annotation is extremely time-consuming, difficult to obtain, and with high dependence on the annotator. The use of these techniques allows researchers to assist in the identification and analysis in four areas: Common causes of death, relevant body locations, personal belongings terminology, and correlations between actors such as doctors and police officers involved in the disappearances. This paper analyzes almost 3000 textual reports of missing persons in S ã o Paulo city during the Brazilian dictatorship through unsupervised algorithms of information extraction in Portuguese, identifying named entities and relevant terminology associated with these four criteria. The analysis allowed us to observe terminological patterns relevant for people identification (e.g., presence of rings or similar personal belongings) and automate the study of correlations between actors. The proposed system acts as a first classificatory and indexing middleware of the reports and represents a feasible system that can assist researchers working in pattern search among autopsy reports. Keywords: information extraction; named entity recognition; terminology extraction; autopsy reports 1. Introduction The development and improvement over the last few decades of natural language processing (hereafter NLP) algorithms, both in performance and precision in some relevant tasks (named entity recognition, open information extraction, part-of-speech tagging, among others), has allowed a more systematic coverage and application of these approaches to domains where large textual sources are common and highly specific terminology and discursive structures exist. Thus, it is common to find applications of NLP algorithms to legal, administrative, or medical reports with varying levels of supervision. Specifically, there are some available ad hoc corpora [ 1 , 2 ], reference indicators [ 3 , 4 ], and applications for these tasks in the medical domain [ 5 – 7 ], although with few applications of NLP to the treatment of forensic reports (autopsies and related reports). However, large investigation efforts into natural catastrophes, war massacres, or political conflicts generally involve an open universe of victims Information 2019,10, 231; doi:10.3390/info10070231 www.mdpi.com/journal/information Information 2019,10, 231 2 of 15 and a high volume of forensic reports. In these contexts, professionals often require software-assisted treatment that allows them to jointly analyze a large volume of reports, looking for certain patterns in them. In addition to this need (given by the high volume of documents), most of these reports contain information about unsolved cases, which makes accurate and reliable processing of high humanitarian importance. For these reasons, the textual analysis and processing of reports like these is crucial. In some Latin America countries like Brazil, there has been a studied [ 8 – 11 ] systematic strategy by the state, using the omission of information in forensic analysis or related forensic practices as a bureaucracy system to force people disappearances. This strategy possibly makes it difficult to identify the forensic cases tagged as NN (a category used for tagging person cases who we consider as potential missing persons without identification) cases, since the information presented in the forensic reports are vague or even filled with all information that could help to individualize that body. These situations have been categorized as negligent forensic practices [12]. In this paper, we present an analysis based on unsupervised information extraction algorithms in Portuguese of a collection of about 3000 forensic reports of persons buried as NN during 1971–1975, the most repressive period of the dictatorship. Our goal was to create a system that can act as a middleware to analyze the information contained in the reports based on the criteria established by anthropologists and forensic experts, who subsequently can perform a preliminarily validation of the results. The paper is organized as follows: The remainder of this Introduction section presents the historical context, analysis criteria, and motivation for this work, as well as a review of existing initiatives of natural language application in similar forensic contexts. Section 2describes the materials and methods employed, including the particularities of the natural language suite Linguakit [ 13 ] for Portuguese, which was used as the basis for information extraction, as well as the forensic corpus analyzed. Section 3presents the results obtained according to the expert criteria adopted: (1) Common causes of death, (2) relevant body locations, (3) personal belongings terminology, and (4) correlations between actors. Due to the limited output visualization options of the Linguakit suite, these results are subsequently treated and visualized with Google Data Studio [ 14 ] dashboards. Section 4discusses the overall application and their forensic and technological implications. Finally, Section 5summarizes the contributions and details future directions of work. 1.1. Historical Context and Motivation On March 31 1964, making use of a coup and the force of the tanks in the streets, the military of Brazil institutionalized execution, torture, and murder. The civilian president, Jo ã o Goulart, was exiled, people were arrested, and had their political and civil rights suspended. According to different studies made by the Amnesty Commission (the Amnesty Commission was installed in the Ministry of Justice on August 28 2001 and aims to examine and assess the amnesty applications, issuing an opinion intended to subsidize the Minister of Justice in the decision on the granting of Amnesty Policy) (2019), more than 75,000 persons demand compensation from the state because they were directly affected by the repression. This number shows the impact of the dictatorship and the consequences today in Brazilian society [15]. In 1990, the mass grave of Perus came to public attention [ 9 ]. A long investigation was carried out with the documentation of the legal medicine institutions of S ã o Paulo [ 16 ], specifically about the names of the politically disappeared, who were buried with false names in the cemetery of Perus. However, it is recognized by the final report of the Brazilian national Truth Commission [ 17 ] that from the year 1973, the strategy of disappearance became systematic [ 18 ] and even documents related to death were produced (albeit with false information). The suspicion is that some of these cases could have been buried as NN, a category commonly used by institutions in cases where identity of a body is not known [ 9 , 19 ]. The forensic anthropology analysis carried out in the 1990s and 2000s were conducted by different research groups that ultimately did not complete the work and further Information 2019,10, 231 3 of 15 contributed to making more identification even more difficult [ 12 ]. Since 2014, an agreement between institutions has made significant progress [20]. Since the beginning of this last investigation in the Perus mass grave, only two people have been identified. The huge volume of cases only in Perus, not to mention the whole of S ã o Paulo city, highlights the difficulty of the document analysis task, the possible impact on Brazilian society, and the need for software assistance. The present article aims to advance in this direction by applying named entity recognition and information extraction algorithms. Our final goal is to provide the experts with a simple software assistance system that allows them to analyze a corpus of forensic reports according to their own criteria and take, as a starting point, the textual information automatically extracted with unsupervised techniques. In the next section, common algorithms used and their applications to similar domains are described. 1.2. Automatic Information Extraction of Medical, Anthropological, and Forensic Information The medical domain constituted one of the first domains for massive application of NLP studies for a number of reasons, including their narrative tradition in reporting or the controlled output formats, vocabularies, and linguistic structures and constructions. As we described in the introduction, it is common to find specific NLP corpora, metrics, and developments for medical purposes. However, it is difficult to find similar works for forensic information, especially in interdisciplinary contexts in which the authors of the reports are not only doctors, but also police officers, anthropologists, or archaeologists, which to a greater or lesser extent contributes to the collaborative reporting activities. There are some recent information extraction applications in analyzing separately police reports [ 21 ] and medical autopsies [ 22 ], with different levels of supervision and mainly in English. There are also some attempts for Portuguese [ 22 ], in which we can find a good review of other cross-domain approaches for NLP tasks, especially information extraction, but without references to previous works on forensic applications. The challenge of extracting information from different contributors from various backgrounds and based on expert criteria is presented in all revised works. These current efforts are clearly aligned with our proposal here and constitute a good basis to continue working on the same direction. 2. Materials and Methods In this section, we describe the characteristics of the analyzed corpus, the conceptualization of the underlying information, and the NLP suite used for the unsupervised information extraction applications, which constitutes the pipeline of the proposed system. 2.1. The “Disappearances in São Paulo” Corpus The corpus we adopted is composed of forensic reports compiled over many years of research about disappearances during the Brazilian military dictatorship in S ã o Paulo. It includes a total of 2980 forensic reports of cases categorized as NN. It is necessary to emphasize that the reports contained in the corpus come from a great diversity of authors (doctors in their majority), who in turn worked from various sources of information coming in from police stations or testimonies from local people, etc. This implies that, although a certain degree of homogeneous treatment is necessary in order to build a computational solution, the reports do not share a common philosophy or make up a “designed” corpus; quite the contrary, they respond to different administrative practices of this past political regime. This heterogeneity presents an additional challenge, both for the human expert and for the computer system that must assist and analyze the texts. As shown in Figure 1, each forensic report corresponds to one missing person case and contains information about first police impressions during the moment of finding the body, including an initial description of the body context and situation; and medical information from the autopsy and related analysis. Reports were written on paper by the doctors in charge and signed by two doctors. Reports usually follow a narrative schema that includes body location, person profile (gender, age, skin color, Information 2019,10, 231 4 of 15 etc.), primary and secondary hypothesis about cause of death, identification information (personal belongings, body marks, etc.), and administrative information (responsible institutions, doctors and police officers involved in each case). Information 2019, 10, 231 4 of 15 Figure 1. Sample of a forensic report (blurred to protect personal details of a missing person) in the “Disappearances in São Paulo” corpus. One of biggest challenges from these reports is the non-systematic way in which some aspects of the reports are described. For example, the place of death is very ambiguous in some cases, sometimes appearing with just street, road, river, but not with the exact place. The same happens for clothes descriptions or death causes. Due to the high volume of reports and the mentioned heterogeneity, the first step in our work was to construct a conceptual model of the major concepts and relationships underlying the corpus. Figure 2 shows the conceptual model (using the object-oriented paradigm [23–25]) that was created to represent the major concepts, properties, and relationships that are mentioned in the corpus. The model is expressed in ConML [26,27], a conceptual modeling language that allows for expressing models in humanities and social sciences domains [28–30]. We chose ConML for two major reasons. Firstly, it is one of the very few modelling languages that focuses on conceptual modelling rather than the specification of software systems (such as UML [31]), while maintaining a significant degree of formalization, which makes it a good candidate for software system implementations. Secondly, ConML incorporates the capacity to represent “soft” issues such as temporality, subjectivity, and vagueness, which are especially relevant to the humanities, whereas other modelling languages lack this ability. The model contains two main categories: Person and institution (modelled as classes in ConML terminology), as well as the actors involved in the reports and their possible roles, including direction roles of institutions and labor relations over time. Also, the necessary attributes have been added to cater for the disappeared persons profile, including the location where the body was found, the cause of death, a legal medicine description about contextual information, police department in charge, initial evidences, and physical description (clothes, personal objects, etc.). Once the conceptual model was ready, a data model was obtained to prepare the necessary software storage for the textual information extracted from the reports. Figure 1. Sample of a forensic report (blurred to protect personal details of a missing person) in the “Disappearances in São Paulo” corpus. One of biggest challenges from these reports is the non-systematic way in which some aspects of the reports are described. For example, the place of death is very ambiguous in some cases, sometimes appearing with just street, road, river, but not with the exact place. The same happens for clothes descriptions or death causes. Due to the high volume of reports and the mentioned heterogeneity, the first step in our work was to construct a conceptual model of the major concepts and relationships underlying the corpus. Figure 2shows the conceptual model (using the object-oriented paradigm [ 23 – 25 ]) that was created to represent the major concepts, properties, and relationships that are mentioned in the corpus. The model is expressed in ConML [ 26 , 27 ], a conceptual modeling language that allows for expressing models in humanities and social sciences domains [ 28 – 30 ]. We chose ConML for two major reasons. Firstly, it is one of the very few modelling languages that focuses on conceptual modelling rather than the specification of software systems (such as UML [ 31 ]), while maintaining a significant degree of formalization, which makes it a good candidate for software system implementations. Secondly, ConML incorporates the capacity to represent “soft” issues such as temporality, subjectivity, and vagueness, which are especially relevant to the humanities, whereas other modelling languages lack this ability. The model contains two main categories: Person and institution (modelled as classes in ConML terminology), as well as the actors involved in the reports and their possible roles, including direction roles of institutions and labor relations over time. Also, the necessary attributes have been Information 2019,10, 231 5 of 15 added to cater for the disappeared persons profile, including the location where the body was found, the cause of death, a legal medicine description about contextual information, police department in charge, initial evidences, and physical description (clothes, personal objects, etc.). Once the conceptual model was ready, a data model was obtained to prepare the necessary software storage for the textual information extracted from the reports. Information 2019, 10, 231 5 of 15 Figure 2. Conceptual model representing the major concepts, properties, and relationships that are mentioned in the corpus. Then, we designed a pipeline architecture to apply information extraction algorithms to the selected corpus, aiming to extract the information stipulated by the data model and without losing sight of the four criteria defined by the experts that guided the analysis. In particular, we found that having an explicit conceptual model greatly helped us to focus the extraction and analysis efforts around the necessary concepts, properties, and relationships, acting as a set of guidelines that established what semantic fields we were interested in, what kinds of characterization was expected for each one, and what connections were expected to appear. In the next sections, we detail this NLP pipeline and the subsequent analysis of results. 2.2. Unsupervised Information Extraction in Portuguese: Linguakit Suite As well as the rest of NLP tasks and algorithms, the development of methods and resources for Portuguese are increasing day by day. Some important examples are HAREM and Second HAREM [32], Linguakit [13], or SIEMÊS [33] algorithms and resources for unsupervised named entity recognition, joint with well-known suites such as FreeLing [34] or Standford CoreNLP [35] for Portuguese and related supervised initiatives based on conditional random fields [36]. It is important to mention here similar works only focused on semantic relation extraction [37]. Most of these NLP resources and algorithms developed for Portuguese and based on machine learning methods require a huge effort to annotate and establish a training set, plus the subsequent training phase. This dependence on annotating information constituted our first handicap in using neural-based algorithms for the NLP analysis, since the contextual annotation of forensic corpus is extremely time-consuming, difficult to obtain due to the needed for experts in the field and in contact with the bodies and the locations, and highly dependent on the annotator for some of the parts. While the medical information (such as cause of death) usually presents a greater degree of agreement and objectivity, descriptions such as previous situation of the body, clothes and personal belongings, or socio-cultural descriptions of the person present a greater degree of variation among annotators. Precisely, it is these more variable details that are especially relevant in terms of a possible Figure 2. Conceptual model representing the major concepts, properties, and relationships that are mentioned in the corpus. Then, we designed a pipeline architecture to apply information extraction algorithms to the selected corpus, aiming to extract the information stipulated by the data model and without losing sight of the four criteria defined by the experts that guided the analysis. In particular, we found that having an explicit conceptual model greatly helped us to focus the extraction and analysis efforts around the necessary concepts, properties, and relationships, acting as a set of guidelines that established what semantic fields we were interested in, what kinds of characterization was expected for each one, and what connections were expected to appear. In the next sections, we detail this NLP pipeline and the subsequent analysis of results. 2.2. Unsupervised Information Extraction in Portuguese: Linguakit Suite As well as the rest of NLP tasks and algorithms, the development of methods and resources for Portuguese are increasing day by day. Some important examples are HAREM and Second HAREM [ 32 ], Linguakit [ 13 ], or SIEM Ê S [ 33 ] algorithms and resources for unsupervised named entity recognition, joint with well-known suites such as FreeLing [ 34 ] or Standford CoreNLP [ 35 ] for Portuguese and related supervised initiatives based on conditional random fields [ 36 ]. It is important to mention here similar works only focused on semantic relation extraction [37]. Most of these NLP resources and algorithms developed for Portuguese and based on machine learning methods require a huge effort to annotate and establish a training set, plus the subsequent training phase. This dependence on annotating information constituted our first handicap in using neural-based algorithms for the NLP analysis, since the contextual annotation of forensic corpus Information 2019,10, 231 6 of 15 is extremely time-consuming, difficult to obtain due to the needed for experts in the field and in contact with the bodies and the locations, and highly dependent on the annotator for some of the parts. While the medical information (such as cause of death) usually presents a greater degree of agreement and objectivity, descriptions such as previous situation of the body, clothes and personal belongings, or socio-cultural descriptions of the person present a greater degree of variation among annotators. Precisely, it is these more variable details that are especially relevant in terms of a possible identification of the person or detection of negligent forensic practices. Thus, unsupervised methods offer us the best alternative in our case, as they remove the need for human annotators. Specifically, the Linguakit suite was evaluated and compared to supervised approaches [ 38 , 39 ] for the four analyzed languages (English, Portuguese, Spanish, and Galician), using different corpora, with results close to those of FreeLing and Stanford CoreNLP, and clearly surpassing OpenNLP. Thus, we determined that Linguakit constituted a good candidate for the necessary linguistic analysis and information extraction. The Linguakit suite presents a modular structure that is depicted in Figure 3, highlighting the sub-modules that were used in this work. Information 2019, 10, 231 6 of 15 identification of the person or detection of negligent forensic practices. Thus, unsupervised methods offer us the best alternative in our case, as they remove the need for human annotators. Specifically, the Linguakit suite was evaluated and compared to supervised approaches [38,39] for the four analyzed languages (English, Portuguese, Spanish, and Galician), using different corpora, with results close to those of FreeLing and Stanford CoreNLP, and clearly surpassing OpenNLP. Thus, we determined that Linguakit constituted a good candidate for the necessary linguistic analysis and information extraction. The Linguakit suite presents a modular structure that is depicted in Figure 3, highlighting the sub-modules that were used in this work. Figure 3. Linguakit modules used for building the natural language processing (NLP) system, and the sequential pipeline and interdependencies between modules output and criteria of analysis taken. Firstly, we treated the input reports from the corpus with some tools of the basic analysis module. This module is highly dependent on the language of the source. Due to the specialization of Linguakit in Portuguese (also Galician and Spanish), this module presents a very high performance on text segmentation into sentences, subsequent tokenization process, and splitting (with basic rules of splitting, such as separation in case of contractions) for Portuguese. Based on these previous outputs, the deep analysis module was used for its lemmatizer. Then, data were disambiguated by the PoS tagger. This disambiguation process allowed the identification of proper names through named entity recognition (NER) and their subsequent named entity classification (NEC). The joint output between the NER entities recognized and their classification in NEC offered the first valid output, making it possible to achieve some of our goals related to extraction information from the forensics reports and according to expert criteria: We had a list of the named entities present in each report, especially actors (doctors and police officers), institutions (mainly hospitals names and police offices), and some locations such as street names. This information helped us address criteria 2 (relevant body locations) and 4 (correlations between actors) defined by the experts. However, this first analysis was not enough to obtain non-named entities that are however relevant to the experts, and which often appear in narrative sections of the forensic reports. To address this, we carried out a keyword and multiword combined analysis, with a filter on semantic information for three topic areas: Death causes, personal belongings found on the bodies, and nonFigure 3. Linguakit modules used for building the natural language processing (NLP) system, and the sequential pipeline and interdependencies between modules output and criteria of analysis taken. Firstly, we treated the input reports from the corpus with some tools of the basic analysis module. This module is highly dependent on the language of the source. Due to the specialization of Linguakit in Portuguese (also Galician and Spanish), this module presents a very high performance on text segmentation into sentences, subsequent tokenization process, and splitting (with basic rules of splitting, such as separation in case of contractions) for Portuguese. Based on these previous outputs, the deep analysis module was used for its lemmatizer. Then, data were disambiguated by the PoS tagger. This disambiguation process allowed the identification of proper names through named entity recognition (NER) and their subsequent named entity classification (NEC). The joint output between the NER entities recognized and their classification in NEC offered the first valid output, making it possible to achieve some of our goals related to extraction information from the forensics reports and according to expert criteria: We had a list of the named entities present in each report, especially actors (doctors and police officers), institutions (mainly hospitals names and Information 2019,10, 231 7 of 15 police offices), and some locations such as street names. This information helped us address criteria 2 (relevant body locations) and 4 (correlations between actors) defined by the experts. However, this first analysis was not enough to obtain non-named entities that are however relevant to the experts, and which often appear in narrative sections of the forensic reports. To address this, we carried out a keyword and multiword combined analysis, with a filter on semantic information for three topic areas: Death causes, personal belongings found on the bodies, and non-NER locations (i.e., words that refer to places or locations without a proper name but that are potentially relevant, such as hill, river, etc.). This analysis responds mainly to criteria 1 (causes of death), 2 (relevant body locations), and 3 (personal belongings terminology) of the experts. We used extractors of relevant terms (keywords and multiword expressions) from the information extraction module and an application based on them, the semantic annotator or entity linking (hereafter EL). The final application allowed us to link the terms mentioned in the text with the concepts of the ontological and encyclopedic database DBpedia. Linguakit Entity Linking module was evaluated for Portuguese language on this task [ 40 ], giving rise to similar results to state-of-the-art EL systems, such as DBpedia Spotlight [ 41 ]. The final use of the EL module responded to the need of having a link between the information extracted from the reports to a reference ontology for future applications in the same domain or for a re-analysis of the current corpus or their future versions. Note that, due to the modular architecture of Linguakit, we obtained an output from each submodule used. In the case of NER and NEC outputs, the list of named entities identified and their classification is directly usable for our middleware system to classify, conceptualize (following the ConML model), and visualize the information of the corpus. In the case of the final part of the pipeline, based on focalized keywords, multiword, and their semantic links, the middleware layer of the systems acts as an aggregator of the information extracted from the three modules for each forensic report. We describe the final outputs and their interpretation in the Results section. 3. Results This section describes the results of the forensic corpus analysis, organized in two parts. First, we show the results corresponding to the algorithmic outputs of the pipeline, organized by expert criteria, also including additional information about grammatical properties and the study of frequencies by output. Subsequently, these outputs are visualized through data analytics dashboards, in order to better assist forensic experts in the global analysis of the corpus. 3.1. Results Analysis We organized the analysis of the results following the criteria defined by experts. Regarding criterion 1, the experts need assistance on extracting common causes of death, the terminology employed for their description, and links to the specific set of reports related to each family of terms. Thus, we performed a combined keywords and multiword analysis for the entire corpus (2890 reports), looking in the Histórico and Local da morte part of the reports. This process produced two sets of words about cause of death terminology: 100 keywords and 665 multiword combinations. Figure 4shows the first 60 entries for each list. The system combined the entries in function of the relevance ranking obtained from Linguakit and offered the ranked terms to the expert. The resultant ranked list provides the experts with an anchoring terminology link to the semantic components of the death causes, which, combined with the DBPedia entity linking analysis, constitutes a first ontological reference for the underlying death causes associated with the corpus. Regarding criterion 2, software assistance focused on identifying relevant body locations. The information extraction process for this criterion was divided into two different parts: On the one hand, locations were found as named entities, such as names of hospitals (i.e., Hospital de as Cl í nicas or Instituto Paulista) offered by Linguakit NEC. On the other hand, we performed an analysis of the Historico part of the report (that inherits information from the initial location for each body) by Information 2019,10, 231 8 of 15 looking up semantic links of the associated entities in DBPedia corresponding to locations terminology (for example rio Tietê). Figure 5shows part of both outputs obtained for criterion 2 analysis. Regarding criterion 3, the extraction and characterization of personal belongings is performed from the Vestes part of the reports. We performed a combined keywords and multiword analysis for the entire corpus (2890 reports). This process resulted in two sets of words about clothes and personal belongings (hats, bags, rings, etc.) terminology: 100 keywords and 1762 multiword combinations were extracted. Information 2019, 10, 231 8 of 15 Figure 4. Linguakit output for keywords and multiword analysis of causes of death in the ‘Disappearances in São Paulo’ corpus. Figure 5. Linguakit output for keywords and multiword analysis of historical information (Histórico) in the ‘Disappearances in São Paulo’ corpus. Figure 6 shows the first 40 entries of each list. The system combined the entries in function of the relevance ranking obtained from Linguakit and offered the ranked terms to the expert. In a first approach, the experts envisioned that the personal belongings analysis would offer them some clues towards identification of bodies, such as special materials, wedding rings, etc. They defined a subset of them that could provide these clues. However, in most cases, the experts realized that the terms Figure 4. Linguakit output for keywords and multiword analysis of causes of death in the ‘Disappearances in São Paulo’ corpus. Information 2019, 10, 231 8 of 15 Figure 4. Linguakit output for keywords and multiword analysis of causes of death in the ‘Disappearances in São Paulo’ corpus. Figure 5. Linguakit output for keywords and multiword analysis of historical information (Histórico) in the ‘Disappearances in São Paulo’ corpus. Figure 6 shows the first 40 entries of each list. The system combined the entries in function of the relevance ranking obtained from Linguakit and offered the ranked terms to the expert. In a first approach, the experts envisioned that the personal belongings analysis would offer them some clues towards identification of bodies, such as special materials, wedding rings, etc. They defined a subset of them that could provide these clues. However, in most cases, the experts realized that the terms Figure 5. Linguakit output for keywords and multiword analysis of historical information (Hist ó rico) in the ‘Disappearances in São Paulo’ corpus. Information 2019,10, 231 9 of 15 Figure 6shows the first 40 entries of each list. The system combined the entries in function of the relevance ranking obtained from Linguakit and offered the ranked terms to the expert. In a first approach, the experts envisioned that the personal belongings analysis would offer them some clues towards identification of bodies, such as special materials, wedding rings, etc. They defined a subset of them that could provide these clues. However, in most cases, the experts realized that the terms extracted by this analysis are also useful for an additional purpose: The description of the belongings are very poor in most of the reports (with generalist terms such as “algod ã o”, “calça”, etc.) with an absence of description about gender-specific clothes and belongings, sizes, specific textures, materials, or print patterns on clothes. This is very unusual and goes clearly against good practices in forensic descriptions, especially in cases that deal with someone who is going to be buried unidentified (NN). In these cases, one of the most important aspects for recognition by relatives and later the identification is the description of personal belongings. Thanks to our work, the experts decided to perform a cross-analysis confronting which reports presented these unusual characteristics and who signed each one, in order to look for patterns on authorship by doctors that signed negligently described reports. This cross-analysis offers some responses regarding possible implications of doctors with unresolved disappearances and negligent forensic practices, with the discovering of pairs of co-working doctors that is necessary to deeply analyze in the near future. Information 2019, 10, 231 9 of 15 extracted by this analysis are also useful for an additional purpose: The description of the belongings are very poor in most of the reports (with generalist terms such as “algodão”, “calça”, etc.) with an absence of description about gender-specific clothes and belongings, sizes, specific textures, materials, or print patterns on clothes. This is very unusual and goes clearly against good practices in forensic descriptions, especially in cases that deal with someone who is going to be buried unidentified (NN). In these cases, one of the most important aspects for recognition by relatives and later the identification is the description of personal belongings. Thanks to our work, the experts decided to perform a cross-analysis confronting which reports presented these unusual characteristics and who signed each one, in order to look for patterns on authorship by doctors that signed negligently described reports. This cross-analysis offers some responses regarding possible implications of doctors with unresolved disappearances and negligent forensic practices, with the discovering of pairs of co-working doctors that is necessary to deeply analyze in the near future. Figure 6. Linguakit output for keywords and multiword analysis of body description: Clothes and personal belongings (Vestes) in the ‘Disappearances in São Paulo’ corpus. Finally, criterion 4 responds to the correlation of doctors’ analysis. Thus, we took the NEC output of Linguakit that automatically classify PERSONS and automatically checked the names of the doctors from a list that the experts had. Thus, we established a matrix of signatory doctors for each report, thus having access to information about how many reports each pair of doctors had signed together, in what year, and working for which organizations. The next section shows the visualization obtained from the outputs, which we consider to be prototypes for a future system that will allow experts to analyze and interpret the obtained information more deeply. 3.2. Visualizing Results in an Interactive Dashboard It is common that the results provided by NLP algorithms presents an output format in free text or, in some cases, in a textual-based structure output stored in formats such as XML, JSON, or similar. Although most of the current software systems can process these formats in an efficient way, it is necessary to remember here that the final users of the information contained on the corpus under use are forensic experts from different background with knowledge generation necessities, but with no Figure 6. Linguakit output for keywords and multiword analysis of body description: Clothes and personal belongings (Vestes) in the ‘Disappearances in São Paulo’ corpus. Finally, criterion 4 responds to the correlation of doctors’ analysis. Thus, we took the NEC output of Linguakit that automatically classify PERSONS and automatically checked the names of the doctors from a list that the experts had. Thus, we established a matrix of signatory doctors for each report, thus having access to information about how many reports each pair of doctors had signed together, in what year, and working for which organizations. The next section shows the visualization obtained from the outputs, which we consider to be prototypes for a future system that will allow experts to analyze and interpret the obtained information more deeply.