A Comparison of Automated Journal Recommender Systems

A Comparison of Automated Journal Recommender Systems Elias Entrup1[0000−0002−7380−1189], Ralph Ewerth1,3[0000−0003−0918−6297], and Anett Hoppe1,3[0000−0002−1452−9509] 1TIB – Leibniz Information Centre for Science and Technology, Hannover, Germany 2L3S Research Center, Leibniz University Hannover, Germany [email protected] Abstract. Choosing the right journal for an article can be a challenge. Automated manuscript matching can help authors with the decision by recommending suitable journals based on user-defined criteria. Several approaches for efficient matching have been proposed in the research literature. However, only a few actual recommender systems are available for end users. In this paper, we present an overview of available services and compare their key characteristics such as input values, functionalities, and privacy. We conduct a quantitative analysis of their recommendation results: (a) examining the overlap in the results and pointing out the similarities among them; (b) evaluating their quality with a comparison of their accuracy. Due to the providers’ lack of transparency about the used technologies, the results cannot be easily interpreted. This highlights the need for openness about the used algorithms and data sets. Keywords: Scientific publishing ·Recommender systems. 1 Introduction The ever-growing number of journals and requirements by funding agencies make it increasingly difficult for researchers to find journals for their manuscripts. Apart from several publication guides [2,26,3,31], the automated recommendation of journals is an active field of study [32,39,41,23]. An overview is provided in [1]. While recommendation approaches based on e.g. co-author networks [23] exist, the majority relies on semantic similarity of the user input to already published scientific articles. Most of the proposed systems do not run in a production mode available to end users. Two prior articles compare available journal recommendation services: In [13], seven services are compared for features, and illustrative query results are presented. The analysis includes the services provided by Clarivate, Cofactor (since archived [8]), Edanz, Elsevier, IEEE, JANE, JournalGuide, and Springer. In [22] the seven recommendation services by Edanz, Elsevier, Enago, IEEE, JANE, JournalGuide, and Springer are compared. The usefulness of these services is analysed in comparison to the publication habits of 15 interviewed researchers. None of the above-mentioned research provides a quantitative comparison of 2 E. Entrup et al. Table 1: List of recommender systems, used abbreviation, the provider, and the scope which describes the subgroup of journals suggested. Recommender Name Abbreviation Provider Scope Bibliometric and Semantic Open Access Recommender Network [4] B!SON TIB and SLUB Open Access Charlesworth Author Services Journal Finder [6] Charlesworth ASJF Charlesworth Author Services All eContent Pro Journal Finder [9] eContent Pro JF eContent Pro All Edanz Journal Selector [10] Edanz JS Edanz (M3) All Elsevier Journal Finder [11] Elsevier JF Elsevier Publisher Food Science and Technology Abstracts Journal Finder [14] FSTA JF FSTA/IFIS Food / Health Institute of Electrical and Electronics Engineers Publication Recommender[17] IEEE PR IEEE Publisher Journal / Author Name Estimator [36] JANE The Biosemantics Group Medicine Jot [37] Jot Townsend Lab Medicine Journal Guide [18] Journal Guide Research Square All MDPI Journal Finder [21] MDPI JF MDPI Publisher Researcher Journal Finder [24] Researcher JF Researcher App All Researcher.Life Journal Finder [25] Researcher.Life JF Researcher.Life All Sage Journal Recommender [28] Sage JR Sage Publishing Publisher ScienceGate Journal Finder [30] ScienceGate JF ScienceGate Publisher Springer Journal Suggester [34] Springer JS Springer Nature Publisher Taylor & Francis Journal Suggester [35] T&F JS Taylor & Francis All Trinka Journal Finder [38] Trinka JF Trinka AI All Wiley Journal Finder [40] Wiley JF Wiley Publisher Web of Science / EndNote Manuscript Matcher [7] WoS MM Clarivate All journal recommender systems; both only include a subset of the available services and compare them using examples or expert evaluations. In this paper, we analyse the 20 currently available journal recommender systems (as of June 6, 2023). We provide an overview of input options, as well as filter and search features. In contrast to previous work, we perform a quantitative evaluation by measuring the accuracy and the number of overlapping results. As a result, we draw conclusions about how well the services perform and complement each other. The paper is organised as follows: Section 2 describes the services selected for the comparison in this paper. A feature comparison with a description of the scope, input, and filters of the services follows. The quantitative analysis of overlapping results and accuracy is presented in Section 3. Section 4 summarises our findings and derives implications for users. A Comparison of Automated Journal Recommender Systems 3 2 Selection and Qualitative Comparison The recommender services in this study were found using “journal recommender”, “manuscript matcher” and “journal finder” as a query for Google and Bing, and evaluating the results on the first three pages. This comparison only considers journal recommendation services that offer a form of automatic manuscript matching. It excludes services that only offer to filter journals. We only consider services that are currently online and that work with automated (not expert) recommendations. The search resulted in 20 recommender systems presented in Table 1. In the following, we will abbreviate their names as indicated. 2.1 Description of Services As shown in Table 1, seven out of 20 services only deliver results that are part of the publisher providing the tool. Of the rest, one is focused on open access and two on medicine. The Charlesworth ASJF, JANE, Jot, Researcher JF, and Trinka.AI JF include pre-print servers in their results. Only B!SON and Jot are open-source. B!SON, the Elsevier JF, JANE, Jot, and the WoS MM have been described in research papers. The B!SON recommender uses Elasticsearch, a neural network, and bibliographic coupling to recommend journals [12,5]. The Charlesworth Journal Finder claims that its search is powered by Researcher JF. The results, however, are different. The Elsevier JF uses BM 25 to find one million similar articles and averages the scores for each journal [19]. JANE uses Lucene’s MoreLikeThis algorithm to find the 50 most similar articles to the user input [29], sums the scores per journal and normalises them. Jot is based on JANE and adds counting of the journal appearances in a user-provided list of references [15]. The WoS MM averages the results of a Support Vector Machine and a Lucene k-Nearest-Neighbors search [27]. 2.2 Search Input While attributes such as full text [16] or authors [20] have been used in research to suggest journals, most services use title and abstract. Keywords and subject are also used by a few services. B!SON works with references by parsing for DOIs in the text the user enters (copied from the PDF or a structured format like bibtex); Jot expects a bibliography file in the RIS format. The Charlesworth ASJF, Edanz JS, IEEE PR, JANE, and Researcher JF use a single input field for several attributes at once. The ScienceGate JF first suggests several, editable keywords based on the title and abstract which are then used for the recommendations. 2.3 Filtering, Sorting and Other Features Most services offer filter and sorting options for the score, title, publisher, publication time, open access or journal impact factor. The Charlesworth ASJF, eContentPro JF, Researcher JF, T&F JS, and Wiley JF have few to no filter, or sorting options. 4 E. Entrup et al. In the following, we will list noteworthy features of the systems: B!SON facilitates the search with an already published article by fetching the inputs via e.g. Crossref. Elsevier JF offers to enter the author’s organisation to get personalised publishing options based on existing agreements. It also detects if the input data belong to an article already published by Elsevier. The IEEE PR can filter venues to publish before a specified date and also searches for conferences (not considered in this paper). JANE allows searching for similar articles and authors who published similar work. Jot provides a two-dimensional visualisation with the “prospect” (estimated chance of acceptance) on the Xaxis and an impact metric (e.g. CiteScore) on the Y-axis. Journal Guide has a comparison function to create an overview of selected journals from the result list. 2.4 Transparency and Privacy Only B!SON and Jot are open source, but several recommender systems show which similar articles led to the recommendation of a journal: B!SON, Edanz JS, JANE, Jot, Journal Guide, Researcher.Life JF, Sage JR, and Trinka.AI JF. Most services do not publicise which journals are in their data set and if it is up-to-date. The websites often, at least, indicate the number of journals included. Both the Journal Guide and JANE have the option to scramble the entered abstract on the client side for privacy. All systems offer an encrypted TLS connection; Jot, however, features an expired certificate at the time of writing. The majority of recommender systems are free and can be used anonymously. However, the WoS MM and the Trinka.Ai JF only work with an account. Researcher.Life JF requires an account for advanced features such as viewing similar articles. Similarly, the eContent Pro JF requires the name and e-mail address for a mandatory sign-up to their e-mail communications. The T&F JS explicitly states that they store the submitted abstracts and which results the user clicks on. The Trinka.AI JF also stores the input along with the generated results so the user can review them later. There is no option to delete searches. Only B!SON and the Edanz JS promise to not store the user inputs. 3 Quantitative Evaluation In the following, we perform a quantitative comparison of the accuracy and the overlap of the results. We used smaller article test sets to avoid getting blocked. 3.1 Comparison of Independent Recommender Systems We test the publisher-independent recommender systems with 50 articles from the only journal we found in all recommenders: “New Biotechnologies” (ISSN 1876-4347). Similarly to research on web search engine results [33], we present the average overlap of the top 15 results based on the ISSNs in Table 2. A Comparison of Automated Journal Recommender Systems 5 Table 2: Comparing the average overlap of results for the publisher-independent recommenders systems B!SON Charlesworth ASJF eContentPro JF Edanz JS FSTA JF JANE Jot Journal Guide Researcher JF Researcher.Life JF ScienceGate JF Trinka.AI JF WoS MM B!SON 15.0 1.5 0.0 1.3 0.5 2.6 2.9 2.0 2.9 2.1 1.9 4.5 2.0 Charlesworth ASJF 1.5 7.0 0.0 1.3 0.6 2.7 3.6 2.8 5.6 1.5 1.7 2.8 2.3 eContentPro JF 0.0 0.0 5.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Edanz JF 1.3 1.3 0.0 15.0 0.4 3.6 3.6 1.5 2.7 2.1 2.6 4.2 3.6 FSTA JF 0.5 0.6 0.0 0.4 4.6 0.6 0.6 0.4 0.8 0.6 0.8 0.8 0.6 JANE 2.6 2.7 0.0 3.6 0.6 15.0 8.9 3.3 3.7 2.6 2.3 4.4 3.5 Jot 2.9 3.6 0.0 3.6 0.6 8.9 14.7 4.1 5.0 2.6 2.4 5.1 3.7 Journal Guide 2.0 2.8 0.0 1.5 0.4 3.3 4.1 14.4 3.4 1.8 1.5 2.9 1.9 Researcher JF 2.9 5.6 0.0 2.7 0.8 3.7 5.0 3.4 15.0 2.4 2.9 4.9 4.3 Research.Life JF 2.1 1.5 0.0 2.1 0.6 2.6 2.6 1.8 2.4 14.4 2.4 3.9 2.7 ScienceGate JF 1.9 1.7 0.0 2.6 0.8 2.3 2.4 1.5 2.9 2.4 15.0 4.2 3.5 Trinka.AI JF 4.5 2.8 0.0 4.2 0.8 4.4 5.1 2.9 4.9 3.9 4.2 14.6 4.5 WoS MM 2.0 2.3 0.0 3.6 0.6 3.5 3.7 1.9 4.3 2.7 3.5 4.5 15.0 The Charlesworth ASJF, eContentPro JF, and FSTA JF provide fewer results, the rest of the services usually provide the 15 results that were considered. Some queries did not return any or only few results. The prominent overlap between Charlesworth ASJF and Researcher JF confirms that Charlesworth ASJF’s recommendations are based on Researcher JF (see Section 2.1). A similar effect can be observed with JANE and Jot. The eContentPro JF and FSTA JF share the least results with the other systems. At least for FSTA JF, this might be caused by its very specific scope. The other systems usually share two to four results, with Trink.AI JF showing the highest overlaps with other services. 3.2 Accuracy We further test the accuracy (precision) of the recommender systems. To ensure a fair comparison, we test with articles from journals in their data set (i.e. test the Elsevier JF only with Elsevier articles). Each system is tested on 100 articles, coming from 100 different journals to broaden the scope of testing. Articles from this year are excluded so that we can assume that the article should be in the training set. As most systems do not disclose the included journals, we used test queries to identify a list of journals in their data set. We use the API of the scientific database Dimensions3to retrieve the corresponding test articles. We also assume that the correct journal is the one where the article was published. 3https://docs.dimensions.ai/dsl/ 6 E. Entrup et al. Table 3: Recommender systems and their accuracy considering the first and the first ten results. Name Acc@1 Acc@10 B!SON 0.20 0.88 Charlesworth ASJF 0.21 0.77 eContentPro JF 0.03 0.16 Edanz JS 0.16 0.54 Elsevier JF 0.35 0.86 FSTA JF 0.07 0.29 IEEE PR 0.26 0.68 JANE 0.83 0.96 Jot 0.19 0.93 Journal Guide 0.38 0.98 MDPI JF 0.48 0.88 Researcher JF 0.07 0.49 Researcher.Life JF 0.15 0.48 Sage JR 0.17 0.69 Springer JS 0.97 0.98 T&F JS 0.48 0.91 Trinka.AI JF 0.07 0.41 ScienceGate JF 0.10 0.35 Wiley JF 0.19 0.59 WoS MM 0.12 0.48 The systems might take other factors into account apart from the semantic match, e.g. possible impact. Having the test articles potentially in the training set is a limitation. Nevertheless, high accuracy can indicate how much the system relies on finding a similar article. The results are shown in Table 3. As JANE is checking for similar articles [29], the accuracy is high because it will usually find the article in question in its data set. Journal Guide and Springer JS also yield high accuracy. The reason for eContentPro JF’s, FSTA JF’s, and ScienceGate JF’s low accuracies are unclear. 4 Conclusions In this paper, we systematically compared 20 journal recommendation services. We found that most of them use the title and abstract to find the bestfitting journal. Apart from publisherspecific services, 13 independent services exist. Many try to inform the user how a match was calculated, but few have published their source code, recommendation approach, or data sources. We tested the accuracy of the services and to what degree they delivered the same results. The accuracy varies widely with the Acc@10 ranging from 16% to 98%. While for most recommender systems two to four results are shared, a higher overlap validates the shared recommendation approach of some services. We derive the following advice: (a) Users should look beyond the first suggestion. (b) For the medical domain, Jot provides more features than JANE and can be recommended. (c) For open-access publications, B!SON and Journal Guide can be recommended. B!SON is more transparent but both services have a high accuracy and number of sorting and filter options. (d) Otherwise, Journal Guide or publisher-specific services can be used. Background knowledge is still required for the final decision. Declaration of Competing Interests The authors were part of the B!SON project. A Comparison of Automated Journal Recommender Systems 7 References 1. Ajmal, S., Muzammil, M.B.: PVRS: Publication Venue Recommendation System A Systematic Literature Review. In: International Conference on Computing Engineering and Design, ICCED 2019, Singapore, April 11-13, 2019. pp. 1–6. https://doi.org/10.1109/ICCED46541.2019.9161106 2. Babor, T.F., Stenius, K., Pates, R., Miovsk´y, M., O’Reilly, J., Candon, P. (eds.): Publishing Addiction Science: A Guide for the Perplexed. Ubiquity Press (2017), https://www.jstor.org/stable/j.ctv3t5qxw 3. Bahadoran, Z., Mirmiran, P., Kashfi, K., Ghasemi, A.: Scientific Publishing in Biomedicine: How to Choose a Journal? International Journal of Endocrinology and Metabolism 19(1), e108417 (Nov 2020). https://doi.org/10.5812/ijem.108417 4. B!SON: B!SON - the Open-Access journal recommender, https://service.tib.eu/ bison, accessed on Jun 6, 2023 5. B!SON: How it works (2023), https://service.tib.eu/bison/how, accessed on Feb 2, 2023 6. Charlesworth Author Services: Journal Finder |Find Journals |Charlesworth Author Services, https://www.cwauthors.com/Journal-Finder, accessed on Jun 6, 2023 7. Clarivate: Web of Science Master Journal List - WoS MJL by Clarivate, https:// mjl.clarivate.com/home, accessed on Jun 6, 2023 8. Cofactor: Journal Selector Tool archived (2021), https://cofactorscience.com/ journal-selector-tool-archived, accessed on Feb 2, 2023 9. eContent Pro: Journal Identifier Database, https://www.econtentpro.com/ journal-finder, accessed on Jun 6, 2023 10. Edanz: MY JOURNAL SELECTOR (Oct 2015), https://www.edanz.com/ journal-selector, accessed on Jun 6, 2023 11. Elsevier: Elsevier®JournalFinder, https://journalfinder.elsevier.com/, accessed on Jun 6, 2023 12. Entrup, E., Eppelin, A., Ewerth, R., Hartwig, J., Tullney, M., Wohlgemuth, M., Hoppe, A.: B!SON: A Tool for Open Access Journal Recommendation. In: Silvello, G., Corcho, O., Manghi, P., Di Nunzio, G.M., Golub, K., Ferro, N., Poggi, A. (eds.) International Conference on Theory and Practice of Digital Libraries, TPDL 2022, Padua, Italy, September 20–23, 2022. pp. 357–364. Lecture Notes in Computer Science, Springer International Publishing, Cham. https://doi.org/10.1007/978-3031-16802-4 33 13. Forrester, A., Bj¨ork, B.C., Tenopir, C.: New web services that help authors choose journals. Learned Publishing 30(4), 281–287 (2017). https://doi.org/10.1002/leap.1112, eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/leap.1112 14. FSTA: Journal Recommendation Service, http://www.fstajournalfinder.com/#/ home/suggest, accessed on Jun 6, 2023 15. Gaffney, S.G., Townsend, J.P.: Jot: guiding journal selection with suitability metrics. Journal of the Medical Library Association 110(3), 376–380 (Dec 2022). https://doi.org/10.5195/jmla.2022.1499, number: 3 16. Ghosal, T., Chakraborty, A., Sonam, R., Ekbal, A., Saha, S., Bhattacharyya, P.: Incorporating Full Text and Bibliographic Features to Improve Scholarly Journal Recommendation. In: ACM/IEEE Joint Conference on Digital Libraries, JCDL 2019, Urbana-Champaign, Illinois, June 2-6, 2019. pp. 374–375. Champaign, IL, USA. https://doi.org/10.1109/JCDL.2019.00077 8 E. Entrup et al. 17. IEEE: IEEE Publication Recommender, https://publication-recommender.ieee. org/home, accessed on Jun 6, 2023 18. JournalGuide: JournalGuide - Home, https://www.journalguide.com/, accessed on Jun 6, 2023 19. Kang, N., Doornenbal, M.A., Schijvenaars, R.J.: Elsevier Journal Finder: Recommending Journals for your Paper. In: ACM Conference on Recommender Systems, RecSys 2015, Vienna Austria, September 16-20 2015. pp. 261–264. https://doi.org/10.1145/2792838.2799663 20. Klemi´nski, R., Kazienko, P., Kajdanowicz, T.: Where should I publish? Heterogeneous, networks-based prediction of paper’s citation success. Journal of Informetrics 15(3), 101200 (Aug 2021). https://doi.org/10.1016/j.joi.2021.101200 21. MDPI: MDPI |Journal Finder, https://www.mdpi.com/about/journalselector, accessed on Jun 6, 2023 22. Nam, N.D., Trung, T., Trung, N.T., Thao, T.P.T.: Manuscript Matcher: A Tool for Finding the Best Journal. In: International Multi-Conference on Complexity, Informatics and Cybernetics, IMCIC 2022, Virtual Event, March 8 - 11, 2022. pp. 50–55. Virtual Conference. https://doi.org/10.54808/IMCIC2022.01.50 23. Pradhan, T., Pal, S.: A hybrid personalized scholarly venue recommender system integrating social network analysis and contextual similarity. Future Generation Computer Systems 110, 1139–1166 (Sep 2020). https://doi.org/10.1016/j.future.2019.11.017 24. Researcher: Researcher |Journal Finder, https://journalfinder.researcher-app. com/, accessed on Jun 6, 2023 25. Researcher.Life: Researcher.Life Journal Finder: Journal Suggester For Your Manuscript, https://researcher.life/journal, accessed on Jun 6, 2023 26. Rison, R.A., Shepphird, J.K., Kidd, M.R.: How to choose the best journal for your case report. Journal of Medical Case Reports 11(1), 198 (Jul 2017). https://doi.org/10.1186/s13256-017-1351-y 27. Rollins, J., McCusker, M., Carlson, J., Stroll, J.: Manuscript Matcher: A Content and Bibliometrics-based Scholarly Journal Recommendation System. In: Mayr, P., Frommholz, I., Cabanac, G. (eds.) Workshop on Bibliometric-enhanced Information Retrieval co-located with the European Conference on Information Retrieval, BIR@ECIR 2017, Aberdeen, UK, April 9th, 2017. CEUR Workshop Proceedings, vol. 1823, pp. 18–29. CEUR-WS.org, http://ceur-ws.org/Vol-1823/paper2.pdf 28. Sage: Journal Recommender, https://journal-recommender.sagepub.com/, accessed on Jun 6, 2023 29. Schuemie, M.J., Kors, J.A.: Jane: suggesting journals, finding experts. Bioinformatics 24(5), 727–728 (Mar 2008). https://doi.org/10.1093/bioinformatics/btn006 30. ScienceGate: Journal Finder |Sciencegate, https://www.sciencegate.app/ journal-finder, accessed on Jun 6, 2023 31. Shokraneh, F., Ilghami, R., Masoomi, R., Amanollahi, A.: How to Select a Journal to Submit and Publish Your Biomedical Paper? BioImpacts 2(1), 61–68 (Mar 2012). https://doi.org/10.5681/bi.2012.008 32. Son, H.T., Tan Phong, H., Dac, N.H.: An efficient approach for paper submission recommendation. In: IEEE REGION 10 CONFERENCE, TENCON 2020, Osaka, Japan, November 16-19, 2020. pp. 726–731. IEEE, Osaka, Japan. https://doi.org/10.1109/TENCON50793.2020.9293909 33. Spink, A., Jansen, B.J., Blakely, C., Koshman, S.: A study of results overlap and uniqueness among major Web search engines. Information Processing & Management 42(5), 1379–1391 (Sep 2006). https://doi.org/10.1016/j.ipm.2005.11.001 A Comparison of Automated Journal Recommender Systems 9 34. Springer: Springer Journal Suggester, https://journalsuggester.springer.com/, accessed on Jun 6, 2023 35. Taylor & Francis: Taylor & Francis Journal Suggester, https://authorservices. taylorandfrancis.com/publishing-your-research/choosing-a-journal/ journal-suggester/, accessed on Jun 6, 2023 36. The Biosemantics Group: Journal / Author Name Estimator, https://jane. biosemantics.org/, accessed on Jun 6, 2023 37. Townsend Lab: Search, https://jot.publichealth.yale.edu/search, accessed on Jun 6, 2023 38. Trinka AI: Trinka - Grammar Checker For Academic & Technical Writing, https:// cloud.trinka.ai/journal-finder, accessed on Jun 6, 2023 39. Vara, N., Mirzabeigi, M., Sotudeh, H., Fakhrahmad, S.M.: Application of k-means clustering algorithm to improve effectiveness of the results recommended by journal recommender system. Scientometrics (May 2022). https://doi.org/10.1007/s11192022-04397-4 40. Wiley: Wiley SJF - Search a journal, https://journalfinder.wiley.com/search? type=match, accessed on Jun 6, 2023 41. ZhengWei, H., JinTao, M., YanNi, Y., Jin, H., Ye, T.: Recommendation method for academic journal submission based on doc2vec and XGBoost. Scientometrics 127(5), 2381–2394 (May 2022). https://doi.org/10.1007/s11192-022-04354-1