BioImage.IO Chatbot: A Community-Driven AI Assistant for Integrative Computational Bioimaging

nature methods Volume 21 | August 2024 | 1368–1370 | 1368 https://doi.org/10.1038/s41592-024-02370-y Correspondence BioImage.IO Chatbot: a community-driven AI assistant for integrative computational bioimaging B ioimaging is a rapidly evolving field with vast and complex data, presenting both opportunities and challenges. Researchers use a variety of tools, such as ImageJ1, alongside specialized AI tools2. The embrace of open science has led to the development of community-based resources such as the BioImage Informatics Index, BioImage Archive, image.sc forum and BioImage Model Zoo3, providing a wealth of tools and data but also creating an overwhelming landscape. Managing diverse data formats, complex workflows and AI models often requires coding or scripting, posing significant challenges for biologists without extensive programming experience. Meanwhile, large language model (LLM)-based conversational assistants such as ChatGPT, powered by GPT-4 with tool call and vision capabilities and advanced reasoning frameworks such as ReAct4, are ideal for helping bioimaging users navigate complex ecosystems and solve bioimage analysis tasks5. The BioImage.IO Chatbot is a communitydriven platform leveraging GPT-4, retrieval augmented generation6 (RAG) and advanced AI agents 7 to help users navigate bioimaging tools, databases and services. Powered by a BioImage community knowledge base built from community-contributed documentation, the chatbot enables efficient semantic search using RAG. Technical documents are split into chunks, generating language embeddings for each segment, which are stored in a vector database to allow the retrieval of information for augmenting responses. This dynamic resource includes documentation from tools like ImageJ 1 and deepImageJ 8 , as well as databases like bio.tools and the ImageJ Wiki. This setup allows the chatbot to deliver precise, context-aware answers, stay current with the latest information without constant fine-tuning, and minimize hallucinations in responses (Fig. 1a). The BioImage.IO Chatbot goes beyond information retrieval by incorporating an extension mechanism enabling enhancements through in-browser ImJoy9 extensions or remote Hypha3 services. These extensions allow the chatbot to perform actions like making function calls, generating code, interacting with online databases and services such as the BioImage Informatics Index and the image. sc forum, generating Python code for image analysis and running AI models (Fig. 1a). When a user’s query is received along with their profile and conversation history, the LLM-powered agent selects tools from a list of extensions, executes them and iteratively observes the outputs in a ReAct loop (Fig. 1b). The ReAct approach combines reasoning and acting with LLMs, prompting them to generate reasoning traces and actions, enabling dynamic adaptation to external environments. This approach not only makes it possible to query documentation, books, online databases and remote services (Fig. 1c) but also enhances the chatbot’s ability to handle complex inquiries by refining search keywords, initiating multi-hop tool calls and dynamically adjusting action plans to provide reliable answers in diverse settings (Supplementary Video 1). To further enhance its utility, we provide specific assistants by grouping different extensions and assigning different goals: a “BioImage seeker” (Melman) for general Q&A, a “BioImage tutor” (Nina) for educational content and a “BioImage analyst” (Bridget) to facilitate bioimage analysis on users’ own data. Leveraging GPT-4’s code generation, toolcalling, vision capabilities and the ReAct loop design, our chatbot assistant automates bioimage analysis, as depicted in Fig. 1d and Supplementary Videos 2 and 3. Based on user requests, it iteratively generates and executes code in an in-browser Python code interpreter to load and display local images, runs AI models such as Cellpose for image segmentation via the BioEngine server, and creates statistical reports. Similar to human programmers, the chatbot autonomously solves analysis tasks by consulting documentation, inspecting images and fixing code problems based on error stack traces. Additionally, GPT-4’s vision capabilities provide visual feedback, allowing the chatbot to examine user images and analysis results, such as segmentation outputs or plots. This enables continuous performance improvement and paves the way for fully autonomous analysis. The BioImage.IO Chatbot offers various flexible integration modes (Fig. 1e). It can be embedded into third-party websites, allowing dynamically loaded custom extensions via ImJoy or Hypha tailored to specific contexts. For example, coupled with model search extensions, it serves as an assistant for the helpdesk of the BioImage Model Zoo (https:// bioimage.io). Additionally, the chatbot supports the development of customized chat assistants (OpenAI GPTs), enabling users to create tailored solutions for publishing to the OpenAI GPT Store. We are acutely aware of the challenges presented by LLMs 10 , including inconsistent answers, biases from training data and the generation of plausible but incorrect information. By using RAG, tool-calling and code generation, the chatbot sources information from reputable databases and established tools, reducing biases and hallucinations. In this work, we used closed-source LLMs such as OpenAI GPT-4, despite their inherent limitations, and we are actively exploring opensource alternatives with similar capabilities. Our commitment to open-source principles promotes transparency, community engagement and continuous improvement. In summary, the BioImage.IO Chatbot marks a substantial advancement in AI-assisted computational bioimaging, fostering an inclusive community that promotes innovation and shared knowledge. We anticipate that it will become an essential tool for the life sciences, streamlining complex bioimaging tasks and empowering researchers with the collective expertise of the global community. Reporting summary Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article. Check for updates nature methods Volume 21 | August 2024 | 1368–1370 | 1369 Correspondence Here are three cell segmentation models in the bioimage model zoo: 1. Live Cell Segmentation Boundary Model - Description: Cell segmentation for phase-contrast microscopy… Find cell segmentation models in the bioimage model zoo Navigating the BioImage Model Zoo Please Inspect segmentation results visually I’ve successfully run the CellPose model on your image using the BioEngine Service. Here’s the result: Segment the image at /mnt/nuclei.tif using CellPose Integrated chat assistant Custom ChatGPT assistant Documentation and books Online databases and services User’s profile User’s query Conversation context AI models and analysis tools BioImage seeker Tool selection Tool execution User query BioImage.IO Chatbot ChatGPT —--------------- —---- —- —-- — —------ —- —---- —---- – —-- —--- —-- —----- —- —- Hi, I am Melman, how can I help you today? BioImage Model Zoo Hi, I am Melman, how can I help you today? Melman d AI-assisted bioimage analysis Mount folder Code interpreter BioImage.IO Chatbot Conversational assistants c Intelligent information retrieval assistant a BioImage community knowledge base and tools b LLM-powered AI agents and assistants The PML (Promyelocytic Leukemia) protein, also known as MYL, RNF71, or TRIM19…Here are some cell images: Tell me about PML protein and show some cell images. Exploring the Human Protein Atlas BioImage.IO e BioImage.IO Chatbot integration and customization Vector database Code generation and execution imagej wiki, napari, ilastik, deepimagej, imjoy, bioimage.io... BioImage Archive, Image.sc, Human Protein Atlas, bio.tools... Cellpose, stardist, scikit-image, BioEngine, imagej.js... Code interpreter Custom extensions API call Extensions Tools ReAct loop Observations Thoughts GPT-4 BioImage tutor Nina BioImage analyst Bridget Answer LLM agent Fig. 1 | Integration of BioImage.IO Chatbot for enhanced computational bioimaging support. a, The BioImage.IO Chatbot integrates a communitydriven knowledge base, drawing from resources such as documentation, books, online databases and analysis tools, providing reliable, up-to-date context information and support for users’ interaction with the chatbot. b, The chatbot leverages GPT-4-powered AI agents to dynamically select and execute tools from unified extensions, adjusting plans based on user queries, profile and chat history. Assistants such as Melman, Nina and Bridget offer specialized support. c, The chatbot serves as an intelligent information retrieval assistant, searching AI models and databases such as the BioImage Model Zoo and Human Protein Atlas to answer bioimaging questions with community-created sources. d, The chatbot can generate and execute Python code in the user’s browser for local image processing and Cellpose segmentation via BioEngine, using GPT-4’s vision capabilities for autonomous closed-loop analysis workflows. e, Custom extensions allow the chatbot to be integrated into other project websites via ImJoy or Hypha, with the API supporting the creation of customized OpenAI GPTs for tailored user assistants. (This figure includes icons designed by Freepik.). nature methods Volume 21 | August 2024 | 1368–1370 | 1370 Correspondence Data availability The data used to support the BioImage.IO Chatbot are either publicly accessible from their original sources or available via the chatbot GitHub repository (https://github.com/ bioimage-io/bioimageio-chatbot). The prebuilt knowledge bases are created by compiling a list of community data sources, detailed in the manifest file within the same repository. Code availability The source code for the BioImage.IO Chatbot is available at https://github.com/bioimage-io/ bioimageio-chatbot. In addition to the raw code, the repository contains detailed documentation, usage examples and other supplementary material for the chatbot. Wanlu Lei1,2, Caterina Fuster-Barceló 3,4, Gabriel Reder 5, Arrate Muñoz-Barrutia 3,4 & Wei Ouyang 5 1Department of Intelligent Systems, KTH Royal Institute of Technology, Stockholm, Sweden. 2Ericsson Inc., Santa Clara, CA, USA. 3Bioengineering Department, Universidad Carlos III de Madrid, Leganes, Spain. 4Bioengineering Division, Instituto de Investigación Sanitaria Gregorio Marañón, Madrid, Spain. 5Department of Applied Physics, Science for Life Laboratory, KTH Royal Institute of Technology, Stockholm, Sweden. e-mail: [email protected] Published online: 9 August 2024 References 1. Schneider, C. A., Rasband, W. S. & Eliceiri, K. W. Nat. Methods 9, 671–675 (2012). 2. Stringer, C., Wang, T., Michaelos, M. & Pachitariu, M. Nat. Methods 18, 100–106 (2021). 3. Ouyang, W. et al. Preprint at bioRxiv https://doi.org/ 10.1101/2022.06.07.495102 (2022). 4. Yao, S. et al. Preprint at arXiv https://doi.org/10.48550/ arXiv.2210.03629 (2022). 5. Royer, L. A. Nat. Methods https://doi.org/10.1038/s41592024-02310-w (2024). 6. Lewis, P. et al. Proc. NIPS '20 793, 16 (2020); https://doi.org/ 10.5555/3495724.3496517 7. Hong, S. et al. in Twelfth International Conference on Learning Representations https://openreview.net/ forum?id=VtmBAGCN7o (2024). 8. Gómez-de-Mariscal, E., García-López-de-Haro, C., Munoz-Barrutia, A. & Sage, D. Nat. Methods 18, 1192–1195 (2021). 9. Ouyang, W., Mueller, F., Hjelmare, M., Lundberg, E. & Zimmer, C. Nat. Methods 16, 1199–1200 (2019). 10. Farquhar, S., Kossen, J., Kuhn, L. & Gal, Y. Nature 630, 625–630 (2024). Acknowledgements We thank C. Rueden for his suggestions on accessing ImageJ wiki documentation and image.sc forum, as well as for testing and reporting bugs, providing valuable feedback during the review process. We also thank M. Kalas for his advice on accessing the bio.tools metadata and M. Hartley for guidance on accessing the BioImage Archive API. We appreciate the AI4Life consortium members’ efforts in creating and improving the BioImage Model Zoo and its documentation. Additionally, we are grateful to F. Jug for testing our system and providing valuable feedback on its design. We also thank W. Xu for his support in implementing and testing chatbot extensions. This work was partially supported by the European Union’s Horizon Europe research and innovation program under grant agreement number 101057970 (AI4Life project) awarded to A.M.-B. and W.O., by the SciLifeLab & Wallenberg Data Driven Life Science Program (grant: KAW 2020.0239) and the Göran Gustafsson Prize (grant: 2317) awarded to W.O., and by the Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación, under grant PID2019109820RB-I00, MCIN/AEI/10.13039/501100011033/, co-financed by the European Regional Development Fund (ERDF), ‘A way of making Europe’, awarded to A.M.-B. Views and opinions expressed are those of the authors only and do not necessarily reflect those of the European Union. Neither the European Union nor the granting authority can be held responsible for them. The authors used the language model ChatGPT developed by OpenAI to assist in structuring and drafting this paper. Author contributions W.O. led conceptualization and design of the project, with supporting code and design contributions from W.L., C.F.-B. and A.M.-B. The development and implementation of the BioImage.IO Chatbot were carried out by W.L., G.R. and W.O. C.F.-B. and A.M.-B. performed testing. C.F.-B. organized the documentation and user interaction design. The manuscript was written by W.O. with input from the other authors. Funding and project administration were managed by A.M.-B. and W.O. Competing interests W.O. is a co-founder of Amun AI AB, a commercial company that builds, delivers, supports and integrates AI-powered data management systems for academic, biotech and pharmaceutical industries. W.L. is an employee of Ericsson Inc.; however, Ericsson Inc. did not influence the study design. Additional information Supplementary information The online version contains supplementary material available at https://doi.org/10.1038/s41592-024-02370-y. Peer review information Nature Methods thanks Curtis Rueden and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.