A deep-learning algorithm to classify skin lesions from mpox virus infection

Nature Medicine | Volume 29 | March 2023 | 738–747 738 nature medicine Article https://doi.org/10.1038/s41591-023-02225-7 A deep-learning algorithm to classify skin lesions from mpox virus infection Alexander H. Thieme 1,2,3,4 , Yuanning Zheng1,2, Gautam Machiraju 5, Chris Sadee 1,2, Mirja Mittermaier 4,6, Maximilian Gertler 7, Jorge L. Salinas8, Krithika Srinivasan8, Prashnna Gyawali1, Francisco Carrillo-Perez 1,2,9, Angelo Capodici 1,2,10, Maximilian Uhlig11, Daniel Habenicht 12, Anastassia Löser13, Maja Kohler 14,15, Maximilian Schuessler 1, David Kaul3, Johannes Gollrad3, Jackie Ma 16, Christoph Lippert 17,18, Kendall Billick 19, Isaac Bogoch20, Tina Hernandez-Boussard 1,2,21, Pascal Geldsetzer 22,23,24 & Olivier Gevaert1,2,24 Undetected infection and delayed isolation of infected individuals are key factors driving the monkeypox virus (now termed mpox virus or MPXV) outbreak. To enable earlier detection of MPXV infection, we developed an image-based deep convolutional neural network (named MPXV-CNN) for the identification of the characteristic skin lesions caused by MPXV. We assembled a dataset of 139,198 skin lesion images, split into training/validation and testing cohorts, comprising non-MPXV images (n = 138,522) from eight dermatological repositories and MPXV images (n = 676) from the scientific literature, news articles, social media and a prospective cohort of the Stanford University Medical Center (n = 63 images from 12 patients, all male). In the validation and testing cohorts, the sensitivity of the MPXV-CNN was 0.83 and 0.91, the specificity was 0.965 and 0.898 and the area under the curve was 0.967 and 0.966, respectively. In the prospective cohort, the sensitivity was 0.89. The classification performance of the MPXV-CNN was robust across various skin tones and body regions. To facilitate the usage of the algorithm, we developed a web-based app by which the MPXV-CNN can be accessed for patient guidance. The capability of the MPXV-CNN for identifying MPXV lesions has the potential to aid in MPXV outbreak mitigation. The monkeypox virus (now termed mpox virus or MPXV), a double-stranded DNA virus belonging to the Orthopoxvirus genus and causative agent of a zoonotic disease, has caused an ongoing outbreak with more than 28,700 confirmed cases in 93 countries as of 5 August 2022. The World Health Organization (WHO) has declared this outbreak a Public Health Emergency of International Concern 1 . Animal-to-human transmission was generally assumed and confirmed in numerous recent MPXV outbreaks. Sustained human-to-human transmission was considered limited as infection chains in the human populations were short in endemic regions of Central and West Africa 2 . This outbreak showed for the first time sustained human-to-human community transmission in nonendemic countries3. Cases were reported primarily in men who have sex with men and in some cases in women and children4–9. Modeling by the European Centre for Disease Prevention and Control identified undetected infections and delayed isolation as key parameters that drive MPXV outbreaks 10 . With WHO case definitions 11 , a significant proportion of infections remained undetected5 such as a person with a characteristic vesicular-pustular rash without a history of contact with a confirmed infection. Therefore, multiple authors have suggested a review and broadening of case definitions5,12. Received: 5 August 2022 Accepted: 19 January 2023 Published online: 2 March 2023 Check for updates A full list of affiliations appears at the end of the paper. e-mail: [email protected] Nature Medicine | Volume 29 | March 2023 | 738–747 739 Article https://doi.org/10.1038/s41591-023-02225-7 Table 1 | Number of skin lesion images per category and per data source in the MPXV and non-MPXV datasets used for training and testing the MPXV-CNN MPXV dataset Non-MPXV dataset Category Publications (n = 75) Encyclopedia (n = 4) News articles (n = 13) Social media (n = 1) Prospective cohort Total Danderm DermIS HDA Fitzpatrick 17k DermNet DermNet NZ PADUFES-20 Esteva Total All 380 42 25 202 63 712 3,437 6,589 2,662 16,577 19,289 14,018 2,298 121,170 186,040 Excluded 31 0 1 4 0 36 5 0 48 52 01,973 045,440 47,518 Included 349 42 24 198 63 676 3,432 6,589 2,614 16,525 19,289 12,045 2,298 75,730 138,522 Training 254 42 24 198 0518 0 0 0 0 0 12,045 0 0 12,045 Testing 95 0 0 0 63 158 3,432 6,589 2,614 16,525 19,289 02,298 75,730 126,477 Age Child (<18 years) 35 6 8 7 0 56 –979 – – – a39 –1,018 Adult (≥18 years) 292 32 11 183 63 581 –2,557 – – – a2,259 –4,816 Unknown 22 4 5 8 0 39 3,432 3,053 2,614 16,525 19,289 12,045 075,730 132,688 Sex Male 277 22 12 184 63 558 –2,593 – – – b741 –3,334 Female 19 2 1 4 0 26 –2,520 – – – b753 –3,273 Unknown 53 18 11 10 092 3,432 1,476 2,614 16,525 19,289 12,045 804 75,730 131,915 Skin tone (Fitzpatrick type) I 7 0 1 19 027 – – – 2,941 – – 153 –3,094 II 87 16 672 26 207 – – – 4,796 – – 876 –5,672 III 115 0 5 49 27 196 – – – 3,296 – – 392 –3,688 IV 32 22 324 081 – – – 2,775 – – 62 –2,837 V30 0 0 27 10 67 – – – 1,527 – – 10 –1,537 VI 78 4 9 7 0 98 – – – 628 – – 1 – 629 Unknown 0 0 0 0 0 0 3,432 6,589 2,614 562 19,289 12,045 804 75,730 121,065 Region of body Head 55 11 156 2125 –1,443 – – – – – – 1,443 Neck 2 0 0 0 1 3 – 96 – – – – – – 96 Torso 50 12 316 889 –705 – – – – – – 705 Upper extremity 62 9 5 59 26 161 –916 – – – – – – 916 Lower extremity 33 1 2 12 12 60 –813 – – – – – – 813 Anogenital 103 4 0 35 9151 –223 – – – – – – 223 Anal 16 0 0 12 028 – 5 – – – – – – 5 Perianal 10 0 0 1 3 14 –18 – – – – – – 18 Genital 77 4 0 22 6109 –106 – – – – – – 106 Unknown 0 0 0 0 0 0 – 94 – – – – – – 94 Multiple body regions 27 1 9 11 452 –110 – – – – – – 110 Unknown or zoomed in 17 4 4 9 1 35 3,432 2,283 2,614 16,525 19,289 12,045 2,298 75,730 134,216 Originc Europe 110 0 0 65 0175 c c c – – c– – – Africa 70 0 5 1 0 76 – – – – – c– – – Asia 6 0 3 1 0 10 – – – c– – – – – South America 7 0 0 28 035 – – – c– – – – – North America 41 0 6 92 63 202 – – – – – c– – – Antarctica 0 0 0 0 0 0 – – – – – – – – – Australia 3 0 0 0 0 3 – – – – – c– – – Unknown 112 42 10 11 0175 3,432 6,589 2,614 16,525 19,289 12,045 2,298 75,730 138,522 Nature Medicine | Volume 29 | March 2023 | 738–747 740 Article https://doi.org/10.1038/s41591-023-02225-7 Artificial intelligence (AI)-assisted case definitions have not been explored so far but could represent a solution. Deep convolutional neural networks (CNN) have shown promise in classifying skin lesions in dermatology13–20 with some authors reporting above expert-level accuracy 14 . In recent studies, the majority of MPXV infections (up to 95.2%) were associated with skin lesions 4,5,21 which appear in different stages over the course of the disease. Informing individuals who are worried about having been infected with MPXV as to whether their skin lesions likely stems from an MPXV infection or not could accelerate appropriate care-seeking and improve the adoption of behaviors to reduce onward transmission. This could be accomplished through the integration of an image-based CNN into an app that allows users to analyze an image of their skin lesion. The aim of this study was, therefore, to develop and evaluate the performance of a CNN for the detection of MPXV skin lesions (MPXV-CNN) in photographic images and to integrate the MPXV-CNN into an app. To identify biases and weaknesses, we evaluated the performance of the MPXV-CNN in multiple large image datasets for different skin tones20 and locations of the skin lesion. We also specifically evaluated the performance of the model in classifying MPXV skin lesions versus other acute skin diseases and differential diagnoses with skin lesions of similar appearance, including varicella, drug-induced allergies, impetigo, measles, molluscum contagiosum, orf, scabies and syphilis22. Results Sample characteristics The image characteristics were summarized in Table 1. We constructed a new dataset of photographic images of skin diseases (n = 139,198) originating from multiple publicly available sources and institutional data as follows: 676 images of MPXV skin lesions (MPXV dataset) aggregated from publications of the scientific literature, encyclopedia articles, news articles, social media and prospectively collected MPXV skin lesion images of patients of the Stanford University Medical Center (prospective cohort) and 138,522 images of non-MPXV skin lesions (non-MPXV dataset) from five public dermatological repositories (Danderm, DermIS, Hellenic Dermatological Atlas (HDA), DermNet, DermNet NZ), two public datasets (PAD-UFES-20 (ref. 23), Fitzpatrick 17k 24 ) and one institutional dataset (Esteva 13 ). Image screening and filtering were performed as described in Fig. 1 and Methods. The following metadata was made available per image: diagnoses for Danderm, DermIS, HDA, DermNet, DermNet NZ, PAD-UFES-20, Fitzpatrick 17k, Esteva and the prospective cohort; skin tone for PAD-UFES-20, Fitzpatrick 17k and the prospective cohort; body region for DermIS and the prospective cohort; age group for DermIS, PAD-UFES-20 and the prospective cohort; sex for DermIS, PAD-UFES-20 and the prospective cohort. We mapped diagnoses of all non-MPXV sources to a uniform taxonomy of 2,013 skin diagnoses previously developed at our institute13. Uniform diagnoses could be associated with 94.5% (130,852 of MPXV dataset Non-MPXV dataset Category Publications (n = 75) Encyclopedia (n = 4) News articles (n = 13) Social media (n = 1) Prospective cohort Total Danderm DermIS HDA Fitzpatrick 17k DermNet DermNet NZ PADUFES-20 Esteva Total Lesions (N) N = 0 (rash) 5 0 0 4 0 9 – – – – – – – – – N = 1 118 16 687 30 257 – – – – – – – – – N = 2 38 10 242 20 112 – – – – – – – – – N = 3 26 6 0 18 353 – – – – – – – – – 4 ≤ N ≤ 5 16 3 0 13 537 – – – – – – – – – 6 ≤ N ≤ 10 30 3 1 12 450 – – – – – – – – – N > 10 116 415 21 1157 – – – – – – – – – Unknown 0 0 0 1 0 1 3,432 6,589 2,614 16,525 19,289 12,045 2,298 75,730 138,522 Duration of presence <7 d 49 6 0 12 067 – – – – – – – – – ≥7 d 80 0 1 43 3127 – – – – – – – – – Unknown 220 36 23 143 60 482 3,432 6,589 2,614 16,525 19,289 12,045 2,298 75,730 138,522 Coalesced lesions Yes 132 11 14 27 2186 N/A N/A N/A N/A N/A N/A N/A N/A N/A No 212 31 10 167 61 481 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A 5 0 0 4 0 9 N/A N/A N/A N/A N/A N/A N/A N/A N/A 2022 MPXV outbreak associated Yes 264 26 24 198 63 575 N/A N/A N/A N/A N/A N/A N/A N/A N/A No 85 16 0 0 0 101 N/A N/A N/A N/A N/A N/A N/A N/A N/A MPXV clade Clade 1 38 0 0 0 0 38 N/A N/A N/A N/A N/A N/A N/A N/A N/A Clade 2 303 26 21 198 63 611 N/A N/A N/A N/A N/A N/A N/A N/A N/A Unknown 816 3 0 0 27 N/A N/A N/A N/A N/A N/A N/A N/A N/A aNo classification per image available, but the database owners reported the following ratios: child 14% and adult 86%. bNo classification per image available, but the database owners reported the following ratios: 48% male and 52% female. cNo classification per image is available for non-MPXV repositories and datasets, however the origin of most images can be assigned to the following continents: Danderm—Europe, DermIS—Europe, HDA—Europe, Fitzpatrick 17k—South America and Asia, DermNet—unknown, DermNet NZ—Europe, Africa, North America, Australia, PAD—unknown. All, number of all available skin lesion images; excluded, number of excluded images; included, number of images included in this study; N/A, not applicable; training, number of images used for training the MPXV-CNN; testing, number of images used for testing the MPXV-CNN;–, not available. Table 1 (continued) | Number of skin lesion images per category and per data source in the MPXV and non-MPXV datasets used for training and testing the MPXV-CNN Nature Medicine | Volume 29 | March 2023 | 738–747 741 Article https://doi.org/10.1038/s41591-023-02225-7 138,522) of skin lesion images in the non-MPXV dataset. All evaluations on non-MPXV diagnoses were pooled analyses on the entire non-MPXV dataset. Frequency tables for uniform diagnoses in the training and testing non-MPXV datasets are collated in Supplementary Tables 1–11. Algorithm performance in the training cohort We used images of MPXV skin lesions (n = 518) and non-MPXV skin lesions (n = 12,045) for the training and validation of the MPXV-CNN (Methods: Data splitting). We performed stratified fivefold cross-validation, wherein in each fold, images from 80% of patients were used for training and 20% for validation. The cross-validation was repeated five times. In the validation dataset, the sensitivity was 0.83 (s.d.: 0.01), specificity was 0.965 (s.d.: 0.002) and the area under curve (AUC) was 0.967 (s.d.: 0.003; Fig. 2a). Performance results for other architectures than ResNet34 can be found in Supplementary Table 12. Algorithm performance in the testing cohort After we evaluated the MPXV-CNN using cross-validation, we trained a final model on images (n = 12,563) from the entire training cohort. The final model was evaluated using images from an external testing cohort (Methods: Data splitting). The testing cohort contained 158 MPXV images and 126,477 non-MPXV images. Sensitivity was 0.91, specificity 0.898 (Fig. 2b) and the AUC 0.966 (Fig. 2c). Specifically, sensitivity was 0.89 in MPXV skin lesion images prospectively collected from patients (n = 63 images from 12 patients, all male) of the Stanford University Medical Center and 0.92 in other MPXV skin lesion images (Extended Data Fig. 1). The false-positive rates (FPRs) in non-MPXV skin lesions of the seven dermatological repositories and databases varied between 3.4% and 22.0% (Extended Data Fig. 2). Variation in algorithm performance by image characteristics We evaluated the performance of the MPXV-CNN in regard to the following image characteristics: number of MPXV skin lesions, duration of the presence of the MPXV skin lesion(s) and coalescing of MPXV skin lesions. We observed a high detection performance of MPXV lesions with a duration of the presence of less than 7 d (true-positive rate (TPR) = 95.7%; Extended Data Fig. 3) which demonstrates the early detection ability of the MPXV-CNN. Also, MPXV skin lesions with a duration of the presence of 7 d or more were detected reliably (TPR = 84.6%) illustrating the ability of the MPXV-CNN to recognize skin lesions in different disease stages. The observed median number of skin lesions in the testing cohort was two (interquartile range: (8)). We evaluated the performance in regard to the number of MPXV lesions visible in each skin lesion image. If at least one skin lesion was present, we observed a high detection performance with TPRs ranging from 81.8% (6–10 lesions) to 100% (4–5 lesions; Extended Data Fig. 4). For images showing an MPXV rash without a visible MPXV skin lesion, the detection rate was low (TPR = 33.3%) with a limited number of available images in this category (n = 3). The observed TPR was higher in images showing coalesced (95.5%) versus noncoalesced (91%) MPXV skin lesion images (Supplementary Fig. 1). Variation in algorithm performance by skin disease Because MPXV skin lesions present as acute skin disease, we assessed the performance in classifying MPXV skin lesions versus acute and chronic skin diseases. The testing cohort contained 38,875 images for acute and 85,148 images for chronic skin diseases. For the classification of MPXV versus other acute skin diagnoses, the specificity was 0.886 (Extended Data Fig. 5) and AUC was 0.962 (Fig. 2c). Identification of images for the MPXV dataset Scientific literatur Databases (n = 2) Encyclopedias (n = 4) PubMed: monkeypox AND (rash OR exanthem OR case report OR skin lesions) Google Scholar: (monkeypox AND skin AND lesions) OR (monkeypox AND photo) OR (monkeypox AND case report) Identification Images excluded (n = 36): PubMed (n = 14) Google Scholar (n = 17) Encyclopedias (n = 0) Social media (n = 4) News articles (n = 1) Prospective cohort (n = 0) Records with image data retrieved (n = 294) PubMed (n = 24) Google Scholar (n = 51) Encyclopedias (n = 4) Social media (n = 202) News articles (n = 13) (Social) media Social media (n = 1) News articles (n = 13) Social media: #monkeypox OR monkeypox positive Screening Records assessed for eligibility/image data (n = 140,205) PubMed (n = 235) Social media (n = 133,323) Google scholar (n = 6,630) Encyclopedias (n = 4) Included Images retrieved (n = 712) PubMed (n = 166) Google Scholar (n = 214) Encyclopedia (n = 42) Social media (n = 202) News articles (n = 25) Prospective cohort (n = 63) Images included (n = 676) PubMed (n = 152) Google Scholar (n = 197) Encyclopedia (n = 42) Social media (n = 198) News articles (n = 24) Prospective cohort (n = 63) Identification of images for the non-MPXV dataset Repositories (n = 5) Danderm DermIS HDA DermNet DermNet NZ Datasets (n = 3) Esteva Fitzpatrick 17k PAD-UFES-20 Images assessed for eligibility (n = 186,040) Danderm (n = 3,437) DermIS (n = 6,589) HDA (n = 2,662) DermNet (n = 19,289) DermNet NZ (n = 14,018) Esteva (n = 121,170) Fitzpatrick 17k (n = 16,577) PAD-UFES-20 (n = 2,298) Images included (n = 138,522) Esteva (n = 75,730)Danderm (n = 3,432) Fitzpatrick 17k (n = 16,525)DermIS (n = 6,589) PAD-UFES-20 (n = 2,298)HDA (n = 2,614) DermNet (n = 19,289) DermNet NZ (n = 12,045) Images excluded (n = 47,518) Danderm (n = 5) DermIS (n = 0) HDA (n = 48) DermNet (n = 0) DermNet NZ (n = 1,973) Esteva (n = 45,440) Fitzpatrick 17k (n = 52) PAD-UFES-20 (n = 0) Clinical Prospective cohort (n = 1) News articles (n = 13) Fig. 1 | Flow diagram for the MPXV and non-MPXV image datasets. The flow diagram showed the identification and screening procedures of images to create the MPXV and non-MPXV datasets. MPXV images were collected from publications of the scientific literature, encyclopedia articles, new articles, social media and a prospective cohort of patients from the Stanford University Medical Center, while non-MPXV images originated from eight repositories and datasets. Nature Medicine | Volume 29 | March 2023 | 738–747 742 Article https://doi.org/10.1038/s41591-023-02225-7 For the classification of MPXV versus chronic skin lesions, the specificity was 0.900 (Extended Data Fig. 5) and AUC was 0.967 (Fig. 2c). We also evaluated the FPRs by the category of the non-MPXV skin disease and observed the highest FPRs for the category genodermatoses and supernumerary growths (15.7%; Supplementary Fig. 2). The number of different skin diseases with at least one available image in the non-MPXV dataset, Esteva, DermNet, DermIS, DermNet NZ, HDA, Fitzpatrick 17k, Danderm, DermNet NZ and PAD-UFES was 809, 792, 496, 458, 310, 297, 220, 178 and 6, respectively. When evaluating the performance of the MPXV-CNN in individual skin diseases with at least 50 available images, the highest FPRs were observed for the following acute skin diseases: orf (42.9%), tinea ringworm groin (39.7%) and varicella (34.6%) (Extended Data Fig. 6). We also observed a comparatively high FPR of 26.9% in images with sunburn. We observed the highest FPRs in the following chronic skin diseases: Ehlers–Danlos syndrome (47.7%), lichen planus actinicus (34%) and prurigo nodularis (27%; Extended Data Fig. 7). We found a low number of images (n = 20) for the Ehlers–Danlos syndrome in the training database (Supplementary Table 7). The FPR for eight differential diagnoses of MPXV was highest with orf (42.9%), followed by varicella (34.6%) and molluscum contagiosum (27.3%) (Supplementary Fig. 3). FPRs for common skin diseases such as cherry angioma, skin tags, dermatofibroma, acne vulgaris, eczema, rosacea and allergic contact dermatitis were 26.7%, 17.9%, 16.0%, 16.0%, 16.5%, 7.6% and 6.5%, respectively (Supplementary Table 2). Frequency tables and FPRs of all diagnoses in the non-MPXV dataset and per repository are available in Supplementary Tables 1–11. Variation in algorithm performance by body region The performance also varied by body region of the skin lesion, with the lowest TPR at the head (TPR = 78.9%) and a high detection performance for other body regions ranging from TPR = 80.5% (upper extremities) to TPR = 100% including the anogenital body region (Extended Data Fig. 8). For MPXV skin lesion images with an ‘unknown’ body region, meaning that these images were zoomed in without visible cues of the body region, a high classification performance (TPR = 100%) could be observed (Extended Data Fig. 8). The highest FPR in non-MPXV images was observed in images showing multiple body regions (19.1%). For other body parts, the FPRs were generally low ranging from 3.6% for the anogenital to 8.8% for the torso body region (Supplementary Fig. 4). Variation in algorithm performance by population We evaluated the performance of the MPXV-CNN in regard to the following population characteristics: skin tone, age group and sex. The TPRs varied by skin tones, with the lowest performance in Fitzpatrick type III (TPR = 85.7%) and ranging from TPR = 88.9% to TPR = 100% in other skin tones with very limited data for type 1 (n = 7) and type VI (n = 1; Extended Data Fig. 9). We observed low FPRs for type I to IV on the Fitzpatrick scale ranging from 7.4% for type I to 9.3% for type IV and higher FPRs for type V (12.1%) and 6 (13.9%; Extended Data Fig. 10). A higher FPR could be observed in children (6.8%) versus adults (4%; Supplementary Fig. 5) and male (9.7%) versus female (7.3%) individuals (Supplementary Fig. 6). Explanation maps SHapley Additive exPlanations (SHAP) were a method to explain the prediction of an instance by computing the contribution of each feature (for example, pixel) to the prediction 25 . The SHAP method computed Shapley values from coalitional game theory. By calculating SHAP values, we were able to visualize which portions of an image the MPXV-CNN was focusing on to make a specific prediction. In the MPXV images correctly classified by MPXV-CNN, we found that the regions with high feature importance overlapped with the areas of MPXV skin lesions (Fig. 3). Correspondence between positive SHAP values and the location of the MPXV skin lesion(s) (Fig. 3a–g) and the perilesional inflammation could be observed (Fig. 3c–f). Personalized recommendation system for patient guidance We developed a prototype of a personalized recommendation system (PRS) for MPXV patient guidance implemented as a web-based app named ‘PoxApp’ which could be used on web-enabled devices such as smartphones (Figs. 4 and 5). PoxApp was released as open-source on Github 26 and published online by Charité—Universitätsmedizin Berlin in June 2022 (ref. 27 ) and Stanford University in August 2022 (ref. 28 ). The PRS combined a survey (Fig. 4b,d,e) with picture-taking of a skin lesion (Fig. 4c). The survey consisted of seven items regarding symptoms, risk contacts, sexual behavior and location (Supplementary Figs. 7–14). The PRS estimated the risk of an MPXV infection using a mobile version of the MPXV-CNN (MobileNet V3) and a decision tree (Supplementary Fig. 15). Personalized recommendations provided information on MPXV testing, postexposure vaccination and quarantine (Fig. 4f). MPXV testing was recommended if the MPXV-CNN detected an MPXV skin lesion or criteria derived from WHO case definitions for suspected and probable MPXV cases were met. Postexposure vaccination was recommended if the user encountered a risk contact within the past 21 d. Local healthcare offerings for MPXV testing and vaccination were shown based on the zip code provided by the user. We invited users to participate in a study to donate their data comprising survey answers AUC = 0.967 ± 0.003 False-positive rate False-positive rate All (AUC = 0.966) Acute (AUC = 0.962) Chronic (AUC = 0.967) Predicted MPXV MPXV Actual True-positive rate True-positive rate Non-MPXV Non-MPXV 0.91 0.9 1.0 0.8 0.6 0.4 0.2 0 0 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.09 0.102 0.898 a b c 1.0 0.8 0.6 0.4 0.2 0 0 0.2 0.4 0.6 0.8 1.0 0.2 0.4 0.6 0.8 1.0 Fig. 2 | Performance diagrams of the MPXV-CNN for the validation and testing cohorts. a, ROC curve derived from repeated fivefold cross-validation on the validation cohort (AUC = 0.967 ± 0.003). b, Confusion matrix on the testing cohort showing the ratios of TPs (0.91), TNs (0.898), FPs (0.102) and FNs (0.09). c, ROC curve of the testing cohort that included MPXV skin lesions and either acute non-MPXV skin lesions (AUC = 0.962), chronic non-MPXV skin lesions (AUC = 0.967) or all non-MPXV skin lesions (AUC = 0.966). FPs, false positives; FNs, false negatives; ROC, receiver operating characteristic; TPs, true positives; TNs, true negatives. Nature Medicine | Volume 29 | March 2023 | 738–747 743 Article https://doi.org/10.1038/s41591-023-02225-7 and skin lesion images. In July 2022, we announced PoxApp to a national mailing list addressed to infectious diseases specialists. Users could find PoxApp via popular search engines and links provided by a variety of institutes such as the German National Center for Disease Control, the Ministry of Foreign Affairs, Federal Center for Health Education and Local Departments of Health. Discussion We report the first proof-of-concept of an MPXV-CNN able to classify MPXV skin lesions using photographic images. The MPXV-CNN showed a high classification performance in the validation and testing datasets. We observed a sensitivity of 0.89 in prospectively collected MPXV images from patients of the Stanford University Medical Center and an overall sensitivity of 0.91 and specificity of 0.898 in the whole testing dataset. The MPXV-CNN achieved a high detection performance in MPXV skin lesions that were present for less than 7 d demonstrating its early detection capabilities. Classification performance was robust across various skin tones and body regions, and in MPXV images with a varying number of lesions with and without coalescing. Explanations of the model with SHAP demonstrated that MPXV-CNN identified the locations of MPXV skin lesions in images and their perilesional inflammation. We performed detailed analyses and identified several parameters that impacted the performance, including the body region of the skin lesion, skin tones and non-MPXV diagnoses. The TPR for skin lesions at the head was lower compared to other body locations. This might be related to the complex facial anatomy and the presence of hair. MPXV-CNN’s best performance was achieved in the anogenital and lower extremities regions with TPR of 100% and 85.7% and FPRs of 3.6% and 3.8% which could be considered preferred locations for classification if a patient has multiple lesions. When testing performance across different body regions, we observed the highest FPR for images showing multiple body regions. It is, thus, preferable to avoid taking images at a distance. We generally observed high TPRs ranging from 85.7 to 100% across all skin tones with the lowest values in skin tone Fitzpatrick type III and very limited data for type VI. In addition, we observed higher FPRs in skin tones with Fitzpatrick type V (12.1%) and 5 (13.9%), which may be due to the challenging detection of perilesional inflammation in the darker-pigmented skin tones. In addition, we evaluated the FPRs of diagnoses in non-MPXV skin lesions using a uniform taxonomy of 2,031 skin diseases and a pooled analysis across the entire non-MPXV dataset. Because MPXV causes acute skin lesions, we specifically evaluated the classification performance of the MPXV-CNN when compared to other acute skin diseases. We observed a high performance with a specificity of 0.886 and an AUC of 0.962. The classification performance compared to chronic skin diseases was nearly identical with a specificity of 0.900 and an AUC of 0.967. While the FPRs were low in common diagnoses such as acne, eczema, rosacea and allergic contact dermatitis, we also identified common diagnoses with relatively high FPRs such as in cherry angioma which could substantially reduce the classification performance of the MPXV-CNN in elderly patients. Acute diseases with the highest FPRs were orf, tinea ringworm groin and varicella. Genetic skin disorders such as Ehlers–Danlos syndrome and neurofibromatosis yielded worse performance and could be defined as an exclusion criterion when the MPXV-CNN should not be used. Presumably, the performance could be improved by adding more images of these diagnoses to the training dataset. We conducted a preliminary analysis of known differential diagnoses and found the highest FPR in orf which is known to be hardly distinguishable from MPXV by human experts. For non-MPXV images in the testing cohort, we observed a higher FRP in male versus female individuals. For MPXV images in the testing cohort, sex-based analyses could not be performed due to the nonavailability of data for female patients. However, MPXV images without visible sexual anatomy such as zoomed-in images or images of the extremities had a high classification performance. Additionally, SHAP explanations showed that the MPXV-CNN specifically used the region of the image that contained the skin lesion and there is no evidence that MPXV lesions have a difference in appearance between male and female patients. The main limitation of our study is related to the current scarcity of MPXV photographic images. Due to a lack of public datasets with MPXV images, we created a new dataset from publications of the scientific literature, encyclopedia articles, news articles, social media and a prospective cohort. This approach, however, is prone to biases. –0.02 –0.0075 0.0075 0 –0.01 0 0.01 0.02 –0.02 –0.015 –0.010 0.010 0 0 –0.010 –0.008 –0.004 0.004 0.008 0 0.010 0 0.015 –0.01 0 0.01 0.02 SHAP value SHAP value SHAP value SHAP value SHAP value SHAP value SHAP value SHAP value Image a b c d e f g Fig. 3 | SHAP analysis of the MPXV-CNN. Photographic images of MPXV skin lesions (top) are shown with the corresponding SHAP analysis (bottom) overlaid on the original image to highlight the discriminative image regions used for detection (a–g). The MPXV lesions shown represent different stages as follows: early-stage vesicle (a), small pustule (b), umbilicated pustule (c), papule with central necrosis (d), hand with one ulcerated skin lesion (e), pubic region with multiple ulcerated skin lesions (f) and late-stage crusted plaques (g). Positive SHAP values, shown in red, indicated areas of the image that contributed to the prediction of MPXV skin lesion, whereas negative SHAP values, shown in blue, indicated areas that detracted from the prediction. All MPXV lesions shown in a–g were part of the testing dataset and were classified correctly by the MPXVCNN. Photo credit (a–g): UK Health Security Agency, licensed under the Open Government License 3.0. Nature Medicine | Volume 29 | March 2023 | 738–747 744 Article https://doi.org/10.1038/s41591-023-02225-7 a b c d e f Fig. 4 | Screenshots of PoxApp. a, Screenshots of the start screen are shown. b, Question regarding the presence of new lesions. c, Prompt for taking a photograph of the skin lesion. d, Question regarding further symptoms. e, Question regarding close contacts with infected individuals. f, A personalized recommendation computed from the information provided and the MPXV-CNN classification of the skin lesion image. Nature Medicine | Volume 29 | March 2023 | 738–747 745 Article https://doi.org/10.1038/s41591-023-02225-7 Authors might report pictures not of typical, but of extraordinary cases, such as patients with a generalized exanthem or superinfected lesions. Additionally, because MPXV is endemic in Africa, a significant proportion of individuals in the MPXV dataset had darkly pigmented skin. We diversified our dataset by incorporating up-to-date publications on case reports and media articles related to the current MPXV outbreak, which provided images from regions where the virus was not previously endemic. For the same reason, we integrated photos of individuals reporting an MPXV infection and sharing their pictures on social media. To prove the performance of the MPXV-CNN, we used prospectively collected images of patients with a laboratory-confirmed MPXV infection as a testing cohort. To compensate for any biases that might be present in the MPXV-negative images, we performed our analyses on a high number of images from eight different image repositories and datasets. As pointed out by the WHO, AI has great potential for neglected tropical infections such as MPXV, but ethical and privacy considerations for AI tools have to be carefully taken into account, such as where user data are stored and data stewardship 29 . As with any infectious disease, and as is the case with MPXV, recognizing early symptoms to guide the patient toward a timely diagnosis is critical, potentially preventing severe disease, complications and secondary infections 30 . Therefore, the most benefit of an MPXV-CNN may be generated by integrating the algorithm into a mobile app usable by the public. This approach however raises concerns and comes with significant challenges. A mobile app, that takes a photo of a skin lesion as only input and returns a probability of a MPXV infection, is not sufficient in regard to the guidance for a user. Such a system could be dangerously mistaken as a substitute for a medical test such as a PCR test for MPXV or medical evaluation and treatment. Predictions of the MPXV-CNN need to be evaluated in context with a variety of factors influencing the pretest probability for an infection such as further symptoms reported by the users, close contact with infected individuals and the incidence of infectious cases at the location of the user, or factors that increase the probability for severe diseases such as pregnancy or immune compromise. A system was needed that combines the prediction of the MPXV-CNN with expert knowledge of healthcare professionals considering all the aforementioned factors to generate easy-to-understand recommendations for users. Therefore, we proposed the combination of the MPXV-CNN with a PRS and developed a prototype that (1) asked survey questions to get a clinical picture of the user, (2) provided instructions to mitigate weaknesses of the MPXV-CNN such as taking a picture of the body regions with the highest predictive power and (3) gave easy to understand personalized recommendation based on the estimated risk of infection. At the time of writing, the PRS was evaluated in a prospective trial. Additionally, by integrating the function of a voluntary data donation into such a system, a PRS could become a source of big data for skin lesion images reflecting closely the true distribution of the users’ age, sex, skin tone, ratio of MPXV and non-MPXV skin lesions and non-MPXV diagnoses. However, the MPXV infection status is unknown at the time the user uses the PRS. This limitation can be overcome with modern, semisupervised machine learning techniques that could use large amounts of skin lesion images with unknown infection status for pretraining and would require just a fraction of images with known infection status for learning 31 which could be acquired by recalling the user or by a clinical trial. Further investigations are needed to assess whether the high predictive power of MPXV-CNN obtained from our experiments can be translated into other settings such as an app used by the general public. The high classification performance observed in MPXV images collected from patients is promising. However, a prospective trial with Rash? Photo? Risk contact? More symptoms? Yes No MPXV detected Probable case Yes No No Contact Low risk No 0 1 Phone photograph MPXV-CNN Exposition? Yes Suspected case MPXV detected No Yes b c 1-specificity Sensitivity Early detection pipeline a Evolving model over time dDatabase update Fig. 5 | Components of the PRS for MPXV patient guidance. a, Simplified decision tree for MPXV infection risk stratification derived from WHO case definitions with the addition of an AI-assisted case definition based on predictions of the MPXV-CNN. An IDE was used to create and update the survey for risk stratification (boxes) based on these questions (rhombuses), logical expressions (arrows) and the MPXV-CNN (rhombus with a brain and AI model). An API distributed the most up-to-date survey, logical expressions and MPXV-CNN to web-based apps. b, The web-based app ‘PoxApp’ implemented the PRS for end users allowing them to answer surveys and take photos of their skin lesions and get personalized recommendations, such as MPXV testing or vaccination. c, Component for voluntary data donation with an API to collect, anonymize and store data in a central database. d, New evolving models with higher sensitivity and specificity could potentially be created based on new user data. API, application programming interface. Nature Medicine | Volume 29 | March 2023 | 738–747 746 Article https://doi.org/10.1038/s41591-023-02225-7 patients under real-world conditions and larger datasets of MPXV skin lesion images will be required for this evaluation. In this first version of the MPXV-CNN, predictions will also be made if the image has a low quality such as in low-light conditions or with significant blurriness. New methods like uncertainty quantifications of CNNs could help detect cases where the prediction of the MPXV-CNN should not be used32. Additional evaluations such as the analysis of the MPXV-CNN of multiple images from different body locations of the same patient could help to improve the performance of the MPXV-CNN. Lastly, the ResNet34 architecture researched in this study was not optimized for mobile devices due to its model complexity and the high number of parameters (21.5 million). Additional evaluations will be necessary to compare the performance with mobile-optimized architectures such as EfficientNet33. We propose the following next steps. First, skin lesion images from patients who suspect they are infected with MPXV should be acquired as part of a prospective, multicentered trial. The MPXV and non-MPXV skin lesion images could be used as a testing dataset for next-generation MPXV-CNNs. Second, a prospective, clinical trial on the PRS should be conducted to assess the real-world performance of the MPXV-CNN, risks of misclassifications, compliance of patients to PRS recommendations and cost impact on the healthcare system. Third, efforts for a successful deployment should be made by targeting populations with a high prevalence of MPXV and endemic areas in low-income countries. Fourth, the proposed PRS could be integrated into local early warning systems at a national level that processes additional orthogonal information that enhances the PRS and increases its merit. From a scientific perspective, the combination of imagery data, disease information, demographic data and governmental policies creates a unique multimodal dataset. This first MPXV-CNN could classify photos of skin lesions as being from an MPXV infection or not with a comparatively high degree of discrimination in a testing cohort that included prospectively collected MPXV images of patients. Technologies like the MPXV-CNN can lead the way to AI-assisted case definitions of MPXV and other infectious diseases. We developed an app-based PRS with the integration of a mobile version of the MPXV-CNN that allowed users to upload a photo of their skin lesion and get personalized recommendations. In such a setting, the MPXV-CNN has the potential to accelerate appropriate care-seeking and increase the adoption of behaviors that reduce onward transmission. The images sourced with a PRS could become a rich source of data for the further development and improvement of AI-assisted approaches to address the current and future MPXV outbreaks. Online content Any methods, additional references, Nature Portfolio reporting summaries, source data, extended data, supplementary information, acknowledgements, peer review information; details of author contributions and competing interests; and statements of data and code availability are available at https://doi.org/10.1038/s41591-023-02225-7. References 1. World Health Organization. Second meeting of the International Health Regulations (2005) (IHR) Emergency Committee regarding the multi-country outbreak of monkeypox. https://www.who.int/ news/item/23-07-2022-second-meeting-of-the-internationalhealth-regulations-(2005)-(ihr)-emergency-committeeregarding-the-multi-country-outbreak-of-monkeypox (2022). 2. Beer, E. M. & Rao, V. B. A systematic review of the epidemiology of human monkeypox outbreaks and implications for outbreak strategy. PLoS Negl. Trop. Dis. 13, e0007791 (2019). 3. Vivancos, R. et al. Community transmission of monkeypox in the United Kingdom, April to May 2022. Euro Surveill. 27, 2200422 (2022). 4. Thornhill, J. P. et al. Monkeypox virus infection in humans across 16 countries—April–June 2022. N. Engl. J. Med. 387, 679–691 (2022). 5. Girometti, N. et al. Demographic and clinical characteristics of confirmed human monkeypox virus cases in individuals attending a sexual health centre in London, UK: an observational analysis. Lancet Infect. Dis. 22, 1321–1328 (2022). 6. Perez Duque, M. et al. Ongoing monkeypox virus outbreak, Portugal, 29 April to 23 May 2022. Euro Surveill. 27, (2022). 7. Martínez, J. I. et al. Monkeypox outbreak predominantly affecting men who have sex with men, Madrid, Spain, 26 April to 16 June 2022. Euro Surveill. 27, 2200471 (2022). 8. UK Health Security Agency. Investigation into monkeypox outbreak in England: technical briefing 4. GOV.UK https://www. gov.uk/government/publications/monkeypox-outbreaktechnical-briefings/investigation-into-monkeypox-outbreakin-england-technical-briefing-4 (2022). 9. van Furth, A. M. T. et al. Paediatric monkeypox patient with unknown source of infection, the Netherlands, June 2022. Euro Surveill. 27, 2200552 (2022). 10. European Centre for Disease Prevention and Control. Considerations for contact tracing during the monkeypox outbreak in Europe. https://www.ecdc.europa.eu/en/ publications-data/considerations-contact-tracing-duringmonkeypox-outbreak-europe-2022 (2022). 11. World Health Organization. Disease outbreak news; multi-country monkeypox outbreak in non-endemic countries. https://www. who.int/emergencies/disease-outbreak-news/item/2022-DON385 (2022). 12. Pan, D. et al. Monkeypox in the UK: arguments for a broader case definition. Lancet 399, 2345–2346 (2022). 13. Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017). 14. Haenssle, H. A. et al. Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists. Ann. Oncol. J. Eur. Soc. Med. Oncol. 29, 1836–1842 (2018). 15. Thomsen, K., Iversen, L., Titlestad, T. L. & Winther, O. Systematic review of machine learning for diagnosis and prognosis in dermatology. J. Dermatol. Treat. 31, 496–510 (2020). 16. Hameed, N. et al. Mobile based skin lesions classification using convolution neural network. Ann. Emerg. Technol. Comput. 4, 12 (2020). 17. Popescu, D., El-Khatib, M., El-Khatib, H. & Ichim, L. New trends in melanoma detection using neural networks: a systematic review. Sensors 22, 496 (2022). 18. Jones, O. T. et al. Artificial intelligence and machine learning algorithms for early detection of skin cancer in community and primary care settings: a systematic review. Lancet Digit. Health 4, 466–476 (2022). 19. Liu, Y. et al. A deep learning system for differential diagnosis of skin diseases. Nat. Med. 26, 900–908 (2020). 20. Han, S. S. et al. Augmented intelligence dermatology: deep neural networks empower medical professionals in diagnosing skin cancer and predicting treatment options for 134 skin disorders. J. Invest. Dermatol. 140, 1753–1761 (2020). 21. European Centre for Disease Prevention and Control/WHO Regional Office for Europe. Monkeypox, joint epidemiological overview. https://cdn.who.int/media/docs/librariesprovider2/ monkeypox/monkeypox_euro_ecdc_final_jointreport_2022-07-13. pdf (2022). 22. World Health Organization. Monkeypox. https://www.who.int/ news-room/fact-sheets/detail/monkeypox (2022). Nature Medicine Article https://doi.org/10.1038/s41591-023-02225-7 Extended Data Fig. 3 | True Positive Rates by duration of presence of the MPXV skin lesion in the testing cohort. n, Number of available images per group. Nature Medicine Article https://doi.org/10.1038/s41591-023-02225-7 Extended Data Fig. 4 | True Positive Rates by number of visible MPXV skin lesions N in the testing cohort. n, Number of available images per group; N, Number of visible MPXV skin lesions in the image. Nature Medicine Article https://doi.org/10.1038/s41591-023-02225-7 Extended Data Fig. 5 | Specificity for classifying MPXV skin lesions versus acute and chronic non-MPXV skin diseases. n, Number of available images per group. Nature Medicine Article https://doi.org/10.1038/s41591-023-02225-7 Extended Data Fig. 6 | Top 30 False Positive Rates of acute diagnoses in the testing cohort with at least 50 available images. The full list of diagnoses and False Positive Rates can be found in Supplementary Tables 1–11. n, Number of available images per diagnosis. Nature Medicine Article https://doi.org/10.1038/s41591-023-02225-7 Extended Data Fig. 7 | Top 30 False Positive Rates of chronic diagnoses in the testing cohort with at least 50 available images. The full list of diagnoses and False Positive Rates can be found in Supplementary Tables 1–11. n, Number of available images per diagnosis. Nature Medicine Article https://doi.org/10.1038/s41591-023-02225-7 Extended Data Fig. 8 | True Positive Rates by body region in the testing cohort. n, Number of available images per body region. Nature Medicine Article https://doi.org/10.1038/s41591-023-02225-7 Extended Data Fig. 9 | True Positive Rates by skin tone (Fitzpatrick Type) in the testing cohort. n, Number of available images per group. Nature Medicine Article https://doi.org/10.1038/s41591-023-02225-7 Extended Data Fig. 10 | False Positive Rates by skin tone (Fitzpatrick Type) of non-MPXV images of the Fitzpatrick 17k dataset. The highest False Positive Rates could be observed in skin tone Fitzpatrick Types V and VI. n, Number of available images per group.