Breast cancer surgery with augmented reality

B eas cance su ge y wi h augmen ed eali y Ped o F. Gou eia a , b , * , Joana Cos a a , Ped o Mo gado c , Ronald Ka es a , Da id Pin o a , Ca los Ma ioso a ,Jo ~ ao Anacle o a , Ma a Ma inho a , Daniel Sim~ oes Lopes d , A lindo R. Fe ei a a , b , Vasileios Va ou akis e , , My ian hi Hadjicha alambous e , Ma co A. Sil a g , Nickolas Papanikolaou a , Celes e Al es a , Fa ima Ca doso a , Ma ia Jo~ ao Ca doso a , h a B eas Uni , Champalimaud Clinical Cen e/Champalimaud Founda ion,A enida B asilia, 1400-038, Lisboa, Po ugal b Facul y o Medicine, Lisbon Uni e si y,A enida P o esso Egas Moniz, 1649-028, Lisboa, Po ugal c AI4medimaging,Rua do Pa que Poen e, Lo e 35, 4705-002, B aga, Po ugal d INESC ID, Ins i u o Supe io T ecnico, Lisbon Uni e si y,Rua Al es Redol 9, 1000-029, Lisboa, Po ugal e Depa men o Mechanical &Manu ac u ing Enginee ing, Uni e si y o Cyp us,Dep . o Mechanical &Manu ac u ing Enginee ing Uni e si y o Cyp us, Cyp us Depa men o Medical Physics &Biomedical Enginee ing, Uni e si y College London,Male Place Enginee ing Building, Uni e si y College London, Gowe S ee , London, WC1E 6BT, Uni ed Kingdom g Mic oso Co po a ion (Po ugal),Rua do Fogo de San elmo, Lo e 2.07.02, Lisboa, Po ugal h NOVA Medical School, Campo dos M a i es da P a ia 130, 1169-056, Lisboa, Po ugal a icle in o A icle his o y: Recei ed 26 No embe 2020 Recei ed in e ised o m 18 Janua y 2021 Accep ed 20 Janua y 2021 A ailable online 27 Janua y 2021 Keywo ds: B eas cance B eas conse a i e su ge y 3D b eas model Augmen ed eali y abs ac In oduc ion: Inno a ions in 3D spa ial echnology and augmen ed eali y imaging d i en by digi al high- ech indus ial science ha e accele a ed expe imen al ad ances in b eas cance imaging and he de elopmen o medical p ocedu es aimed o educe in asi eness. P esen a ion o case: A 57-yea -old pos -menopausal woman p esen ed wi h sc een-de ec ed le -sided b eas cance . A e unde going all s aging and p e-ope a i e s udies he pa ien was p oposed o conse a i e b eas su ge y wi h umo localiza ion. Du ing su ge y, an expe imen al digi al and non-in asi e in a-ope a i e localiza ion me hod wi h augmen ed eali y was compa ed wi h he s anda d p e-ope a i e localiza ion wi h ca bon a ooing (ins i u ional p o ocol). The b eas su geon wea ing an augmen ed eali y headse (Hololens) was able o isualize he umo loca ion p ojec ion inside he pa ien ’s le b eas in he usual supine posi ion. Discussion: This wo k desc ibes, o ou knowledge, he fi s expe imen al es wi h a digi al non-in asi e me hod o in a-ope a i e b eas cance localiza ion using augmen ed eali y o guide b eas conse a i e su ge y. In his case, a success ul o e lap o he p e ious s anda d p e-ope a i e ma ks wi h ca bon a ooing and umo isualiza ion inside he pa ien ’s b eas wi h augmen ed e- ali y was ob ained. Conclusion: B eas cance conse a i e guided su ge y wi h augmen ed eali y can pa e he way o a digi al non-in asi e me hod o in a-ope a i e umo localiza ion. ©2021 The Au ho (s). Published by Else ie L d. This is an open access a icle unde he CC BY-NC-ND license (h p://c ea i ecommons.o g/licenses/by-nc-nd/4.0/). 1. In oduc ion B eas conse a i e su ge y (BCS) combined wi h adio he apy has become he ea men o choice o he majo i y o women p esen ing wi h ea ly b eas cance [1]. The aim o BCS is o achie e op imal long- e m local con ol, wi h ee ma gins a e umo *Co esponding au ho . B eas Uni , Champalimaud Clinical Cen e/Champalimaud Founda ion, A enida B asília, 1400-038, Lisboa, Po ugal. E-mail add esses: [email p o ec ed] (P.F. Gou eia), [email p o ec ed] (J. Cos a), [email p o ec ed] (P. Mo gado), [email p o ec ed] (R. Ka es), da id.pin o@ undacaochampalimaud.p (D. Pin o), ca los.ma[email p o ec ed] (C. Ma ioso), joao.anacle o@ undacaochampalimaud.p (J. Anacle o), [email p o ec ed] (M. Ma inho), [email p o ec ed] (D.S. Lopes), a lindo. e ei a@ undacaochampalimaud.p (A.R. Fe ei a), a ou akis. [email p o ec ed] (V. Va ou akis), hadjicha alambous.my ian [email protected] (M. Hadjicha alambous), madasi@ mic oso .com (M.A. Sil a), nickolas.papanikolaou@ esea ch. champalimaud.o g (N. Papanikolaou), [email p o ec ed] (C. Al es), a imaca doso@ undacaochampalimaud.p (F. Ca doso), ma iajoao.ca doso@ undacaochampalimaud.p (M.J. Ca doso). Con en s lis s a ailable a ScienceDi ec The B eas jou nal homepage: www.else ie .com/b s h ps://doi.o g/10.1016/j.b eas .2021.01.004 0960-9776/©2021 The Au ho (s). Published by Else ie L d. This is an open access a icle unde he CC BY-NC-ND license (h p://c ea i ecommons.o g/licenses/by-nc-nd/4.0/ ). The B eas 56 (2021) 14e17 excision and a good cosme ic ou come. Due o widesp ead sc eening an impo an pe cen age o ea ly b eas cance s a e non- palpable. As a consequence, umo localiza ion is needed o assis he su geon du ing su ge y. Cu en modali ies o p e-ope a i e localiza ion o b eas cance lesions a e all in asi e p ocedu es equi ing imaging guidance: wi e-guided, ca bon a ooing, o mo e ecen ly, adioac i e seed localiza ion, adio-occul lesion localiza ion and magne ic seeds [2]. Inno a ions in 3D spa ial echnology and augmen ed eali y (AR) d i en by digi al high- ech indus ial science ha e accele a ed explo a o y esea ch in b eas cance imaging. Al hough ex ensi e li e a u e exis s on 3D fini e elemen s simula ion and b eas issue modeling, ew pape s ha e add essed b eas magne ic esonance imaging (MRI) o 3D scan usion o simula e a eal b eas [3e8]. Recen ly, a pa ien -specific 3D digi al b eas model in eg a ing he b eas o so and umo loca ion was c ea ed and alida ed wi h a MRI o 3D su ace scan usion algo i hm [9]. The spa ial compu ing applied o a b eas cance pa ien , me ging digi al and physical ana omic s uc u es o he b eas wi h umo included in a digi al 3D b eas model, can be isualized h ough AR in he ope a ing hea e . The objec i e o his wo k is o epo a clinical case o a b eas cance conse a i e guided su ge y wi h AR. 2. P esen a ion o case 2.1. Case his o y A 57-yea -old pos -menopausal woman p esen ed wi h a sc een-de ec ed le sided b eas cance . In e io pe ia eola mic ocalcifica ions we e p esen wi h an es ima ed mammog aphy ex ension o 17 mm associa ed o a 15 mm dys ophic a ea on ul- asound (US). A 29 mm b eas MRI nodula lesion was obse ed. A guided acuum-assis ed biopsy wi h clip placemen was pe o med showing an in asi e b eas ca cinoma o non-special ype (NST), g ade 2, ER and PR o 100%, HER2 nega i e wi h a Ki67 o 20% (1 agmen in 10), associa ed wi h in e media e g ade duc al ca ci- noma in si u (DCIS). Bila e al physical examina ion was nega i e o palpable lesions and axilla y lymph nodes. 2.2. Expe imen al p o ocol desc ip ion o p oduce a pa ien -specific 3D digi al b eas model (phan om model) Th ee b eas su ace ma ke s (BSM) we e applied on he pa- ien ’s b eas s (in a-mamma y sulcus) along wi h a 10 cm fiducial ma ke below he xiphoid p ocess (Fig. 1). These e e ence poin s we e anno a ed wi h a black pe manen ma ke a he pa ien ’s skin. A 3D su ace scan o he pa ien was pe o med in he supine posi ion wi h a ms a 90  deg ees (Fig. 1), cap u ing he size and shape o bo h b eas s and o so using a Go!Scan 3D handheld by C ea o mTM. A e 3D su ace scan, fish-oil capsules we e fixed on he h ee BSM ma ks o MRI acquisi ion. A con as -enhanced MRI was pe o med wi h he pa ien in p one posi ion and ma ke s in place. The ins i u ional b eas MRI acquisi ion p o ocol was modi- fied o include a final 5-min Dixon sequence wi h he phased a ay o so MRI coil and he pa ien in he supine posi ion wi h a ms along he body (Fig. 2). Anno a ion, segmen a ion and olume compu a ion o he MRI issue po ions (wi h Dixon sequence in he supine posi ion) we e pe o med and alida ed by wo adiologis s using Ho os R so - wa e 2.4.0 (b eas con ou , b eas issue including malignan u- mo and la issimus do si muscle an e io bo de ). B eas su ace ma ke s we e manually anno a ed on he 3D su ace scan (black pe manen ma ke ) and MRI da a (fish-oil capsules). B eas MRI/3D scan usion was accomplished wi h simple o e lap be ween bo h modali ies, including umo , using BSM o pe o m spa ial alignmen be ween bo h modali ies (Fig. 3). A pa ien -specific 3D digi al b eas model was p ocessed and con- e ed o a poin cloud b eas 3D model, e e ed he ea e as phan om model (Fig. 5), eady o be uploaded o an AR headse : Hololens 1. 2.3. Su ge y wi h AR headse S anda d p e-ope a i e localiza ion wi h ca bon a ooing unde ul asound guidance was pe o med a ge ing he nodula lesion and he clip (mic ocalcifica ions). P eope a i e ma kings we e d awn on he pa ien skin in he s anding posi ion. Du ing su ge y, an expe imen al digi al and non-in asi e in a-ope a i e localiza- ion me hod wi h AR was compa ed wi h he s anda d p e- ope a i e localiza ion wi h ca bon a ooing. The b eas su geon wea ing an AR headse (Fig. 4), synch onized he pa ien -specific 3D digi al b eas model ( iewed h ough he Hololens) wi h he eal pa ien lying down in he ope a ing hea e bed, p ojec ing he umo loca ion inside he le b eas (Fig. 5). A b eas conse a i e su ge y wi h sen inel lymph node biopsy h ough an in e io pe i-a eola incision was pe o med, ollowed by de ec epai wi h local glandula flaps. Fig. 1. 3D su ace scan o he pa ien in he supine posi ion wi h a fiducial ma ke (whi e a ow) as a acking e e ence o he Hololens sys em. Fig. 2. Le b eas cance lesion wi h clip a i ac isible a MRI wi h he pa ien in he supine posi ion. P.F. Gou eia, J. Cos a, P. Mo gado e al. The B eas 56 (2021) 14e17 15 Pa hology: 17 mm umo was emo ed consis ing o in e me- dia e g ade DCIS wi h ee ma gins and one nega i e sen inel lymph node. No in asi e cance was ound. Final TNM s aging was pTis pN0(sn)cM0. 3. Discussion Su geons need o be able o co ela e 2D adiological images wi h he eal loca ion o he umo wi hin he pa ien s b eas o su gical planning, bo h p e and pe -ope a i ely. Despi e all p e- ope a i e and in a-ope a i e localiza ion echniques, accu acy is s ill no pe ec , and he p ocess equi es addi ional exams and in asi e p ocedu es in he p e-ope a i e se ing o expe ise in US echniques in case o an in a-ope a i e se ing. The diagnos ic b eas cance wo kflow includes di e en im- aging modali ies acqui ed in di e en posi ions and wi h di e en b eas comp essions and consequen de o ma ions. As b eas can- ce s a e diagnosed mo e equen ly a an ea ly s age and mos o hem a e non-palpable, localiza ion echniques a e equen ly needed: wi e-guided, ca bon a ooing, adioac i e seed localiza- ion, adio-occul lesion localiza ion and magne ic seeds a e conside ed as s anda d. Howe e , among hese echniques, none has p o en supe io i y in e ms o educing posi i e ma gins [2]. Wo ldwide, e-excision a es ollowing b eas cance conse a i e su ge y exhibi wide a iabili y. Re-ope a ion a es ange om 0% o 70% (by indi idual su geons) in he Uni ed S a es [10], om 17% o 56% in Canada [11], and om 12 o 30% [12] in England. Se e al s a egies ha e been p oposed o imp o e b eas cance su gical ou comes. P eope a i e con as -enhanced b eas MRI has eme ged as a solu ion, bu nei he he COMICE [13] no he MONET [14] ial ound an associa ed benefi . Addi ional s udies a e needed o u he exploi addi ional MRI in o ma ion and echniques o be applied in he su gical hea e [15]. In aope a i e US (as an addi- ional ool o s anda d localiza ion echniques) has been used success ully o some ex en o educe posi i e ma gins, he e o e educing e-ope a ion a es [16,17]. Howe e , he use o in a- ope a i e US equi es a le el o expe ise beyond he s anda d o b eas su geons and could also be ime-consuming wi hin he su gical hea e . These conside a ions s ongly sugges he need o de elopmen o a no el me hodology o enhance he su geon’s isualiza ion o umo localiza ion and consequen ly o imp o e su gical clinical ou comes. This wo k desc ibes a success ul expe imen al es wi h a digi al non-in asi e me hod o in a-ope a i e b eas cance localiza ion. A pa ien -specific 3D digi al b eas model was p ocessed and iewed h ough AR o guide a b eas cance conse a i e su ge y. In his case an o e lap o he p e ious s anda d p e-ope a i e ma ks wi h ca bon a ooing and umo isualiza ion inside he pa ien ’s b eas wi h AR was ob ained (Fig. 5). Pe kins e al. [18] p oposed a simila app oach, bu wi h a di e en me hodology o su gical planning and no sui able o li e su ge y: hey p ojec ed a b eas MRI 3D holog am egis e ed wi h he pa ien in he supine posi ion and applied i wi h six fiducial ma ke s. Ou me hodology inco po a es a b eas MRI o 3D su ace scan usion in o de o p oduce a comple e pa ien -specific 3D digi al b eas model (phan om model) wi h only one fiducial ma ke and h ee in a-mamma y b eas su ace ma ke s, used du ing bo h imaging modali ies egis a ion o op imize image synch oniza ion. The diagnos ic b eas MRI in p one posi ion is pe o med wi h a final 5-min Dixon image sequence in he supine posi ion. The ins i u ional p o ocol needed o be adap ed wi h pa- ien eposi ioning om p one o supine du ing he MRI egis a- ion. B eas MRI o 3D su ace scan usion algo i hms wi h pose ans o ma ion simula ion om p one o supine [9] can po en ially le e age his echnology wi hou he need o change diagnos ic Fig. 3. Pa ien specific 3D digi al b eas model a e b eas MRI o 3D su ace scan image usion wi h umo highligh ed ed on he le b eas . Fig. 4. In ac ion: su geon wea ing Hololens headse a he su gical hea e . Fig. 5. - Su geon iew o he phan om model (wi h umo loca ion) synch onized wi h he pa ien h ough augmen ed eali y. An o e lap is obse ed be ween he wo ca bon a ooing ma ks (whi e a ows) and umo p ojec ion wi h augmen ed eali y in he ed colou . P.F. Gou eia, J. Cos a, P. Mo gado e al. The B eas 56 (2021) 14e17 16 ins i u ional b eas MRI egis a ion p o ocols. A wo s eps p o- spec i e clinical ial (p e-clinical plus clinical s eps) was designed and will s a ec ui ing du ing 2021 o add ess his limi a ion. To imp o e AR use expe ience applied o b eas cance con- se a i e su ge y, 3D space came a ancho s ins alled in he oo o he ope a ing hea e could be used o pe o m eal- ime 3D mapping o he pa ien in he ope a ing hea e , eplacing handheld 3D su ace scan de ices. 3.1. Conclusion B eas cance conse a i e guided su ge y wi h an AR headse , using 3D digi al b eas models, could pa e he way o a digi al, non-in asi e me hod o in a-ope a i e umo localiza ion. The p oposed solu ion has he po en ial o imp o e he su geon’s isualiza ion o he umo while imp o ing pa ien ’s quali y o li e (no pain, no anxie y, no bleeding). Funding This esea ch did no ecei e any specific g an om unding agencies in he public, comme cial, o no - o -p ofi sec o s. Fa ima Ca doso epo s pe sonal financial in e es in o m o consul ancy ole o : Amgen, As ellas/Medi a ion, As aZeneca, Celgene, Daiichi-Sankyo, Eisai, GE Oncology, Genen ech, Glax- oSmi hKline, Mac ogenics, Medscape, Me ck-Sha p, Me us BV, Mylan, Mundipha ma, No a is, Pfize , Pie e-Fab e, p IME Oncology, Roche, Samsung Bioepis, Sanofi, Seagen, Te a. Decla a ion o compe ing in e es No conflic o in e es and financial ela ionships ele an o he con en o his a icle ha e been disclosed by any o he o he au ho s. Re e ences [1] an Dongen JA, Voogd AC, Fen iman IS, Leg and C, Syl es e RJ, Tong D, e al. Long- e m esul s o a andomized ial compa ing b eas -conse ing he apy wi h mas ec omy: Eu opean O ganiza ion o Resea ch and T ea men o Cance 10801 ial. J Na l Cance Ins 2000;92:1143e50. [2] Co si F, So en ino L, Bossi D, Sa ani A, Foschi D. P eope a i e localiza ion and su gical ma gins in conse a i e b eas su ge y. In J Su g Oncol 2013;2013: 793819. [3] Lee A, Schnabel J, Rajagopal V, Nielsen P, Nash M. B eas image egis a ion by combining fini e elemen s and ee- o m De o ma ions2010. [4] Reece GP, Me chan F, Andon J, Kha am H, Ra i-Chanda K, Wes on J, e al. 3D su ace imaging o he human emale o so in up igh o supine posi ions. Med Eng Phys 2015;37:375e83. [5] Bjoe n E, Vasileios V, John HH, S en K, C is ian L, Thomas B, e al. Su ace d i en biomechanical b eas image egis a ion. P ocSPIE2016. [6] Va ou akis V, Eiben B, Hipwell JH, Williams NR, Kesh ga M, Hawkes DJ. Mul iscale mechano-biological fini e elemen modelling o oncoplas ic b eas su ge y-nume ical s udy owa ds su gical planning and cosme ic ou come p edic ion. PloS One 2016;11:e0159766. [7] Salmon R, Nguyen T, Moo e L, Bass B, Ga bey M. Mul imodal imaging o he b eas o e ie e he e e ence s a e in he absence o g a i y using fini e elemen Modeling2017. [8] Du aes M, C oche P, Pages E, G auby E, Lasch L, Rebel L, e al. Su ge y o nonpalpable b eas cance : fi s s ep o a i ual pe -ope a i e localiza ion? Fi s s ep o i ual b eas cance localiza ion. B eas J 2019;25:874e9. [9] Bessa S, Gou eia PF, Ca alho PH, Rod igues C, Sil a NL, Ca doso F, e al. 3D digi al b eas cance models wi h mul imodal usion algo i hms. B eas 2020;49:281e90. [10] McCahill LE, Single RM, Aiello Bowles EJ, Feigelson HS, James TA, Ba ney T, e al. Va iabili y in eexcision ollowing b eas conse a ion su ge y. J Am Med Assoc 2012;307:467e75. [11] Po e G, Waga B, B yan H, Hewi M, Wai E, Dabbs K, e al. Ra es o b eas cance su ge y in Canada om 2007/08 o 2009/10: e ospec i e coho s udy. CMAJ Open 2014;2:E102e8. [12] Jee an R, C omwell DA, T i ella M, Law ence G, Kea ins O, Pe ei a J, e al. Reope a ion a es a e b eas conse ing su ge y o b eas cance among women in England: e ospec i e s udy o hospi al episode s a is ics. BMJ 2012;345:e4505. [13] Tu nbull L, B own S, Ha ey I, Oli ie C, D ew P, Napp V, e al. Compa a i e e ec i eness o MRI in b eas cance (COMICE) ial: a andomised con olled ial. Lance 2010;375:563e71. [14] Pe e s NH, an Esse S, an den Bosch MA, S o m RK, Plaisie PW, an Dalen T, e al. P eope a i e MRI and su gical managemen in pa ien s wi h non- palpable b eas cance : he MONET - andomised con olled ial. Eu J Canc 2011;47:879e86. [15] Sa danelli F, T imboli RM, Houssami N, Gilbe FJ, Helbich TH, Al a ez Beni o M, e al. Sol ing he p eope a i e b eas MRI conund um: design and p o ocol o he MIPA s udy. Eu Radiol 2020. [16] K ekel NM, Haloua MH, Lopes Ca dozo AM, de Wi RH, Bosch AM, de Wid - Le e LM, e al. In aope a i e ul asound guidance o palpable b eas cance excision (COBALT ial): a mul icen e, andomised con olled ial. Lance Oncol 2013;14:48e54. [17] Spick C, Bal ze PA. Diagnos ic u ili y o second-look US o b eas lesions iden ified a MR imaging: sys ema ic e iew and me a-analysis. Radiology 2014;273:401e9. [18] Pe kins S, Lin M, S ini asan S, Wheele A, Ha g ea es B, Daniel B. A mixed- eali y sys em o b eas su gical Planning2017. P.F. Gou eia, J. Cos a, P. Mo gado e al. The B eas 56 (2021) 14e17 17