Continuous design control for machine learning in certified medical systems

This is a sel -a chi ed e sion o an o iginal a icle. This e sion may di e om he o iginal in pagina ion and ypog aphic de ails. Au ho (s): Ti le: Yea : Ve sion: Copy igh : Righ s: Righ s u l: Please ci e he o iginal e sion: CC BY 4.0 h ps://c ea i ecommons.o g/licenses/by/4.0/ Con inuous design con ol o machine lea ning in ce i ied medical sys ems © The Au ho (s) 2022 Published e sion S i bu, Vlad; G anlund, Tuomas; Mikkonen, Tommi S i bu, V., G anlund, T., & Mikkonen, T. (2023). Con inuous design con ol o machine lea ning in ce i ied medical sys ems. So wa e Quali y Jou nal, 31, 307-333. h ps://doi.o g/10.1007/s11219-022-09601-5 2023 Vol.:(0123456789) So wa e Quali y Jou nal h ps://doi.o g/10.1007/s11219-022-09601-5 1 3 Con inuous design con ol o machine lea ning ince i ied medical sys ems VladS i bu1,5 · TuomasG anlund2,3· TommiMikkonen4,5 Accep ed: 1 Sep embe 2022 © The Au ho (s) 2022 Abs ac Con inuous so wa e enginee ing has become commonplace in nume ous ields. Howe e , in egula ing in ensi e sec o s, whe e addi ional conce ns need o be aken in o accoun , i is o en conside ed di icul o apply con inuous de elopmen app oaches, such as de ops. In his pape , we p esen an app oach o using pull eques s as design con ols, and apply his app oach o machine lea ning in ce i ied medical sys ems le e aging model ca ds, a no el echnique de eloped o add explainabili y o machine lea ning sys ems, as a egula- o y audi ail. The app oach is demons a ed wi h an indus ial sys em ha we ha e used p e iously o show how medical sys ems can be de eloped in a con inuous ashion. Keywo ds Machine lea ning· ML· MLOps· CD4ML· Design con ol· Medical so wa e· Regula ed so wa e· Con inuous enginee ing 1 In oduc ion Du ing he la es decade, he Web has silen ly become he dominan pla o m o so wa e applica ions. E ec i ely, his p ocess has made eleasing so wa e so simple and cheap ha o a deg ee, de elopmen and deploymen ac i i ies a e en angled. New pa s o so - wa e a e expe imen ally deployed, and eedback om eleased so wa e is used o assis in Tuomas G anlund and Tommi Mikkonen ha e con ibu ed equally o his wo k * Vlad S i bu [email p o ec ed] Tuomas G anlund [email p o ec ed] Tommi Mikkonen ommi.j.mikk[email p o ec ed]; [email p o ec ed] 1 CompliancePal, Tampe e, Finland 2 Soli a, Tampe e, Finland 3 Tampe e Uni e si y, Tampe e, Finland 4 Uni e si y o Jy äskylä, Jy äskylä, Finland 5 Uni e si y o Helsinki, Helsinki, Finland So wa e Quali y Jou nal 1 3 de elopmen . As poin ed ou in Tai alsaa i e al. (2008), his leads o a new ype o de el- opmen app oach, ad ancing in e olu iona y ashion, whe e so wa e is always on, and upda es a e iny changes in he code. Conc e e models o such con inuous so wa e engi- nee ing (Fi zge ald & S ol,2017) include con inuous deli e y (Humble & Fa ley,2010) and De Ops (Ebe e al.,2016). Howe e , no all so wa e li es on he Web, whe e applica ions can cons an ly e ol e behind he cu ains. Ins ead, nume ous applica ions a e used o powe ac o ies, con ol elec onics, p o ide guidance o d ones, and so on. Fo hese, i is common ha addi- ional conce ns a e added in he de elopmen p ocess. These can be added o he con inu- ous de elopmen p ocess as add-ons o amalgama ions, some imes also e lec ed in hei espec i e names, such as De SecOps (My bakken & Colomo-Palacios,2017) o De Ops used o de elop secu e sys ems, RegOps (D a e al.,2020) o digi alizing he egula o y alue chain, o MLOps (T e eil e al.,2020) o con inuous deli e y o Machine Lea ning (ML) ea u es. Un o una ely, hese app oaches ocus on one pa icula aspec ha is added o he con- inuous so wa e enginee ing pipeline, and do no conside how o in eg a e hem o a big- ge whole. Hence, hei in e ope abili y emains weak. Consequen ly, ela ing egula o y compliance and ML, o ins ance, equi es addi ional conside a ions which a e no a pa o any o - he-shel app oach. As an example, conside he MLOps pipeline isualized in Fig. 1, consis ing o da a ope a ions, execu ed by da a enginee s, da a analysis and ML ope a ions, un by da a scien is s, and de elope s who implemen and deploy he inal applica ion. In con as , RegOps ocuses only on so wa e de elopmen — he inal pa , pe o med by so wa e de elope s — bu o e looks he es (Toi akka e al.,2021; S i bu & Mikkonen,2018). Hence, while MLOps helps in o ming a pipeline o he whole de el- opmen e o , RegOps only suppo s he inal pa s wi h egula o y conside a ions and design con ols. In his pape , we in oduce con inuous design con ols o ML in ce i ied medical sys- ems, co e ing he MLOps pipeline. The p oposed app oach s a s wi h con inuous so - wa e enginee ing p ac ices, which is hen expanded wi h ML and da a p ocessing acili ies. Then, we in oduce he necessa y egula o y p ocesses, which co e bo h so wa e and ML pa s o he de elopmen . To simpli y p esen a ion, de ails o da a ope a ions, which in any case a e o en speci ic o ce ain o ganiza ion (Aho e al.,2020), a e la gely o e looked, al hough p ac ical echniques ha bind hem o con inuous so wa e enginee ing p ac ices aining code Buildlabeled da a model Deploy app code da a enginee s da a scien is s de elope s and ops Fig. 1 Simpli ied MLOps pipeline. Figu e adap ed om G anlund e al. (2021) So wa e Quali y Jou nal 1 3 a e included in he pape . The esul ing model is hen demons a ed wi h an indus ial case s udy ha we ha e used in ou p e ious pape (G anlund e al.,2021), wi h an ex ended discussion ega ding he p oposed imp o emen s. The es o his pape is s uc u ed as ollows. In Sec .2, we in oduce he necessa y backg ound o he pape . In Sec .3, we p opose a solu ion o con inuous design con ols. In Sec .4, we demons a e he solu ion wi h an indus y example. In Sec .5, we discuss he implica ions o he p oposed solu ion. In Sec .6, we d aw some inal conclusions. 2 Backg ound Because o mul i- ace ed na u e o his wo k, i combines se e al di e en esea ch ields, including con inuous so wa e de elopmen , ML, and he landscape o medical egula ions. In he ollowing, we p esen ecen ad ances in hese ields, so ha we can in oduce he p oposed p ac ical pipeline o egula ed MLOps. 2.1 Con inuous so wa e enginee ing p ac ices The co e o con inuous so wa e enginee ing p ac ices is wo old. On one hand, i con- sis s o a mindse whe e he de elope s ake esponsibili y o he whole so wa e as a whole, and, while a single de elope wo ks on a pa icula ea u e, he bigge whole is cons uc ed in e ms o de elope s’ collabo a i e e o . On he o he hand, i includes a oolse ha allows deploying new ea u es o use as soon as hey a e a ailable (Fi zge ald & S ol,2017). The goal is o p oduce con inuous low o alue adding so wa e a i ac s om he de elopmen o he ac ual p oduc ion use, wi h quali y assu ance also happening con inuously as a pa o he low. A pa icula la o o con inuous so wa e enginee ing is called De Ops (Debois,2011; Rajkuma e al.,2016). I can be desc ibed as a se o p ac ices whose goal is o sho en he commi eedback cycle wi hou comp omising quali y (Bass e al.,2015), and o expand he de elopmen eam wi h he ope a o s. In addi ion, he con inuous so wa e enginee - ing oolse is expanded wi h moni o ing capabili ies, ensu ing ha each s akeholde ge s a imely access o wha hey need. Finally, i is impo an o no ice ha he au oma ed pipeline is no abou so wa e going in o p oduc ion wi hou any ope a o supe ision, bu a he he pipeline p o ides a eed- back loop o each o he s akeholde s om all s ages o he deli e y p ocess. Mo eo e , as he so wa e p og esses h ough he pipeline, di e en s ages can be igge ed o example by ope a ions and es eams by he click o a bu on. 2.2 ML li e cycle challenges The ypical p ocess o de eloping a ML applica ion s a s wi h ans o ming inpu da a om a a ie y o sou ces in o a collec ion o labeled da a, a p ocedu e pe o med by da a enginee s. This collec ion is hen used by da a scien is s o pe o m a se ies o expe imen s ha allows hen o build a se o candida e models. The model ha ul ills bes he desi ed c i e ia cap u ed in he unc ional and non- unc ional equi emen s is selec ed o deploy- men . Finally, he selec ed model is inco po a ed in o he so wa e sys em and deployed o he p oduc ion en i onmen , a p ocedu e ha is pe o med by so wa e and ope a ions enginee s. So wa e Quali y Jou nal 1 3 Using con inuous so wa e enginee ing p ac ices h oughou ML applica ion li e cycle is no s aigh o wa d. In Fig.1, we can see ha he de elopmen is pe o med ac oss h ee compe ence clus e s: da a enginee s, da a scien is and so wa e de elope s. The le el o expe ise and skills ela ed o con inuous enginee ing p ac ices a ies among hese special- ies: while so wa e de elope s use ooling o achie e a high deg ee o au oma ion in hei daily wo k, da a scien is and, o a lesse ex en , da a enginee s ha e a less s uc u ed way o wo king. Many da a scien is s a e no awa e o ools like e sion con ol sys ems ha can ack e icien ly changes in he aining code and he da a used o pe o m he expe i- men s, o icke ing sys ems ha enable hem o ack p og ess om he ea u e implemen a- ion o equi emen s. Ins ead, hey ely on bespoke solu ions ha migh no be app op ia e when p ac iced in he de elopmen p ocess o sa e y c i ical sys ems. To inco po a e ML ea u es in so wa e de elopmen , se e al echniques ha e been p oposed. In his wo k, we build on Con inuous Deli e y o ML (Sa o e al.,2019) and ML Model Ca ds (Mi chell e  al.,2019). These echniques a e b ie ly in oduced in he ollowing. 2.2.1 Con inuous deli e y o ML P obably he bes -known MLOps implemen a ion, Con inuous Deli e y o Machine Lea ning (CD4ML) (Sa o e  al., 2019) by Though Wo ks, aims a au oma ing he ML applica ion li e cycle in an end- o-end ashion (Fig.2). In CD4ML, a c oss- unc ional eam p oduces applica ions based on code, da a, and models in small and sa e inc emen s ha can be ep oduced and eliably eleased a any ime, in sho adap a ion cycles. Th ee dis- inc s eps a e included: (i) iden i y sui able da a sou ces and p epa e he da a o aining, (ii) expe imen wi h di e en models o ind he bes pe o ming candida e, and (iii) deploy and use he selec ed model in p oduc ion as a pa o a bigge so wa e sys em. egula o y "locked" bounda y Building E alua ion and Expe imen a ion Packaging Tes ing Deploymen Moni o ing aining da a candida e model selec ed model packaged model packaged model aining code applica ion code alida ion da a es code me ics applica ion bundle machine lea ning pipeline deploymen pipeline da a scien is sde elope s and opsda a enginee s da a sou ce da a labeling pipeline es da a Fig. 2 CD4ML pipelines and a i ac s. Figu e adap ed om G anlund e al. (2021) So wa e Quali y Jou nal 1 3 CD4ML is no he only solu ion ha aims o p e en he accumula ion o hidden ech- nical dep in machine lea ning applica ions (Sculley e al., 2015). While o he solu ions a e hea ily op imized o a pa icula cloud in as uc u e (AWS Solu ions,2021; Google Cloud Solu ions,2021), an ML so wa e s ack implemen a ion (Baylo e al.,2017), o con- cep ual cha ac e is ics o he app oach like MLOps (John e al.,2021), he CD4ML model has he ad an age ha i can be used as a e e ence model in any applica ion domain. Fu - he mo e, i has he necessa y phases documen ed a he app op ia e le el, and he imple- men a ion is based on open sou ce componen s. Hence, i se es as a solid ounda ion o expe imen a ion, and o ansla e he esul s o o he MLOps pipelines. 2.2.2 ML model ca ds ML model ca ds is a amewo k o anspa en communica ion o in o ma ion ha acili- a es he co ec u iliza ion o machine lea ning models (Mi chell e al.,2019). In ended as documen a ion ha accompanies a ained machine lea ning model, he model ca d con- ains in o ma ion abou in ended use case, benchma k e alua ion in a a ie y o ele an condi ions, such as demog aphics o geog aphic loca ion, o any o he in o ma ion ha he c ea o s conside sui able o he p ope use o he ained model. Al hough he o iginal p oposal p esen ed he model ca d as a isual ep esen a ion, ecen de elopmen s1 p opose a p og amma ic mechanism o gene a e he model ca ds, and a machine- eadable se ializa- ion2 ha acili a es he consump ion o model ca ds by sc ip s in ML pipelines. While he machine- eadable se ializa ion o he model ca d is a s ep in he good di ec- ion, he in o ma ion included in he ca d is limi ed o speci ic audiences, like da a scien- is s o machine lea ning enginee s. To ul ill he model ca ds po en ial, he in o ma ion should be ex ended o include he needs o o he s akeholde s. The model ca d me ada a becomes a model om which a ious iews a ge ed a di e en s akeholde s can be de i ed, in a simila ashion as a ious so wa e a chi ec u e iews can be de i ed om a single so wa e model. The app oach will u n he model ca d in o a mus ha e a i ac ha con eys no only he in o ma ion abou he model’s in ended use o pe o mance, bu also wha o he ac i i ies ha e been conduc ed in ela ion wi h he model (e.g., egula o y isk managemen ), o me ada a abou da a se s ha acili a e downs eam es ing (G anlund e al.,2021). 2.3 Medical egula o y landscape The manu ac u ing o medical de ices is s ic ly con olled by au ho i ies, and manu ac- u e s mus con o m o he egion’s egula o y equi emen s in which a medical de ice is being ma ke ed o use. Fo his eason, medical so wa e sys ems mus also be de eloped acco ding o he equi emen s o he a ge a ea. Fo example, he de elopmen is egula ed by Medical De ice Regula ion (MDR) (Eu opean Pa liamen and he Council,2017) and InVi o Diagnos ics Regula ion (IVDR) (Eu opean Pa liamen and he Council,2017) in he EU egion and by Fede al Food, D ug, and Cosme ic Ac (FD&C Ac ) (U.S. Depa - men o Heal h and Human Se ices,2021) in he USA. 1 h ps:// gi hub. com/ enso low/ model- ca d- oolk i 2 Tenso Flow model ca d schema So wa e Quali y Jou nal 1 3 2.3.1 Design con ol The egula o y amewo k aims o ensu e ha a medical de ice is sa e o use and clinically e ec i e o i s in ended medical pu pose. In p ac ice, he e a e ce ain manda o y p o- cesses ha include con ol mechanisms o he whole so wa e li e cycle, including design, de elopmen , and manu ac u ing o he p oduc . The egula o y equi emen s ela ed o hese p ocess phases a e gene ally e e ed o as he Design Con ols (FDA - Cen e o De ices and Radiological Heal h,1997). The pu pose o he Design Con ol p ocess (depic ed in Fig.3), is o p omo e a well- designed de elopmen p ocess ha includes aceabili y be ween p ocess inpu s and ou - pu s a di e en s ages o he p ocess. S a ing om use needs con e ed in o design inpu s, con inuing wi h he design p ocess ha ans o ms he inpu s in o design ou pu s, and inally o ming he esul ing medical de ice. In addi ion, he e iews du ing each s ep o he p ocess e i ies and alida es ha he equi emen s a e me by he implemen a ion, educing he possibili y o design and implemen a ion de ec s. In gene al, hese egula- o y bounda ies a e add essed in so wa e de elopmen wi h dedica ed medical so wa e li e cycle managemen ools, such as Pola ion3. Fo so wa e medical de ices he design con ol is implemen ed in wo laye s, depic ed in Fig.4: he p oduc and sys em de elopmen ac i i ies (IEC 82304 (In e na ional Elec- o echnical Commission, 2016)), and he so wa e de elopmen ac i i ies (IEC 62304 (In e na ional Elec o echnical Commission, 2015)). A he p oduc le el, he iden i- ied use needs a e con e ed o sys em equi emen s ha se e as design inpu s o he so wa e de elopmen p ocess. Du ing so wa e de elopmen , he sys em equi emen s a e ans o med in o high le el so wa e equi emen s ha co e he so wa e sys em and Re iew Use Needs Re iew Design Inpu Re iew Design P ocess Ve i ica ion Re iew Design Ou pu Valida ion Medical De ice Fig. 3 Design con ol p ocess o medical de ices. Figu e adap ed om FDA - Cen e o De ices and Radiological Heal h (1997) 3 h ps:// pola ion. plm. au om a ion. sieme ns. com/ p odu c s/ pola ion- alm So wa e Quali y Jou nal 1 3 a chi ec u al conce ns. La e on, he high le el so wa e equi emen s a e u he dis illed in o low le el so wa e equi emen s ha se e as design inpu o implemen a ion. The a chi ec u al design ac i i y de ines he majo s uc u al componen s o he so - wa e, known as so wa e i ems. I iden i ies hei key esponsibili ies, hei ex e nally isible p ope ies, and he ela ionship among hem. The esul ing so wa e a chi ec u e a i ac ensu es he co ec implemen a ion o he so wa e equi emen s, and is comple e when all so wa e equi emen s can be implemen ed by he iden i ied so wa e i ems. The a chi ec u al decisions a e ex emely impo an o implemen ing e ec i e isk con ol measu es. The p ope unde s anding and accu a e documen a ion o so wa e i ems beha - io a e essen ial o ensu ing ha he so wa e sys em is sa e. De ailed design ac i i ies e ine he iden i ied so wa e i ems du ing a chi ec u e design in o smalle so wa e i ems. When a so wa e i em is no decomposed u he i is called so wa e uni . In he end, he manu ac u e is esponsible o he g anula i y o he so wa e decomposi ion, and should ensu e ha he ac i i y pe o med o he app op ia e de ail o allow a sa e and e ec i e implemen a ion. The esul ing code, es cases and a ious o he a i ac s, such as a chi ec u e and de ailed module design documen a ion, c ea ed du ing he so wa e de elopmen ac i i- ies, se e as he design ou pu s. The e iew o he a i ac s and he es esul p o ide an e ec i e e i ica ion p ocedu e a uni , in eg a ion and sys em le el. The accep ance es s oge he wi h he esul epo s o clinical ials se e as he alida ion p ocedu e. All hese p ocedu es ensu e ha he p ope design con ols ha e been applied du ing de elopmen , esul ing in a medical p oduc ha mee s he use needs 2.3.2 Design con ol o ML: hemissing pa s Al hough he design con ol p ocess o medical so wa e is o he wise well-de ined, he egula o y amewo ks o he majo ma ke a eas, such as he EU and USA, do no cu en ly explici ly add ess he equi emen s ela ed o AI/ML echnologies. While AI/ML-based High le el SW equi emen s/ A chi ec u al design Low le el SW equi emen s/ De ailed design Uni es In eg a ion es Sys em es Accep ance es Sys em equi emen s Use equi emen s Use needs Use needs me Sys em de elopmen ac i i ies So wa e de elopmen ac i i ies Implemen a ion IEC 82304 IEC 62304 Ve i ica ion Ve i ica ion Ve i ica ion Valida ion Fig. 4 Sys em and so wa e de elopmen design con ol ac i i ies So wa e Quali y Jou nal 1 3 medical de ices ha e g ea po en ial o imp o e ca e o indi idual pa ien s, hey ca y some unique isks. Fo example, ce ain ML sys ems may be designed o lea n and op imize hei unc ionali y in eal ime. As a esul , he use - acing isk p o ile o hese sys ems may change o e ime which is an incompa ible idea compa ed o gene al design con ol p ac- ices. E iden ly, he cu en lack o p ecise equi emen s o AI/ML is a se e e sho coming in he legisla ion as i c ea es pa icula challenges and unce ain ies o he manu ac u e s on how o p o e AI/ML de ice sa e y and e iciency when seeking de ice app o als. In gene al, he AI/ML de ices mus comply wi h equi emen s applicable o all medi- cal de ice so wa e. The e o e, manu ac u e s a e equi ed o manage he isks ela ed o hei p oduc s h oughou he de ices’ whole li e cycle (Eu opean Pa liamen and he Council, 2017; G anlund e  al., 2021). In p ac ice, medical so wa e isk managemen ac i i ies a e implemen ed acco ding o he equi emen s o ISO 14971 (In e na ional O ganiza ion o S anda diza ion, 2019) and IEC 62304 (In e na ional Elec o echnical Commission, 2015). Howe e , he isk managemen ac i i ies desc ibed in de ail in he s anda ds men ioned assume ha he de ice’s unc ionali y emains he same as du ing he p oduc de elopmen phase, e en a e deploymen , and does no change o e ime. As a esul , i may be di icul o he AI/ML manu ac u e o p o e he e ec i eness o he implemen ed isk managemen ac i i ies agains he cu en equi emen s. Fu he mo e, speci ic AI/ML models can be complex and da a-in ensi e, so hey may be complica ed o unde s and ully. As a esul , i may no be easy o assess how he model has eached a decision. In addi ion, he quali y o da a plays a signi ican ole and may con- ain biases no isible o a human audi o . Also, speci ic challenges ela ed o model ain- ing, such as o e i ing and unde i ing, mus be conside ed when alida ing he sys em’s clinical e iciency and sa e y. The e a e cu en ly ce ain ongoing e o s o add ess he p oblem o missing AI/ML egula o y guidance. Fo example, in he EU, he heme “A i icial In elligence unde MDR/IVDR amewo k” will be add essed wi hin he o hcoming guidance documen by he Medical De ice Coo dina ion G oup (MDCG) (Medical De ice Coo dina ion G oup, 2021). Fu he mo e, in he USA, he US Food and D ug Adminis a ion (FDA) has eleased an ac ion plan documen “A i icial In elligence/Machine Lea ning (AI/ML)- Based So wa e as a Medical De ice (SaMD) Ac ion Plan” (U.S. Food and D ug Admin- is a ion (FDA),2021). While speci ic and binding egula o y equi emen s a e s ill unde de elopmen , he In e es G oup o he No i ied Bodies o Medical De ices in Ge many (IG-NB) has c ea ed pe haps he mos de ailed and conc e e guideline a ailable cu en ly, “Guideline o A i icial In elligence in Medical De ices” (de Benann en S ellen ü Medizinp oduk ein Deu schland(IG-NB),2021). The IG-NB guideline can be used as a basis o gain an unde s anding o he expec a ion le el o he no i ied bodies ela ed o AI/ ML p oduc s. 3 P oposed solu ion The p oposed design con ol p ocess o CD4ML pipelines aims a o malizing so wa e de elopmen so ha i be used o QMS pu poses. This is achie ed by using pull eques s as basis o e iews ha o ms he design con ol, and using model ca ds me ada a as he design ou pu a i ac ha se es also as an audi ail o egula o y ac i i ies such as clini- cal alida ion and isk managemen . So wa e Quali y Jou nal 1 3 been implemen ed in line wi h he equi emen s o IEC 62304, which go e ns he de el- opmen o so wa e used in medical p oduc s. In p ac ical e ms, he pull eques cons i u es he design con ol mechanism ha ensu e ha he e olu ion o he sys em is sys ema ically e iewed and ha he code baseline is always up o da e om a egula o y pe spec i e. Besides aligning he de elopmen and he egula o y ac i i ies, we iden i ied he model ca d me ada a as an ideal candida e o documen ing no only he model bu also he egula o y speci ic ac i i ies pe o med du ing model’s de elopmen , such as da ase jus i ica ion o clinical pe o mance e alu- a ion, among o he p e-ma ke isk managemen ac i i ies. The model ca d me ada a docu- men , oge he wi h he model code, se es e ec i ely as he design ou pu a i ac . Being machine- eadable, he model ca d me ada a can be used in pipelines o gene a e au oma i- cally addi ional documen s in ended o end use s (e.g., he model ca d), and egula o y au ho i ies (e.g., clinical alida ion epo ). Pos -ma ke moni o ing and main enance ac i - i ies ha iden i y de ia ions om he expec ed model beha io a e iden i ied and cap u ed as bug epo s o eedback and ed in o he eam backlog as equi emen s. The e minol- ogy and de elopmen phase ha moniza ion oge he wi h he design ou pu a i ac s a e desc ibed in Fig.9. 4 Case s udy: O a izio p ocess e ised In ou p e ious wo k, we ha e in oduced O a izio6, CE ce i ied medical so wa e o assessing he isks o join eplacemen su ge ies (G anlund e al.,2021). We use his sys- em o demons a e how o apply con inuous design con ol o ML in ce i ied medical sys ems. The wo k is a concep p o o ype in i s na u e; i builds on expe iences om an indus y sys em, bu he p oposed implemen a ion has no been deployed o indus ial use. Ou p e ious wo k wi h O a izio has in oduced a con inuous aining pipeline. The pipeline allows o o e come egula o y cons ain s associa ed wi h O a izio ML model aining and o simul aneously achie e au oma ion goals associa ed wi h MLOps (G anlund Pe o mance e alua ion Iden i y de ia ions Iden i y isks and mi iga ions De ailed design So wa e uni implemen a ion and e i ica ion So wa e in eg a ion and in eg a ion es ing So wa e sys em es ing So wa e elease Da a p epa a ion Building E alua ion and expe imen a ion Packaging Tes ing Deploymen Moni o ing CD4ML IEC 62304 Main enance Ma ke p e pos So wa e isk managemen (ML) Da ase jus i ica ion Gene a e alida ion epo Fig. 9 Design con ol p ocess 6 h ps:// o a iz. io/ So wa e Quali y Jou nal 1 3 e al.,2021). In addi ion, he pipeline add esses he medical de ice so wa e design con ol equi emen s by design. Fo his pape , we ha e e ised he pipeline by ex ending i wi h a ca e ully selec ed se o model ca d documen s, o demons a e he p oposed solu ion. Figu e 10 p esen s he pipeline o G anlund e al. (2021), wi h he p oposed model ca d ex ensions ma ked wi h a da ke colo . 4.1 Con inuous aining wi hMLOps pipeline The da a used o ain and e- ain he O a izio ML models is gene a ed wi hin he clinical p ocesses o a collabo a ing pa ne hospi al, and, by design, O a izio does no gene a e co esponding da a in p oduc ion use. Because o he sensi i e na u e o da a, access o he hospi al’s compu a ional en i onmen is s ic ly es ic ed. As a esul , he con inuous aining pipeline was designed o ope a e inside he clinical pa ne ’s con olled en i on- men and e ch new da a om he da a s o e based on p e-de ined igge s. Fu he mo e, as he models a e de e minis ic by na u e, hei echnical pe o mance can be alida ed acco ding o he p inciples o clinical e alua ion wi h es da a in a es ic ed en i onmen Da a S o e - scheme - alues Da a Ex ac - con igu able pe ins alla ion Da a Valida ion Da a snapsho Ve i ica ion T aining & Building Packaging & alida ion suppo T igge Pe o mance & alida ion epo Packaged model Audi ail Manual esolu ion Anomaly de ec ed Pe o mance dec eased Isola ed & es ic ed en i onmen (e.g. hospi al's compu a ional en i onmen ) Deploymen pipeline Design and de elopmen ou pu e iew & alida ion Da a enginee s and da a scien is s So wa e de elope s and ops Risk manage s and compliance o ice s Da a Quali y Con ol Risk Managemen Pe o mance equi emen s Da a P epa a ion - anonymous da a Managemen o da a se s Fig. 10 Con inuous aining pipeline wi h model ca ds. A ows indica e da a lows So wa e Quali y Jou nal 1 3 wi hou he need o do he in ended medical use speci ic alida ion in he inal p oduc ion en i onmen . Finally, O a izio was designed o be deployed in a p oduc ion en i onmen wi h i s ML model in a “locked” s a e o egula o y easons. In p ac ice, O a izio’s models a e ained du ing he de elopmen phase, and hei abili y o imp o e he ou come on he ly is disabled in p oduc ion use. Despi e his limi a ion, he pipeline enables he labo ious ask o e- aining o be au oma ed. In addi ion, he ac ha he de elopmen eam does no need access o he es ic ed en i onmen beyond he pipeline’s ins alla ion and main enance is a aluable design ea- u e. Fu he mo e, he pipeline can au oma ically gene a e he equi ed documen a ion needed o assess he model pe o mance in he clinical pe o mance e alua ion. All c ea ed a i ac s a e deli e ed o he de elopmen eam om he isola ed en i onmen . 4.2 Design con ol documen a ion wi hmodel ca ds E en i he o iginal e sion o he pipeline in G anlund e al. (2021) con ains he equi ed design con ol documen a ion, which can be gene a ed au oma ically, he documen a ion o ma has no been based on any gene ally known s anda d. The eason is ha a he ime o implemen ing he pipeline, no such o ma was a ailable (Mi chell e al.,2019). In addi- ion, he selec ed documen empla es ha e been simila o mo e adi ional quali y man- agemen sys em ypes o eco ds a ge ed o egula o y s akeholde s and wi hou he ech- nical abili y o be se ialized. To add ess hese challenges and u he suppo con inuous design con ol o ML, we expanded he pipeline wi h he model ca ds ailo ed o add ess he design con ol documen a ion equi emen s. 4.2.1 Da a se managemen To ensu e he equi ed pe o mance o he models, he selec ed da a se mus be ep esen a- i e o he a ge popula ion. In O a izio’s case, da a sou ces could di e pe ins alla ion, and, as a esul , he p ocedu e o da a ex ac ion mus be con igu able. In addi ion, he e a e many ela ed documen a ion equi emen s, which can be documen ed wi h a model ca d. Fi s ly, he allowed da a sou ces mus be lis ed wi h he speci ic equi emen s and desc ip ions o a da a sou ce. Secondly, he da a inclusion and exclusion c i e ia mus be de ined wi h he p ocedu e o in alid da a managemen . Thi dly, da a p o ec ion policy needs o be de ined wi h clea ins uc ions on how o ensu e da a p o ec ion a la e s ages o da a p ocessing. Finally, po en ial biases in he da a mus be e lec ed and he selec ions made jus i ied acco dingly. 4.2.2 Da a quali y con ol As pa o da a quali y con ol o a sys em using supe ised lea ning, such as O a izio, he labeling and label e i ica ion p ocedu es a e essen ial. Mo eo e , when u ilizing au oma- ion in e-lea ning, he co ec ness o labeling needs o be cons an ly moni o ed. I is pos- sible o use di e en p e-de ined schemas and bounda ies o alida e he da a quali y, bo h in e ms o o ma and con en . As model ca ds a e al eady in a machine- eadable o ma , hey can be used simul aneously as a documen a y and a alida ing a i ac . I he a e o e o in he da a alida ion ises abo e he accep able limi , he e-lea ning canno con inue au oma ically, and he anomaly mus be esol ed manually. So wa e Quali y Jou nal 1 3 4.2.3 Risk managemen In conjunc ion wi h clinical e alua ion, e ec i e isk managemen p o ides a p ac ical ool o he manu ac u e o p o e he sa e y and clinical e iciency o he de ice. As he ML models a e a cen al pa o O a izio, he models ha e a conside able impac on he de ice’s isk p o ile. As a esul , he po en ial isks ela ed o he models need o be ca e ully add essed. When u ilizing model ca ds, he iden i ied model- ela ed haza ds and co espond- ing isk mi iga ions a e documen ed on he model ca d documen . I is wo h no icing ha ce ain isks, bo h in e ms o pa ien sa e y and da a secu i y, a e unique o AI/ML sys ems. These isks include, o ins ance, model d i , he d i in da a dis ibu ion, and isks ela ed o con inuous lea ning sys ems. In addi ion, also isks ela ed o ad e sa ial a acks need o be conside ed. 4.2.4 Pe o mance equi emen s Acco ding o he egula o y equi emen s, manu ac u e s o medical de ices mus docu- men he in ended pu pose o hei de ice, including speci ica ion o indica ions, con- aindica ions, pa ien a ge g oups, and ope a ing pa ame e s and limi a ions (Eu opean Pa liamen and he Council,2017). In addi ion, pe o mance cha ac e is ics, accu acy, and he limi s o accu acy, p ecision, and analy ical pe o mance mus be add essed i applicable. Many o he abo e equi emen s apply di ec ly o O a izio’s ML models. Mo eo e , hey can be con enien ly documen ed wi hin a model ca d. 5 Discussion In his sec ion, we p o ide an ex ended discussion ega ding he p oposed app oach, highligh ing how i b ings bene i s o he in ol ed s akeholde s. In addi ion, we discuss limi a ions o his wo k. 5.1 Aligning ML de elopmen and egula o y p ac ices The use o ML echnologies in ce i ied medical de ices is an eme ging end in a no o- iously conse a i e indus y. Consequen ly, he egula o y p ac ice is no as es ablished as he p ac ices in he de elopmen o adi ional so wa e medical de ices. Ou p oposal b ings oge he he ML applica ion li e cycle, demons a ed using CD4ML, and he medical de ice so wa e li e cycle p ocess s anda d IEC 62304. The esul lowe s he cogni i e ba ie s be ween he machine lea ning model de elope s, such as da a enginee s and da a scien is s, and egula o y p ac i ione s, allowing hem o wo k oge he o e ec i ely de elop medical de ices ha include machine lea ning echnologies. So wa e Quali y Jou nal 1 3 5.1.1 A oiding common ML sys em design p oblems In gene al, ML models cons i u e only a subse o he inal sys em ha inco po a es he espec i e ML echnology and makes i a ailable o he end use s (Sculley e al.,2015). Wi h his in mind, he con inuous design con ol app oach o handling ML in ce i ied medical sys ems de elopmen i s unde he p ocess managemen and ools ca ego y. Table1 desc ibes how he ac i i ies de eloped as pa o he con inuous design con- ol help mi iga e design p oblems, such as he accumula ion o echnical dep , and he ole played by he model ca ds documen s as he audi ail o pe o ming hese ac i i ies. 5.1.2 Answe ing egula o s’ conce ns Al hough he use o ML echnology wi hin he medical de ices is ela i ely new, he egu- la o s a e ac i ely engaged in a dialog wi h he indus y o guide i s adop ion (Food and D ug Adminis a ion,2021). This indica es ha he egula o s a e awa e o he new ech- nologies, and a e conside ing how o bes egula e he de elopmen o medical de ices ha include ML ea u es. Table 2 desc ibes how ou app oach add esses he egula o s’ conce ns. 5.2 Model ca d me ada a asaudi ail The eme ging model ca d ecosys em inc eases he enginee ing ma u i y o machine lea ning model de elopmen . The model ca d me ada a documen p o ides an ex ensible machine- eadable medium in which conce ns ela ed o he model de elopmen can be cap- u ed. In ou wo k, we le e aged he model ca d me ada a documen o cap u e he egu- la o y aspec s, such as in ended use, he sou ces o da a used o aining, o he clinical pe o mance e alua ion, ele an when he machine lea ning model is used in a ce i ied medical de ice. In doing so, we e ined exis ing p ope ies de ined in he Tenso Flow’s model ca d me ada a schema and added new p ope ies when needed (e.g., model_pa ame e s. da a[].x_sou ces). We ound ha he desc ip ion p ope ies de ined in he schema documen as s ing a e no s uc u ed enough and we used Ma kdown o ha e a empla e d i en ep esen a ion o he in ended use o he model (e.g., model_de ails. documen a ion), and o he desc ip ion o he da ase s used in model aining (e.g., model_pa ame e s.da a[].desc ip ion). The app oach allowed us o i e a e as , enabling he eam membe s o ocus on adding con en . The expe ience, backed wi h eedback om o he implemen e s, will allow us o iden i y he ele an in o ma ion ha can e en ually be ex ac ed and o malized in o model ca d schema ex ensions. The combina ion o using a s uc u ed documen wi h he semi-s uc u ed ma kdown desc ip ion is app op ia e o he a ge audience o med by enginee s and egula o y p o- essionals, each ca ego y con ibu ing using speci ic modali ies. Al hough he app oach is e ec i e a collec ing he needed in o ma ion ha se es as an audi ail o he ac i i ies pe o med by he eam membe s, edi ing he me ada a documen using ex edi o s does no p o ide he bes use expe ience o all use s. So wa e Quali y Jou nal 1 3 Table 1 Mi iga e common ML sys em design p oblems (Sculley e al.,2015) wi h con inuous design con ol ML sys em design p oblems Mi iga ion s a egy Model ca d documen ole Con igu a ion: Do we know all con igu a ion op ions and hei e ec s? Con igu a ion managemen is a p ime conce n in he SDLC o medical sys ems (In e na ional Elec o echnical Commission,2015). The exis ing p ac ices es ablished by he medical de ice manu ac u e o con igu a ion and so wa e isk managemen can be expanded o co e he con igu a ions o he ML model in eg a ed in o he medical sys em. Con ains he ML model pa ame e s and hei alid anges o he in ended use. Da a collec ion & Fea u e ex ac ion: Do we know i he inpu da a and ea u es de eloped a e enough o he in ended use? The e iew includes an analysis o he selec ed da a sou ces and he me hods used o ea u e ex ac ion. The model ca d is ex ended o cap u e he sou ce o each da a se . Toge he wi h he jus i ica ion sec ion included desc ip ion, i p o ides he e idence o needed o ul ill he clinical e alua ion. Da a e i ica ion The ele an da a se is used o e i y he in eg a ion o each change eques . The e i ica ion da a se is included in he model ca d, and should be used by he ele an MLOps in eg a ion s ages. Resou ce managemen : Do we know ha he needed ha dwa e and so wa e esou ces a e a ailable o ensu e he model pe o ms co ec ly? The quali y managemen sys em, implemen ed by he manu ac u e o ul ill egula o y equi emen s (In e na ional O ganiza ion o S anda diza ion,2016), ensu es ha he needed esou ces o he p ope unc ioning o he medical p oduc a e alloca ed. The model ca d con ains he in o ma ion abou he special esou ces ha a e needed o un he model. The in o ma ion should be used downs eam when planning he esou ce alloca ion. Se ing in as uc u e: Do we know ha he model is in eg a ed and wo ks p ope ly? ML models a e packaged and in eg a ed in o medical sys ems as lib a ies (see Sec .3.2). As such, hei unc ionali y is exposed ia an API and changes o he ML model a e con ained. - Moni o ing: Can we de ec de ia ions while he model is used wi h p oduc ion da a? Medical de ice manu ac u e s mus es ablish pos -ma ke su eillance p og am (In e na ional O ganiza ion o S anda diza ion,2016; In e na ional Elec o echnical Commission,2015). The p og am should co e he moni o ing he ML componen s in use. The quan i a i e da a included in he model ca d documen se es as inpu o moni o ing componen s ha de ec de ia ions. So wa e Quali y Jou nal 1 3 In he u u e, we plan o explo e wi h ha ing dedica ed edi o s o egula o y p o ession- als so ha hey can in oduce hei con en using mo e amilia app oaches, such as wha you see is wha you ge edi o s. 5.3 Pull eques ascon inuous design con ol The pull eques is he p ac ice ypically used by so wa e de elopmen eams o man- age changes. Ou p oposal ex ends he use o pull eques h oughou he machine lea n- ing de elopmen li e cycle. Besides so wa e enginee s, da a enginee s and da a scien is use he pull eques o manage he e olu ion o he so wa e p oduc s wi hin hei a ea o Table 2 Suppo ing he good ML p ac ice guiding p inciples (Food and D ug Adminis a ion,2021) wi h con inuous design con ol Guiding p inciples Implemen a ion Mul i-disciplina y expe ise is le e aged h oughou he o al p oduc li e cycle The pull eques is he enue o pe o m mul i- disciplina y e iews du ing all de elopmen s ages Good so wa e enginee ing and secu i y p ac ices a e implemen ed ML de elopmen is in eg a ed in o he p oduc and so wa e de elopmen le e aging bes p ac ices and ools Clinical s udy pa icipan s and da a se s a e ep esen a i e o he in ended pa ien popula ion Al hough clinical s udies can be seen as pa ly ou side he scope o p oduc de elopmen , he model ca d documen can be used o documen da a collec ion p o ocols and da a cha ac e is ics ha a e ele an o he in ended pa ien popula ion. In addi ion, con inuous design con ol p omo es aceabili y om clinical s udy da a se s o he inal model T aining da a se s a e independen o es se s Following he con inuous design p ac ice ensu es ha he es da a se is e iewed, e sioned and i is no used du ing model de elopmen Selec ed e e ence da a se s a e based upon bes a ailable me hods I accep ed e e ence da a se s a e a ailable, hei use in he model de elopmen should be p omo ed and documen ed in he model ca d documen Model design is ailo ed o he a ailable da a and e lec s he in ended use o he de ice Con inuous design con ol ensu es, by including he jus i ica ion in he model ca d documen , ha he da a se s a e enough o sa is y he in ended use Focus is placed on he pe o mance o he human-ai eam Tes ing a di e en de elopmen s ages (e.g., aining, in eg a ion, sys em) ensu es ha he di e en s akeholde s’ in e es s a e cap u ed Tes ing demons a es de ice pe o mance du ing clinically ele an condi ions Con inuous es ing in s aging en i onmen s ha mimic he clinically ele an condi ions Use s a e p o ided clea , essen ial in o ma ion The egula o y equi ed documen a ion con ains in o ma ion ha can be collec ed in he model ca d documen s du ing de elopmen . The con inuous design and e iew ac i i ies ensu es ha he end use documen a ion i s hei needs Deployed models a e moni o ed o pe o mance and e- aining isks a e managed The moni o ing s age o he MLOps pipeline, implemen ing common echniques o de ec ing model pe o mance anomalies, oge he wi h he egula o y equi ed pos -ma ke moni o ing p ac ices enable he manu ac u e o iden i y de ia ions and pe o m he necessa y co ec i e ac ions So wa e Quali y Jou nal 1 3 esponsibili y. As he pull eques is linked wi h equi emen s (e.g., design inpu s), and he in oduced changes consis mainly o he machine lea ning model and he model ca d me ada a (e.g., design ou pu s), we ha e an e ec i e quali y ga e ha ensu es ha design e iews a e pe o med sys ema ically and he app op ia e audi ails a e build a e e y i e - a ion h oughou he de elopmen li e cycle. 5.4 Handling model anomalies One o he c i ical ad an ages o u ilizing in e p e able machine lea ning models is ha hey allow o mo e e icien anomaly de ec ion and analysis. The ML model can be in e - p e ed as consis ing o di e en componen s, such as inpu s, ea u es, pa ame e s, and weigh s, and he unde s andabili y o he model inc eases i i can be decomposed in o di e en explainable pa s (Lip on,2018). Ou p oposed app oach p o ides a solid ounda- ion o documen di e en model aspec s o suppo explainabili y, which can, in u n, help he enginee ing eam o pe o m anomaly and oo cause analysis ac i i ies. Fu he mo e, e en i ini ially designed o p omo e egula o y ac i i ies in he o m o an audi ail, he ine-g ained aceabili y p o ides addi ional suppo o he anomaly analysis. Based on he esul s, he eam can de e mine he app op ia e co ec i e ac ions needed o be pe o med and included in subsequen model eleases. 5.5 Sa e con inuous sel ‑lea ning The abili y o lea n a e being deployed o eal-wo ld use is undoub edly one o he c i ical di e ences be ween an AI/ML-enabled sys em and a mo e adi ional ule-based sys em. Howe e , as discussed p e iously, due o he cu en egula o y unce ain ies, manu ac u - e s o medical de ice AI/ML-enabled sys ems may p e e such AI/ML models ha can be deployed in a locked s a e. I is e iden ha such a design app oach can se iously educe he bene i s o AI/ML-enabled echnology. The e o e, al e na i e ye pa ien sa e y ensu - ing design and de elopmen me hods a e needed. A obus and e ec i e isk managemen p ocess is he basis o sa e medical de ice so - wa e de elopmen . As he p ocess s a s wi h isk iden i ica ion (In e na ional O ganiza- ion o S anda diza ion,2019), he de elopmen eam mus be compe en in assessing he p oduc ’s speci ic ML change- ela ed aspec s, pa icula ly when he chosen echnology’s complexi y and opaqueness inc ease. In p ac ice, a c oss- unc ional de elopmen eam should include knowledgeable and expe ienced da a scien is s, in addi ion o he ypical se o clinical and p oduc de elopmen specialis s. Acco ding o he egula ions, medical de ice manu ac u e s mus seek app o al o changes o he app o ed design o a de ice p io o making he change, whe e he change has a subs an ial impac o can a ec he de ice’s con o mi y wi h he gene al sa e y and pe o mance equi emen s (Eu opean Pa liamen and he Council,2017). The e o e, i is clea ha i enabled, sel -lea ning can only occu wi hin a p e-de e mined ole ance and change con ol plan. In addi ion, he manu ac u e is esponsible o demons a ing ha he change ole ance complies wi h he de ice’s in ended use, use en i onmen , use g oups, and o he medical claims p io o placing he de ice on he ma ke . Finally, an essen ial aspec o sel -lea ning and sa e y is he abili y o moni o he de ice’s pe o mance as a pa o he de ice’s pos -ma ke su eillance ac i i ies. Con a y o he i s hough , i can be a gued ha sel -lea ning AI/ML sys ems a e, in ac , mo e ole an agains model d i han he locked sys ems as hey a e cons an ly imp o ing hei So wa e Quali y Jou nal 1 3 pe o mance wi h he new da a. Howe e , moni o ing he cons an ly changing sys em can be mo e di icul as he e a e addi ional aspec s o conside and measu e. The mos impo - an hing is o ensu e ha he de ice’s pe o mance canno dec ease due o an upg ade. 5.6 C oss domain e minology challenges A con lic ing e minology is a common p oblem when se e al domains — such as medical de ice egula o y concep s, da a science, and so wa e enginee ing — a e combined wi hin a single p ojec . This p oblem can lead o miscommunica ion, misunde s andings, and, a wo s , poo decision-making (Vogel,2011). The e minology con lic s we e also eme gen wi hin his pape ’s con ex , pa icula ly ega ding he e m alida ion. Wi hin he ield o ML alone, he e m has been used wi h wo di e en meanings: o da a cu a ion (i.e., da a alida ion) o ML model uning. To make ma e s e en mo e complica ed, in he con ex o medical de ice de elopmen , alida ion means con i ma ion ha he pa icula equi e- men s o speci ic in ended use can be consis en ly ul illed (In e na ional Medical De ice Regula o s Fo um,2022). As a p ac ical solu ion, we p opose a o ing egula o y e mi- nology in he documen s ha demons a e con o mi y, and, in gene al en o cing explici communica ion o a oid con usion. 5.7 In o ma ion secu i y conside a ions The model ca d me ada a documen se es as an e ec i e audi ail o he model de el- opmen . As such, i con ains a ple ho a o in o ma ion ha should be conside ed p i a e, as i migh con ain pe sonal da a, in o ma ion ha is no open o public, o e en c i ical ade sec e s. While he documen should se e as inpu o gene a ing he public echnical documen a ion o he medical de ice, as expec ed by egula ion and applicable s anda ds, manu ac u e s should employ he necessa y in o ma ion managemen p ac ices o ensu e ha he p ope ies classi ied as p i a e a e no included in he model ca d ep esen a ions in ended o public consump ion. 5.8 Limi a ions Ou implemen a ion o he p oposed app oach le e ages exis ing ools and p ocesses widely used by so wa e de elopmen eams, such as Gi o e sion con ol, issues o acking equi emen s and wo k i ems, o pull eques s o e iews and change manage- men . Howe e , we wish o poin ou ha ou implemen a ion has no been exposed o a wide ange o eal li e medical p oduc s, excep O a izio. Fo example, managing he e olu ion o he medical p oduc has been implemen ed using he ea u e-b anch app oach, in which a new b anch is c ea ed om he mainline, o each equi emen , and me ged ollowing a success ul e iew. O he de elopmen models such as unk-based de elop- men (Jø gensen,2001) ha e no been in es iga ed ho oughly, al hough equi alen e iew acili ies a e suppo ed by ools used o his de elopmen s a egy. The e o e, he p o- posed app oach is no in ended o be a model ha sui s any medical p oduc o si ua ion, which one mus ollow in a e ba im ashion. Ra he , we wan o emphasize ha sys ema ic e iews and using he model ca d as he audi ail o egula o y ac i i ies ep esen an e ec i e o m o design con ol ha is compa ible wi h he egula o y equi emen s ha So wa e Quali y Jou nal 1 3 go e n medical de ices ha con ain so wa e. Simila app oaches o ack he model ca d me ada a and pe o ming equi alen ac i i ies will mos likely esul in a sa is ac o y solu- ion om a egula o y pe spec i e. Ope a ions ela ed o da a ha e been o e looked in he pape , because much o he wo k happens in da a enginee s’ own en i onmen , ollowing hei own ways o wo king (Aho e al.,2020). Howe e , we p oposed model ca ds as a mechanism o eco d he ail o p o - enance om da a ope a ions o he model, so ha his pa can be included in he MLOps pipeline as well. The e o e, explo ing he da a ope a ions and hei ela ion o model ca ds is a pa o u u e wo k we plan o ca y ou . 6 Conclusions Theso wa e enginee ing indus y has widely adop ed con inuous de elopmen and deploy- men o new ea u es. These ea u es may include AI/ML unc ions, which ha e become commonplace in nume ous applica ions, calling o deploymen pipelines whe e such unc- ions can be included in mains eam de elopmen ac i i ies. Such con inuous se up o ms a sha p con as o he de elopmen o medical sys ems, whe e design con ols a e o en in e p e ed o equi e wa e all-like de elopmen app oach. In his pape , we p opose using an app oach whe e con inuous design con ol o ML is enabled while de eloping medical sys ems. The p oposed app oach builds on ou ea - lie wo k on MLOps, bu ex ends i wi h he design con ols ha a e explici ly included in he MLOps pipeline. The app oach was demons a ed wi h an indus y sys em, which is in ac i e use. As u u e wo k, we plan o in es iga e da a ope a ions, ela ed o building ML models, in mo e dep h. Acknowledgemen s The au ho s wish o hank p ojec AHMED and associa ed conso ium, unded by Business Finland, o suppo ing his esea ch. Funding Open Access unding p o ided by Uni e si y o Helsinki. Da a a ailabili y Da a sha ing no applicable o his a icle as no da ase s we e gene a ed o analyzed du ing he cu en s udy. Decla a ions Con lic o in e es The au ho s decla e no compe ing in e es . Open Access This a icle is licensed unde a C ea i e Commons A ibu ion 4.0 In e na ional License, which pe mi s use, sha ing, adap a ion, dis ibu ion and ep oduc ion in any medium o o ma , as long as you gi e app op ia e c edi o he o iginal au ho (s) and he sou ce, p o ide a link o he C ea i e Com- mons licence, and indica e i changes we e made. The images o o he hi d pa y ma e ial in his a icle a e included in he a icle’s C ea i e Commons licence, unless indica ed o he wise in a c edi line o he ma e ial. I ma e ial is no included in he a icle’s C ea i e Commons licence and you in ended use is no pe mi ed by s a u o y egula ion o exceeds he pe mi ed use, you will need o ob ain pe mission di ec ly om he copy igh holde . To iew a copy o his licence, isi h p:// c ea i eco mmons. o g/ licen ses/ by/4. 0/.