Evaluating an LLM-Based Phenotype Extractor from Medical Reports Using Combinatorial Testing

✦ 1 / 3 ✦ E v a lu a t i n g a n LLM - B a s ed P he notyp e E xtr ac tor f rom M edica l R e ports U s i n g C om bi n a tor ia l T e st i n g T hi s i s t he r e pl ica t i on p ackage f or t he p a p e r " E v a lu a t i n g a n LLM - B a s ed P he notyp e E xtr ac tor f rom M edica l R e ports U s i n g C om bi n a tor ia l T e st i n g ". T he LLM - ba s ed p he notyp e e xtr ac tor i s a v ai l ab l e a t h ttps :// d o i . or g /10.5281/ z e no d o .17912782, i n c lu di n g a ll c on fig ur a t i on fi l e s , s c r i pts , a n d i nstru c t i ons on h ow to s e t up a n d us e t he e xtr ac tor . R e qu i r e m e nts f or t hi s R e pl ica t i on P ackage B ef or e runn i n g t he s c r i pts , i nst a ll t he f ollow i n g de p e n de n cie s : F or t he M aj or i ty o f pro g r a ms op e r a t i n g w i t h c sv : pip install pandas F or tr a nsl a t i on : pip install deep_translator F or s c r i pts op e r a t i n g on t he HPO ontolo g y : pip install pronto T he HPO ontolo g y , c ont ai n ed i nto t he fi l e hp.obo , i s a lso r e qu i r ed , a n d must be stor ed un de r t he utils di r ec tory . Y ou ca n d ownlo ad i t f rom t he f ollow i n g l i n k : h ttps :// h po . ja x . or g / da t a / ontolo g y U s age F ol de r S tru c tur e T he f ol de r c ont ai ns P yt h on s c r i pts t ha t pro ce ss a n d a n a lyz e da t a . T he CS V fi l e s a r e lo ca t ed i n t he output f ol de r . T he f ollow i n g a r e t he most i mport a nt on e s : merged_dataset.csv : t he m e r ged da t a s e t c ont ai n i n g m edica l r e ports a n d t hei r a sso cia t ed p he notyp e s . merged_dataset_with_parents.csv : t he m e r ged da t a s e t c ont ai n i n g m edica l r e ports a n d t hei r a sso cia t ed p he notyp e s , i n c lu di n g p a r e nt c o de s . 2wise_test_suite.csv : t he c om bi n a tor ia l t e st su i t e ge n e r a t ed f rom t he da t a s e t . T he utils f ol de r c ont ai ns ut i l i ty pyt h on s c r i pts a n d ut i l i ty fi l e s . ✦ 2 / 3 ✦ T he prompts f ol de r c ont ai ns t he prompts us ed f or t he LLM - ba s ed p he notyp e e xtr ac tor d ur i n g t e st i n g . T he results f ol de r c ont ai ns t he r e sults o f t he e xp e r i m e nts CombinatorialPromptTesting.xlsx r e ports t he summ a ry da t a o f t he e xp e r i m e nts . summary_*.csv fi l e s c ont ai n t he output o f each t e st e x ec ut i on . results_analysis.py c ont ai ns t he s c r i pt us ed to a n a lyz e t he r e sults a n d to e xtr ac t summ a ry v a lu e s . T he test_scripts f ol de r c ont ai ns t he s c r i pts us ed to run t he LLM - ba s ed p he notyp e e xtr ac tor on t he ge n e r a t ed t e st ca s e s . D a t a s e t P r e p a r a t i on T hi s r e pl ica t i on p ackage a lr ead y c ont ai ns t he r e sult i n g da t a s e ts ( merged_dataset.csv a n d merged_dataset_with_parents.csv ). T h us , us e rs w h o d on ' t n eed to tr a nsl a t e or m e r ge da t a s e ts ca n s ki p t hi s s ec t i on . save_and_prepare_database.py : I t s a v e s t he da t a s e t n eeded f or f urt he r op e r a t i ons . T he n , i t tr a nsl a t e s a sp ecific da t a s e t ( tr a nsl a t e s 10 rows to E n g l i s h p e r c y c l e ), a n d m e r ge s t he two da t a s e ts i nto a s i n g l e fi l e . I t a lso c ustom i z e s t he da t a s e t b y addi n g diffe r e nt n a m e s f or p a t ie nts , p h ys icia ns , a n d h osp i t a ls . T he s c r i pt us e s som e o f t he f un c t i on a l i t ie s o ffe r ed b y utils/ontology_depth.py . A f t e r runn i n g t hi s pro g r a m , you w i ll ha v e a da t a s e t n a m ed merged_dataset.csv a n d on e n a m ed merged_dataset_with_parents.csv c ont ai n i n g a lso p a r e nt c o de s . T he f un c t i on a l i t ie s o f save_and_prepare_database.py e xplo i ts t he fi l e utils/names.py w hich c ont ai ns t he n a m e s o f p a t ie nts , p h ys icia ns , a n d h osp i t a ls . IPM C r ea t i on generate_ipm.py : I t ge n e r a t e s t he IPM , f or a gi v e n de pt h a n d num be r o f p a r a m e t e rs . T he IPM i s s a v ed i n a fi l e n a m ed hpo.ctw , i n C TW edge f orm a t , a n d hpo.acts i n t he f orm a t c omp a t ib l e w i t h CA ge n . T o p e r f orm t he tr a nsl a t i on , t he s c r i pt us e s t he f un c t i on a l i t ie s o ffe r ed b y utils/ctwedge_to_acts_translator.py . C om bi n a tor ia l s a mpl i n g combinatorial_sampling.py : I t ge n e r a t e s t he fi l e 2wise_test_suite.csv w hich c ont ai ns t he r ea l t e st ca s e s to be e x ec ut ed agai nst t he LLM . T e st e x ec ut i on A f t e r t he da t a s e t ( 2wise_test_suite.csv ) i s r ead y a n d t he LLM - ba s ed p he notyp e e xtr ac tor i s s e t up ( s ee h ttps :// d o i . or g /10.5281/ z e no d o .17912782) a n d stor ed i n t he test_scripts f ol de r , you ca n run t he t e sts us i n g t he s c r i pts i n t he test_scripts f ol de r . I n p a rt ic ul a r , t he c om bi n a tor ia l prompt t e st i n g pro ce ss i s e x ec ut ed t h rou gh t he fi l e test_scripts/PhenotypeExtractorTesting.py , w hi l e t he e x ha ust i v e prompt t e st i n g i s e x ec ut ed t h rou gh t he fi l e test_scripts/PhenotypeExtractorTestingExhaustive.py . ✦ 3 / 3 ✦ A n a lys i s s c r i pts utis/dataset_analysis.py : I t check s h ow m a ny o f t he p he notyp e s i n HPO a r e not c ov e r ed b y t he da t a s e t . I t c ons ide rs t he da t a s e t stor ed i n t he output/merged_dataset_with_parents.csv fi l e , a n d t he c om bi n a tor ia l t e st su i t e i n output/2wise_test_suite.csv .