[Obsolete] Personalized prognosis & treatment using an optimal predictor machine: An example study on conversion from Mild Cognitive Impairment to Alzheimer's Disease

PERSONALIZED PROGNOSIS & TREATMENT USING LUSTED-JAYNES MACHINES: AN EXAMPLE STUDY ON CONVERSION FROM MILD COGNITIVE IMPAIRMENT TO ALZHEIMER’S DISEASE P.G.L. Porta Mana western norway university of applied sciences, norway I. Rye university of oslo, norway A. Vik haukeland university hospital, norway M. Kociński , A. Lundervold , A.J. Lundervold university of bergen, norway A.S. Lundervold western norway university of applied sciences, norway 9th February 2023 Correspondence should be sent to E-Mail: [email protected] Psychometrika Submission February 9, 2023 2 PERSONALIZED PROGNOSIS & TREATMENT USING LUSTED-JAYNES MACHINES: AN EXAMPLE STUDY ON CONVERSION FROM MILD COGNITIVE IMPAIRMENT TO ALZHEIMER’S DISEASE Abstract The present work presents a statistically sound, rigorous, and model-free algorithm – called “Lusted-Jaynes machine” in homage to these two pioneers – for use in personalized medicine. The algorithm is designed first to learn from a dataset of clinical with relevant predictors and predictands, and then to assist a clinician in the assessment of prognosis & treatment for new patients. It allows the clinician to input, for each new patient, additional patient-dependent clinical information, as well as patient-dependent information about benefits and drawbacks of available treatments. We apply the algorithm in a realistic setting for clinical decision-making, incorporating clinical, environmental, imaging, and genetic data, using a data set of subjects suffering from mild cognitive impairment and Alzheimer’s Disease. We show how the algorithm is theoretically optimal, and discuss some of its major advantages for decision-making under risk, resource planning, imputation of missing values, assessing the prognostic importance of predictors, and further uses. Key words: Clinical decision making, Utility theory, Probability theory, Artificial Intelligence, Machine Learning, Base-rate fallacy Psychometrika Submission February 9, 2023 3 1. Introduction: Personalized prognosis, treatment, and computer algorithms 1.0. Prologue: Four unique patients Meet Olivia, Ariel, Bianca, Curtis. 1 These four persons don’t know each other, but they have something in common: they all suffer from a mild form of cognitive impairment, and are afraid that their impairment will turn into Alzheimer’s Disease within a couple of years. This is why each of them recently underwent a wide range of clinical examinations and tests, including brain imaging. Today they are receiving the results. Based on their individual results, on available clinical statistical data, and on other relevant information, their clinician will assess their risk of developing Alzheimer’s Disease. Then, together with the patients and their relatives, the clinician will make a decision among four distinct preventive-treatment options, available to each patient. 2 In these tasks, the clinician will be helped by a computer algorithm. Besides a shared diagnosis of Mild Cognitive Impairment and associated worries, these patients have other things in common – but also some differences. Let’s take Olivia as reference, and list the similarities and differences between her and the other three patients: • Olivia and Ariel have identical results on the clinical and laboratory measures and age. They would also incur similar benefits and losses from the four available treatment options. Ariel, however, comes from a different geographical region, which presents a higher rate of conversion from Mild Cognitive Impairment to Alzheimer’s Disease. And unlike Olivia, Ariel comes from a family with a heavy history of Alzheimer’s Disease. Because of this geographical and family background and some relevant statistics found in some publications, the clinician judges, before seeing the clinical data, that there’s a 65% probability that Ariel’s cognitive impairment will convert to Alzheimer’s Disease. • Olivia and Bianca have identical clinical results and age; they also come from the same geographical region and have very similar family histories. In fact, we shall see that they have the same probability of developing Alzheimer’s Disease. Bianca, however, suffers from several allergies and additional clinical conditions that render some of the treatment options slightly riskier for her. • Olivia and Curtis have different results on all measures included in the clinical and laboratory examinations; Olivia is also more than 10 years older than Curtis. They otherwise come from the same geographical region, have very similar family histories, and would incur similar benefits or 1 These are purely fictive characters but with clinically realistic conditions; any reference to real persons is purely coincidental. 2 In the present paper, we use “prognosis” in a general sense to include also “diagnosis”, and “treatment” quite loosely to mean any course of action a clinician might take, including preventive treatment or even “additional tests”. Psychometrika Submission February 9, 2023 4 losses from the treatment options. Note that the imaging result for Curtis (hippocampal volume) is missing. Considering the similarities and differences among these patients, which of the four available treatments will be optimal for each of them? The clinician will find that, despite the many factors in common among our four patients – even despite Olivia’s, Ariel’s, and Bianca’s identical clinical results, and Olivia’s and Bianca’s identical probability of conversion to Alzheimer’s Disease – the optimal treatment for each patient is different from those for the other three – how come? 1.1. Assistive computer algorithms: personalized input and output In the example above, we said “in these tasks, the clinician will be helped by a computer algorithm”. The need for such computational help is clear from the vast amount of clinical statistical data and the large number of clinical predictors today available to clinicians. But how should such an assistive computer algorithm be designed in order to take fully into account patient differences? Although the example above concerns specifically Alzheimer’s Disease, the differences among patients described there apply more generally to most, if not all, clinical problems of prognosis and treatment. These differences can be broadly categorized as “difference in auxiliary or supplementary tests and background information” (Olivia vs Ariel), “difference in benefit and availability of treatments” (Olivia vs Bianca), “difference in clinical predictors” (Olivia vs Curtis), as schematized in the side figure. Each of these difference categories can affect the clinician’s final choice of optimal treatment. An assistive algorithm should therefore reflect these differences in its input, its output, or both: • In principle, there could be three kinds of input “slots”, where the clinician can input the current patient’s specific values as regards clinical predictors, auxiliary information, and treatment options & benefits. • If input slots are only available for one or two of the categories above, the output should at least be of such a kind as to allow the clinician to integrate the current patient’s specific values of the missing input categories. To appreciate these requirements, one should contrast the input and output of many kinds of machine-learning classification algorithms. These typically only allow the input of a patient’s clinical predictors, with no space for patient-specific auxiliary information or for adjustments of differences in background statistics (think of Olivia vs Ariel). And they typically output only a discrete prognostic label (say, “stable Mild Cognitive Impairment” vs “conversion to Alzheimer’s Psychometrika Submission February 9, 2023 5 Disease”), but no measure of the uncertainty about that label. Unfortunately, such output does not allow the clinician to assess treatment benefits and losses for the current patient, for this assessment depends not on the presence (present or future) of a disease, but on the risk of its presence. We shall discuss these points at length in §§ 3.2 and 3.3. The purpose of the present work is to present an assistive algorithm that meets the requirements above. This algorithm is designed to first learn from a dataset of clinical data with relevant predictors and predictand 3 , and then assist a clinician in the assessment of prognosis & treatment for new patients. It offers these ten features: 1. It can work with clinical predictors comprising any combination of categorical and one-dimensional (continuous, discrete ordinal, unbounded or bounded, uncensored or censored) variates. The predictand can also be any combination of categorical and one-dimensional variates. 2. It treats predictor and predictand variates on equal footing, in the sense that the clinician can at any moment decide to infer some other variate given the rest. 3. It does not require that the current patient be considered in all respects as a member of the population underlying the learning dataset. The patient can be considered a member only conditionally on particular variate values. 4. It accepts three inputs: (a) the clinical-predictor values for the current patient; (b) information about which predictand-predictor relationships learned from the dataset can be generalized to the current patient, and a prior prognostic probability representing auxiliary information; (c) a set of treatment options and their benefits and losses for the current patient. 5. It yields three basic outputs: (a) any prognostic probabilities or likelihoods about predictors and predictand desired by the clinician, given input 4a; (b) final prognostic probabilities, given inputs 4a–4b; (c) optimal treatment, given inputs 4a–4c; 6. Its input and outputs are modular, in the sense that the clinician can, for instance, give inputs 4a– 4b only, get a prognostic probability 5b as output, and then proceed to treatment assessment by other means or algorithms. 7. It works even if predictor data are missing, both in the learning dataset and for the current 3 literally “quantity to be predicted” or, more generally, inferred (cf. measurand in metrology, jcgm 2012, 2.3). We find this term, used in meteorology and climate science, more precise and less obscure or misleading than “dependent variate”, “response variate”, “outcome variable”, or similar. Psychometrika Submission February 9, 2023 6 patient. 8. It can quantify the uncertainty of its own outputs, allowing for sensitivity analyses. For example, it can tell how much a prognostic probability could have been different if the learning dataset had been larger, or whether the optimal treatment could be different if a particular missing predictor for the current patient were available. 9. It can make various kinds of long-term forecasts, such as frequency of prognoses with given probabilities, frequency of prescribed treatments, and similar – provided that the dataset used for its learning can be considered representative of the full population. 10. It is model-free and extracts the maximal amount of information theoretically contained in the learning dataset, and therefore achieves the maximal prognostic power that the predictors can yield. In other words, it is unbeatable. Let us comment on some of these features. We believe that the capability of working with complex predictands, feature 1, is important for a more realistic and nuanced approach to prognosis. In the case of Alzheimer’s Disease, for instance, a simple dichotomy “has disease” vs. “doesn’t have disease” is possibly an oversimplification 4 . Without feature 3, the capability of auxiliary contextual information, the algorithm would be of no use in the often occurring case of patients having peculiar clinical contexts. The capability of dealing with missing data, feature 7, is important for a concrete implementation in a clinical setting, typically afflicted by imputation problems. Feature 8is extremely important for a clinician to assess the reliability of final decisions and honestly inform the patient of the possibility of unwanted outcomes. Finally, features 2and 10, the fact that this algorithm yields the maximal amount of information jointly contained in all variates, makes it valuable in general clinical research. The algorithm can, for example, forecast the maximal accuracy obtainable by any inference algorithm based on the same predictors or a subset of those predictors; and it attains, by construction, that maximal accuracy. Further features of interest in Machine Learning are discussed in the next section. We call this algorithm a Lusted-Jaynes machine, for reasons explained in the next section. It is at the moment available as a collection of scripts 5 in the R programming language (R Core Team, 2023), which we plan to assemble into a clinician-friendly R package soon. The methodology underlying this algorithm has been successfully demonstrated for Alzheimer’s Disease with a smaller number of predictors (Antoniano-Villalobos et al.,2014), is used in many applications in astrophysics (Event Horizon Telescope Collaboration,2019,2022;Del Pozzo et al., 2018), and its advantages in neurocritical care have recently been emphasized (Jawa and Maslove, 4see e.g. Edmonds et al. (2015,2020), whose methods we find, however, inconclusive. 5doi:10.17605/osf.io/zb26t,https://github.com/pglpm/ledley-jaynes_machine. Psychometrika Submission February 9, 2023 7 2023). The next section 2gives an intuitive understanding of the Lusted-Jaynes machine’s underlying principles and workings. The machine’s concrete application is shown in § 3, using the four-patient fictitious scenario of § 1.0 as a concrete example, and subsection 3.4 discusses further applications to general medical research. A summary and discussion is given in § 4. Mathematical details and proofs on which the present work is grounded are given in a companion technical note 6 , which also explains how to use the R scripts. We apologize to readers who may find some discussions or explanations too obvious, or some mathematical details too scarce. We wanted the present work to be accessible to a wide audience, from clinicians and students of medicine to researchers in machine learning and probability theory. 2. The Lusted-Jaynes machine This section can be especially of interest to readers from Machine Learning and Artificial Intelligence. It is largely independent of the next one, which describes the machine’s application. It can be read after § 3by readers who would like to see the machine in action first. 2.1. Underlying theory and characteristics The method to solve clinical decision-making problems such as the one of § 1is none other than Decision Theory: the combination of probability theory and utility theory. It integrates available clinical statistical data with each patient’s unique combination of clinical results, auxiliary information, and treatment benefits, in a mathematical framework, completely determined by basic self-consistency requirements.7 Medicine has the distinction of having been one of the first fields to adopt Decision Theory, with the pioneering work by Ledley – who, incidentally, died of Alzheimer’s Disease (Shah et al.,2013) – and Lusted (Ledley and Lusted,1959a,b,1960;Lusted and Ledley,1960;Lusted,1967), who also promoted its algorithmic implementation (Lusted,1968;Ledley,1959,1960, § 1-5 p. 21). Clinical decision-making is today explained and exemplified in brilliant textbooks for medical students and clinicians (Weinstein and Fineberg,1980;Sox et al.,2013;Hunink et al.,2014). An outline is given in § 3.3. 6https://github.com/pglpm/ledley-jaynes_machine/raw/main/omni-predictor_machine.pdf 7 Jaynes (2003, chs 13–14); von Neumann and Morgenstern (1955); Cox (1946); Savage (1972); Luce and Raiffa (1957); Raiffa and Schlaifer (2000); Raiffa (1970); Lindley (1988); Kreps (1988). Psychometrika Submission February 9, 2023 8 The “Lusted-Jaynes machine” is an algorithmic implementation, as dreamed by Lusted and Ledley 8 , of the main calculations underlying the clinical decision-making process: from the comparison of a patient’s specific predictors with the statistics offered by a clinical database, to the choice of optimal treatment. The name is a homage to Lusted and to Jaynes (2003), who brilliantly explained the inductive logic underlying such a “robot”. Decision theory is also the normative foundation for the construction of an Artificial Intelligence agent capable of rational inference and decision making (Russell and Norvig 2022, part IV; Jaynes 2003, chs 1–2, 13–14). The Lusted-Jaynes machine can therefore be seen as an ideal machine-learning algorithm. It is “ideal” in the sense of being free from special modelling assumptions (this is why we do not call it a “model”) and from limitations of informational output which affect most common machine-learning algorithms; not “ideal” in the sense of being impracticable. Quite the opposite, the present work shows that this ideal machine-learning algorithm can today be used in a wide range of inference problems at insubstantial computational cost. More concretely, the Lusted-Jaynes machine is ideal because it computes the probability distribution over all possible long-run frequency distributions from which the learning dataset can originate, these frequency distributions being joint ones for all predictor and predictand variates. 9 This is the maximum possible amount of information that can be extracted from the learning dataset, in a strict information-theoretic sense. From this probability distribution, the Lusted-Jaynes machine can indeed calculate any quantity outputted by other machine-learning algorithms. For example (for terminology see e.g. Murphy,2012, § 8.6): • “Discriminative” algorithms: the probability p( Y|X )of any set of predictands Y given any set of input predictors X. • “Generative” algorithms: the probability p( X|Y )of any set of input predictors X given any set of predictand values Y. More generally, the machine can compute any joint, marginal, or conditional probabilities p(Z′, Z′′),p(Z′),p(Z′|Z′′)for any desired subsets of variates Z′, Z′′. • Regression or classification: the expected value E( Y|X )of any set of variates Y , given any other set of variates X , including the particular case of Y predictand, and X predictors. The uncertainty or variability around such an average is also automatically computed. • Functional regression: if the predictand Y or any other variate of interest turns out to be a 8cf. the Appendices in Lusted 1968. 9 This goes by the Sibylline technical name of “Bayesian nonparametric density regression”; see e.g. Rodríguez et al. (2009); Bhattacharya and Dunson (2010); and Walker’s (2010) witty overview. Psychometrika Submission February 9, 2023 9 function f of variates X , then their conditional probability will be a delta distribution: p( Y|X ) = δ [ Y−f ( X )]. Thus the Lusted-Jaynes machine always recovers a functional relationship if there is one, as well as its noise distribution. Furthermore, the machine also quantifies the uncertainty of all outputs above. More precisely, it takes into account how the statistical properties of the learning dataset could be different from those of its original population, owing to sampling fluctuations; and it can compute how much any of the outputs above would probably change if more learning data were collected. In the next section we explain intuitively how the Lusted-Jaynes machine computes the general probability distribution over long-run frequencies. A couple of special characteristics brought about by such computation can already be summarized here. First, in contrast to machine-learning algorithms such as neural networks, random forests, Gaussian processes, support-vector machines, or generalized linear models, the Lusted-Jaynes machine does not assume the existence of a function (possibly contaminated by a little noise) from predictors to predictands. This is a very strong assumption, justifiable in the presence of informationally very rich predictors such as images, but otherwise quite unrealistic for many kinds of predictors considered in medicine, especially those that are more readily available and less invasive and, therefore, more desirable. Second, in contrast to algorithms such as neural networks, random forests, support-vector machines, logistic regression, or generalized linear models, the Lusted-Jaynes machine does not do an optimization during the learning phase, searching for the minimum of some objective function. It does a full hypothesis-space survey. The optimization done by most machine-learning algorithms is an approximate form of this survey, based on the assumption or hope that the most relevant portion of the hypothesis space will be around the extremum (MacKay,1992a;Murphy,2012, ch. 16; see also Self and Cheeseman,1987). The underlying necessity of a more extensive survey, however, becomes manifest in many of the obligatory procedures that go together with the training of most machine-learning algorithms, cross-validation being a prominent example (MacKay,1992b). This leads to a third special characteristic of the Lusted-Jaynes machine: it does not need validation sets, test sets, or other data splits; nor does it need cross-validation procedures. Intuitively this is the case because the underlying hypothesis-space survey realizes a sort of full-fledged cross-validation and data partition. It can indeed be proven that one of the internal computations of the machine is mathematically equivalent to doing k -fold cross-validations for all possible data splits and k(Porta Mana,2019;Fong and Holmes,2020). Such flexibility and informationally rich output come, of course, at a computational cost. Until some years ago, the cost would have been prohibitive in all but the simplest inferential problems. But today an inference problem involving 13 variates and 700 datapoints, such as the example considered in the present work, takes less than six hours of computation on an office computer workstation. We discuss computational limitations further in § 4.2. Psychometrika Submission February 9, 2023 16 fig. 1, for example favouring more unimodal distributions or more multimodal distributions. These simple results show the great usefulness of the Lusted-Jaynes machine for general medical research. 3.1. Patient’s clinical information The 12 predictor values for our four patients are reported in table 2, top. Note that Curtis’s value for the Hippocampal Volume is missing; this is not a problem for the Lusted-Jaynes machine. Given these predictor values the Lusted-Jaynes machine can output any probabilities of interest to the clinician. Table 2, bottom, reports three probabilities that are important for the step of the next subsection:15 • p( cAD =Y|predictors ): the probability that the patient will convert to Alzheimer’s Disease, given the patient’s specific predictors and that the patient comes from the same population as the learning dataset. • p( predictors |cAD =Y ): the probability that a patient who will convert to Alzheimer’s Disease would have these specific predictor values. In other words, the likelihood 16 of conversion to Alzheimer’s Disease, given the predictors. • p( predictors |cAD =N ): the probability that a patient who will remain with stable Mild Cognitive Impairment would have these specific predictor values. In other words, the likelihood of stable Mild Cognitive Impairment, given the predictors. The Lusted-Jaynes machine can also answers other questions of interest to the clinician. For instance, what could be the value of Curtis’s Hippocampal Volume? The answer is given in fig. 4, which also shows the full-population distribution as comparison (dashed grey); with 95% probability Curtis’s value is between 2.8 and 5.3, with a median of 3.8. And what is the frequency of conversion to Alzheimer’s Disease among the subpopulation having Olivia’s, Ariel’s, or Bianca’s predictors? The answer is given in the histogram of fig. 5: with 95% probability, the fraction of this subpopulation that eventually converts to Alzheimer’s Disease is between 0.19 and 0.43; this uncertainty range is due to the limited size of the learning dataset. The probability p( cAD =Y|predictors )is equal to the average of such a distribution (e.g. Bernardo and Smith,2000, §§ 4.2–4.3), provided the patient and dataset can be considered as belonging to the same population. 15 All relative uncertainties of the results caused by numerical computation error are below 0.8%, Curtis’s two likelihoods being an exception at 2%. 16p(A|B)is the probability of Agiven B, as well as the likelihood of Bgiven A(Good,1950, § 6.1). Psychometrika Submission February 9, 2023 17 Olivia Ariel Bianca Curtis Age 75.4 75.4 75.4 63.8 Sex F F F M HV/10−34.26 4.26 4.26 [missing] APOE4 N N N Y ANART 18 18 18 15 CFT 21 21 21 14 GDS 3 3 3 2 RAVLT-imm 36 36 36 20 RAVLT-del 5 5 5 0 RAVLT-rec 10 10 10 3 TMTA 21 21 21 36 TMTB 114 114 114 126 p(cAD =Y|predictors)0.302 0.302 0.302 0.703 p(predictors |cAD =Y)/10−12 8.97 8.97 8.97 1.14 p(predictors |cAD =N)/10−12 18.6 18.6 18.6 0.343 Table 2: Predictor values for the four patients (see § 3.0), and resulting conditional probabilities. 2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 0 50 100 150 200 250 300 350 400 450 500 550 600 650 Curtis's HV probability density Curtis Whole population Figure 4: Probability distribution for Curtis’s Hippocampal Volume (green). The full-population distribution (dashed grey) is also plotted for reference. 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0 6.5 frequency of conversion to AD, given predictor values probability density Figure 5: Probability distribution for the frequency of conversion to ad in the subpopulation having Olivia’s predictors. The red vertical line is the value of the probability p( cAD =Y|predictors ). Psychometrika Submission February 9, 2023 18 3.2. Assessment of relevant subpopulation and auxiliary information Rationale As already mentioned, and as will be argued more concretely in the next section, the clinician needs a probability in order to choose a treatment or other course of action for the current patient. This probability is computed by generalizing associations between predictors and predictand hidden in a dataset of similar patients, as discussed in § 2. The way this generalization is made, however, can differ from patient to patient in two respects: • Only some particular directed associations can be generalized to the current patient, whereas others would be inappropriate to generalize. In some cases, for example when the learning dataset is artificially assembled with balancing or stratification methods, some associations cannot be generalized to any patients at all. • There can be additional information available for the current patient, for instance some clinical predictors not included in the learning dataset, or other “softer” information such as family history or geographic background. There is no sharp separation between these two items. The presence of additional information often automatically implies that some associations cannot be generalized from the learning dataset to the current patient. Psychometrika Submission February 9, 2023 19 Let us explain with a familiar example why particular associations cannot be generalized. Most students of medicine learn about the base-rate fallacy (Bar-Hillel,1980;Jenny et al.,2018;Sprenger and Weinberger, 2021;Matthews,1996). Consider a large set of clinical trials, illustrated in the upper table on the side, where each dot represents, say, 10 000 patients. In this sample dataset it is found that, among patients having a particular value “+” of some predictors (left column), 71.4% of them (or 5/7, upper square) eventually developed a disease. The fallacy lies in judging that a new real patient from the full population, who has predictor value “+”, also has a 71.4% probability of developing that disease. In fact, this probability will in general be different. In our example, it is 33.3% (5/15), as can be seen in the lower table illustrating the full population. This difference would be noticed as soon as the inappropriate probability was used to make prognoses in the full population. A similar situation happens for the predictor value “−”. There is a discrepancy in the conditional frequencies of predictand given predictors, between the sample dataset and the full population, because the proportion of positive vs negative disease cases in the latter has some value, 16.7%/83.3% in our example, whereas the samples for the trials (dashed line in the lower table) were hand-chosen so as to have a 50%/50% proportion. This sampling procedure is called “class balancing” in machine learning (Provost,2000;Drummond and Holte,2005; Weiss and Provost,2003). More generally this discrepancy can appear whenever a population and a sample dataset from it do not have the same frequency distribution for the predictand. In this case, we cannot rely on the probabilities of “predictand given predictors” obtained from the sample dataset, which we symbolically write as p(predictand |predictors,dataset)(1) A little counting in the side figure reveals, however, that other frequencies may be relied upon. Consider the full population. Among all patients who developed the disease, 83.3% of them (or 5/6, upper row) had the predictor value “+”, while among those who did not develop the disease, 33.3% (or 1/3, lower row) had the predictor value “ − ”. And these frequencies are the same in the sample dataset. These frequencies from the clinical trials can therefore be used to make a prognosis using Bayes’s theorem. For brevity, denote the predictors by X , the predictand by Y , the dataset or trials Psychometrika Submission February 9, 2023 20 by D, and the full-population base rate by B. Bayes’s theorem yields p(Y|X, D, B) = p(X|Y, D)·p(Y|B) P Y p(X|Y, D)·p(Y|B)(2) In our example we find p(Y= + |X= +, D, B) = p(X= + |Y= +, D)·p(Y= + |B) p(X= + |Y= +, D)·p(Y= + |B)p(X= + |Y=−, D)·p(Y=− | B) ≈0.833 ·0.167 0.833 ·0.167 + 0.333 ·0.833 = 0.33 (3) which is indeed the correct full-population frequency. If the samples of the clinical trials had been chosen with the same frequencies as the full population (no “class balancing”), then the probability p( predictand |predictors,dataset )from the dataset would be the appropriate one to use. But the probabilities p( predictors |predictand,dataset )together with Bayes’s theorem as in eq. (2) would also lead to exactly the same probability. We thus see that using the probabilities p(predictors |predictand,dataset) from the dataset is preferable to using p( predictand |predictors,dataset ). The former yield the same results as the latter when use of the latter is appropriate, and allow us to apply corrections when use of the latter is inappropriate. The superiority of using p( predictors |predictand,dataset )probabilities (called “generative” in machine learning, see e.g. Murphy,2012, § 8.6) is illustrated with a toy example in table 3. The use of dataset probabilities different from p( predictand |predictors,dataset )can be necessary even when the dataset has statistics identical with the population it is sampled from. Typical cases are the prognosis of a patient that comes from a peculiar subpopulation or even from a different population (Lindley and Novick 1981;Quintana et al. 2017;Sox et al. 2013, ch. 4; Hunink et al. 2014, ch. 5). For instance, the first case happens when the clinician has additional information not included among the predictor variates, such as the result of an additional clinical test, or family history; the second case happens when the patient comes from a different geographical region. There is of course no sharp distinction between these two cases. What is important is that, in either case, it can still be possible to use statistical information from the sample dataset to make prognoses. It is sufficient that some conditional statistics may be applicable to the specific patient. For a patient coming from a different region, for example, it may be Psychometrika Submission February 9, 2023 21 Table 3: superiority of the “predictors | | |predictand” (or “generative”) approach We split our learning dataset into two subsets: •One with 361 subjects and a ratio of 29.9%/70.1% of subjects with cAD =Yvs cAD =N. • One with 343 subjects and a ratio of 63.3%/36.7% of subjects with cAD =Y vs cAD =N . This subset is used as a fictive full population. This partition was made with no systematic sampling of any variates except the predictand cAD. After training on the learning dataset, we make a prognosis for each of the 343 “new” patients, through four separate approaches: (a) using the probabilities p( predictand |predictors,dataset ), as typical of machine-learning algorithms; (b) using p( predictors |predictand,dataset )together with the base rate, as explained above; (c) tossing a coin; (d) always prognosing “ cAD =Y ”, which guarantees 63.3% correct prognoses owing to the base rate of the full population. Finally, the accuracies (number of prognoses giving more than 50% probability to the correct outcome) of these four approaches are calculated. Here are the results from lowest to highest: predictand |predictors coin toss always predict conversion predictors |predictand & base rate 37.3% 50% 63.3% 73.2% The “predictand | predictors” approach (“discriminative” in machine-learning parlance) leads to worse results than a coin toss because of its underlying base-rate fallacy. The “predictors | predictand” approach (“generative” in machine-learning parlance) leads to better results than simply always prognosing the most common base-rate outcome; this shows that the dataset can still provide useful statistical information despite its mismatched base rate. Inference algorithms that only yield “predictand | predictors” outputs, unlike the Lusted-Jaynes machine, are incapable of extracting this useful information. Psychometrika Submission February 9, 2023 22 judged that the conditional probabilities p( predictand |predictors,dataset )still apply. In other words, the patient may still be considered a member of the subpopulation having those specific predictor values. Using more technical language we say that a new patient can be considered exchangeable with the patients constituting the dataset, but only conditional on particular variates. See Lindley (2014, especially around §§ 7.3, 8.6; 1981) for a clear and logically impeccable presentation not obscured by technical language (more technical references are de Finetti 1930,1937;Dawid 2013;Bernardo and Smith 2000, §§ 4.2–4.3, 4.6; see also Malinas and Bigelow 2016,Sprenger and Weinberger 2021 about confounding and Simpson’s paradox, to which this topic is tightly related). This topic is complex and of extreme importance for inference, but its detailed study is not the goal of the present work. Our main point here is that population variability and auxiliary clinical information are important factors that differentiate patients, and a personalized approach ought to take them into account. The method here presented does this naturally, allowing a great flexibility in selecting which statistical features of the sample dataset should be used for each new patient, and the integration of auxiliary clinical information in the form of a prior probability. As discussed in § 3.1, the Lusted-Jaynes machine allows us to quickly calculate conditional probabilities p( Y|X, dataset ) for any desired variate subsets Yand Xrequired by the patient’s relevant population. Application to the example study In our example of § 1.0, all statistics of the dataset are considered relevant for Olivia, Bianca, and Curtis. For these patients the clinician can therefore use Bayes’s theorem with the likelihoods of table 2and the dataset conversion rate of 0 . 463 – or equivalently directly the probabilities p(cAD =Y|predictors,dataset)provided in the same table. For Ariel, however, the clinician judges that a different base rate or prior probability of conversion should be used, equal to 65%, because of her different geographical origin and family history. In her case the clinician uses Bayes’s theorem with the likelihoods of table 2and the prior probability of 0.65. The final probabilities of conversion to Alzheimer’s Disease for our four patients are reported in table 4. Note how the final probability for Ariel is higher than that for Olivia and Bianca, even if the predictor data are the same for these three patients. Psychometrika Submission February 9, 2023 23 Olivia Ariel Bianca Curtis initial probability p(cAD =Y|aux info)0.463 0.65 0.463 0.463 final probability p(cAD =Y|predictors,dataset,aux info)0.302 0.47 0.302 0.703 Table 4: Final probabilities of conversion computed from dataset and auxiliary information 3.3. Assessments of treatments and benefits; final decision Rationale A crucial point in clinical decision-making is this: the clinician needs to assess, not the presence (present or future) of a disease, but the risk of its presence. Is there a difference between these two problems? and why is the difference important? In clinical practice, we can rarely diagnose or prognose a medical condition with full certainty. Perfect classification is therefore impossible. But also a “most probable” classification, which may be enough in other contexts, is inadequate in clinical ones. The problem is that the clinician has to decide among different courses of action, such as different treatments, more tests, and so on, and the optimal one depends on how probable the medical condition is, not just on whether it is more probable than not. Two examples illustrate this point. Suppose there is a dangerous treatment that extends the patient’s lifetime by 1 year if the disease is on its course, but shortens the patient’s lifetime by 5 years if the disease is not present. Also suppose that some algorithm tells the clinician whether the disease’s presence is “more probable than not”, given some predictor values; in which case the clinician administers the dangerous treatment. It turns out that 60 out of 100 treated patients having these same predictor values eventually develop the disease, so “more probable than not” is correct. However, the final result is that the clinician has added 1 × 60 = 60 years but also subtracted 5×40 =240 years from the combined lifespans of the treated patients! The conclusion is that the treatment cannot be prescribed just because the disease is “more probably present than not”. As an opposite example, suppose that a less dangerous treatment extends the patient’s lifespan by five years if the disease is on its course, but shortens it by one month if the disease is not present. In this case, it may be advisable to undergo the treatment even if the disease is less probably present than not. If the clinician administer the treatment to 100 similar patients, and 20 of them develop the disease, then the clinician has added 5 × 20 = 100 and subtracted 1 12 × 60 = 5 years to their combined lifespans. In both examples, it is clearly important to assess the probability – having precise connections with the population frequency – that the patient will develop the disease. In the first example, the Psychometrika Submission February 9, 2023 24 treatment should only be administered if the probability is higher than 83.3%; in the second, it can be administered if the probability is at least 1.6%. The Lusted-Jaynes machine, as explained in the previous sections, tells the clinician the specific probability for the current patient. But the choice between treatments depends not only on the probability of the medical condition. Here is where differences between patients vary and matter the most. Consider again the second example above, about the less dangerous treatment. Let us add that the treatment would extend the lifespan by five years, but would also somewhat worsen the quality of life of the patient and of the patient’s family. Suppose our patient is quite old and tired, has had a happy life, and is now looking with a peaceful mind towards death as a natural part of life. Such a patient may prefer to forego the bother of the treatment and the additional five years, even if the probability for the disease is quite high. The benefits of the different treatments, and the probability thresholds at which one treatment becomes preferable to another, must therefore be judged and quantified primarily by the patient. Utility theory and maximization of expected utility allow clinician and patient to make such judgements and decisions in a coherent way (Sox et al. 2013;Hunink et al. 2014;Lusted 1968; see also the clear and charming exposition by Lindley 1988, and O’Hagan et al. 2006). We summarize the main, patient-dependent procedure for decision-making, and show how our computations so far fit perfectly with it. The clinician first assesses and lists the mutually exclusive courses of action available for the specific patient. These could be preventive or curative treatments, more tests, doing nothing, and so on. Often there are sequences of decisions available, but the utility framework can be applied to them as well (see references above and Raiffa,1970). In the present work we are calling these heterogeneous alternatives simply “treatments” for simplicity (see footnote 2, p. 3). The list treatments is already patient-dependent: some alternatives may not be medically suitable (say, owing to allergies or other clinical conditions), some may be economically too costly, and so on. Each treatment will have different consequences, which additionally depend on the patient’s unknown clinical condition of interest. A treatment may have some consequences if the patient has or will develop the disease, and different consequences otherwise. The patient quantifies, with the clinician’s guidance, the benefits and costs – technically called “utilities” – of such possible consequences. The quantification of utilities is not within the scope of the present work. The references cited above offer guidelines and rules for numerically translating factors such as quality of life and expected lifespan into utilities. The treatments, uncertain clinical conditions, and the quantified utilities U of their consequences can be organized into a table of this form: Psychometrika Submission February 9, 2023 25 clinical condition aclinical condition b. . . treatment αUαa Uαb . . . treatment βUβa Uβb . . . . . . . . . . . . . . . which can be compactly represented by a so-called utility matrix Uij ), the row index i enumerating the treatments, and the column index j the clinical conditions. Note that the number of possible treatments and clinical conditions do not need to be equal; generally, they are not. The expected utility ¯ Ui of a treatment i is calculated as the expectation of its utilities Uia, Uib, . . . with respect to the probabilities p(a),p(b), . . . of the clinical conditions a, b, . . . : ¯ Ui:=Uia p(a) + Uib p(b) + · · · (4) Note that this corresponds to a matrix multiplication between the matrix of utilities and the vector of probabilities. Finally, the recommended treatment is the one having maximal expected utility. Application to the example study At present there are no cures for Alzheimer’s Disease, although some recent pharmacological agents are shown to delay onset of pathology related to Alzheimer’s Disease 17 . But for the sake of our case study let us imagine that in the near future there are three mutually exclusive treatment options for prevention or retardation of the disease; call them β , γ , δ , the simple option of “no treatment” being denoted by α . The clinical conditions to be considered are just two: the patient will convert to Alzheimer’s Disease ( cAD =Y ), or will remain with stable Mild Cognitive Impairment (cAD =N). We have therefore 4 × 2=8possible consequences of the four treatments depending on the two clinical conditions. Our four patients and the clinician quantify the utilities, arriving at the utility matrices shown in table 5, top. Olivia, Ariel, and Curtis quantify the benefits of the treatments in exactly the same way, but Bianca’s quantification differs slightly, because of the interaction of the treatments with several allergies and additional clinical conditions, as explained in § 1.0. The probabilities for the two medical conditions are those found in the previous subsection, table 4. For brevity, we denote just by p( cAD )the probability of conversion given a patient’s predictor values, 17 e.g. lecanemab, a monoclonal antibody infusion given every two weeks, targeting amyloid beta plaques; see https://www.fda.gov/news-events/press-announcements/fda-grants-accelerated-approval-alzheimersdisease-treatment. Psychometrika Submission February 9, 2023 32 0.50 0.54 0.58 0.62 0.66 0.70 0.74 0.78 accuracy (expected utility) APOE4 Sex GDS Age ANART TMTA CFT HC TMTB APOE4+HC+Age+Sex RAVLT−rec RAVLT−imm RAVLT−del all minus RAVLT−del cognitive+Age+Sex all minus RAVLT−imm all minus HC all minus GDS all minus Age all minus APOE4 all minus TMTA all minus Sex all minus CFT all minus ANART all minus RAVLT−rec all minus TMTB all 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 0.18 0.20 0.22 mutual information/Sh GDS Sex APOE4 Age ANART TMTA TMTB CFT HC APOE4+HC+Age+Sex RAVLT−rec RAVLT−imm RAVLT−del all minus RAVLT−del all minus RAVLT−imm all minus TMTB all minus RAVLT−rec all minus HC cognitive+Age+Sex all minus TMTA all minus CFT all minus Age all minus GDS all minus ANART all minus Sex all minus APOE4 all Figure 7: Expected accuracy for the next new patient (left), and mutual information (right), of several sets of predictors for the prognosis of conversion to Alzheimer’s Disease. Each graph has been vertically ordered according to increasing values; the two rankings agree within the respective uncertainties. The all predictor set is mathematically guaranteed to be optimal according to both metrics and has therefore been ranked first. Bars show the uncertainty interval ( ± two standard deviations). 2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Curtis's unknown HV Curtis's probability of conversion to AD Figure 8: Probability of cAD =Y for Curtis, given Curtis’s known predictors and different possible values of his unknown Hippocampal Volume. The thinner curves are 100 probable samples of how this probability would change with a larger learning dataset. Compare this figure with fig. 3, p. 15, bottom-right. Psychometrika Submission February 9, 2023 33 The Lusted-Jaynes machine shows that the omission of any one of the 12 predictors, except RAVLT-del and possibly RAVLT-imm , does not lead to an appreciable decrease in accuracy (relative decrease of 0.3% or less) or in mutual information (relative decrease of less than 3%). This puts the prognostic-importance analysis of Rye et al. (2022) into perspective. The exact quantification of these subtle differences is computationally quite expensive, and we did not carry it out further. 4. Discussion Which requirements does a personalized approach to prognosis and treatment impose on assistive computational technology? This is an important question, because with the increasing amount of statistical clinical data and clinical predictors available for medical care, assistive computational technology is today not merely a useful option, but a necessity in clinical practice. In the present work we started from the perspective of the clinician’s ultimate task, decisionmaking under risk, and saw that patients’ differences relevant to prognosis and treatment can be approximately divided into three categories: • differences in the values – and availability – of a core set of clinical predictors, for which we have population-wide statistical information; • differences in the availability and values of auxiliary and usually semi-quantitative clinical information, such as geographical or family background; • differences in the availability and values or “utilities” of clinical courses of action, such as preventive treatments or further tests; such values can have a highly variable, patient-dependent subjective component. Luckily there is a theory that takes into account and integrates these differences towards the final goal: Decision Theory, which is the subject of several good textbooks on clinical decision-making (Weinstein and Fineberg,1980;Sox et al.,2013;Hunink et al.,2014) after the pioneering work of Lesley & Lusted (1959a;1959b;1960;1960;1968) (a summary and references were given in § 3.3). Decision-making under risk requires any assistive algorithm to work, explicitly or implicitly, in terms of probabilities, having precise connections with population statistics (§ 3.3). Without this condition the integration of patient-dependent treatment utilities would be impossible. The handling of these probabilities should moreover be enough flexible to take into account peculiar but common subpopulations of patients having special contexts or auxiliary information (§ 3.2), and the common possibility of missing values for some clinical predictors (§ 3.1). Most, if not all, popular machine-learning algorithms either do not meet these requirements, or they do so at the Psychometrika Submission February 9, 2023 34 Table 6: Summary of the clinician’s patient-dependent inputs and the Lusted-Jaynes machine’s outputs. Input data and final results that distinguish Ariel, Bianca, Curtis from Olivia are in red. Olivia Ariel Bianca Curtis Clinician’s patient-dependent inputs Predictor values Age 75.4 75.4 75.4 63.8 Sex F F F M HV/10−34.26 4.26 4.26 [missing] APOE4 NNNY ANART 18 18 18 15 CFT 21 21 21 14 GDS 3 3 3 2 RAVLT-imm 36 36 36 20 RAVLT-del 5 5 5 0 RAVLT-rec 10 10 10 3 TMTA 21 21 21 36 TMTB 114 114 114 126 Additional information auxiliary info none family history, base rate none none applicable dataset subpopulation all predictor |predictand all all prior probability of conversion 0.463 0.65 0.463 0.463 Available treatments and utilities treatment α treatment β treatment γ treatment δ cAD N Y     10 0 9 3 8 5 0 10     cAD N Y     10 0 9 3 8 5 0 10     cAD N Y     10 0 8 3 7 5 0 10     cAD N Y     10 0 9 3 8 5 0 10     Outputs of Lusted-Jaynes machine p(cAD =Y|predictors,dataset)0.302 0.302 0.302 0.703 p(predictors |cAD =Y,dataset)/10−12 8.97 8.97 8.97 1.14 p(predictors |cAD =N,dataset)/10−12 18.6 18.6 18.6 0.343 final probability of conversion p( cAD =Y|predictors,dataset,aux info ) 0.302 0.47 0.302 0.703 exp. utility treatment α exp. utility treatment β exp. utility treatment γ exp. utility treatment δ Optimal treatment 6.98 7.19 7.09 3.02 β 5.27 6.16 6.58 4.73 γ 6.98 6.49 6.40 3.02 α 2.97 4.78 5.89 7.03 δ Psychometrika Submission February 9, 2023 35 cost of unrealistic modelling assumptions. Unfortunately they tend to overly simplify the problem of decision-making under risk, as if it were a simple classification or regression task. We presented an assistive algorithm, the “Lusted-Jaynes machine”, that meets all these requirements (§ 2) and carries out the calculations required by decision theory. This algorithm is moreover model-free, not making a-priori assumptions about functional dependencies or particular distributions in the variates. The inference principles on which it is based have recently been recommended for the study of Alzheimer’s Disease (Temp et al.,2021; see also asa,2016,2019), and have been successfully demonstrated in a simpler predictor setting (Antoniano-Villalobos et al.,2014). We showed its application in an example of prognosis and treatment of conversion from Mild Cognitive Impairment to Alzheimer’s Disease for four different patients, where all three categories of differences listed above appeared. The patients were fictitious but the underlying learning database, originating from adni, is real, and was explored in a previous work (Rye et al.,2022). The Lusted-Jaynes machine was also shown to have uses that go beyond individual clinical decision-making but are still of importance to personalized medicine. For instance, it can assess the maximum possible prognostic power of particular sets of predictors, potentially allowing us to discard clinical predictors that are too invasive or expensive and yet prognostically unimportant. In actual deployment, we would recommend the hospital, medical centre, or clinician using a Lusted-Jaynes machine to keep a database of incoming patients, with their predictor values, adding the true values of their predictand later in time, once they become known. The Lusted-Jaynes machine can then be retrained on such local database when the latter reaches a size comparable to the original one’s, and periodically retrained afterwards. All inferences would thus become increasingly more reliable, because the machine would base them on updated population statistics that are characteristic to the specific hospital. 4.1. Counters to possible critiques Any inference or decision-making algorithm aspiring to take into account patient differences must perforce have some open “input slots” for such differences. We saw that the Lusted-Jaynes machine requires inputs about a patient’s specific predictors, relevant statistical relations and auxiliary data, and treatment utilities. The most difficult input to quantify is probably the third: translating benefits and drawbacks of different treatments into numbers. On this complex topic we refer the reader to specially dedicated textbooks on clinical decision making, for example Sox et al.’s (2013) and Hunink et al.’s (2014). But some readers may wonder: “can all these additional inputs be avoided?”, fearing that errors could sneak in through them. Psychometrika Submission February 9, 2023 36 This question is answered by a mathematical theorem at the very core of decision theory 20 , which is too seldom emphasized: Any decision we make, either (A) comes explicitly or implicitly through some set of utilities and maximization of their expectations, or (B) is logically inconsistent. There is no third alternative. Thus the choice is not between using utilities or not using utilities, but between choosing them explicitly or letting them be chosen in a way we do not know. If we use a decision-making algorithm that does not ask us for utilities, then the algorithm is internally supplying utilities not chosen by us (and probably divorced from our specific problem), or, worse, is committing logical inconsistencies. The first advantage of explicitly operating through utilities, probabilities, decision theory, is that we are, at the very least, sure of not acting in a self-contradictory way. The second advantage is that the utilities used to arrive at a decision appear openly in front of us. We can analyse and change them if we find them inappropriate to a specific problem. If they are hidden, it is more difficult to analyse which are inappropriate and how they should be changed. The fact that an algorithm works according to decision theory is also an assurance of striving towards theoretical optimality. This point has very subtle consequences. Consider a non-optimal algorithm that leads to saving 85 000 patients out of 100 000. Given these numbers it might be deemed a success. But what if a theoretically optimal algorithm leading to 95 000 saved patients is feasible? What shall we say to the families of the 10 000 patients who could have been saved but weren’t? The most difficult input to quantify is probably the third: translating benefits and drawbacks of different treatments into numbers. On this complex topic we refer the reader to specially dedicated textbooks on clinical decision making, for example Sox et al.’s (2013) and Hunink et al.’s (2014). 4.2. Range of application of Lusted-Jaynes machines First let us emphasize, even if it is obvious, that the quality of the results obtained with the Lusted-Jaynes machine depends on the quality of the learning dataset. Any peculiar sampling biases (or numerical errors) in the dataset that are unknown to the clinician will affect the final results. This is of course true for any inference algorithm. But we saw that the Lusted-Jaynes machine allows the clinician to correct for particular sampling biases present in the dataset, if they are known. The range of application of the Lusted-Jaynes machine has two kinds of bounds: computational and theoretical. The fact that the Lusted-Jaynes machine extracts all available information from the dataset makes it computationally expensive (see § 2). At present it cannot be used with high-dimensional 20 Savage (1972); Luce and Raiffa (1957); Raiffa and Schlaifer (2000); Atkinson et al. (1964); Ferguson (1967); Lindley (1988,1977); Kreps (1988); Bernardo and Smith (2000); Pratt et al. (1996); Lindley (2014); Pettigrew (2019) Psychometrika Submission February 9, 2023 37 predictors: if our dataset had included a predictor such as a 128 × 128 × 128 greyscale mri image, the learning stage would have taken around 100 years. Approximate but much faster algorithms such as neural networks and random forests are thus, at present, still the only options with such predictors. There is, however, the interesting possibility of combining these fast algorithms together with a Lusted-Jaynes machine, as a post-processor of their raw output. The machine extracts useful information usually hidden in their output at a low computational cost (Dyrland et al.,2022b); this information can then be used for clinical decision-making as illustrated in the present work. The sole assumption underlying the Lusted-Jaynes machine’s inference and its practical use with new patients, is that the latter can be assumed to come, at least in some respects, from the same population as the learning dataset (in probability-theory jargon, partial or conditional exchangeability applies; see § 3.2). This precludes using the Lusted-Jaynes machine to forecast how the statistics of the full population could change in the future. However, the machine can be used for time-dependent (longitudinal) inferences within a stable population, such as forecasts of the future time of disease onset, expected lifelength, and similar. For instance, if data about the time of conversion to Alzheimer’s Disease were available in the dataset, the Lusted-Jaynes machine could forecast not only whether, but also when the conversion could take place (cf. e.g. De la Cruz-Mesía et al.,2007). Finally, the machine is not meant to handle sequences of decisions in a clinical decision tree (Sox et al.,2013, ch. 6; Hunink et al.,2014, ch. 1) – it would be impossible in a personalized approach, because such a tree is fully patient-dependent – but it could be used in individual decision branches. A. Further mathematical and computational details about the example application of the Lusted-Jaynes machine The Lusted-Jaynes machine surveys the space of possible distributions of frequencies of all 13 variates discussed in § 3.0, for the full population of patients from which the dataset originates (see §3.2). In the present study, it does so by mathematically representing a generic joint frequency distribution F(Y, X1, X2, . . . )as a convex mixture of appropriate kernels products: F(Y, X1, X2, . . . ) = X i wiK(Y|υi)K(X1|ξi 1)K(X2|ξi 2)· · · , along the ideas in Dunson and Bhattacharya (2011) and Ishwaran and Zarepour (2002); see also Rossi (2014); Rasmussen (1999). This representation uses a total of 1535 independent parameters ( wi,υi,ξi 1, . . . ), with roughly 190 parameters for each continuous or integer variate. As a crude intuition, it is as if we divided the range of each variate into 190 bins, and considered all possible frequency histograms over these. The actual parametrization is smarter, incrementally using Psychometrika Submission February 9, 2023 38 parameters to represent less and less smooth traits of the distribution. We indeed expect the distribution for a full population to have some degree of smoothness, owing to physical and biological reasons. Actually, the number of parameters used is in principle infinite, because the machine gives a warning if the data indicate that more parameters are needed. In the present study, the data indicate, on the contrary, that fewer than 250 parameters would be enough. Note that the machine constructs the kernels K ( | )and their product automatically, depending on how many and what kinds of variates the dataset comprises. The probability of a candidate frequency distribution F is determined by its “fit” F ( D )of the data Dand a prior-expectation factor p(F), as explained in § 2.2: p(F|D)∝F(D) p(F). Finally, the predictive conditional probability for any two sets of variates Z′, Z′′ , given the dataset, is given by the expectation over the unknown F , as required by the probability calculus and de Finetti’s theorem (Bernardo and Smith,2000, § 4.6): p(Z′|Z′′, D) = ZF(Z′|Z′′) p(F|D) dF where the conditional frequencies F(Z′|Z′′):=F(Z′, Z′′)/F(Z′′). The frequency-space survey and the calculation of the probabilities p( F|D )for the populationfrequency distributions was done via Gibbs sampling (Neal,1993, ch. 4; MacKay,2005, § 29.5; Casella and George,1992) with the R package Nimble (de Valpine et al.,2021), using 1024 independent Markov chains. Stationarity was assessed by common diagnostic measures (Gilks et al.,1998), especially integrated autocorrelation time (Christen and Fox,2010) and Hellinger distance (Boone et al.,2014), as well as visual inspection. An automated method for stationarity check was developed, to be discussed in future publications. Conflict of Interest Statement The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. Author Contributions The authors were too immersed in the development of the present work to keep a detailed record of who did what. Psychometrika Submission February 9, 2023 39 Funding The study was supported by grants from the Trond Mohn Research Foundation, grant number BFS2018TMT0, and from The Research Council of Norway, project number 294594. Acknowledgements PGLPM thanks Soledad Gonzalo Cogno and Iván Davidovich for inspiring discussions; Maja, Mari, Miri, Emma for continuous encouragement and affection; Buster Keaton and Saitama for filling life with awe and inspiration; and the developers and maintainers of Nimble, L A T E X, Emacs, AUCT E X, Open Science Framework, R, Inkscape, LibreOffice, Sci-Hub for making a free and impartial scientific exchange possible. Computations underlying the Lusted-Jaynes machine were initially performed on resources provided by Sigma2 – the National Infrastructure for High Performance Computing and Data Storage in Norway (project NN8050K). Clinical-data collection and sharing for this project was funded by the Alzheimer’s Disease Neuroimaging Initiative (ADNI) (National Institutes of Health Grant U01 AG024904) and DOD ADNI (Department of Defense award number W81XWH-12-2-0012). ADNI is funded by the National Institute on Aging, the National Institute of Biomedical Imaging and Bioengineering, and through generous contributions from the following: AbbVie, Alzheimer’s Association; Alzheimer’s Drug Discovery Foundation; Araclon Biotech; BioClinica, Inc.; Biogen; Bristol-Myers Squibb Company; CereSpir, Inc.; Cogstate; Eisai Inc.; Elan Pharmaceuticals, Inc.; Eli Lilly and Company; EuroImmun; F. Hoffmann-La Roche Ltd and its affiliated company Genentech, Inc.; Fujirebio; GE Healthcare; IXICO Ltd.; Janssen Alzheimer Immunotherapy Research Development, LLC.; Johnson & Johnson Pharmaceutical Research Development LLC.; Lumosity; Lundbeck; Merck Co., Inc.; Meso Scale Diagnostics, LLC.; NeuroRx Research; Neurotrack Technologies; Novartis Pharmaceuticals Corporation; Pfizer Inc.; Piramal Imaging; Servier; Takeda Pharmaceutical Company; and Transition Therapeutics. The Canadian Institutes of Health Research is providing funds to support ADNI clinical sites in Canada. Private sector contributions are facilitated by the Foundation for the National Institutes of Health (www.fnih.org). The grantee organization is the Northern California Institute for Research and Education, and the study is coordinated by the Alzheimer’s Therapeutic Research Institute at the University of Southern California. ADNI data are disseminated by the Laboratory for Neuro Imaging at the University of Southern California. Software-availability Statement The R scripts used for this study can be found in the Open Science Framework project doi: 10.17605/osf.io/zb26t (see also the repository https://github.com/pglpm/ledley-jaynes_mac Psychometrika Submission February 9, 2023 40 hine). We hope to assemble them into an R package soon. References Alvarez-Melis, D. and Broderick, T. (2015). A translation of “The characteristic function of a random phenomenon” by Bruno de Finetti. arXiv doi: 10.48550/arXiv.1512.01229 . Transl. of de Finetti (1929) Antoniano-Villalobos, I., Wade, S., and Walker, S. G. (2014). A Bayesian nonparametric regression model with normalized weights: A study of hippocampal atrophy in Alzheimer’s disease. J. Am. Stat. Assoc. 109, 477–490. doi:10.1080/01621459.2013.879061 Arnborg, S. and Sjödin, G. (2001). On the foundations of Bayesianism. Am. Inst. Phys. Conf. Proc. 568, 61–71. doi: 10.1063/1.1381871 , http://www.stats.org.uk/bayesian/ArnborgSjodin200 1.pdf asa (2016). ASA statement on statistical significance and p -values. Am. Stat. 70, 131–133. Ed. by R. L. Wasserstein. doi: 10.1080/00031305.2016.1154108 . See also introductory editorial in Wasserstein and Lazar (2016) and discussion in Greenland et al. (2016) asa (2019). Moving to a world beyond “ p < 0 . 05”. Am. Stat. 73, 1–19. Ed. by R. L. Wasserstein, A. L. Schirm, N. A. Lazar. doi:10.1080/00031305.2019.1583913 Atkinson, F. V., Church, J. D., and Harris, B. (1964). Decision procedures for finite decision problems under complete ignorance. Ann. Math. Stat. 35, 1644–1655 Bar-Hillel, M. (1980). The base-rate fallacy in probability judgments. Acta Psychol. 44, 211–233. doi:10.1016/0001-6918(80)90046-3 Bernardo, J. M., Bayarri, M. J., Berger, J. O., Dawid, A. P., Heckerman, D., Smith, A. F. M., et al. (eds.) (2011). Bayesian Statistics 9 (Oxford: Oxford University Press). doi: 10.1093/acprof: oso/9780199694587.001.0001 Bernardo, J.-M. and Smith, A. F. (2000). Bayesian Theory (New York: Wiley), repr. edn. doi: 10.1002/9780470316870. First publ. 1994 Bhattacharya, A. and Dunson, D. B. (2010). Nonparametric Bayesian density estimation on manifolds with applications to planar shapes. Biometrika 97, 851–865. doi:10.1093/biomet/asq044 Boone, E. L., Merrick, J. R., and Krachey, M. J. (2014). A Hellinger distance approach to MCMC diagnostics. J. Statist. Comput. Simul. 8, 833–849 Psychometrika Submission February 9, 2023 41 Casella, G. and George, E. I. (1992). Explaining the Gibbs sampler. Am. Stat. 46, 167–174. doi: 10.1080/00031305.1992.10475878 Cherry, C. (ed.) (1961). Information Theory (London: Butterworths) Christen, J. A. and Fox, C. (2010). A general purpose sampling algorithm for continuous distributions (the t-walk). Bayesian Anal. 5, 263–281 Clayton, A. and Waddington, T. (2017). Bridging the intuition gap in Cox’s theorem: A Jaynesian argument for universality. Int. J. Approximate Reasoning 80, 36–51. doi: 10.1016/j.ijar.2016. 08.002 Cover, T. M. and Thomas, J. A. (2006). Elements of Information Theory (Hoboken, USA: Wiley), 2 edn. doi:10.1002/0471200611. First publ. 1991 Cox, R. T. (1946). Probability, frequency, and reasonable expectation. Am. J. Phys. 14, 1–13. doi: 10.1119/1.1990764 Cox, R. T. (1961). The Algebra of Probable Inference (Baltimore: The Johns Hopkins Press) Damien, P., Dellaportas, P., Polson, N. G., and Stephens, D. A. (eds.) (2013). Bayesian Theory and Applications (Oxford: Oxford University Press). doi: 10.1093/acprof:oso/9780199695607.001. 0001 Dawid, A. P. (2013). Exchangeability and its ramifications. In Damien et al. (2013), chap. ch. 2. 19–29. doi:10.1093/acprof:oso/9780199695607.003.0002 de Finetti, B. (1929). Funzione caratteristica di un fenomeno aleatorio. In Atti del Congresso Internazionale dei Matematici:, ed. S. Pincherle (Bologna: Zanichelli), vol. 6. 179–190. https: //www.mathunion.org/icm/proceedings , http://www.brunodefinetti.it/Opere.htm . Transl. in Alvarez-Melis and Broderick (2015). See also de Finetti (1930) de Finetti, B. (1930). Funzione caratteristica di un fenomeno aleatorio. Atti Accad. Lincei: Sc. Fis. Mat. Nat. IV, 86–133. http://www.brunodefinetti.it/Opere.htm . Summary in de Finetti (1929) de Finetti, B. (1937). La prévision: ses lois logiques, ses sources subjectives. Ann. Inst. Henri Poincaré 7, 1–68. http://www.numdam.org/item/AIHP_1937__7_1_1_0 . Transl. in Kyburg and Smokler (1980), pp. 53–118, by Henry E. Kyburg, Jr. Psychometrika Submission February 9, 2023 48 Neal, R. M. (1993). Probabilistic Inference using Markov Chain Monte Carlo Methods. Tech. Rep. CRG-TR-93-1, University of Toronto, Toronto. http://www.cs.utoronto.ca/~radford/review. abstract.html,https://omega0.xyz/omega8008/neal.pdf O’Hagan, A., Buck, C. E., Daneshkhah, A., Eiser, J. R., Garthwaite, P. H., Jenkinson, D. J., et al. (2006). Uncertain Judgements: Eliciting Experts’ Probabilities (Chichester: Wiley). doi: 10.1002/0470033312 Paris, J. B. (2006). The Uncertain Reasoner’s Companion: A Mathematical Perspective (Cambridge: Cambridge University Press), repr. edn. doi: 10.1017/CBO9780511526596 . See also Snow (1998) Petersen, R. C., Aisen, P. S., Beckett, L. A., Donohue, M. C., Gamst, A. C., Harvey, D. J., et al. (2010). Alzheimer’s Disease Neuroimaging Initiative (ADNI): Clinical characterization. Neurology 74, 201–209. doi:10.1212/WNL.0b013e3181cb3e25 Pettigrew, R. (2019). Epistemic utility arguments for probabilism. In Stanford Encyclopedia of Philosophy, ed. E. N. Zalta (Stanford: The Metaphysics Research Lab). https://plato.stanfo rd.edu/archives/win2019/entries/epistemic-utility. First publ. 2011 Pólya, G. (1954). Mathematics and Plausible Reasoning: Vol. I: Induction and Analogy in Mathematics (Princeton: Princeton University Press). https://archive.org/details/Induction_And_ Analogy_In_Mathematics_1_,doi:10.1515/9780691218304 Pólya, G. (1968). Mathematics and Plausible Reasoning: Vol. II: Patterns of Plausible Inference (Princeton: Princeton University Press), 2 edn. First publ. 1954 Porta Mana, P. G. L. (2019). A relation between log-likelihood and cross-validation log-scores. Open Science Framework doi: 10.31219/osf.io/k8mj3 ,hal: hal-02267943 , arXiv doi: 10.48550/arX iv.1908.08741 Pratt, J. W. (1961). Book review: Testing Statistical Hypotheses, E. L. Lehmann. J. Am. Stat. Assoc. 56, 163–167 Pratt, J. W., Raiffa, H., and Schlaifer, R. (1996). Introduction to Statistical Decision Theory (Cambridge, USA: MIT Press), 2nd pr. edn. First publ. 1995 Provost, F. (2000). Machine Learning from Imbalanced Data Sets 101. Tech. Rep. WS-00-05-001, AAAI, Menlo Park, USA. https://aaai.org/Library/Workshops/2000/ws00-05-001.php Psychometrika Submission February 9, 2023 49 Quintana, M., Viele, K., and Lewis, R. J. (2017). Bayesian analysis: Using prior information to interpret the results of clinical trials. J. Am. Med. Assoc. 318, 1605–1606. doi: 10.1001/jama.201 7.15574 R Core Team (2023). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna. https://www.R-project.org. First released 1995 Raiffa, H. (1970). Decision Analysis: Introductory Lectures on Choices under Uncertainty (Reading, USA: Addison-Wesley), 2nd pr. edn. First publ. 1968 Raiffa, H. and Schlaifer, R. (2000). Applied Statistical Decision Theory (New York: Wiley), repr. edn. First publ. 1961 Rasmussen, C. E. (1999). The infinite Gaussian mixture model. Adv. Neural Inf. Process. Syst. (NIPS) 12, 554–560. https://www.seas.harvard.edu/courses/cs281/papers/rasmussen-1999a.pdf Rodríguez, A., Dunson, D. B., and Gelfand, A. E. (2009). Bayesian nonparametric functional data analysis through density estimation. Biometrika 96, 149–162. doi:10.1093/biomet/asn054 Rosenkrantz, R. D. (1977). Inference, Method and Decision: Towards a Bayesian Philosophy of Science (Dordrecht: Reidel) Rossi, P. E. (2014). Bayesian Nonand Semi-parametric Methods and Applications (Princeton: Princeton University Press). doi:10.1515/9781400850303 Russell, S. J. and Norvig, P. (2022). Artificial Intelligence: A Modern Approach (Harlow, UK: Pearson), fourth global ed. edn. http://aima.cs.berkeley.edu/global-index.html , https: //archive.org/details/artificial-intelligence-a-modern-approach-4th-edition . First publ. 1995 Rye, I., Vik, A., Kocinski, M., Lundervold, A. S., and Lundervold, A. J. (2022). Predicting conversion to Alzheimer’s disease in individuals with Mild Cognitive Impairment using clinically transferable features. Sci. Rep. 12, 15566. doi:10.1038/s41598-022-18805-5 Savage, L. J. (1972). The Foundations of Statistics (New York: Dover), 2nd rev. and enl. ed. edn. First publ. 1954 Savage, L. J., Bartlett, M. S., Barnard, G. A., Cox, D. R., Pearson, E. S., and Smith, C. A. B. (1962). The Foundations of Statistical Inference: A Discussion (London: Methuen). With a discussion including H. Ruben, I. J. Good, D. V. Lindley, P. Armitage, C. B. Winsten, R. Syski, E. D. Van Rest, G. M. Jenkins Psychometrika Submission February 9, 2023 50 Self, M. and Cheeseman, P. C. (1987). Bayesian prediction for artificial intelligence. In Proceedings of the Third Conference on Uncertainty in Artificial Intelligence (UAI’87), eds. J. Lemmer, T. Levitt, and L. Kanal (Arlington, USA: AUAI Press). 61–69. Repr. in arXiv doi: 10.48550/arXiv.1304. 2717 Shah, S., Beck, J. R., and Pauker, S. G. (2013). In memoriam: Robert Steven Ledley, DDS, MS (physics), 1926–2012. Med. Decis. Making 33, 731–733. doi:10.1177/0272989X1348794 Shannon, C. E. (1948). A mathematical theory of communication. Bell Syst. Tech. J. 27, 379–423, 623– 656. https://archive.org/details/bstj27-3-379 , https://archive.org/details/bstj274-623,http://math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf Smith, J. E. and Winkler, R. L. (2006). The optimizer’s curse: Skepticism and postdecision surprise in decision analysis. Manag. Sci. 52. doi:10.1287/mnsc.1050.0451 Snow, P. (1998). On the correctness and reasonableness of Cox’s theorem for finite domains. Comput. Intell. 14, 452–459. doi:10.1111/0824-7935.00070 Snow, P. (2001). The reasonableness of possibility from the perspective of Cox. Comput. Intell. 17, 178–192. doi:10.1111/0824-7935.00138 Sox, H. C., Higgins, M. C., and Owens, D. K. (2013). Medical Decision Making (New York: Wiley), 2 edn. doi:10.1002/9781118341544. First publ. 1988 Sprenger, J. and Weinberger, N. (2021). Simpson’s paradox. In Stanford Encyclopedia of Philosophy, ed. E. N. Zalta (Stanford: The Metaphysics Research Lab). https://plato.stanford.edu/arc hives/sum2021/entries/paradox-simpson Temp, A. G. M., Lutz, M. W., Trepel, D., Tang, Y., Wagenmakers, E.-J., Khachaturian, A. S., et al. (2021). How Bayesian statistics may help answer some of the controversial questions in clinical research on Alzheimer’s disease. Alzheimer’s Dement. 17, 917–919. doi:10.1002/alz.12374 Tribus, M. (1969). Rational Descriptions, Decisions and Designs (New York: Pergamon). doi: 10.1016/C2013-0-01558-7 Van Horn, K. S. (2003). Constructing a logic of plausible inference: a guide to Cox’s theorem. Int. J. Approximate Reasoning 34, 3–24. doi:10.1016/S0888-613X(03)00051-3 von Neumann, J. and Morgenstern, O. (1955). Theory of Games and Economic Behavior (Princeton: Princeton University Press), 3rd ed., 6th pr. edn. https://archive.org/details/in.ernet.d li.2015.215284. First publ. 1944 Psychometrika Submission February 9, 2023 51 Walker, S. G. (2010). Bayesian nonparametric methods: motivation and ideas. In Hjort et al. (2010), chap. ch. 1. 22–34 Wasserstein, R. L. and Lazar, N. A. (2016). The ASA’s statement on p -values: Context, process, and purpose. Am. Stat. 70, 129–133. doi: 10.1080/00031305.2016.1154108 . See asa (2016) and discussion in Greenland et al. (2016) Weinstein, M. C. and Fineberg, H. V. (1980). Clinical Decision Analysis (Philadelphia: Saunders) Weiss, G. M. and Provost, F. (2003). Learning when training data are costly: The effect of class distribution on tree induction. J. Artif. Intell. Res. 19, 315–354. doi:10.1613/jair.1199 Woodward, P. M. (1964). Probability and Information Theory, with Applications to Radar (Oxford: Pergamon), 2 edn. doi:10.1016/C2013-0-05390-X. First publ. 1953