Statistical Literacy: Introducting Observational Causation

Statistical Literacy: Introducing Observational Causation Milo Schield New College of Florida. Sarasota FL Abstract Most students are more interested in causal inference than in population inference. Most students are more interested in observational causation than in randomized-experimental causation. They want to use observational statistics as evidence for causal connections. This paper summarizes the basic quantitative needs of today's students. It presents three common techniques for analyzing observational data: multivariate OLS regressions, Rubins causal method and Pearl’s Directed Acyclic Diagrams. This paper argues that collectively they have some unique conceptual prerequisites. Six prerequisite topics or skills are identified. This paper argues that statistical educators need to support the development and teaching of an introductory course designed to help today’s students deal with claims involving observational causation. Statistical Literacy is proposed as one such course. Key Words: Observational studies, statistical education 1. Our Mission ASA Fellow Richard De Veaux claimed that statistical educators are “teaching the wrong things, the wrong way, in the wrong order.” (Rossman, 2016). Assuming this is a true, consider this followup question. “How do statistical educators decide what to teach?”. The answer is simple: they ask faculty in their client disciplines. Client disciplines are those majors that require a statistics course. Why are we doing this? Of those taking introductory statistics, most are not majoring in statistics. Thus, statistical educators typically ask the faculty in client disciplines what statistical topics and techniques these faculty want their students to know. Claim #1: This is the wrong question. By asking what techniques and topics client disciplines want their students to know, their answers are typically restricted to existing topics and techniques— typically those involving chance. This is arguably measurement bias. Asking the wrong question limits the utility of the answer. Claim #2: This is the wrong audience. By focusing on client disciplines, this selection ignores those in the non-quantitative disciplines: those disciplines that do not require a statistics course. This is arguably selection bias. Statistical educators should be investigating the statistical needs of students in non-quantitative majors. As to the right question, statistical educators be asking, “What are your goals in using statistics? What do use statistics for?” Some faculty in quantitative majors may say ‘generalization’ (confidence intervals) and ‘decision making’ (statistical inference). Others may say ‘prediction’. But if questioning just this group involves selection bias, it may also introduce measurement bias. To avoid both measurement and selection bias, consider how statistics are used in the everyday media. In a study of 160 articles, 79% of the articles were based on samples, but only 3% presented a p-value (1% gave a margin of error). In a study of 850 articles, only 3% used ‘statistically significant’. (Raymond and Schield, 2008). What was more common? Causation. Of the 160 news stories, 62% were judged to use association as evidence for causation. In the study of 850 news stories, 67% used causal grammar. What caused an observed association? How did it happen? Solving the problem of causation is easiest in ideal experiments where scientists can manipulate and repeat on the same or similar subject. Solving the problem of causation is most difficult when scientists can only observe. Yet much, if not most, of human knowledge is obtained by simply observing. The search for causation is fundamental to science, technology, philosophy and in everyday life. The search is simplest when a condition or event is both necessary and sufficient for a given outcome in a controlled situation where the treatment can be repeated at will on the same subject in the same condition or the subjects are homogeneous. Such situations seldom involve people where related factors confound the analysis. Fisher statistically controlled for these confounders by using random assignment to give the treatment and control groups statistically similar mixtures of each confounder. Such situations are either impossible, illegal or immoral in many situations. Observational studies are central to disciplines such as sociology, social work, economics, business, and social psychology. Observational studies are central to some of the social sciences (sociology and economics), the physical sciences (astronomy, geology, archeology, epidemiology, wildlife biology and space physics), and the professions (business, education, nursing and social work). Indeed, observational studies have been the basis for much – if not most – of our knowledge. For additional insight, note the information presented in the appendices to this paper. If statistical educators are to meet the quantitative needs of college students, they need to rethink their mission using this kind of data. 2. Analyzing Observational Causation What techniques are used to analyze observational causation. Arguably, here are the three leading approaches: • Multivariable OLS regression • Rubin’s Propensity Scores • Pearl’s Directed Acyclic Graphs (DAG) These three methods were united in a grant funded by the W. M. Keck Foundation. Schield was the project PI. As part of that grant, Rubin attempted to teach his method to Harvard undergraduates. According to his TA’s, that presentation was too advanced. Students lacked the foundation to deal with the ideas being presented. Pearl presented his methods in a workshop/seminar at UCLA. In talking with the attendees (most of whom had advanced degrees), the most common response was that they had difficulty following the presentation. Again, they lacked the foundation to deal with the ideas being presented. Schield focused on developing materials that would provide the foundation for students to analyze and evaluate statistics obtained from observational studies and thereby prepare students for the more advanced topics and methods involved in analyzing observational causation. (Schield, 2001). 3. Necessary Ideas and Skills This paper (with the hindsight of almost 25 years) tries to identify what foundation is necessary for college students to deal with observational causation – and to be able to study these various approaches. Here are six key topics: #1: Confounding. “The greatest challenge to drawing causal inferences in observational studies is the existence of potential confounding variables, not all of which can be specified, measured, or modeled.” (Ejima et al., 2020) “Confounding and variation are two major obstacles in analyzing data.” (Tintle et al, 2021) Confounding is the ‘elephant’ in statistical education. 80% of 100 statistics textbooks did not include confound (or its derivative forms) in the index. (Schield, 2021a). Students don’t understand ‘confound,’ ‘confounded’, confounding or confounder. Most students have never used these words. Most students have never read or heard these words. They may understand that a confounder can confuse. But other things can confuse. They don’t understand what is different about how a confounder confuses. #2: Study Design: Observational studies vs. randomly controlled trials (RCT). Although observational studies are much more common than randomly controlled trials, the latter is more commonly taught by statistical educators. Here is Fisher’s comment about observational studies, “That’s not an experiment; that’s an experience.” Moore (1998) said, “I will say just one big thing: We are still too narrow. Our teaching has moved in the right direction. Yet we still often take big ideas for granted in our rush to present technical material. We less often commit the mortal sin of omitting the distinction between observation and experiment.” #3: Conditional Probability: this is tricky. Small changes in wording (syntax) can create big changes in meaning (semantics). Suppose a pie chart of US teenagers shows a 12% slice labelled “suicide”. Does that mean that 12% of US teens die from suicide? No. More likely it means that “12% of US teen deaths involve suicide.” The placement of die or deaths is critical. Putting ‘deaths in the subject makes it the whole; putting ‘die’ in the predicate make it part. How could the birth rate be increasing and decreasing for the same country at the same time. Consider these two conditional probabilities: (1) The percentage of women 15-24 who gave birth. (2) The percentage of women who gave birth. The first could be increasing over time (women are having more children) while the second might be decreasing (women are living longer). #4: Confusion of the Inverse. “The basic problem is that people confuse the conditional probability P(A|B) with the conditional probability P (B|A).” Utts (2003) This is one of the seven topics that educated citizens should know. Moore (2001) argued that “it is, at best, premature to teach the ideas and methods of Bayesian inference” in a first statistics. In the Q&A following his dialog with Berry, Moore argued that the difficulties with conditional probability and the confusion of the inverse were big reasons for not teaching Bayesian methods. #5: Take into account (control for) something quantitatively. Most students have no idea of what this means. They aren’t likely to encounter this idea until they take Statistics 201 or Econometrics. And even then, they may have no idea of what is happening in the process. They enter data into the computer, a program does something, and the student views the results. Students just memorize this phrase. #6: Work multivariable problems. Math teachers recognize that students don’t really understand a mathematical idea until they can work problems. Students need to be able to work multivariable problems without computers and algebra to really understand what it means to ‘take something into account (control for something) quantitatively. Working multivariable problems in statistics is arguably more valuable for many (if not most) college students than college algebra. 3. What is Needed A different course – a very different course – is needed to prepare students to think critically about observational causation. It must avoid the extremes of cynicism and naivete. It should be accessible to almost all college students. Moore (2001) argued that “Thinking of statistics as a liberal art helps us balance our essential technical expertise with the will to expand on it rather than be limited by it. A corollary is that the first course in statistics is not primarily intended to develop statisticians.” One such course (Statistical Literacy) is being taught at the University of New Mexico (Math 1300) where it satisfies a mathematics requirement in their General Education curriculum and is required by all students majoring in statistics. Schield (2021b, 2022). A new half-semester version of this course is now required of all incoming students at New College Florida. Students value this course. About half of those taking this course at UNM and NCF agree or strongly agree that confounder-based Statistical Literacy should be required by all college students for graduation. This Statistical Literacy course was presented at the 2024 General Education meeting hosted by the AAC&U. in their innovative topics track. (Schield 2024). 5. Conclusion If statistical educators make it their mission to help students analyze and evaluate statistics involving claims about observational causation, they should identify the core ideas that are needed to prepare students for this task. They should review those courses that share these goals. Statistical Educators should review the GAISE guidelines and if necessary, modify them to include a track for observational causation. Acknowledgements To Joel Best who pioneered in recognizing the essential role of context in reading, interpreting and evaluating statistics used as evidence. To Donald Rubin and Judea Pearl for agreeing to participate in a joint venture because we shared a vision that observational causation was extremely important and needed to be supported. References Moore, D. S. (1997). Bayes for Beginners? Some Reasons to Hesitate. The American Statistician, 51(3), 254–261. https://doi.org/10.1080/00031305.1997.10473972 Moore, D. S. (1998) Statistics Among the Liberal Arts. 1998 American Statistical Association. Journal of the American Statistical Association 1998, Vol. 93, No. 444, Theory and Methods. Copy at https://www.stat.purdue.edu/~dsmoore/articles/LibArts.pdf Moore, D. S. (2001) Undergraduate Programs and the Future of Academic Statistics. The American Statistician, 2001, pp. 1–6. Raymond R., and M. Schield (2008). Numbers in the News: A Survey. 2008 ASA Proceedings of the Section on Statistical Education. P. 2848-2855. www.statlit.org/pdf/2008RaymondSchieldASA.pdf Rossman, A., & De Veaux, R. (2016). Interview With Richard De Veaux. Journal of Statistics Education, 24(3), 157–168. https://doi.org/10.1080/10691898.2016.1263493 Schield, M. (2001). Augsburg Grant Proposal to the W. M. Keck Foundation. Copy at www.StatLi2001-Schield-Keck-Proposal-2a.pdf Schield, M. (2021a). “Statistical Literacy: Teaching Confounding”. USCOTS. Copy at www.StatLit.org/pdf/2021-Schield-USCOTS.pdf Schield, M. (2021b). Statistical Literacy Approved for General Education at UNM. ASA JSM Proceedings of the Section on Statistics and Data Science Education. Pgs.1192-1202. https://statlit.org/pdf/2021-Schield-ASA.pdf Schield, M. (2022). Statistical Literacy UNM Math 1300: First Year Results. ASA JSM Proceedings of the Section on Statistics and Data Science Education. Pgs. 1235-1265. https://statlit.org/pdf/2022-Schield-ASA.pdf Schield, M. (2024). Statistical Literacy for General Education. AAC&U 2024 GEPA Conference. Invited presentation. www.statlit.org/pdf/2024-Schield-AACU-GEPA.pd Utts, J. (2003). What Educated Citizens Should Know About Statistics. The American Statistician, May 2003, Vol. 57, No. 2. DOI: 10.1198/0003130031 https://ics.uci.edu/~jutts/AmerStat2003.pdf Appendix A: Mathematics Required and Type of Data Used by Major Students in the majors on the left are generally required to take introductory statistics. Students in the majors on the right are generally required to take Calculus or Discrete/Finite Math. Figure 1: Bachelor's Degrees: Statistically Based vs. Calculus-Based However, some on the left (Economics, Finance) may take calculus in place of statistics while others (History) may not be required to take a particular math course. Some of those on the right may not be required to take calculus. For simplicity, an estimated 50% of today's college graduates will be required to take an introductory statistics course. While 15% of college graduates will be required to take Calculus or Discrete/Finite mathematics. The remainder (35%) have no specific mathematics requirement as part of their major. Figure 2: College Students by Type of Mathematics Required Figure 3 presents the kinds of studies commonly encountered by various majors. Studies are classified as observational (researcher is passive observer) or experimental (researcher or nature is active). Experiments are classified as ideal (laboratory), clinical trials (randomized controlled trials) or quasi-experiments. Quasi-experiments can involve researcher doing (price changes and sales/responses over time) or nature doing (COVID positives by state over time). Computer and Information science majors are included since they can use computer simulations to conduct population inference. These majors were not included in Figure 1 since they may take Discrete or Finite Mathematics. Figure 3: Most Common Statistical Studies by Statistically Based Majors. The typical introductory statistics course focuses on random sampling (surveys) and randomized controlled trials (RCT). This matches the statistical needs of almost half (47%) of those taking statistics. These assignments are very crude. For example, psychologists encounter observational studies when random assignment is impossible or immoral, or when they are dealing with clinical or social psychology. Now consider the statistical needs of all college students. Those in non-quantitative majors see mostly statistics based on observational studies in the everyday media. Those in science and engineering are more likely to deal with ideal or laboratory experiments. This allows the maximum control by the researcher. Experiments can typically be repeated to eliminate coincidence and most confounders. However, these science-engineering majors are most likely to see statistics from observational studies when they observe the everyday media as citizens. Figure 4 shows the results for all college students. Figure 4: Statistical Studies Commonly Encountered by All Students. Based on the primary kind of study encountered and summarize, 54% of all students deal primarily with observational studies, 26% deal primarily with clinical trials, and 20% deal primarily with quasi-experiments. Political science and education majors also deal with quasiexperimental studies. In order to simplify and avoid the perception of unjustified precision, the estimates shown in Figure 5 are used. Figure 5: Distribution of College Students by Most Common Study Encountered. Although subjective and crude, these estimates give an empirical basis for deciding what should be taught in introductory statistics in order to meet the statistical needs of all college students. Appendix B: Typed of Inductive Inference and Type of Student Design Many of the different types of inductive inference can be classified into four categories as shown in Figure 6. Figure 6: Wheel of Inference Appendix C: Typed of Inductive Inference and Type of Student Design Statisticians focus mainly on generalization and prediction. Non-statisticians focus mainly on explanation/causation and on specification. One way to classify college students’ statistical needs is shown in Figure 7. Figure 7: Distribution of Students by Primary Data Usage and Type of Inference These subjective estimates should be replaced by data obtained analyzing journal articles by quantitative majors. These estimates assume that the 30% of students in non-quantitative majors are most likely to encounter observationally based statistics and that these statistics are primarily used to explain, to argue about causation, or to make evaluations.