What Do We Really Mean by Rigor in Information Systems Research?

This is a self-archived version of an original article. This version may differ from the original in pagination and typographic details. Author(s): Title: Year: Version: Copyright: Rights: Rights url: Please cite the original version: CC BY-NC-ND 4.0 https://creativecommons.org/licenses/by-nc-nd/4.0/ What Do We Really Mean by Rigor in Information Systems Research? © Authors, 2022 Published version Soliman, Wael; Siponen, Mikko Soliman, W., & Siponen, M. (2022). What Do We Really Mean by Rigor in Information Systems Research?. In Proceedings of the 55th Hawaii International Conference on System Sciences (HICSS 2022) (pp. 6583-6592). University of Hawai'i at Manoa. Proceedings of the Annual Hawaii International Conference on System Sciences. https://doi.org/10.24251/HICSS.2022.797 2022 What Do We Really Mean by Rigor in Information Systems Research? Wael Soliman University of Jyvaskyla [email protected] Mikko Siponen University of Jyvaskyla [email protected] Abstract The term “rigor” entered the information systems (IS) vernacular nearly four decades ago to reflect an ideal that would help transform IS into a coherent research field. Today, rigor is often both claimed and demanded by IS authors as evidence for the worthiness of research. However, it seems that we, as an IS community, lack both a shared understanding of what this ideal represents or what qualifies as attaining this ideal. In this paper, we analyze the usage of the term “rigor” in four leading IS journals, aiming to grasp some of its meanings within the IS community. The findings reveal that “rigor” in IS has multiple meanings, denotes a variety of referents, and is used for various purposes. Yet, even if the exact meaning of rigor is unclear or ambiguous, many IS researchers are dissatisfied with the current level of rigor and demand more. In contrast, we argue that our research endeavors could benefit from relaxing, rather than intensifying, the need for rigor. 1. Introduction Scientific rigor (sometimes referred to as academic rigor, methodological rigor, analytic rigor, etc.)—or simply rigor—is one of the most discussed criteria for determining the quality of scientific research in IS. For example, 31% of the articles published in Management Information Systems Quarterly (MISQ) between 2005 and 2015 mention the term “rigor” in one form or another (e.g., methodological rigor, theoretical rigor, rigorous design, rigorous analysis, etc.). The quest for rigor, however, has been a central concern for IS scholars for much longer. In fact, as early as the 1980s, the term “rigor” was popularized as part of a broader question concerning whom we, as a discipline, should emulate to improve our academic posture. The first time the term appeared in an IS outlet was likely in Keen’s [1] influential article. In this article, he emphasized the 1 The notion of “relevance” (or practical relevance) generally points to a debate in IS about the importance of making IS research more accessible to non-academics. A critical assessment of what IS scholars important role “analytic rigor” played in positioning Operations Research (OR) as an established discipline, which, Keen argued, was lacking in the IS. Rigor, in this sense, entered the IS vocabulary as an ideal that could transform Management IS from a collection of ideas— looking “muddled, messy, and fraudulent”—into a coherent field [1, p. 10]. Nearly two decades after Keen’s article [1], the question “Whom we should emulate?” re-emerged in MISQ as a central theme in a series of “Issues and Opinions” articles addressing the rigor-vs-relevance debate [2–6]. While the contributing authors generally agreed that we should pay attention to practical relevance 1 without jeopardizing our hard-won rigorous image [5], the debate dedicated much of its attention to discussing which scientific model we should emulate. Should we emulate the works of natural science [2]; emulate our colleagues in medicine and law schools [6]; conduct research “in a way that emulates inquiry in the professions” [6, p. 29]; or should we adopt a model that facilitates a “dialogue between the technical and social” [5, p. 26]? More generally, how should senior faculty provide methodological guidance to junior scholars [4]? Worryingly, it seems that the quest for legitimacy that began during the 1980s and 1990s has turned rigor into a “fetish” in IS, be it methodology fetish [7] or theory fetish [8]. But what does rigor actually mean in IS? To help clarify this issue, in this paper, we analyze the use of “rigor” in 393 articles published between 2005 and 2015 across four leading IS journals. In alphabetical order, these journals are as follows: European Journal of Information Systems (EJIS), Information Systems Research (ISR), Journal of Management Information Systems (JMIS), and Management Information Systems Quarterly (MISQ). The early results of our analysis imply that 1) the concept of rigor has not only become part of the philosophical vernacular in IS, but 2) numerous IS authors generally also take it as self-evident that rigor is of utmost importance in IS. For example, we found that 29% of the articles highlight the importance of rigor, mean by relevance in various IS writings deserves its own analysis and is beyond the scope of this article. Proceedings of the 55th Hawaii International Conference on System Sciences | 2022 Page 6583 URI: https://hdl.handle.net/10125/80137 978-0-9981331-5-7 (CC BY-NC-ND 4.0) while 23% express their dissatisfaction with the current level of rigor and say that more rigor is needed. What, then, counts as rigor? We argue that despite all the publications about, and references to, rigor, the IS literature does not reflect a shared understanding of what the term rigor means, what it refers to, or what qualifies as having attained rigor. The main concern here is that what one author means by rigor can be completely different from the meaning attributed by another author. For instance, Rivard warns that deficiency in conceptual clarity “can be particularly harmful because readers will themselves ascribe meanings to constructs, with the risk of ending up with as many meanings as there are readers” [9, p. vii]. As a result, there is a risk that different IS authors will fail to understand each other and, therefore, will fail to accurately communicate regarding rigor. Another concern is that IS readers often fail to understand what authors mean when they say rigor. This is particularly the case when authors refer to rigor but do not explain what it means for them or why it is necessary. These two concerns lead to another greater concern regarding something that should be avoided in any scientific community: using empty jargon. The use of “rigor” in the majority of publications in the field is often little more than empty jargon. Despite the generally held belief among those in the field that something meaningful is being communicated, often only an obscure claim is presented. For example, many top IS journal authors have repeatedly claimed that their studies are “rigorous” without presenting evidence as to why this is so or what this means in the context of the studies. Authors and reviewers should not assume that a study is “rigorous” (however this is defined) simply because the claim has been made. Finally, there is the risk that rigor has turned into a kind of dogma. Consequently, for numerous IS authors, rigor means the strict, to-the-letter application of something that someone has proposed. In scientific research, however, we should understand the evidence and why we are applying something in our own work. Simply following something because it has been proposed or accepted is tantamount to dogmatism. 2. Research approach The research at hand is a “concept-centric review” [10], with the purpose of conducting a “critical assessment” [11] of the usage of the term “rigor” in top IS journals. Although uncommon, this type of review is needed, since its ultimate aim is “to critically analyze the extant literature on a broad topic to reveal weaknesses, contradictions, controversies, or inconsistencies” [10, p. 189]. To this end, we adopted a multi-staged research approach as follows: (1) setting the research scope; (2) search and extraction; (3) analysis; and (4) reporting. First, regarding the research scope, the following two research questions guided this research inquiry: (a) What do scholars in top IS journals mean by rigor? and (b) What messages do authors convey by talking about rigor? To make the research manageable, we decided to set two boundaries regarding the literature sources and the search timeframe. Specifically, we limited our search queries to four leading IS journals— MISQ, ISR, JMIS, and EJIS—and set the timeframe to a full decade. Given that the project started in mid-2016, it was deemed appropriate to set the timeframe of the study between 2005 and 2015. In the second stage, search and extract, a search query was conducted for each journal individually, inclusively extracting all articles containing possible variations of the word RIGO* (e.g., rigor, rigour, rigorous, rigorously, etc.). Once the articles were identified, they were extracted from the journal’s database, and paragraphs containing the word rigor (or its variations) were uploaded to our database. During this stage, we extracted a total of 412 articles from the databases (157 articles from MISQ, 105 from EJIS, 100 from JMIS, and 49 from ISR). Articles in which the word rigor (or its variations) appeared only in the references were excluded from further analysis, thus leaving us with a total of 393 articles (as shown in Table 1). Table 1. Breakdown of articles containing “rigor” per journal between 2005 and 2015 Journal Total number of published articles Number of articles containing RIGO* Percentage of articles containing RIGO* MISQ 470 144 31% ISR 549 47 9% EJIS 561 105 19% JMIS 521 97 19% TOTAL 2101 393 19% The third stage, analysis, began immediately after the extraction of the first set of articles. The analytical strategy was guided by the two research questions mentioned above. Tabulation and coding were central to the analysis process. Both a priori and posteriori codes were instrumental in the analysis. The a priori codes were codes (or themes) that we knew to be relevant to our analysis before starting the analysis and included research approach, number of occurrences, definition of rigor, and references used. Conversely, posteriori codes were themes that emerged as an important dimension while undertaking analysis of the content. Such themes included “referents” and “messages”. Each of these themes, in turn, was composed of multiple codes. For Page 6584 instance, the “referent” theme refers to the focal point of interest that authors consider as the object of interest regarding rigor. Codes under the “referent” theme included science, methodology, theory, analysis, findings, etc. Thus, we noted that while the research questions guided the overall research direction and scope, the analysis process itself was iterative and largely inductive, especially in its early stages. The eventual analysis protocol emerged by constantly comparing what was learnt from the article at hand with what was learnt from the previous article(s). The fourth stage, reporting, concerned consolidating the overall findings and presenting some illustrative examples. Considering the space limitation, we will only focus on presenting some of the key findings. 3. Findings 3.1. Overview In this section, we provide a general overview of some of the notable descriptive statistics. First, we start with a brief discussion of how variations of the term rigor have evolved over time (see Figure 1). Clearly, MISQ is the leader in terms of the usage of rigor for the overall period, and its standing (in %) has been relatively steady over the years. EJIS takes the second spot in terms of overall usage; however, there seems to have been a notable decline in its position since 2012, when it held the highest spot, with 47% of publications in that year. Interestingly, JMIS follows the opposite trend compared to EJIS. Whereas EJIS reached the peak of its usage of the term in 2012, JMIS’ usage reached its lowest point in 2012 and began climbing from that point onward, taking the leading spot in 2015. Finally, ISR generally ranked the lowest in terms of its publications containing the term rigor, i.e., three percent or less in the last four years (2012–2015). Figure 1. Usage of the term “rigor” in IS journals (EJIS, ISR, JMIS, MISQ) between 2005 and 2015 (per year) Next, we present an overview of the basis of the term rigor in terms of research approaches (i.e., article type). Jointly, the largest share of articles containing the term rigor (36%) belonged to articles that were quantitative in nature. The second spot was shared almost equally amongst editorials (16%), qualitative research (13%), and conceptual articles (12%). The remaining 22% were distributed amongst action and design research (8%), meta-analysis articles (6%), mixed-methods research (4%), and others (4%). Altogether, 15 articles were categorized under “others”, since they did not fit easily into the existing categories. Specific research approaches in the “others” category included computer simulations [12], text mining [13], and research method guidelines [14]. Figure 2. Usage of the term “rigor” in IS journals (EJIS, ISR, JMIS, MISQ) between 2005 and 2015 (per research approach) Breaking down the statistics by journal provides a more detailed view of the distribution and presents the data from a different perspective (see Figure 2). For instance, rigor was most frequently encountered in quantitative articles published in ISR, JMIS, and MISQ. Surprisingly, however, in EJIS, the editors’ commentaries led the list, followed by qualitative research articles. It would be interesting to extrapolate what might explain these trends; however, such a discussion is beyond the scope of the current report. In order to gain further insights on the most influential writings in shaping the IS community’s conception of rigor, we conducted a rigor-focused bibliographic analysis. Specifically, for each article, we carefully read all paragraphs containing the term rigor (and its variants) and extracted all references used in these paragraphs to support claims about rigor. Conducting this procedure for all 393 articles revealed that nearly one-third of the reviewed articles (131) did Page 6585 not cite any reference to support their claims regarding rigor. The remaining articles (177) collectively cited 323 references, of which only three references were cited five times or more. Namely, Hevner et al.’s [15] design science guidelines was cited 13 times; Benbasat and Zmud’s [2] and Davenport and Markus’ [6] commentaries on the rigor-vs-relevance debate were cited 11 times and seven times, respectively. 3.2. What is this thing called rigor? Now we turn to the main research question regarding what IS scholars mean by rigor. Our analysis shows that there seems to be a broad implied agreement that rigor is about the strict conformity to the principles of academic or scientific research. Despite this general agreement, there is a clear lack of shared understanding as to what these principles are. To a majority of researchers, rigor is “the stringent application of research methods” [16, p. 289] or “conformity to norms” [17]. From this perspective, rigor is achieved by closely following the methodological steps provided in a research guideline article [18–20]. Table 2. Definition of rigor from selected IS articles Article Definition Referent [21] “For behavioural IS research, statistical significance is established as a clear and common measure of its results’ rigour” [p. 470] Method [20] In interpretive research, rigor is achieved by following the methodological guidelines provided by [25, 26, 27 and 28]. Method [19] “To ensure rigor, we closely followed Dube and Pare’s [13] suggested guidelines for positivist exploratory case study research” [p. 145]. Method [26] “[T]he papers that are published here [i.e., an MISQ special issue] are rigorous in that they have a strong theory base and use formal analysis” [p. 210]. Theory and method [27] In DSR, rigor is about conducting a “project [that] is guided by … theory” [p. 144]. Theory [28] In DSR, rigor is about demonstrating “how well the artifact works, not to theorize about or prove anything about why the artifact works” [p. 742]. Method, not theory 2 A referent is defined as “[t]he thing in the world that a word or phrase denotes or stands for” (www.en.oxforddictionaries.com). We use the term referent to capture the target of interest in texts containing the term rigor. For instance, when an article makes a claim regarding “scientific rigor” [79, p. 526], the referent is coded as “science”. In addition to the emphasis on method, many researchers also believe that rigor requires adopting or developing a strong theory [21–23]. Table 2 provides a few examples reflecting the emphasis on method and/or theory in defining rigor by various authors from different research genres. A closer look reveals that when talking about rigor, different authors allude to different referents 2 in academic/scientific research. Thus, we decided to capture all referents in the dataset. By capturing the referents in this way, we were able to gain a better and more comprehensive understanding of what rigor is said to mean in leading IS writings. As Figure 3 illustrates, rigor targets different referents, and these referents occupy different positions at the various stages in a typical life cycle of a research article. Not only do IS scholars see rigor in method and theory, but rigor is also claimed (as well as demanded) in research design, research questions, literature reviews, data collection, data analysis, interpretation, reporting of findings and conclusions, as well as in contributions. These were not the only referents identified in our analysis. Other less frequent rigor referents included definitions, conceptual argumentation, sampling, prediction, and coding, among many others. Figure 3. Referents of “rigor” in IS journals (EJIS, ISR, JMIS, MISQ) between 2005 and 2015 Based on this extensive reading of the top-ranked IS publications, a rough characterization for (scientific) rigor can be proffered: the demonstrability of satisfying the quality standards demanded of academic research [32–34], which is reflected by at least a) formulating a rigorous research question [35]; b) adopting and/or developing a rigorous theory [21–23]; c) applying a rigorous research methodology [16, 17, 36]; d) conducting a rigorous literature review [37–39]; e) relying on rigorously collected empirical data [40–42]; f) subjecting the data to rigorous analysis and interpretation [43–45]; g) offering rigorous contributions [46, 47]; and h) when submitting research Similarly, when an article makes a claim regarding “analytical rigor” [80, p. 374] or “rigorous analysis” [81, p. 188], the referent is coded as “analysis”. Page 6586 findings to a journal subjecting it to a rigorous review process [48–50]”. The answer to our first research question points to a startling dilemma: while there is a general agreement among IS scholars that rigor denotes a strict adherence to some quality standards, there is also a lack of agreement (as well as shared understanding) as to what these standards are, even among members of the same research tradition. We return to this issue in the discussion section. 3.3. What messages do authors convey by referring to rigor? The second question we address here relates to the “message”. Specifically, we sought to capture the essence of the intended message where the authors mentioned the term rigor (and its variants). To this end, we devised two generic questions for each paragraph containing a variation of the term. The first question addressed whose work the paragraph was referring to. This question captures whether the authors were referencing their own work (e.g., our analysis, our research, etc.) or the work of others (e.g., research in general, future researchers, etc.). The second question addressed the conclusion conveyed from the paragraph (e.g., rigor is achieved, more rigor is needed, etc.). Answering these questions revealed that the paragraphs containing rigor had been instrumental in conveying the following four core messages: (1) our work is rigorous; (2) rigor is an important ideal; (3) current research lacks rigor, and more rigor is needed; and (4) rigor can be achieved without jeopardizing practical relevance. While additional messages were identified, due to space limitations, we focus only on these four popular messages. One or more of these messages may have appeared in the same article. We briefly discuss these messages below. Our work is rigorous. We identified 191 articles (49% of all articles) conveying this message, and over half of these (101 articles) used no references. This message was utilized by authors who sought to communicate to their readers that they had achieved rigor in their work. For example, Zhu et al. [51, p. 515] argued for the rigor of their work: “based on a rigorous empirical analysis of a unique international data set”. Elliot [52, p. 203] claimed the rigor of their research by stating that “benchmarks for research integrity and quality from another source were … reviewed and applied”. In contrast, Armstrong et al. [53, p. 720] claimed rigor of their work by arguing that their test results “demonstrate that our model meets or exceeds the rigorous standards expected in IS research”. As these examples demonstrate, the central message being communicated is that the work at hand has achieved (or exceeded) the standards of rigor. Interestingly, however, different authors had different conceptions of, and criteria for, meeting these standards. Rigor as an important ideal. Of the analyzed articles, 115 articles (29%) conveyed this message. The main point here is that the authors in this case were not making a specific argument about their own work; rather, their messages concerned rigor as an important scientific ideal/value that researchers should achieve. For example, Smith et al. [54] emphasized the value that rigorous empirical research could add in the domain of privacy research, arguing that “there are many theoretical developments in the body of normative and purely descriptive studies that have not been addressed in empirical research on privacy. Rigorous studies that either trace processes associated with, or test implied assertions from, these value-laden arguments could add great value” [p. 989]. In a similar vein, McLaren et al. [40] stressed the role played by rigorous hypothesis testing in producing stronger IS theories, maintaining that the “coupling of carefully designed research artifacts with rigorous hypothesis testing research has great potential to produce stronger IS theories” [p. 926]. Once again, the authors’ messages that “rigor is important” were clear. However, it was often unclear what rigor meant, or why the proposed conception of rigor should be considered the definitive definition. Current research is lacking rigor. Twenty-three percent of the reviewed work (90 articles) conveyed this message. This message was often used by authors who expressed their dissatisfaction with the level of rigor in IS research. For example, Berger [55] justified the explorative nature of their work on account of a lack of existing rigorous research [p. 107]. Shanks et al. [56] conveyed a similar message regarding the lack of rigor they observed prior to their research. They maintained that “[a] small amount of theoretical work has been undertaken to evaluate the merits of these alternative representations […], but much still needs to be done. Moreover, to the best of our knowledge, no rigorous empirical evaluation of alternative representations of part–whole relations has so far been undertaken” [56, p. 554]. Rigor can be achieved without jeopardizing relevance. Considering that a major part of what has been written about rigor was from the lens of the rigorversus-relevance conversation, one of the recurrent themes in the reviewed articles reflected the argument/notion that both rigor and relevance are important and that a balance between them can be achieved. This argument frequently appeared in editorials or in editors’ comments. For instance, Paul [57] noted: “We [EJIS editors] wish to do everything we can to promote relevance without loss of rigor […] [W]e seek to publish high-quality rigorous papers that are critical, relevant, pluralist and with impact on our Page 6587 readership” [p. 208]. Sambamurthy [58] offered a similar message in an ISR editorial: “As scholars in the IS community of practice expand the range of their inquiry and push the ‘boundaries’ of their phenomenological, theoretical, and methodological choices, ISR will provide them a forum for creative dialog while preserving rigor and quality […] The review processes will emphasize relevance to practice and the organizational realities of information systems as equally important as academic rigor and theoretical contributions” [p. 3]. This implies that rigor is important, while it simultaneously remains unclear what “without loss of rigor” really means in practical terms. 4. Discussion The findings of our analysis of “rigor” suggest that IS authors allude to “rigor” to refer to vastly different ideas in their writings. These ideas include scientific rigor, methodological rigor, analytical rigor, as well as the need to provide rigorous understanding, rigorous conceptualizations, and rigorous definitions. Nevertheless, the IS community as a whole seems to lack a shared definition of rigor that can be used to communicate about the idea effectively. Although, we proposed a rough characterization of rigor (see section 3.2), which is emphasizing the importance of adhering to quality standards, this description fails as a scientific definition. It leaves unanswered what such standards are, largely because (in general) the IS papers on rigor do not explain their positions or stances adequately. Many authors seem to use the term rigor rhetorically. For example, an article might begin by emphasizing the importance of rigor (message #2), then move on to criticize the insufficiency of rigor levels in the field (message #3); eventually, perhaps, declaring that the authors’ rigorous work has remedied the situation (message #1). Such practice, which is not necessarily deliberate, raises several concerns. Beginning with the less serious concerns, many IS authors repeatedly claimed that their studies were “rigorous”. At the same time, they often did not provide explicit evidence as to why this is so. Often it also seemed somewhat unclear what rigor referred to. The IS community should not assume that a study is “rigorous” (however this is defined) simply because the claim has been made. The more serious concern is the risk that rigor has turned into a kind of dogma for many in the field. Namely, in the articles we reviewed, IS scholars asked that certain protocols be followed in the name of rigor. Yet, at the same time, often what the authors believed “rigor” to be was unclear or inconsistent [59]. Likely, the most common idea of rigor, based on our readings, is “the strict following of instructions”. This definition may lead authors in top IS journals to feel it is incumbent on them to show how they rigorously followed particular method guidelines, even if the guidelines themselves may not be evidence based [60] . Next, we discuss two possible implications of this phenomenon. 4.1. Defending the importance of rigor and why increased rigor is needed The first implication would put premium on rigor, require it, or even demand more rigor. Thus, for example, advocates of this approach would demand applying a strong theoretical foundation [21–23], conforming to published research methodology principles [16, 17, 36], conducting a rigorous literature review [37–39], relying on rigorously collected data [40–42], and so on (see section 3.2). At present, this approach poses problems that those wishing to defend rigor must address. While the required justification depends on the specific rigor claims, our generic observation is that defending “rigor” is not straightforward and generally requires in-depth examination and explanation. Two examples of the kinds of problems facing claims of rigor are briefly presented next. Example 1: Theory-ensures-rigor claim. Consider, for example, a view according to which using a theory or strong theoretical foundation ensures rigor. For instance, in King and Torkzadeh’s [29, p. 210, emphasis added] introduction to the special issue on IS offshoring, they noted: “the papers that are published here are rigorous in that they have a strong theory base and use formal analysis.” The same is observed in the evaluation criteria in another special issue on agile software development [61], where rigor in the selected articles “is shown through the sound theoretical base upon which the studies have been conceived and the findings drawn” [p. 283, emphasis added]. The problem here is that what counts as sound theoretical base (or rigor in this case) remain vacuous. More precisely, the problem is that anyone advocating rigor through the use of a sound theory must first outline what counts as “theory”, or sound theory in this case. Without clearly defining these terms, they remain unclear and the resultant discussion runs the risk of becoming practicably useless. To illustrate this phenomenon, while Chen et al. [27] argued that the use of theory ensured the rigor of their design science research project, Goes [69, p. 6], in his editorial commentary on design science research, has rejected the idea that “an article without an explicit tie to theory is less rigorous”. Such debate runs the risk of being functionally unusable unless what constitutes a theory is explicitly outlined. Page 6588 Furthermore, merely defining theory is not enough to defend the claim that sound or strong theory ensures rigor. Such claims of rigor should also explain why the claimed characteristics of theory are necessary in any good scientific research. Consider, for example, Briggs et al. [63, p. 4], who provide different rigor criteria for “four modes of inquiry: exploratory, theoretical, experimental, and AS/E [applied science/engineering] research.” For the authors [63], “[t]heoretical research produces only one kind of model, [namely] theories that predict and explain, also known as general and covering laws, received theory […]” [pp. 4-5]. The problem here is that most theories in any science are not “covering laws”, a concept introduced in Hempel’s deductive nomological explanation [64, see also 65]. These covering laws are statements of strictly universal form, such as “all metals are conductors of electricity” [66, p. 153]. Perhaps no statement in IS should be deemed a strictly universal claim in terms of covering law [64]. In this example [63], the claim “theory ensures rigor”, and theory is covering law, would be totally misleading in IS. Example 2: Rigor as strict adherence to published research methodology principles. Dubé and Paré [67] provide comprehensive criteria to determine whether a “positivist case study” has been conducted with rigor. To Dubé and Paré [67], a “necessary factor” in determining whether a “positivist case study” has achieved rigor is whether the case researchers have “adopted and implemented the attributes that leading case research methodologists have identified as contributing to rigor” [p. 599]. The methodological principles and guidelines Dubé and Paré [67] base their evaluation of rigor on are those proposed by Benbasat et al. [68], Eisenhardt [69], Lee [70], and Yin [71]. Based on these four writings, Dubé and Paré [67] identified a total of 34 attributes (i.e., criteria) to assess the rigor of positivist case research in terms of research design, data collection, and analysis [p. 606]. The implication of this framing (i.e., evaluating research against a checklist) is that case study researchers who fail to prove that they have rigorously followed those 34 criteria will be deemed dubious. Such a conclusion is likely to be drawn despite Dubé and Paré’s [67] warnings that 1) researchers should not treat this list as a “cookbook recipe” for how to conduct rigorous research; 2) rigorously adhering to those rules does not necessarily make a good study; and 3) there is “an everexisting tension between the desire for detail and brevity”, especially in qualitative research, leaving researchers no choice but to omit some details [pp. 6278]. Yet, and despite those warnings, Dubé and Paré [67] express their disappointment at the case study research they reviewed for not rigorously following those guidelines. Their disappointment meant that case researchers could improve their posture by following the guidelines more rigorously. But if rigorous adherence does not necessarily lead to better quality, and if those guidelines should not be strictly followed as a cookbook recipe, then why would ignoring some of them be disappointing? What would the advocates of “more rigor” need to do here? In the case of research method guidelines [67], it is somewhat clear what rigor means— adhering to the 34 identified attributes. That being said, as with the example of theory, the advocates of rigor as strict compliance to methods must justify how these attributes improve case study research. This means providing evidence at the level of each attribute to how it is necessary in obtaining some identified preferred scientific outcome. Justifying the attributes merely under the authority of “leading case research methodologists” is an authority argument, and reasoning should be evidence based, not authority based. 4.2. A confutation of rigor Straub’s [72] general assessment is that many in the IS community have over-emphasized rigor at the expense of intellectual content and that exciting, good, and/or frame-breaking research deserves to be published even if the methodology is “minimally acceptable” (p. vii). In addition to this, the problem as previously noted is the vacuousness of the concept rigor. The critics of rigor do not lack examples to make their cases that many tenets of rigor are too ambiguous to be acceptable in their current form. Rigor as “strict compliance” is an example of such. This can be misleading, as even the most painstaking work “could be rigorously wrong” [59, p. 626], which can lead to dogmatism if strict compliance is expected for its own sake. The same problems that advocates of rigor must overcome, as outlined in section 4.1, can be seen as reasons for rejecting the rhetoric of rigor. In addition, the critics of rigor can point to further problems with the term or concept. For instance, critics can point to cases where some alleged rigorous method principles/guidelines were not only lacking evidence to support their claims, but the guidelines had the potential to hinder the progress of science [60]. Such hindrance may occur, for example, if the method guidelines require meeting some generic or universal criterion in the name of rigor that does not fit a particularized research setting [60]. For example, theory development is widely portrayed as a creative guessing activity, by hypothetico-deductivists [64], Feyerabend [73], and many others, as opposed to a rule-governing activity. Against this background, non-strict compliance is not necessarily synonymous with bad research [59], [67], Page 6589 [72]; indeed, such is advocated for in the theory development phase by the conventional scientific method (hypothetico-deductivism) [64]. 4.3. Where does “rigor” come from? Although we pointed to Keen’s [1] vision as one of the earliest writings on “rigor”, tracing the origins of the idea and how it emerged in IS is difficult. One possibility is that the term “rigor” was taken from the mid-20th-century philosophy of science [64]. At that time, Rudolf Carnap, Carl G. Hempel, Hans Reichenbach, and later Jaakko Hintikka made “logical rigor” a central task for professional philosophers [74– 77]. Generally, this task is no longer widely undertaken by philosophers of science [74]. For example, even the logician van Benthem [78] reports how “the famous classic The Structure of Science [79] explains how science differs from ‘common sense’, in its standards of rigor […]. While I assiduously learnt all these criteria by heart as a student, they now seem unconvincing to me—and largely based on ignorance of the delicate workings of common sense…” [p. 784]. Continuing, van Benthem [78] writes: “I would now think that science is the exercise of certain qualities of our common sense reasoning, but taken further in isolation, and also importantly, simplified in that many subtle features of actual reasoning and communication are put out of play” [p. 784]. What if van Benthem [78] is correct, and rigor comes to IS from the philosophers’ method of “logical analysis” and logical reconstructions for philosophical purposes [80], [81]? If such is the case, the point in science may not be “rigor”, but activities which violate the classical stand on rigor. This, however, cannot be known before we better understand what rigor actually means. 5. Conclusion In this paper, we analyzed the use of “rigor” in IS literature. Central to this project was a desire to answer what IS authors meant by “rigor” and what messages authors conveyed when discussing “rigor”. The findings suggest that IS authors mentioned “rigor” to refer to various ideas in their writings. Not only have IS scholars claimed and demanded rigor in method and theory, but also in research design, research questions, literature reviews, data collection, data analysis, interpretation, findings, conclusions, as well as in contributions. Yet, in terms of rigor, the IS community itself seems to lack an adequate understanding of what it means and, especially, why it is needed. Furthermore, the most common message authors conveyed by mentioning “rigor” was the claim that their work was rigorous and that more rigor is needed in the field. Ultimately, this article is not intended to prevent authors from claiming rigor or demanding it. However, when invoked, rigor should be defined and explained clearly, including the criteria to ensure it is attained. Moreover, it is important to provide evidence in support of the criteria for rigor. This evidence cannot be based on authority arguments but must account for how each principle in the criteria is necessary in terms of science. If the principle in question is not necessary, then researchers must ask why rational actors should require rigor. In such circumstances, the risk of dogmatism is also apparent. Given the present state of “rigor” in the field of IS, its advocates must more effectively justify rigor in general. Before this can happen, perhaps we in the field should reconsider how we claim and demand rigor. 6. References [1] P. G. W. Keen, “MIS research: Reference disciplines and a cumulative tradition,” in International Conference on Information Systems (ICIS), 1980, pp. 9–18. [2] I. Benbasat and R. W. Zmud, “Empirical research in information systems: The practice of relevance,” MIS Q., vol. 23, no. 1, pp. 3–16, 1999. [3] A. S. Lee, “Rigor and relevance in MIS research: Beyond the approach of positivism alone,” MIS Q., vol. 23, no. 1, pp. 29–33, 1999. [4] L. M. Applegate and J. L. King, “Rigor and relevance: Careers on the line,” MIS Q., vol. 23, no. 1, pp. 17–18, 1999. [5] K. Lyytinen, “Empirical research in information systems: On the relevance of practice in thinking of IS research,” MIS Q., vol. 23, no. 1, pp. 25–27, 1999. [6] T. H. Davenport and M. L. Markus, “Rigor vs. relevance revisited: Response to Benbasat and Zmud,” MIS Q., vol. 23, no. 1, pp. 19–23, 1999. [7] D. S. Hovorka et al., “Scholarly commentaries on Hirschheim’s ‘Against Theory,’” J. Assoc. Inf. Syst., vol. 20, no. 9, pp. 1356–1387, 2019. [8] J. Iivari, “Editorial: A critical look at theories in design science research,” J. Assoc. Inf. Syst., vol. 21, no. 3, pp. 502–519, 2020. [9] S. Rivard, “The ions of theory construction,” MIS Q., vol. 38, no. 2, pp. iii–xiii, 2014. [10] J. Webster and R. Watson, “Analyzing the past to prepare for the future: Writing a literature review,” MIS Q., vol. 26, no. 2, pp. xiii–xxiii, 2002. [11] G. Paré et al., “Synthesizing information systems knowledge: A typology of literature reviews,” Inf. Manag., vol. 52, no. 2, pp. 183–199, 2015. [12] D. Kwon, W. Oh, and S. Jeon, “Broken Ties: The Impact of Organizational Restructuring on the Stability of Information-Processing Networks,” J. Manag. Inf. Syst., vol. 24, no. 1, pp. 201–231, 2007. [13] M. Indulska, D. Hovorka, and J. Recker, “Quantitative approaches to content analysis: Identifying conceptual drift across publication outlets,” Eur. J. Inf. Syst., vol. 21, pp. 49–69, 2012. [14] V. Venkatesh, S. A. Brown, and H. Bala, “Bridging the Page 6590