Epistemic Beliefs and Googling

Frontline Learning Research Vol. 7 No 3 (2019) 27 - 63 ISSN 2295-3159 1 Corresponding author: Tore Ståhl, Arcada University of Applied Sciences, Jan-Magnus Janssons plats 1, FIN-00560 Helsinki, [email protected] DOI: 10.14786/flr.v7i3.417 Epistemic Beliefs and Googling Tore Ståhl a aArcada University of Applied Sciences Helsinki; University of Tampere, Finland. Article received 26 September 2018 / Article revised 4 April / Accepted 5 July / Available online 18 July Abstract With the introduction of internet as a source of information, parents have observed youngsters’ tendency to prefer internet as a source, and almost a reluctance to learn in advance since “you can look it up when needed”. Questions arise, such as ‘Are these phenomena symptoms of changing beliefs about knowledge and learning? Is it at all possible to learn on a deeper level simply by looking up the basic facts, without memorizing them?’ Within an existing line of investigation, epistemic beliefs have been described as a set of dimensions. Although internet-based information and internet as a source of information have been acknowledged, studies so far have not explored how dealing with internet-based information relates to other epistemic beliefs dimensions. To capture how users view internet-based information per se but also in relation to other epistemic beliefs, I suggest three new dimensions, out of which the most crucial is labelled ‘Internet reliance’. Offloading memory using memory aids is not a new phenomenon but the ‘Internet reliance’ dimension indicates that especially internetreliant users may be confusing external information with personal knowledge, with all the risks it may entail. Besides including beliefs about learning, this study also challenges earlier assumptions regarding uncorrelated dimensions. Keywords: epistemic beliefs; internet; constructivism; outsourcing knowledge; factor analysis. Ståhl 28 | F L R 1 Introduction and aim of study During the last decade, most people will have heard youngsters respond to a question with the acronyms JFGI or GIYF (“Just F…g Google It” and “Google Is Your Friend”, see https://en.wiktionary.org/wiki/JFGI). For most adults, expecting a proper answer, this response was surprising, puzzling and perhaps even offensive. The response is, however, an illustration of the gap between the parent generation’s “You should know this”-view on knowledge, and the young generation’s stance “I’ll look it up when I need it”. With the introduction of easy and ubiquitous access to information over internet, the attitude of looking it up when one needs it became common, especially among frequent internet-users. Given that the young generation born after the mid 1980’s grew up surrounded by information and communications technologies (hereafter ICT), the interesting question is, has the easy and ubiquitous access to information actually influenced their view on knowledge, knowing and learning? During the first decade of this millennium, the so-called Digital Natives of the Net generation were supposed to hold characteristics such as being constantly on-line, being ICT savvy and being at home on social media (e.g. Prensky, 2001; Siemens, 2005). Indeed, the youngsters differ from their parent generation in that they lack a personal history of the time before mobile phones, internet and search engines (Gunter, Rowlands, & Nicholas, 2009, p. 3), not to mention smart phones. Large parts of the youngsters within this cohort embrace the opportunities provided by ICT, e.g. preferring internet-based information instead of books (cf. OSF, 2010; Purcell et al., 2012, p. 4). Still, several studies have pointed out the heterogeneity within the generation (cf. Jones & Hosein, 2010; van den Beemt, Akkerman, & Simons, 2011). Also among the students participating in the present study, large differences occurred regarding both self-reported ICT and media use patterns and performance-based ICT skills (Ståhl, 2017). Within education, the easy and ubiquitous access to information raises concerns about how and upon which information students build their knowledge, since they seem to accept the veracity of on-line information too easily, and lack the skills of thinking critically and synthesizing the information found on-line (Purcell et al., 2012, pp. 26-27). The vast popularity of search engines (with covert operating logics) in combination with users’ lacking critique has considerable epistemic implications, as demonstrated in the theoretical work and the studies cited below (section Knowledge and information in the internet era). The present study will build upon the above studies that confirm the existence of the JFGI phenomenon. Existing self-report instruments for measuring epistemic beliefs are not capable of capturing signs indicating internet-induced changes in the views of knowledge and learning. Especially the Digital Natives’ ways of dealing with knowledge and learning have been described in literature (some examples in section Hypothesized dimensions) but so far, this topic has been scarcely approached from an epistemic point of view. This topic calls for empirical investigation, which requires instruments. This paper will describe how the existing dimensions (Structure and Certainty of knowledge, Innate learning ability and Omniscient authority) are extended with the new dimensions Constructivist approach, Internet reliance and Learning by dialogue. Creating a validated instrument requires more than one round and therefore, the aim of this endeavour is an initial exploration of how new dimensions might contribute to a better description of how today’s higher education learners in an internet-saturated context view knowledge and learning. Contemporary research regarding epistemic beliefs largely subscribes to epistemic beliefs being limited to beliefs about knowledge, and not about learning. The present study will deviate from this view by exploring also views about learning. Doing so, this study contributes to the discussion by looking beyond the knowledge dimensions of epistemic beliefs, and by describing the connection between beliefs about knowledge and beliefs about learning, a connection that is necessary to illuminate consequences for educational practice. Ståhl 29 | F L R 2 Personal knowledge, external information To provide a rationale for the present study, this section will 1) review some studies regarding knowledge, information and epistemic beliefs in the internet era, 2) review epistemic beliefs as a research area, 3) review some arguments regarding learning as part of epistemic beliefs, and 4) discuss why domain specificity and justification of knowledge where omitted from the study at this stage. 2.1 Knowledge and information in the internet era George Siemens tried to grasp the impact of technology and the decreasing half-life of knowledge by introducing connectivism as a new learning theory for the digital age. He suggested supplementing the existing forms of propositional (knowing-that) and procedural (knowing-how) knowledge with ‘knowing-where’ and ‘knowing-who’, i.e. an understanding of where to find knowledge. According to Siemens, since we cannot experience everything or store all knowledge ourselves, we store knowledge in other people and in non-human appliances. The key is connectedness, and the knowledge is distributed (Downes, 2007, p. 84; Siemens, 2005). Connectivism was apparently neither a learning nor a knowledge theory but rather a pedagogical view but still, the connectivist ideas resemble the concept of distributed mind, which suggests that knowledge can reside in people, in tools, and in cultural settings, and that the potential lies in the combination of those (cf. Shaffer & Clinton, 2006). The results of an experimental study by Sparrow and her team suggest that internet has become a kind of extension to our individual memory system. If the net is available, we do not bother to memorize the information itself but rather, where to find the information, as when youngsters respond: “JFGI!” We are becoming increasingly symbiotic with our computer-based tools, growing into interconnected systems that remember less by knowing information than by knowing where to find the information. (Sparrow, Liu, & Wegner, 2011) The concept of the extended mind (Clark & Chalmers, 1998) suggests that human cognition may extend beyond the brain and include elements from social and technological environments (cf. Siemens, 2005). Applying the concept to the context of the web opens up for the concept of the web-extended mind, which includes the idea that “… the informational and technological elements of the web can, at least on occasion, constitute part of the material supervenience base for (at least some of) a human agent’s mental states and processes” (Smart, 2012, p. 451). The mere existence of the web does not automatically make it part of a person’s extended mind but in addition, three criteria need to be met: the availability criterion, the trust criterion and the accessibility criterion (Clark & Chalmers, 1998; Smart, 2012). Considering the development, that has taken place within the web and smart phone contexts since Smart wrote his article, we have reason to suspect that users often regard these criteria as met, and too easily incorporate on-line information into their personal body of knowledge: due to internet capable smartphones, the availability and the accessibility criteria are easily met. The problematic part is the trust criterion: on-line information is too easily endorsed and too rarely subject to critical scrutiny (Purcell, Brenner, & Rainie, 2012, pp. 10-11). This is especially problematic since e.g. Google made personalized search in 2009 the default option for all users (Simpson, 2012, p. 437). The personalization of search results performed by search engines means that the results are tailored to what will probably interest the enquirer, and that those hits that do not fit the enquirer’s profile are ranked down or even omitted. According to Thomas Simpson (2012), the epistemic significance of search engines lies in their acting as surrogate experts, firstly as they assist the enquirer in finding sources and secondly as they orient the enquirer to supposedly relevant sources of information (the expert role also discussed by Fisher, Goddu, & Keil, 2015, below). The problematic aspect here is that by filtering and ranking the results, the search engine implies a judgment about what is relevant, without the enquirer having neither insight into, nor the possibility to influence the criteria for judgement. As Simpson (2012, p. 427) puts it: “… objectivity may require telling enquirers what they do not want to hear, or are not immediately interested in” (my emphasis) Ståhl 30 | F L R (also see Hinman, 2008). Therefore, Simpson regards personalization as an actual threat to objectivity. By leaving out relevant voices, the tailored search results contribute to an epistemic bubble, and the operating logics of search engines combined with the enquirers’ ignorance increases the risk of the enquirer being trapped in an epistemic bubble or even an echo chamber (Nguyen, 2018). The complexity of the objectivity problem is illustrated by the findings of Purcell, Brenner, & Rainie (2012): although a majority in their study disapproved search engines collecting information about their searches, 23-29% thought that using the information for personalizing search results was a positive feature (pp. 19-21). Further, on average two thirds of the participants believed that the information provided by search engines was fair and unbiased: the younger, the more they relied on search engines’ objectivity (pp. 10-11). A further aspect, illustrating the objectivity problem, is the ritualization described by Bhatt & MacKenzie (2019), i.e. students’ information seeking practices being largely motivated by adhering to what they call the rules of the game. These rules can be appropriate in the beginning to induce students to the knowledge creation practices of the discipline but when detained too long, they may inhibit the development of students’ information seeking skills and trust in their own capacity to consider the justification of the information they find. In an experimental study, Fisher et al. (2015) highlight the risks embedded in ubiquitous access to information, which may blur the boundaries between personal knowledge and external information, thus creating an illusion of possessing personal understanding. Further, their results suggest that some individuals tend to regard internet as an expert regardless of domain. These results pose a true challenge for education at all levels, at least if we consider personal and integrated knowledge, instead of loose bits of information, as the objective of education and learning. Miller & Record (2013) discuss the covert operating logics of search engines and their epistemic implications using a framework building upon a responsibilist account of justified belief. According to this, an epistemically responsible enquirer will aim at having true beliefs and will therefore perform all the necessary actions to collect sufficient evidence to support his belief, such as checking a broad enough range of e.g. web pages and comparing them to other types of sources (cf. Bråten, Brandmo, & Kammerer, 2018). There are, however, three cases where the enquirer may fail to acquire justification for his belief: 1) the enquirer neglects performing a proper search, 2) the enquirer performs a proper enquiry, but the results do not support his belief or 3) the activity to justify his belief is not possible, e.g. due to lack or impracticability of a technology. Assuming that an enquirer is literate enough to avoid the first case, he can still fail as in cases 2 and 3. In cases of internet searches the problem is that, due to the covert search logics, the enquirer may not even know that he has failed. He may believe that he has performed a proper search but, due to the search engine’s filtering and ranking, the results may not provide the full picture of facts required to justify or rule out the belief. Furthermore, due to the covert operating logic, it is impracticable (case #3) for the enquirer to assess the quality of the set of sources provided by the search engine. As shown above, the past decades’ technological development has induced changes in how individuals acquire information, and blurred the boundaries between personal knowledge and external information. The problem is not about using external memory aids or systems for offloading information (Säljö, 2012). As Säljö explains, man started developing external symbolic storages and artificial memory systems thousands of years ago, and memory aids such as Otto’s physical notebook (Smart, 2012) or address books in smartphones are everyday tools used to offload information from our memory. However, there is a risk that (especially young) users not only offload information but perhaps even outsource cognitive processes, since they may lack the epistemic competencies and practices required in this new information ecology (cf. Bhatt & MacKenzie, 2019; Fisher et al., 2015; Säljö, 2012; Sparrow et al., 2011). To provide a rationale for the approach of this study, the following sections will briefly review 1) epistemic beliefs as a research area, 2) how epistemic beliefs may relate to learning and 3) dimensions and tools for measuring epistemic beliefs. These sections also aim at explaining how this study was delimited and why some aspects, albeit frequently discussed in other studies, were not included in this study. Ståhl 31 | F L R 2.2 Epistemic beliefs as a research area William G. Perry’s (1970) study of college students’ ideas regarding Source and Certainty of knowledge is commonly regarded as the starting point for research on epistemic beliefs or personal epistemology Over the past decades, epistemic beliefs have been conceptualized in different ways (cf. Schraw, 2013). Some researchers conceive them as broad and developing stage-like. Other researchers conceive them as a set of more or less independent dimensions expressing beliefs about knowledge and learning, Marlene Schommer (1990; 1993) being the first in this line of research. The term ‘epistemic beliefs’ will be used here since the study will focus on the respondents’ (implicit and unconscious) views of knowledge, not their theories of knowledge or epistemology (cf. Kitchener, 2002; Hofer, 2008, p. 5). The works during the 1990ies of Marlene Schommer (1990; 1993, later Schommer-Aikins) and Barbara K. Hofer and Paul R. Pintrich (1997) in developing research around epistemological theories are important to acknowledge. During the first decade of this century, research around epistemic beliefs increased and extended from Perry’s original North American, white, elite, male college students context to other age groups and geographical and cultural contexts. For extensive overviews, please see the works by Hofer & Pintrich (2002), Niessen, Vermunt, Abma, Widdershoven, & van der Vleuten (2004), deBacker, Crowson, Beesley, Thoma, & Hestevold (2008) and Khine (2008). Further, the more recent works by Schraw (2013) Greene, Sandoval, & Bråten (2016), Bernholt, Gruber, & Moschner (2017) and Knight et al. (2017), out of which the four latter where not yet available at the time for planning this study. Domain-specificity and domain differences have been issues throughout the years. The initial assumption, that one’s epistemic beliefs are general across domains, has been questioned and instead, it has been suggested that one can hold different epistemic beliefs, depending on the field of knowledge one is dealing with (Muis, Bendixen, & Haerle, 2006). The longitudinal study by Trautwein & Lüdtke (2007), albeit focusing on the certainty dimension only, confirmed the hard-soft difference but also that students aiming at certain college programmes differed regarding their beliefs already at the end of their upper secondary education. In their large review, Muis et al. (2006) noted that empirical research had been presented in support for both domain general and for domain specific epistemic beliefs respectively, and that they may co-exist and possibly interact. The suggestions by Muis et al. were strongly supported by both Hofer (2006) and Alexander (2006). To conclude, I acknowledge the co-existence of and interaction between domain-general and domainspecific epistemic beliefs. The question regarding domain-generality vs. domain-specificity was, however, not the focus of the present study. The development of self-report instruments for measuring epistemic beliefs has encountered several challenges. In his review article, Schraw notes that there has been disagreement about the underlying conceptual structure, and replications of exploratory factor analyses (hereafter EFA and CFA will be used for exploratory and confirmatory factor analysis, respectively) have often failed. Common problems have been that items load in an unexpected manner often resulting in less factors or another factor structure than anticipated in the underlying conceptual model, too few items loading per factor and the resulting model showing a low explanation score. (Schraw, 2013) In the present study, I will subscribe to the line of research that considers the concept of epistemic beliefs as multidimensional. Assuming that hitherto described dimension sets are not sufficient to describe epistemic beliefs in the new information ecology, I attempt to introduce some new dimensions. The aim of testing new dimensions required starting on a general level and therefore, the questionnaire items (except for the internet-related items) did not refer to any specific discipline or context (section Instrument construction). 2.3 Epistemic beliefs and learning Alongside with motivation and cognitive styles, the concept of epistemic beliefs is an important factor affecting learning and study success. Hofer & Pintrich (1997) called for more research to understand how students’ epistemic beliefs may influence learning performance. Further, they suggested that the type of learning tasks may shape the students’ epistemic beliefs, as shown later by Kienhues, Bromme, & Stahl (2008). Ståhl 32 | F L R Brownlee, Walker, Lennox, Exley, & Pearce (2009) approached the topic of epistemic beliefs qualitatively, and their results highlight that first-year students may hold subjectivist or objectivist core beliefs that may decrease their ability to engage in critical thinking, required in higher education. Walker et al. (2009) also approached first-year students and identified some students being at risk of having difficulties in higher education due to their naïve beliefs about learning and knowing. There is also evidence suggesting cultural differences. Zhang & Watkins (2001) observed that Chinese students’ cognitive-developmental patterns were the opposite of the patterns observed in the U.S. sample. Their results also indicate that epistemic beliefs are not static but developing (cf. Kienhues et al., 2008). Further, Hofer (2008, pp. 11-12) observed differences between Japanese and US college students such that US students had more sophisticated beliefs about the factors describing certainty, simplicity, source and justification of knowledge. Education is moving towards methods of teaching and learning that often involve using Internet-based resources (e.g. the flipped classroom, Knewton, 2011). These methods require more self-regulation from part of the student, and e.g. Bråten (2008, pp. 369-370) highlights the risk that students with naïve epistemic beliefs may tend to over-reliance towards internet-based resources. Regarding the changes in teaching methods, it is worth noting that the teachers’ choices of pedagogical activities and learning settings are also influenced, perhaps unconsciously, by the teacher’s own epistemic beliefs (Palmer & Marra, 2008, p. 337). An overall awareness regarding epistemic beliefs is called for among teachers at all levels of education. An interesting attempt to support this awareness is the theoretical model between epistemic beliefs and self-regulation suggested by Muis (2007), where epistemic beliefs facilitate selfregulation and play a crucial role in all four phases (Task definition, Goal setting, Enactment and Evaluation) of the learning process. An example from a constructivist education context (PBL) is the study by Otting, Zwaal, Tempelaar, & Gijselaers (2010), where the results showed a connection between conceptions of expert knowledge and traditional conceptions of teaching and learning on one hand, and on the other hand a connection between learning effort and a constructivist conception of teaching and learning. The examples above illustrate that there is much going on within the educational context, most importantly that education is moving from being teacherand subject-centred towards being more studentand learning-centred. The development of the technological structures around ICT is increasingly beyond control of the educational system. However, learning analytics is an area where education is actively applying ICT: the core characteristic is the generation of high-resolution data about various types of [learning] actions (Knight, Wise, & Chen, 2017), and applying knowledge from multidisciplinary perspectives such as business intelligence, web analytics and data mining for analysis purposes (Ferguson, 2012). Thus, learning analytics can generate real-time individual and group performance information with potential to support teachers’ decision-making (Knight, Wise, & Chen, 2017). Knight et al. (2017) present a novel approach as they explore how students’ epistemic beliefs predict e.g. students search behaviour (traced using learning analytics methods). Their results did not show a convincing predictive value, whereas the results by Pieschl, Stallmann, & Bromme (2014) were a bit more encouraging. This issue is further commented in section Internet-specific epistemic beliefs. 2.4 Dimensions and measurement Within the line of investigation that regards epistemic beliefs as multidimensional, self-report instruments have been developed to capture the dimensions of epistemic beliefs. In her original 63-item Schommer Epistemological Questionnaire (SEQ), Schommer (1990; 1998) suggested the dimensions Simple knowledge, Certain knowledge, Innate ability, Quick learning and Omniscient authority. Using EFA, Schommer managed to extract four but not the Omniscient authority dimension. Thus, the dimensions described views on both knowledge and learning. Several authors (e.g. Hofer & Pintrich, 1997) have criticized Schommer for not performing factor analysis on the 63 original items but using 12 subscale scores (packages) Ståhl 33 | F L R based on those items, as variables. Still, Schommer’s questionnaire has been the starting point for a large part of later development regarding questionnaire-based instruments (for an overview, please see Niessen et al., 2004), out of which the following instruments, besides the SEQ, were used as reference in the present study: - Wood & Kardasch (2002) developed the Epistemological Beliefs Survey (EBS) containing 38 items, out of which 32 stemmed from or resembled items in the SEQ, and covering two SEQ dimensions. - Schraw, Bendixen, & Dunkle (2002) developed the Epistemic Beliefs Inventory (EBI) containing 28 items, out of which 17 stemmed from or resembled items in the SEQ. EBI reflected the same dimensions as the SEQ. - Moschner, Gruber, & Studienstiftungsarbeitsgruppe EPI (2005) developed the 43-item Fragebogen zur Erfassung epistemischer Überzeugungen (Questionnaire for capturing epistemic beliefs, hereafter FEE) containing nine items from SEQ. FEE included the three SEQ dimensions Certainty of knowledge, Learning ability and Omniscient authority. Additionally, the FEE proposed five new dimensions labelled Social aspects of knowledge, Value of knowledge, Culture related aspects of knowledge, Gender related approaches to knowledge and Reflective nature of knowledge. 2.4.1 Knowledge, knowing and learning The discussion whether epistemic beliefs should be limited to beliefs about knowledge and knowing, or whether beliefs about learning should be included, has been ongoing throughout the decades. Hofer & Pintrich (1997) recommended excluding beliefs about learning for the sake of clarity of the concept of epistemic beliefs. Instead, they retained Certainty and Simplicity of knowledge (describing nature of knowledge) and proposed the dimensions Source of knowledge and Justification for knowing to describe the nature of knowing. Schommer introduced an embedded systemic model that included Beliefs about Ways of Knowing, interplaying with Beliefs about Knowledge and Beliefs about Learning, i.e. beliefs about knowledge and learning as separate constructs but within the same system (Schommer-Aikins, 2004). Sandoval (2005) warned for conflation of the concepts. Although beliefs about knowledge will probably influence one’s beliefs about learning, Sandoval proposed that they should be investigated as separate constructs. In a comment to the discussion, Elby (2009) suggested that it is too early to decide and therefore, views on learning should at least for the time being be included in the concept of epistemic beliefs for further empirical and theoretical development. For the present study, data were collected regarding beliefs about both knowledge and learning and consequently, the analyses include both aspects. This approach is also supported by previous research presented in the section Epistemic beliefs and learning. 2.4.2 Internet-specific epistemic beliefs The point of departure for this study, the tendency not to look up information until needed and to rely on internet-based sources, is close to the research regarding internet-specific epistemic beliefs by Bråten, Strømsø and their teams. In 2005, they developed the Internet Specific Epistemic Questionnaire (ISEQ: Bråten, Strømsø, & Samuelstuen, 2005), which was based on the four dimensions described by Hofer & Pintrich (1997) and thus omitting learning dimensions. In performing EFA, Bråten et al. used Maximum Likelihood (hereafter ML) as extraction method together with an oblique rotation method but did, however, extract only two factors. They labelled the first one General Internet Epistemology, which included beliefs concerning the certainty and simplicity of Internet-based knowledge, as well as beliefs concerning the Internet as a source of knowledge, i.e. three dimensions in one factor. The second factor was labelled Justification for Knowing and described whether internet-based knowledge claims could be accepted without critical evaluation, or should they be critically evaluated using multiple sources, reasoning and prior knowledge. Ståhl 34 | F L R All eighteen ISEQ items referred to internet and thus, all questions connected explicitly and exclusively to the internet context. Further, when reading the ISEQ General Internet Epistemology items it seems obvious that they do not actually reflect the certainty or structure of knowledge (cf. corresponding items in Table 1) but rather, they mainly express the coverage and availability of information on the internet. Thus, the ISEQ seems to leave questions open about the respondent’s beliefs regarding certainty and simplicity of knowledge in general, about the beliefs regarding other sources of knowledge, and how these beliefs relate to each other; unanswered questions constituting a research gap. In a subsequent study, Bråten & Strømsø (2006) applied parts of the SEQ (Schommer, 1990), but not the ISEQ, to explore the connection between epistemic beliefs and internet-based search and communication activities. It turned out e.g. that students who believed in quick learning tend to overlook the importance of critically evaluating web-based resources. In another study, based on 17 out of 18 items in the ISEQ item set, the authors extracted only three factors (using ML and Direct Oblimin): Certainty and source of knowledge, Justification for knowing and Structure of knowledge (Strømsø & Bråten, 2010). The ISEQ has also been applied in other contexts and for other purposes: Karimi (2014), exploring the connection between internet-specific epistemic beliefs and grammar achievement, extracted the same three factors as Strømsø & Bråten (2010), although with Varimax rotation. Chiu, Liang, & Tsai (2013) used a Chinese translation of the ISEQ, and applied an EFA method (apparently with oblique rotation) but upon only twelve items. These authors did, however, not extract ISEQ dimensions as described by Bråten et al. (2005) but instead, the four dimensions originally suggested by Hofer & Pintrich (1997), i.e. Certainty, Simplicity and Source of knowledge and Justification for knowing, but using items specifically denoting an internet-based context. Kammerer & Gerjets (2012) applied ISEQ to categorize users for comparison, but they only used eight items attributed to the ISEQ-dimension Certainty and Source of knowledge and thus, did not test the factor structure proposed in the original ISEQ. The study by Knight et al. (2017) exemplifies a research approach linking epistemic beliefs with log data analytics. They used the ISEQ in an extensive study to explore whether the two-factor ISEQ scores could predict e.g. trustworthiness ratings of internet-based sources or traced search behaviour. According to their results, the factor scores did not predict search behaviour, and they had only small predictive value for trustworthiness rating. The approach by Knight et al. is interesting and relevant but raises the question: Could the connections to search behaviour have turned out differently had they not used the two-factor ISEQ, where the General Internet Epistemology factor contains a mix of Certainty, Structure and Source of knowledge? E.g. the results by Pieschl et al. (2014), indicate that students’ epistemic beliefs influence how they approach complex tasks. To conclude, epistemic beliefs have been explored also in relation to internet-based information, but the picture is disparate. The studies referred to above, as well as many other studies, suffer from the problems addressed by Schraw (2013). The studies published prior to the present data collection (Bråten et al., 2005; Bråten & Strømsø, 2006; Strømsø & Bråten, 2010) focused on beliefs about internet-based information without actually relating these beliefs to beliefs about knowledge based on other information sources. The studies referred to above also leave the question open, whether internet should be regarded as an authority or knowledge source, or a specific context (cf. Grossnickle Peterson, Alexander, & List, 2017, p. 262). 2.4.3 Justification for knowing Hofer & Pintrich (1997) introduced Justification for knowing as a dimension, which was later supported by several researchers. Both Alexander (2006) and Greene, Azevedo, & Torney-Purta (2008) have noted that this dimension is least developed, and that exploring justification is more challenging than exploring other dimensions. This assumption seems well founded considering the complexity of the justification aspect, e.g. in terms of the responsibilist account of justified belief suggested by Miller & Record (2013) (see section Personal knowledge, external information). Ståhl 35 | F L R Greene et al. (2008) point out two aspects that are part of the challenge in investigating the justification dimension. First, considering the number of different kinds of justification identified in philosophy, justification as part of the epistemic beliefs model will probably require to be described by multiple factors rather than one single factor. Further, Greene et al. suggest that a person needs to have a sophisticated ontology of a domain before issues of justification, such as critical thinking, become relevant. This seems congruent both with Bloom’s original cognitive process dimensions and especially with the knowledge dimensions described later by Krathwohl (2002): issues of justification are probably far more relevant when applying, analysing or evaluating conceptual knowledge than when recalling facts. The above suggestion by Greene et al. comes to expression in a recent study by Bråten, Brandmo & Kammerer (2018), where they delimit the context to internet and the domain to that of educational topics within teacher education. Their study focuses solely on the justification dimension, approaching it as a threedimensional concept including justification by authority, justification by multiple sources and justification against prior personal knowledge and reasoning. As a result, they present the validated Internet-Specific Epistemic Justification Inventory (ISEJ). Against the background of the considerations referred above, and the fact that epistemic beliefs in the new information ecology was totally uncharted territory, it seemed appropriate to leave the Justification dimension outside this investigation. Hence, the FEE instrument (Moschner et al., 2005) was chosen as a starting point (see section Instrument construction). 2.5 Research questions Capturing all dimensions of epistemic beliefs (or cognition, cf. Greene et al., 2008) while at the same time adding and testing new dimensions would be both adventurous and beyond this study. Therefore, while acknowledging that epistemic beliefs consist of multiple dimensions developing over time, this study adopts a narrow focus on capturing a snapshot of the participants’ current epistemic beliefs, including beliefs in internetbased information. Thus, the Justification dimension as well as the topics regarding subject-, domain-, discipline-, cultureor gender-specificity of epistemic beliefs (see e.g. DeBacker et al., 2008) are beyond the scope of this study. The approach of this study is openly explorative in testing whether it is possible, overall, to extend the existing instruments and their dimension sets with new dimensions of epistemic beliefs, and specifically to capture such ways of relating to knowledge that have become common among frequent internet users during the past decades. Further, this study will explore the relation between existing epistemic dimensions and those describing internet-based knowledge and knowing. Apart from ISEQ (Bråten et al., 2005), this study does not aim to explore how individuals justify internet-based information, but rather to explore whether and to which extent individuals rely on and prefer internet-based information sources, and how this preference relates to other epistemic dimensions. The investigation is framed in a single research question: (How) can the set of epistemic beliefs dimensions be extended so that it also expresses a googling attitude? The research question is openly phrased since, although research on epistemic beliefs has been going on for some time, the proposed dimensions are on uncharted territory. For the sake of clarity, I will use the term original dimensions for those dimensions described in or stemming from Schommer’s SEQ (1990). Hypothesized dimensions will be used to denote suggested dimensions until their existence has been confirmed, after which they are denoted as novel dimensions or scales in the proposed model, which is the endpoint of the present study. Ståhl 42 | F L R with two non-substantial options, ‘don’t know’ and ‘don’t understand’. This is partly supported by Muis et al. (2006, p. 25), noting that it has not been empirically studied what individuals actually think as they fill out questionnaires. Providing both options was especially important when introducing new items, since these options provided information regarding comprehensibility, potentially valuable when considering items to exclude (cf. Finch, Immekus, & French, 2016, p. 144). Further, the non-substantial options were placed on both sides of the substantial options in order not to distort the visual midpoint of the Likert-type response format (cf. Tourangeau, Couper, & Conrad, 2004). In survey presentation, it was necessary to prevent fatigue effect and satisficing (cf. Cape, 2010), and any effect where question context or order might influence question interpretation (cf. Martin, 2005, p. 726; Tourangeau et al., 2004). Therefore, a progress indicator was included and the items were distributed over twelve pages containing four to six items each, which also improved readability. Further, to prevent inter-item influence, each subscale’s items were distributed over different pages (e.g. the page in Figure 1 containing items from five subscales) and the survey service was set to randomise item order within each page. 3.3 Research data and sample characteristics The present study is based on data collected in 2012, where 371 students chose to complete the survey ‘Me and my knowledge’. Only those cases containing substantial responses to more than 70% of the items where retained for further analyses (n = 348). The 23 excluded cases had responded only to first-page items and were therefore regarded as dropouts. The complete data set with 371 cases exhibited missing values increasing from 4.2% up to 11.7% on page level, whereas this trend in the 348-case subsample developed from 2.5% to 7.0%. This, together with the dropouts, indicates that most respondents who started the survey also completed it, and that an actual fatigue effect was avoided. On item level, the portion of missing values ranged from 1.4% to 10.1%, where the two Certainty of knowledge items k13_2F44 and k14_2F49 (Table 1) showed the highest portions of missing values, mostly ‘don’t know’ responses. The highest ‘don’t understand’ portions occurred for three items representing the dimensions Just-in-time learning (k04_1), Constructivist approach (k07_2) and Connectivist approach (k08_5). Since the questionnaire applied a Likert-type response format producing data on an ordinal scale, it is not meaningful to analyse distribution or assess normality on item level (cf. Carifio & Perla, 2007) but instead, analysis of the actual scales is postponed to the discussion section. For those calling for an item level analysis it can be mentioned that for each item, the response value ranged over the whole scale (1..6). The items showed a standard deviation between 1.02 and 1.66, a skewness between -1.23 and 0.91 and a kurtosis between -1.26 and 1.38. The criterion of the skewness and kurtosis value being within the range ±1 was met regarding 57 and 55 items, respectively. The Shapiro-Wilks test suggested non-normal distribution, whereas the KolmogorovSmirnov test suggested normal distribution throughout all items. A visual inspection of histograms, normal QQ plots and box plots showed that the items were approximately normally distributed. For the items showing skewness or kurtosis outside the ±1 range, the deviation was minor and further, the sample size was large enough to reduce a possible detrimental effect (cf. Hair, Black, Babin, & Anderson, 2010). Based on the aforementioned criteria, the items were considered as normally distributed. The current 348-case subsample holds students from twelve degree programmes, both domestic and international students (86.8% / 13.2%), and a gender distribution holding 66% female students. The age average was 21.7 with 91% being born in 1986-1995. For this study, sample demographics should be reviewed in relation to access to internet resources. Internet and publicly available search engines were launched already in the mid 1990’ies and during the following ten years, search engine use was established (http://www.searchenginehistory.com/). 2011-2012 were the very years when internet services, previously available via computers, became truly ubiquitous due to 3G/4G-connected smartphones becoming everyday tools, and Finnish net operators offering affordable 3G/4G-subscriptions including generous mobile data. The mobile phone prevalence within both cohorts was close to 100%. Smartphone as a concept was not yet established and thus, the corresponding survey item was phrased “My mobile phone is connected to the internet”. From 2011 to 2012, the portion of users across the Ståhl 43 | F L R cohorts having an internet-connected phone increased among domestic students from 48.7 to 81.3% and among international students from 60.7 to 90.6%, within the total cohorts from 50.2 to 82.5%. This corresponds well with the national statistics, according to which 53% of those aged 16-24 had a smartphone in the spring of 2011 (OSF, 2011). At the time of data collection, the respondents had been exposed to computers, mobile phones and internet for in average 12, 10 and 9 years respectively. To conclude, the sample can be regarded a rather typical Net Generation cohort. 3.4 Analysis methods Fabrigar, Wegener, MacCallum, & Strahan (1999) argue that Principal Component analysis is not a true method of factor analysis. They recommend the use of Maximum Likelihood, as later supported by Osborne (2014, p. 9) and Finch et al. (2016, p. 131). Thus, the analysis procedure starts with an EFA with ML as extraction method, followed by a validation procedure including EFA and CFA on split halves of the sample (Fabrigar et al., 1999; Fokkema & Greiff, 2017; Knight et al., 2017; Leal-Soto & Ferrer-Urbina, 2017; Osborne, 2014, pp. 6, 119-120; Tang, 2010). For all statistical tests, a significance level of .05 was used and in EFA and CFA procedures, the absolute loading value .32 was used as the threshold when assessing item (non-)loadings and cross-loadings (cf. Finch et al., 2016, p. 143). In table and diagram presentations, loadings <.32 are generally not displayed although during EFA procedures, low loadings were not suppressed since that may cause loss of valuable information (such as item k14_5, Table 3). The SPSS software package (SPSS, 2016b) was used for EFA procedures, and the CFA procedures were performed using the Amos software package (SPSS, 2016a). To build analysis on true data, missing data values were not imputed, since any kind of replaced or imputed values are, after all, only estimates. This choice was made at the cost of listwise deletion reducing the number of cases in EFA, and missing values thwarting the use of modification indices to support refinement in CFA. 4 Results In this section, the results are presented together with analyses, since some results inform the subsequent steps. The reasoning behind e.g. item disposal, number of factors and factor labelling will be presented in conjunction with EFA on the complete item set. 4.1 Original and hypothesized dimensions of epistemic beliefs 4.1.1 Replicating original dimensions For replication purposes, EFA was first performed on the 27 items associated with the original dimensions. Dysfunctional items (zero, low and cross-loading) were stepwise discarded (cf. Finch et al., 2016, pp. 143-144). A model based on 18 items showed good fit indices and was interpreted as a successful replication (despite ML extraction and Promax rotation). 4.1.2 Emerging dimensions The 60-item questionnaire contained 33 items that were associated with six hypothesized dimensions: Reflective nature of learning, Connectivist approach, Just-in-time learning, Constructivist approach, Internet reliance and Valuing diversity (Table 1). Seeking inspiration from Bråten et al. (2005) and Trautwein & Lüdtke (2007) who analysed only one or two factors, this item subset was initially factor analysed separately in order to identify dysfunctional items. Ståhl 44 | F L R The EFA on the hypothesized dimensions was truly exploratory, including different rotation methods, varying the number of extracted factors and stepwise reduction of dysfunctional items (Finch et al., 2016, pp. 143-144; Osborne, 2014, pp. 17, 30-33). Using ML, Promax rotation and Listwise deletion, the EFA resulted in a four-factor model based on 23 items (n=191). The model showed good fit indices (Eigenvalues 6.35 .. 1.5, 51% of variance explained; KMO=.864, Bartlett's Chi-Square=1397, df=253, Sig.<.000, Goodness-of-fit Test Chi-Square=182.7, df=167, Sig.=.192) and reflected three of the hypothesized dimensions: Constructivist approach, Connectivist approach, Internet reliance and a fourth one, now labelled Learning by dialogue. Each item loaded strongly on one factor without cross-loadings. Throughout the various models, Constructivist approach and Connectivist approach correlated strongly, and several of the other factors correlated weakly with each other (Table 2). Table 2 Factor correlation matrix, four-factor model based on 23 new items Inter-factor correlations Constructivist approach Connectivist approach Internet reliance Learning by dialogue Constructivist approach .516 .067 .269 Connectivist approach .516 .287 .194 Internet reliance .067 .287 .030 Learning by dialogue .269 .194 .030 4.1.3 An extended set of dimensions Since inter-factor correlation occurred in all the previous analyses, EFA on the complete item set were performed using ML extraction and Promax or Oblimin as oblique rotation methods (cf. Finch et al., 2016, p. 133; Knight et al., 2017; Osborne, 2014, pp. 30-33; Strømsø & Bråten, 2010). The model was stepwise refined by removing low-loading and cross-loading items while simultaneously assessing their conceptual relevance, their communality estimates and their internal consistency within the anticipated scale. During the process, the six reversely phrased items occurring in four dimensions (Table 1) were discarded due to dysfunctionality. Thus, within all the hypothesized dimensions, the items were unidirectional. The refinement procedure boiled down to a model with seven factors and 26 items that fit the data reasonably well (Table 3). Both original and hypothesized dimensions appeared distinctly without crossloadings (except for items k10_7 and k14_5), each dimension loaded on at least 3 items, and 25 out of 26 items loaded (>.32) on the anticipated factor. Ståhl 45 | F L R Table 3 The proposed EFA model based on 26 items Item # Item label Constructivi st approach Omniscient authority Internet reliance Structure of knowledge Learning by dialogue Certainty of knowledge Learning ability k14_1F48 After thorough consideration, I can often see a problem with new eyes .822 k14_4F51 New experiences cause me to view knowledge in another way .628 k09_2 Knowledge is about seeing a matter from diverse perspectives, building up the whole .546 k09_1F28 When I learn new things I often see my previous knowledge in a new light .446 k07_2 Learning is about recognizing patterns and connections between concepts and phenomena .382 k07_4 Knowing is not an end state, but an on-going process .341 k05_2 The value of knowledge lies in its ability to explain an issue from multiple perspectives .323 k09_3F29 I have to accept the answers from a teacher as true .822 k12_5F42 Teachers are almost always right .520 k14_7 I seldom or never question authorities .475 k12_6 I learn things quicker from Internet pages than from books .888 k13_6 Internet sources usually provide me with a clearer picture of subjects than do books .719 k14_5 I can get almost all the information I need to know about a subject from one or two Internet sources (.30) (.25) (.23) k11_8 It bothers me when teachers do not tell me the answers to complicated problems .699 k13_7 I find it annoying to listen to teachers who cannot make their mind up about what they believe .557 k10_7 It bothers me when a teacher does not say clearly what I am supposed to know in an examination .364 .537 k11_9 I prefer topics where most problems have only one right answer .448 k03_5 I create knowledge by interacting with others .753 k03_7 I like discussing with people who have varying opinions .519 k04_3 When I need to learn about something new, I often ask several friends about their opinion .405 k03_8 A true fact today will also be a true fact tomorrow .613 k04_2F13 There are scientific facts that will never change .587 k04_4F04 All experts within a field have the same understanding regarding the basic issues of that field .424 a) k09_5F31 Each person needs to learn for her-/himself how she/he learns .723 k13_4F47 It is possible to learn how to learn .557 k11_1F33 A focussed use of learning techniques will lead to better results .464 Ståhl 46 | F L R Factor Correlation Matrix Constr. approach Omnisc. authority Internet reliance Structure of knowl. Learning by dial. Certainty of knowl. Learning ability Constructivist approach .031 -.021 .115 .282 -.023 .506 Omniscient authority .031 .081 .319 -.062 .376 .030 Internet reliance -.021 .081 .143 -.026 .143 -.110 Structure of knowledge .115 .319 .143 -.010 .221 .205 Learning by dialogue .282 -.062 -.026 -.010 -.075 .195 Certainty of knowledge -.023 .376 .143 .221 -.075 -.005 Learning ability .506 .030 -.110 .205 .195 -.005 Table footnotes ML, Promax rotation converged in 16 iterations; Listwise deletion, n=195, Eigenvalues 4.85 .. 1.01, 59.3% of variance explained; KMO=.782, Bartlett's Chi-Square=1468, df=325, Sig. <.000, Goodness-of-fit Test Chi-Square=169.8, df=164, Sig.=.361 a) Item loading not consistent with hypothesized dimension but conceptually coherent. From this model, several items were dropped due to loading weakly or inconsistently with the hypothesized dimension. The retained original items loaded on the same factors as in the EQ, except for item k04_4F04 that was originally associated with Omniscient authority. The current loading on Certainty of knowledge can be regarded as conceptually coherent. The issues regarding which items to discard, how to decide on the number of factors, and how to label the subscales require some comments. As Osborne (2014, pp. 17-18) notes, EFA is a low-stakes procedure and expressly exploratory. Accordingly, I entered the process with 60 items, a hypothesized underlying conceptual model, and used the statistical package (SPSS, 2016b) to provide suggestions for a factor model. During EFA iterations, the dysfunctional items were eventually revealed and discarded. Besides varying extraction and rotation, the search for an adequate number of factors included extracting factor sets ranging from two factors below up to two factors above the number suggested by the scree plot elbow (Osborne, 2014, p. 18). This method provided valuable information: increasing the number of factors caused related items to split over several factors, whereas reducing factors caused items to pile up on one factor, usually then holding items from dimensions that at the end turned out to correlate. Thus, the search for a factor model included weighing of theory, scree plot, item loadings and communalities, eigenvalues, internal consistencies and conceptual considerations. During the EFA iterations, the hypothesized dimensions did not turn out quite as anticipated, which is only part of the nature in explorative work (cf. Osborne, 2014, p. 17). In most of the explored models, the five hypothesized dimensions boiled down to three (Table 3). The dimension Learning by dialogue holds items from the suggested dimensions Valuing diversity and Connectivist approach, whereas the dimension Constructivist approach besides its own items also holds items from the hypothesized dimensions Valuing diversity and Reflective nature of learning. All the three items originally associated to Internet reliance consistently loaded on that dimension (Table 1). Retaining the items k14_5 and k10_7 violates the rule of using only strong, single-loading items and requires commenting. The internal consistency test showed that deleting the item k14_5 entailed a slightly improved alpha value, but at the cost of reducing the factor Internet reliance to only two items. Further, since the item communality value was reasonably good, the connection to Structure of knowledge occurred also in the path diagram, and the CFA indicated that discarding the item impaired fit indices, there was enough support for retaining the item. As expected, the item k10_7 loaded strongly on the Structure of knowledge dimension, but surprisingly also on Constructivist approach. The item was retained since discarding it would have impaired the Structure of knowledge internal consistency considerably. The split half EFA suggested single-loading on Structure of knowledge, whereas the CFA suggested a connection to Constructivist approach and indicated that discarding the item impaired fit indices. Ståhl 47 | F L R Three of the original dimensions (except Learning ability) correlated weakly with each other. Further, Constructivist approach correlated strongly (.506) with Learning ability and weakly (.282) with Learning by dialogue. 4.2 Evaluating the extended instrument For the purpose of evaluating the stability of the model presented in Table 3, the data set was randomly split into two equal halves A and B, that were subject to EFA and CFA, respectively (cf. Fokkema & Greiff, 2017, p. 401). 4.2.1 Exploratory split half An EFA was performed with 26 items on the split half A using the same methods as in the initial model. The model arrived at (Table 4) did not show a one-to-one correspondence to the initial model (Table 3) but resembled it strongly, with 22 out of 26 items loading as anticipated. All the hypothesized dimensions were reflected in the seven factors, although two of the Constructivist approach items loaded on the Learning ability factor (a). Further, two items loaded on unexpected factors (b), and the Learning by dialogue item k03_5 caused a Heywood case. Still, the fit indices suggested that the proposed model, appearing almost similar in both Oblimin and Promax rotation, fit also the split data set fairly well. Ståhl 48 | F L R Table 4 Exploratory Factor Analysis on split half A Item # Item label Learning by dialogue Constructivi st approach Internet reliance Omniscient authority Structure of knowledge Learning ability Certainty of knowledge k03_5 I create knowledge by interacting with others 1.046 k03_7 I like discussing with people who have varying opinions .306 k14_1F48 After thorough consideration, I can often see a problem with new eyes .889 k14_4F51 New experiences cause me to view knowledge in another way .653 k09_2 Knowledge is about seeing a matter from diverse perspectives, building up the whole .618 k07_4 Knowing is not an end state, but an on-going process .451 k09_1F28 When I learn new things I often see my previous knowledge in a new light .369 k12_6 I learn things quicker from Internet pages than from books .914 k13_6 Internet sources usually provide me with a clearer picture of subjects than do books .703 k09_3F29 I have to accept the answers from a teacher as true .751 k12_5F42 Teachers are almost always right .640 k14_7 I seldom or never question authorities .467 k04_3 When I need to learn about something new, I often ask several friends about their opinion (.217) .334 b) k10_7 It bothers me when a teacher does not say clearly what I am supposed to know in an examination -.693 k11_8 It bothers me when teachers do not tell me the answers to complicated problems -.632 k13_7 I find it annoying to listen to teachers who cannot make their mind up about what they believe -.584 k11_9 I prefer topics where most problems have only one right answer -.436 (.320) k09_5F31 Each person needs to learn for her-/himself how she/he learns .755 k11_1F33 A focussed use of learning techniques will lead to better results .649 k07_2 Learning is about recognizing patterns and connections between concepts and phenomena .402 a) k13_4F47 It is possible to learn how to learn .358 k05_2 The value of knowledge lies in its ability to explain an issue from multiple perspectives .337 a) k03_8 A true fact today will also be a true fact tomorrow .575 k04_4F04 All experts within a field have the same understanding regarding the basic issues of that field .573 k04_2F13 There are scientific facts that will never change .454 k14_5 I can get almost all the information I need to know about a subject from one or two Internet sources (.229) .448 b) Ståhl 49 | F L R Factor Correlation Matrix Learning by dial. Constr. approach Internet reliance Omnisc. authority Structure of knowl. Learning ability Certainty of knowl. Learning by dialogue .210 -.051 -.010 .267 .169 -.147 Constructivist approach .210 .048 .010 -.152 .494 -.054 Internet reliance -.051 .048 .076 -.140 -.059 .157 Omniscient authority -.010 .010 .076 -.259 .131 .363 Structure of knowledge .267 -.152 -.140 -.259 -.208 -.126 Learning ability .169 .494 -.059 .131 -.208 -.063 Certainty of knowledge -.147 -.054 .157 .363 -.126 -.063 Table footnotes a) Item loading not consistent with hypothesized dimension but conceptually coherent b) Item loading not consistent with hypothesized dimension, vague conceptual coherence ML, Oblimin rotation converged in 12 iterations; Listwise deletion n=96, Eigenvalues 5.00 .. 1.06, 62.9% of variance explained; KMO=.706; Bartlett's Chi-Square=928, df=325, Sig. <.000; Goodness-of-fit Test Chi-Square=179.5, df=164, Sig.=.193 As in the initial model, the Constructivist approach factor correlated strongly with the Learning ability factor (.494) and Omniscient authority correlated with Certainty of knowledge (.363). 4.2.2 Confirmatory split half CFA was performed on the same 26 items as the previous EFA but on the split half B (n=174) of the data set. In the first step, conceptually irrelevant and low connections between latent variables were removed, which resulted in an initial model with partly insufficient fit indices. Assessing model fit and choice of cut-off criteria (in brackets) follow the recommendations by Schreiber, Nora, Stage, Barlow, & King (2006) and Hooper, Coughlan, & Mullen (2008). Since the data set contained empty cells, it was not possible to utilize the feature where the Amos software would provide suggestions for modification (SPSS, 2016a). Instead, model refinement was performed manually, partly following loadings and correlations indicated in the previous EFA models (Tables 3 and 4), and partly by adding and removing connections based on conceptual considerations in an exploratory manner. Thus, some connections between latent variables, although weak, were retained, under the condition that they were conceptually defensible and contributed to improving fit indices. Then again, in some cases conceptually defensible connections had to be discarded if their loading value was low (mainly <.32) and retaining them impaired the fit indices. The procedure resulted in a conceptually defensible path diagram (Figure 2), similar to the EFA models (Tables 3 and 4) and reasonable although not perfect fit indices (Chi-square/df=1.504, RMSEA=.054, TLI=.821, CFI=.853, PCLOSE=.260). The item k13_6 caused a minor Heywood case (1.01), which was accepted since any attempt to manipulate constraints impaired fit indices. Ståhl 50 | F L R Figure 2. Simplified CFA path diagram based on 26 items and split half B data set (cut-off criteria in brackets). Chi-square/df=1.504 (<2), RMSEA=.54 (<.60), TLI=.821 (≥.95), CFI=.853 (≥.95), PCLOSE=.260 (≥.05). Ståhl 51 | F L R 5 Discussion 5.1 Construct validity After the seven dimensions had been identified (Table 3), they appeared stable throughout the succeeding analyses. In some cases, items associated with the dimensions Constructivist approach and Learning ability cross-loaded. This phenomenon is conceptually coherent considering the strong correlation between these factors that, in turn, is possibly due to a latent second-level variable. The internal replications by exploratory and confirmatory factor analyses on randomized split halves provide information speaking in favour of the proposed factor model. Since the suggested construct holds good or reasonable fit indices and behaves in a consistent manner throughout the different analyses, it can be regarded as holding initial construct validity. Initial meaning here that the present study was only a first attempt to launch the hypothesized dimensions, and further research (with new data and adjusted items) is still required to test the generalizability of the construct (Finch et al., 2016, pp. 127-128). Further testing should also involve a diverse student population with regards to domains and cultural background. 5.2 Content validity In general, the factors reflect both original and hypothesized dimensions. In all EFA models (Tables 3 and 4) as well as in the CFA path diagram (Figure 2), the highest loading on each factor occurred on one of the anticipated items. Regarding the original dimensions, it turned out that 12 out of the 27 items reflected the anticipated construct and one item loaded differently than in previous studies (see rightmost column in Table 1). Within the hypothesized scales, 13 items where included in the novel scales. To answer the question if and to which extent the factors actually describe the dimensions, this section presents comments regarding each dimension in the model (Table 3, Figure 2). The original dimensions retained their original labels, and the labelling of the novel dimensions is commented in section Novel dimensions. Choices regarding factor model and number of factors were discussed in section An extended set of dimensions. 5.2.1 Original dimensions Learning ability Three of the four items in the Learning ability dimension proved stable across most analyses and models, whereas the item k12_9 was dropped at an early stage. In some EFA models, this factor attracted items from the Constructivist approach dimension, which is consistent with the strong correlation between these dimensions (Table 3, Figure 2). Omniscient authority In her earliest studies, Schommer (1990; 1998) reported this dimension as difficult to capture, whereas Schraw et al. (2002, p. 267) and Moschner et al. (2005) were able to identify this dimension. In this sample, the authority dimension manifested clearly in all models, and the items k09_3F29, k12_5F42 and k14_7 loaded consistently on the Omniscient authority factor. Structure of knowledge Throughout the analyses, most of the Structure of knowledge items loaded as expected. Item k12_7 often loaded on the Learning ability factor but for this item, a connection to learning is not far-fetched; combining information across sources may express an active stance towards learning, rather than a view of the Structure of knowledge. Thus, this item may be an example of a phrasing containing something that might be Ståhl 58 | F L R they show between-groups variations congruent with the googling attitude they are expected to express. A connection between epistemic beliefs and academic performance has been suggested (e.g. Aditomo, 2018). If the instrument for measuring epistemic beliefs can be developed to measure more precisely, it will probably have a predictive value in assessing each student’s epistemic competence in relation to study context, and a value for teaching practices in supporting students’ epistemic competencies by appropriate choice of learning activities. On this point, the picture is disparate with both encouraging (Pieschl et al., 2014) and discouraging (Knight et al., 2017) results and thus, epistemic beliefs as predictors of learning behaviour seems an underresearched area. However, net-based learning environments (LMS, VLE) having started to include learning analytics features will provide better possibilities to investigate the connection between students’ epistemic dimensions and trace data from authentic learning contexts, i.e. courses. There are indicators suggesting that epistemic beliefs dimensions should be measured on a sufficiently fine-grained level, since coarsely composed dimensions as the General Internet Epistemology (Knight et al., 2017), will blur the picture. The study by Trautwein & Lüdtke (2007), focusing on the certainty dimension, is an interesting initiative in this line. The recent study by Bråten, Brandmo & Kammerer (2018) expresses what we might call increased granularity: besides focusing only on the Justification dimension, they divide it into three sub-dimensions, justification by authority, multiple sources and personal knowledge. These examples, together with earlier replication problems (Schraw, 2013) expose a challenging tension: should we measure epistemic beliefs as a set of dimensions or as separate constructs? The proposed model arrived at (Table 3) and confirmed by internal replication (Table 4 and Figure 2) shows fit indices that are not ideal but sufficient to encourage further development. Despite deficiencies, the model provides an interesting input to the debate whether epistemic beliefs should include only views on knowledge, or also views on learning. The CFA path diagram (Figure 2) provides an illustration to this debate: two groups of latent variables, the upper group describing views on learning, and the lower one describing views on knowledge. It is not far-fetched to imagine two second-level latent variables, influencing views on knowledge and views on learning, respectively (cf. section Correlating dimensions). The correlations within the two groups of latent variables, especially the strong correlation between Constructivist approach and Learning ability, also point in this direction, and exploring second-level latent constructs is a topic for further investigation. Topics dealing with the instrument itself include 1) developing the instrument such that each dimension would be represented by more than only three items (cf. Carifio & Perla, 2007), 2) improving items with low loadings, and 3) exploring the discarded items regarding common features that might have contributed to their dysfunctionality. In addition to these topics, the functionality of the model should be tested by exploring how well the dimensions distinguish different learners. This will be done by exploring if and to which extent dimensional group differences can be identified e.g. across users representing different digital orientations or study domains. The extensions to the epistemic beliefs instrument and the proposed (but not validated) model are, needless to say, only a beginning. Considering the twenty years of history with SEQ and its successors gives an idea of the work that still lies ahead. Ståhl 59 | F L R Keypoints The novel dimension Internet reliance may help in identifying learners that are over-reliant towards Internetbased resources. Beliefs about learning contribute to describing one’s epistemic orientation, although they are not regarded as part of the epistemic beliefs concept. Although assumed to develop independently, the epistemic beliefs dimensions correlate when using an appropriate rotation method. The novel dimensions contribute to an epistemic awareness and to adapting instruction and learning practices to learners’ epistemic orientations. An increasingly international learning context and multicultural student body requires awareness about culturally induced epistemic orientations. Acknowledgments This research was funded by Föreningen Konstsamfundet, Koulutusrahasto and Svenska Kulturfonden. I am grateful for the support provided by Arcada through Filip Levälahti and all participating students during the data collection process, and for the support from the MEDA project through Matteo Stocchetti. I am grateful also for the feedback provided by my supervisors Marita Mäkelä and Eero Sormunen, and by anonymous reviewers to earlier drafts of this paper. References Aditomo, A. (2018). Epistemic beliefs and academic performance across soft and hard disciplines in the first year of college. Journal of further and Higher Education, 42(4), 482-496. doi:10.1080/0309877X.2017.1281892 Alexander, P. A. (2006). What Would Dewey Say? Channeling Dewey on the Issue of Specificity of Epistemic Beliefs: A Response to Muis, Bendixen, and Haerle (2006). Educational Psychology Review, 18(1), 55-65. doi:10.1007/s10648-006-9002-7 Anderson, S., & Balsamo, A. (2008). A Pedagogy for Original Synners. In T. McPherson (Ed.), Digital Young, Innovation, and the Unexpected (1st ed., pp. 241-259). Cambridge, MA: The MIT Press. doi:10.1162/dmal.9780262633598.241 Bernholt, A., Gruber, H., & Moschner, B. (Eds.). (2017). Wissen und Lernen. Wie epistemische Überzeugungen Schule, Universität und Arbeitswelt beeinflussen [Knowing and learning. The influence of epistemic beliefs on schools, universities and working life]. Münster: Waxmann Verlag. Bhatt, I., & MacKenzie, A. (2019). Just Google it! Digital literacy and the epistemology of ignorance. Teaching in Higher Education, 24(3), 302-317. doi:10.1080/13562517.2018.1547276 Bråten, I. (2008). Personal Epistemology, Understanding of Multiple Texts, and Learning Within Internet Technologies. In M. S. Khine (Ed.), Knowing, Knowledge and Beliefs: Epistemological Studies across Diverse Cultures (pp. 351-376). Dordrecht: Springer. doi:10.1007/978-1-4020-6596-5_17 Bråten, I., Brandmo, C., & Kammerer, Y. (2018). A Validation Study of the Internet-Specific Epistemic Justification Inventory With Norwegian Preservice Teachers. Journal of Educational Computing Research, (OnlineFirst), 1-24. doi:10.1177/0735633118769438 Bråten, I., & Strømsø, H. I. (2006). Epistemological beliefs, interest, and gender as predictors of Internetbased learning activities. Computers in Human Behavior, 22(6), 1027-1042. doi:10.1016/j.chb.2004.03.026 Bråten, I., Strømsø, H. I., & Samuelstuen, M. S. (2005). The Relationship between Internet-Specific Epistemological Beliefs and Learning within Internet Technologies. Journal of Educational Computing Ståhl 60 | F L R Research, 33(2), 141-171. doi:10.2190%2FE763-X0LN-6NMF-CB86 Brownlee, J., Walker, S., Lennox, S., Exley, B., & Pearce, S. (2009). The first year university experience: using personal epistemology to understand effective learning and teaching in higher education. Higher Education, 58(5), 599-618. doi:10.1007/s10734-009-9212-2 Cape, P. (2010). (2010). Questionnaire Length, Fatigue Effects and Response Quality Revisited. Paper presented at the Re:Think 2010: The ARF 56th Annual Convention, NY. Retrieved from https://www.surveysampling.com/ Carifio, J., & Perla, R. J. (2007). Ten Common Misunderstandings, Misconceptions, Persistent Myths and Urban Legends about Likert Scales and Likert Response Formats and their Antidotes. Journal of Social Sciences, 3(3), 106-116. doi:10.3844/jssp.2007.106.116 Chiu, Y., Liang, J., & Tsai, C. (2013). Internet-specific epistemic beliefs and self-regulated learning in online academic information searching. Metacognition and Learning, 8(3), 235-260. doi:10.1007/s11409-0139103-x Clark, A., & Chalmers, D. (1998). The extended mind. Analysis, 58(1), 7-19. doi:10.1093/analys/58.1.7 DeBacker, T. K., Crowson, H. M., Beesley, A. D., Thoma, S. J., & Hestevold, N. L. (2008). The Challenge of Measuring Epistemic Beliefs: An Analysis of Three Self-Report Instruments. Journal of Experimental Education, 76(3), 281-312. doi:10.3200/jexe.76.3.281-314 Downes, S. (2007). An Introduction to Connective Knowledge. Paper presented at the Media, Knowledge and Education: Exploring New Spaces, Relations and Dynamics in Digital Media Ecologies, ed. T. Hug, innsbruck university press, Innsbruck, Austria, 2007, June 25-26, pp. 77-102. Elby, A. (2009). Defining Personal Epistemology: A Response to Hofer & Pintrich (1997) and Sandoval (2005). Journal of the Learning Sciences, 18(1), 138-149. doi:10.1080/10508400802581684 Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the Use of Exploratory Factor Analysis in Psychological Research. Psychological Methods, 4(3), 272-299. Ferguson, R. (2012). Learning analytics: drivers, developments and challenges. International Journal of Technology Enhanced Learning, 4(5), 304-317. doi:10.1504/IJTEL.2012.051816 Finch, W. H., Immekus, J. C., & French, B. F. (2016). Applied Psychometrics Using SPSS and AMOS. Charlotte, NC: Information Age Publishing Inc. Fisher, M., Goddu, M. K., & Keil, F. C. (2015). Searching for Explanations: How the Internet Inflates Estimates of Internal Knowledge. Journal of Experimental Psychology, General, 144(3), 674-687. doi:10.1037/xge0000070 Fokkema, M., & Greiff, S. (2017). How Performing PCA and CFA on the Same Data Equals Trouble. European Journal of Psychological Assessment, 33(6), 399-402. doi:10.1027/1015-5759/a000460 Greene, J. A., Azevedo, R., & Torney-Purta, J. (2008). Modeling Epistemic and Ontological Cognition: Philosophical Perspectives and Methodological Directions. Educational Psychologist, 43(3), 142-160. doi:10.1080/00461520802178458 Greene, J. A., Sandoval, W. A., & Bråten, I. (Eds.). (2016). Handbook of Epistemic Cognition. New York: Routledge. doi:10.4324/9781315795225. Retrieved from http://ebookcentral.proquest.com/ Grossnickle Peterson, E., Alexander, P. A., & List, A. (2017). The argument for epistemic competence. In A. Bernholt, H. Gruber & B. Moschner (Eds.), Wissen und Lernen. Wie epistemische Überzeugungen Schule, Universität und Arbeitswelt beeinflussen (pp. 255-270). Münster: Waxmann Verlag. Gunter, B., Rowlands, I., & Nicholas, D. (2009). The Google Generation. Are ICT innovations changing information-seeking behaviour? Cambridge: Chandos Publishing. Retrieved from http://ebookcentral.proquest.com/ Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2010). Multivariate Data Analysis: A Global Perspective (7th ed.). Upper Saddle River (N.J.): Prentice Hall. Head, A. J., & Eisenberg, M. B. (2010). How today’s college students use Wikipedia for course-related research. First Monday, 15(3). doi:10.5210/fm.v15i3.2830 Hinman, L. M. (2008). Searching Ethics: The Role of Search Engines in the Construction and Distribution of Knowledge. In A. Spink, & M. Zimmer (Eds.), Web Search - Multidisciplinary Perspectives (pp. 67-76). Berlin, Heidelberg: Springer. doi:10.1007/978-3-540-75829-7_3. Retrieved from https://ebookcentral.proquest.com/ Ståhl 61 | F L R Hofer, B. K. (2006). Beliefs About Knowledge and Knowing: Integrating Domain Specificity and Domain Generality: A Response to Muis, Bendixen, and Haerle (2006). Educational Psychology Review, 18(1), 67-76. doi:10.1007/s10648-006-9000-9 Hofer, B. K. (2008). Personal Epistemology and Culture. In M. S. Khine (Ed.), Knowing, Knowledge and Beliefs: Epistemological Studies across Diverse Cultures (pp. 3-22). Dordrecht: Springer. doi:10.1007/978-1-4020-6596-5_1 Hofer, B. K., & Pintrich, P. R. (1997). The Development of Epistemological Theories: Beliefs about Knowledge and Knowing and Their Relation to Learning. Review of Educational Research, 67(1), 88140. doi:10.2307/1170620 Hofer, B. K., & Pintrich, P. R. (Eds.). (2002). Personal Epistemology: The Psychology of Beliefs About Knowledge and Knowing. Mahwah, N.J: L. Erlbaum Associates. doi:10.4324/9781410604316 Hooper, D., Coughlan, J., & Mullen, M. (2008). Structural Equation Modelling: Guidelines for Determining Model Fit. Electronic Journal of Business Research Methods, 6(1), 53-60. Retrieved from http://www.ejbrm.com/volume6/issue1/p53 Jones, C., & Hosein, A. (2010). Profiling University Students' Use of Technology: Where is the Net Generation Divide? International Journal of Technology, Knowledge & Society, 6(3), 43-58. doi:10.18848/1832-3669/cgp/v06i03/56097 Kammerer, Y., & Gerjets, P. (2012). Effects of search interface and Internet-specific epistemic beliefs on source evaluations during Web search for medical information: an eye-tracking study. Behaviour & Information Technology, 31(1), 83-97. doi:10.1080/0144929X.2011.599040 Karimi, M. N. (2014). EFL students' grammar achievement in a hypermedia context: Exploring the role of Internet-specific personal epistemology. System, 42, 1-11. doi:10.1016/j.system.2013.10.017 Khine, M. S. (Ed.). (2008). Knowing, Knowledge and Beliefs: Epistemological Studies across Diverse Cultures. Dordrecht: Springer. doi:10.1007/978-1-4020-6596-5. Retrieved from https://ebookcentral.proquest.com/ Kienhues, D., Bromme, R., & Stahl, E. (2008). Changing epistemological beliefs: The unexpected impact of a short-term intervention. British Journal of Educational Psychology, 78(4), 545-565. doi:10.1348/000709907x268589 Kitchener, R. F. (2002). Folk epistemology: An introduction. New Ideas in Psychology, 20(2–3), 89-105. doi:10.1016/S0732-118X(02)00003-X Knewton. (2011). The Flipped Classroom Infographic. Retrieved from http://www.knewton.com/flippedclassroom/, 03.08.2012 Knight, S., Rienties, B., Littleton, K., Mitsui, M., Tempelaar, D., & Shah, C. (2017). The relationship of (perceived) epistemic cognition to interaction with resources on the internet. Computers in Human Behavior, 73, 507-518. doi:10.1016/j.chb.2017.04.014 Knight, S., Wise, A. F., & Chen, B. (2017). Time for Change: Why Learning Analytics Needs Temporal Analysis. Journal of Learning Analytics, 4(3), 7–17. doi:10.18608/jla.2017.43.2 Krathwohl, D. R. (2002). A revision of Bloom's taxonomy: An overview. Theory into Practice, 41(4), 212218. doi:10.1207/s15430421tip4104_2 Kuhn, D., & Weinstock, M. (2002). What Is Epistemological Thinking and Why Does It Matter? In B. K. Hofer, & P. R. Pintrich (Eds.), Personal Epistemology: The Psychology of Beliefs About Knowledge and Knowing (pp. 121-144). Mahwah, N.J: L. Erlbaum Associates. Leal-Soto, F., & Ferrer-Urbina, R. (2017). Three-factor structure for Epistemic Belief Inventory: A crossvalidation study. PLoS ONE, 12(3), 1-16. doi:10.1371/journal.pone.0173295 Martin, E. (2005). Survey Questionnaire Construction. In K. Kempf-Leonard (Ed.), Encyclopedia of Social Measurement (pp. 723-732). New York: Elsevier. doi:10.1016/B0-12-369398-5/00433-3. Retrieved from http://www.sciencedirect.com/ Miller, B., & Record, I. (2013). Justified belief in a digital age: On the epistemic implications of secret internet technologies. Episteme, 10(02), 117-134. doi:10.1017/epi.2013.11 Moschner, B., Gruber, H., & Studienstiftungsarbeitsgruppe EPI. (2005). Epistemologische Überzeugungen. Forschungsbericht Nr. 18. Regensburg: Universität Regensburg, Lehrstuhl für Lehr-Lern-Forschung. Retrieved from https://portal.uni-regensburg.de/48/ Ståhl 62 | F L R Muis, K. R. (2007). The Role of Epistemic Beliefs in Self-Regulated Learning. Educational Psychologist, 42(3), 173-190. doi:10.1080/00461520701416306 Muis, K. R., Bendixen, L. D., & Haerle, F. C. (2006). Domain-Generality and Domain-Specificity in Personal Epistemology Research: Philosophical and Empirical Reflections in the Development of a Theoretical Framework. Educational Psychology Review, 18(1), 3-54. doi:10.1007/s10648-006-9003-6 Nguyen, C. T. (2018). Echo chambers and epistemic bubbles. Episteme, (FirstView, Sept 13, 2018). doi:10.1017/epi.2018.32 Niessen, T., Vermunt, J., Abma, T., Widdershoven, G., & Van der Vleuten, C. (2004). On the nature and form of epistemologies: Revealing hidden assumptions through an analysis of instrument design. European Journal of School Psychology, 2(1-2), 39-64. Osborne, J. W. (2014). Best Practices in Exploratory Factor Analysis. Retrieved from http://pareonline.net/ OSF. (2010). Use of information and communications technology. Official statistics Finland. Retrieved from http://www.stat.fi/til/sutivi/2010/sutivi_2010_2010-10-26_tie_001_en.html OSF. (2011). Use of information and communications technology by individuals. Official statistics Finland. Retrieved from http://www.stat.fi/til/sutivi/2011/sutivi_2011_2011-11-02_tie_001_en.html Otting, H., Zwaal, W., Tempelaar, D., & Gijselaers, W. (2010). The structural relationship between students' epistemological beliefs and conceptions of teaching and learning. Studies in Higher Education, 35(7), 741-760. doi:10.1080/03075070903383203 Palmer, B., & Marra, R. M. (2008). Individual Domain-Specific Epistemologies: Implications for Educational Practice. In M. S. Khine (Ed.), Knowing, Knowledge and Beliefs: Epistemological Studies across Diverse Cultures (pp. 325-350). Dordrecht: Springer. doi:10.1007/978-1-4020-6596-5_16 Perry, W. G. (1970). Forms of intellectual and ethical development in the college years: A scheme. New York: Holt, Rinehart and Winston. Personal Data Act of 22.4.1999. Retrieved from http://www.finlex.fi/fi/laki/ajantasa/1999/19990523 Pieschl, S., Stallmann, F., & Bromme, R. (2014). High School Students' Adaptation of Task Definitions, Goals and Plans to Task Complexity - The Impact of Epistemic Beliefs. Psychological topics, 23(1), 3152. doi:10.31820/pt Prensky, M. (2001). Digital Natives, Digital Immigrants Part 1. On the Horizon, 9(5), 1-6. doi:10.1108/10748120110424816 Purcell, K., Brenner, J., & Rainie, L. (2012). Search Engine Use 2012. Washington, DC: Pew Research Center. Retrieved from http://www.pewinternet.org/2012/03/09/search-engine-use-2012/ Purcell, K., Rainie, L., Heaps, A., Buchanan, J., Friedrich, L., Jacklin, A., . . . Zickuhr, K. (2012). How Teens Do Research in the Digital World. Washington DC: Pew Research Center. Retrieved from http://www.pewinternet.org/2012/11/01/how-teens-do-research-in-the-digital-world/ Säljö, R. (2012). Literacy, Digital Literacy and Epistemic Practices: The Co-Evolution of Hybrid Minds and External Memory Systems. Nordic Journal of Digital Literacy, 7(1), 5-19. Retrieved from http://www.idunn.no/ts/dk/2012/01/art08 Sandoval, W. A. (2005). Understanding students' practical epistemologies and their influence on learning through inquiry. Science Education, 89(4), 634-656. doi:10.1002/sce.20065 Schommer, M. (1990). Effects of Beliefs About the Nature of Knowledge on Comprehension. Journal of Educational Psychology, 82(3), 498-504. doi:10.1037/0022-0663.82.3.498 Schommer, M. (1993). Epistemological Development and Academic Performance Among Secondary Students. Journal of Educational Psychology, 85(3), 406-411. doi:10.1037/0022-0663.85.3.406 Schommer, M. (1998). The influence of age and education on epistemological beliefs. British Journal of Educational Psychology, 68(4), 551-562. doi:10.1111/j.2044-8279.1998.tb01311.x Schommer-Aikins, M. (2004). Explaining the Epistemological Belief System: Introducing the Embedded Systemic Model and Coordinated Research Approach. Educational Psychologist, 39(1), 19-29. doi:10.1207/s15326985ep3901_3 Schraw, G. (2013). Conceptual Integration and Measurement of Epistemological and Ontological Beliefs in Educational Research. ISRN Education, vol. 2013, 1-19. doi:10.1155/2013/327680 Schraw, G., Bendixen, L., & Dunkle, M. E. (2002). Development and Validation of the Epistemic Belief Inventory (EBI). In B. K. Hofer, & P. R. Pintrich (Eds.), Personal Epistemology: The Psychology of Ståhl 63 | F L R Beliefs About Knowledge and Knowing (pp. 261-275). Mahwah, N.J: L. Erlbaum Associates. Schreiber, J. B., Nora, A., Stage, F. K., Barlow, E. A., & King, J. (2006). Reporting Structural Equation Modeling and Confirmatory Factor Analysis Results: A review. The Journal of Educational Research, 99(6), 323-337. doi:10.3200/joer.99.6.323-338 Shaffer, D. W., & Clinton, K. A. (2006). Toolforthoughts: Reexamining Thinking in the Digital Age. Mind, Culture, and Activity, 13(4), 283-300. doi:10.1207/s15327884mca1304_2 Siemens, G. (2005). Connectivism: A Learning Theory for the Digital Age. International Journal of Instructional Technology and Distance Learning, 2(1), 3-10. Siemens, G. (2006). Knowing knowledge, elearnspace. Retrieved from http://www.elearnspace.org/ Simpson, T. W. (2012). Evaluating Google as an Epistemic Tool. Metaphilosophy, 43(4), 426-445. doi:10.1111/j.1467-9973.2012.01759.x Smart, P. R. (2012). The Web-Extended Mind. Metaphilosophy, 43(4), 446-463. doi:10.1111/j.14679973.2012.01756.x Sparrow, B., Liu, J., & Wegner, D. M. (2011). Google Effects on Memory: Cognitive Consequences of Having Information at Our Fingertips. Science, 333(6043), 776-778. doi:10.1126/science.1207745 SPSS. (2016a). Amos 24.0 [computer software]. Chicago, IL: SPSS Inc., IBM Corporation. SPSS. (2016b). SPSS 24.0 [computer software]. Chicago, IL: SPSS Inc., IBM Corporation. Ståhl, T. (2017). How ICT savvy are Digital Natives actually? Nordic Journal of Digital Literacy, 12(3), 89108. doi:10.18261/ISSN.1891-943X-2017-03-04 Ståhl, T., & Mildén, P. (2017). Applying the FEE to explore epistemic beliefs among students. In A. Bernholt, H. Gruber & B. Moschner (Eds.), Wissen und Lernen. Wie epistemische Überzeugungen Schule, Universität und Arbeitswelt beeinflussen (pp. 59-97). Münster: Waxmann Verlag Strømsø, H. I., & Bråten, I. (2010). The role of personal epistemology in the self-regulation of internet-based learning. Metacognition and Learning, 5(1), 91-111. doi:10.1007/s11409-009-9043-7 Tang, J. (2010). Exploratory and Confirmatory Factor Analysis of Epistemic Beliefs Questionnaire about Mathematics for Chinese Junior Middle School Students. Journal of Mathematics Education, 3(2), 89105. Tourangeau, R., Couper, M. P., & Conrad, F. (2004). Spacing, Position, and Order: Interpretive Heuristics for Visual Features of Survey Questions. Public Opinion Quarterly, 68(3), 368-393. doi:10.1093/poq/nfh035 Trautwein, U., & Lüdtke, O. (2007). Epistemological beliefs, school achievement, and college major: A large-scale longitudinal study on the impact of certainty beliefs. Contemporary Educational Psychology, 32(3), 348-366. doi:10.1016/j.cedpsych.2005.11.003 van den Beemt, A., Akkerman, S., & Simons, P. R. J. (2011). Patterns of interactive media use among contemporary youth. Journal of Computer Assisted Learning, 27(2), 103-118. doi:10.1111/j.13652729.2010.00384.x Walker, S., Brownlee, J., Lennox, S., Exley, B., Howells, K., & Cocker, F. (2009). Understanding first year university students: personal epistemology and learning. Teaching Education, 20(3), 243-256. doi:10.1080/10476210802559350 Wood, P., & Kardash, C. (2002). Critical Elements in the Design and Analysis of Studies of Epistemology. In B. K. Hofer, & P. R. Pintrich (Eds.), Personal Epistemology: The Psychology of Beliefs About Knowledge and Knowing (pp. 231-260). Mahwah, N.J: L. Erlbaum Associates. Zhang, L., & Watkins, D. (2001). Cognitive development and student approaches to learning: An investigation of Perry's theory with Chinese and U.S. university students. Higher Education, 41(3), 239261. doi:10.1023/A:1004151226395