Continuous Assurance for the Digital Transformation of Internal Auditing

Copyright © 2022 by Author/s and Licensed by IADITI. This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Journal of Information Systems Engineering and Management 2022, 7(1), 11681 e-ISSN: 2468-4376 https://www.jisem-journal.com/ Research Article Continuous Assurance for the Digital Transformation of Internal Auditing Cátia Barros1, Rui Pedro Marques 1 * 1 University of Aveiro, Portugal * Corresponding Author: [email protected]@ua.pt Citation: Cátia Barros, Rui Pedro Marques (2022). Continuous Assurance for the Digital Transformation of Internal Auditing. Journal of Information Systems Engineering and Management, 7 (1), 11681. https://doi.org/10.55267/iadt.07.11681 ARTICLE INFO ABSTRACT Received: 10 Dec. 2021 Accepted: 26 Jan. 2022 As a result of the increasing demand on the credibility of the financial information disclosure, internal auditing has been playing an increasingly important role in organizations. Currently, t here is a need to ensure the compliance of organizational transactions in real time in order to increase the reliability of the information and to reduce risks. In this context, the concept of Continuous Assurance has emerged, allowing to reduce potential errors and risks and obtain useful information in real time, supporting more effectively the decision making and internal auditing. This paper aims to understand the importance and the use of Continuous Assurance services from the perspective of the intern al auditor. So, the methodology used is qualitative and the questionnaire survey was used to collect data from interna auditors in Portugal. The main results demonstrate, among other findings, that the Continuous Assurance services are considered by the in ternal auditors as very important. Despite this, its implementation in the organizations does not follow the importance assigned to it, and is still far from being fully deployed. The dimension of the audit department is an influential factor in the use of some Continuous Assurance services. Keywords: Digital Transformation, Internal Audit, Continuous Assurance, Risk Management INTRODUCTION In recent years we have seen constant changes: in the economy, in business transactions, in information systems, then the business environment has to adapt daily to these transformations. In this context, internal auditing has been playing an increasingly important role within organizations. The internal auditing not only verifies the accounting system and the transparency of the information provided by the financial statements, but also helps to implement control systems and fundamental procedures inherent in the activity of the organization. According to IIA (2021), internal auditing is an independent, objective assurance and consulting activity designed to add value and improve an organization’s operations. It helps an organization accomplish its objectives by bringing a systematic, disciplined approach to evaluate and improve the effectiveness of risk management, control, and governance processes. Furthermore, with the most technological developments, auditing has also been digitally transformed, allowing the evolution of traditional auditing to a continuous and real-time auditing. Information systems have changed the way organizations operate, transactions are increasingly automated, and human intervention in these transactions also changes, notably as regards the roles, responsibilities and qualifications required. Thus, the auditor's work has also been adapted to introduce supporting technologies that provide not only reliable but real-time information on the business processes to be audited. We can explain continuous audit as a systematic process of activities that intends to provide credibility and assurance in continuous information simultaneously, or in a short period of time after the occurrence of relevant events. But despite the numerous advantages in its application, its implementation is not always easily accepted by the organization, due to the high costs of implementing the technologies and information systems necessary for its process, the lack of procedures and practices of the control system and the lack of qualified professionals in continuous auditing (Santos et al., 2019). The Continuous Assurance concept emerged with the development of continuous auditing, based on a set of services and technologies that enable auditors to proactively conduct real-time management of transactions that are being executed simultaneously, giving them the ability to intervene in the completion of transactions, correcting them and informing those responsible for the state of execution (Vasarhelyi et al., 2010). Barros and Marques / J INFORM SYSTEMS ENG, 7 (1), 11681 / J INFORM SYSTEMS ENG , 7 (1), 11689 2 / 12 The main objective of this study is to explore, from the internal auditor's perspective, the degree of importance and use of Continuous Assurance services. In addition to being a recent, innovative concept and there being no evidence of the existence of studies of this type, as we may have clues in the review made, it is also a topic that may have an impact on the future of organizations, since it provides a new and broad vision in which will be the digital transformation of internal audit. Its relevance is justified by the fact that internal auditing should be a constant and active presence of the organizations, providing effective management of them. The internal auditor should be aware that transformation of the audit because it not only changes your routine as your way of thinking. This chapter is divided into five main sections, including this introduction. Next, the literature review is presented with the main concepts and models which support this work, as well as some related works more pertinent. Then it presents the methodology that led to this research. Before the conclusions, the results are presented and discussed. LITERATURE REVIEW Continuous Assurance Continuous Assurance is defined as the application of emerging information and communication technologies to the standard techniques of auditing, both mandatory periodic auditing and internal auditing (Vasarhelyi et al., 2010). In that view, the same authors add that Continuous Assurance appears as a pacesetter of the digital transformation of auditing as a pioneer in the evolution of transactional auditing from manual techniques to automated methods. Furthermore, Continuous Assurance emerges as a set of services which aims to restore the credibility of auditing, simultaneously allowing organizations to meet the requirements of regulations. Hence, it can diagnose the company’s viability and allegations of fraud and illegal acts, assessing the economy, efficiency and effectiveness of organizations (Murcia et al., 2008; Vasarhelyi et al., 2010). Moreover, Continuous Assurance has provided a change in the auditing practice for the maximum possible degree of automation. Given the emphasis on the transformation of the entire auditing system, the development of Continuous Assurance requires a fundamental reassessment of all aspects of auditing, in particular on how data is made available to the auditor, how alerts are managed, what kind of reports are issued, and how often and to whom they are sent (Marques, 2019) The terms Continuous Assurance, Continuous Auditing and Continuous Monitoring are sometimes used indiscriminately in literature. Therefore, it is crucial to understand what characterizes and distinguishes them, but mostly to understand how they can be relate to, complementing each other (Figure 1). Continuous Assurance is a statement on the adequacy and effectiveness of controls and integrity of information. Continuous monitoring of controls is at the center of Continuous Assurance strategies; however, the audit activity ensures that management activities are appropriate and effective so that organizations have a greater level of certainty about the effective operation of controls, about risk management and about the integrity of information used for decision making (Alles et al., 2003; Coderre, 2005; Kuhn and Sutton, 2006). Figure 1. Continuous monitoring, auditing and assurance (adapted from (Coderre, 2005)) In turn, Continuous Auditing refers to activities undertaken to provide warranty and credibility to operations, besides giving a more timely character to issues of control and management of risk. Continuous Monitoring is responsible for constantly monitoring and evaluating business transactions and their related controls, enabling a real-time view on effectiveness of controls and on integrity of transactions ((Littley et al., 2010; Minnaar et al., 2008). Evaluating the combined results of Continuous Monitoring and audit procedures, auditors can provide Continuous Assurance. However, only recently has an effort been made to understand their differences and distinguish between the concepts. In 2006, a survey (PricewaterhouseCoopers, 2006) concluded that Continuous Assurance triggered corporate sensitivity to its adoption because in 2005 only 35% had a continuous auditing or monitoring processes in place or were planning to develop one, and this value increased to 50% in 2006. It is interesting to observe that 56% of respondents said their continuous auditing processes include both manual and automated elements, 41% indicated their processes are entirely manual, and 3% reported having fully automated processes. Another study by Institute of Internal Auditors and ACL (ACL, 2006) also showed similar results: 36% of surveyed organizations confirmed they implemented a Continuous Assurance approach in all their business processes or simply in some selected areas, and 39% intended to implement in the near future. However, it also states that regardless of the reasons that organizations may have had to neglect the continuing auditing in the past, the recent regulations, the stimulus for real-time monitoring and reporting of financial information and the ability to automate the traditional audit methods have strongly encouraged its adoption. According to the subsection Related Works of this paper, we Barros and Marques / J INFORM SYSTEMS ENG, 7 (1), 11681 / J INFORM SYSTEMS ENG , 7 (1), 11689 3 / 12 can see that the implementations designated as providers of Continuous Assurance services, in the period 2002-2019, were still few and with some limitations regarding the diversity of services that Continuous Assurance should be expected to offer. This shows that this area is still developing and maturing. Also, other studies (Marques and Santos, 2017a, 2017b) also revealed with bibliometric studies that Continuous Assurance is not mature enough and is growing. Objectives and Components The objectives of Continuous Assurance, making it advantageous, are divided into four levels, which are difficult to define in a mutually exclusive way, but which serve to illustrate the functional dependence of Continuous Assurance on auditing (Alles et al., 2004):  Level 1: Evaluation of Organizational Transactions - at this level, it is intended to evaluate organizational transactions, and analyze and verify the atomic actions of transactions execution. With the use of corporate systems, such as ERP (Enterprise Information System) systems, it is possible to analyze, aggregate and evaluate data in order to classify and monitor organizational transactions. At this level, input data should be tested in order to verify whether they are valid, and whether procedures in execution of transaction are consistent with the sequence of established operations Contrary to traditional auditing, with Continuous Assurance it is possible to do this check in real time due to automation and integration of audit procedures. This transaction control can use the formal specification of workflow of processes defined in the ERP systems as a standard behavior of transaction. Thus, it is possible to verify whether transactions have been executed in compliance with all provided steps to foresee the flow of transactions, and whether these are missing or failing.  Level 2: Compliance of Performed Operations - at this level it is intended to ensure that procedures applied in the execution of organizational transactions are appropriate (for example, are consistent with the rules, norms, or standards set by the organization or by external regulatory entities).  Level 3: Quality of Estimates and the Consistency of Aggregate Data - in some businesses, estimates and forecasts are often used because the measurement or direct determination of some information is difficult and expensive to obtain. A simple and economical alternative is to use a formal model to automatically get an estimate, then the auditor’s task will be reduced only to verifying the acceptability of this model, which can be done only once and off-line, based on knowing whether the used parameter values in model are reasonable. This level of Continuous Assurance includes automation of analytical procedures based on internal and external parameters. The use of analytical procedures in an automated system of Continuous Assurance increases efficiency and effectiveness of auditing.  Level 4: Evaluation of Organizational Decisions - the auditing carried out by using ERP systems and advanced financial instruments must incorporate complex and high-level assessments, which are especially important for decision making. Continuous Assurance and the current analytic technology allow the extensive gathering of exogenous evidence, which provides crucial input into these judgments. Continuous Assurance may use, for example data warehousing and data mining as tools that facilitate automation of some of these decisions, improving the quality of high-level decisions and decreasing the audit risk. Continuous Assurance is divided into three distinct components, but complementary (Vasarhelyi et al., 2010):  Continuous Controls Monitoring (CCM): Consists of a set of procedures to monitor the operation of internal control mechanisms;  Continuous Data Assurance (CDA): Verifies the integrity of the data circulating in organizational information systems;  Continuous Risk Monitoring and Assessment (CRMA): Measures the risk dynamically and allows to sustain an auditing plan. Model for Evaluating an Information System with Continuous Assurance A model composed of four dimensions were proposed in order to evaluate an information system with Continuous Assurance services (Marques et al., 2016). It intends to be regarded as a referential set of requirements when developing this type of information systems. Marques et al. (2016) used the Delphi method to validate the model in order to ensure the relevance of inclusion of these dimensions and requirement and also a set of metrics to be included in each dimension. Furthermore, this model was successfully used in a more comprehensive research project (Marques et al., 2015). It is composed of the dimensions Monitoring, Compliance, Estimation and Reporting, which comprise all objectives and components of a system with Continuous Assurance services (Marques et al., 2016):  Dimension Monitoring relates with the objectives of level 1 and the component CCM. In this dimension, the following metrics may be found to assess whether the system can: monitor the various operations of a process as soon as they occur; identify an irregular (unforeseen or inconsistent) operation as soon as it occurs; verify whether the operations were processed at all the previous steps as required; detect lack of operations; and assess the continuity and completeness of transactions.  Dimension Compliance includes the features of the component CDA and the objectives of level 2. The metrics associated to this dimension aims to assess whether the system can: recognize which known execution pattern was or has been followed by each organizational transaction monitored; ascertain which rules, conditions and procedures were fulfilled and unfulfilled in the organizational transactions monitored; detect potential errors; inhibit inappropriate events or behaviors; and help compliance with existing laws, policies, norms and procedures.  Dimension Estimation covers the objectives of level 3 and the functions of the CRMA component. The metrics of this dimension assess whether the system can: estimate, given the current situation, what the possible results of the organizational transaction execution will be; and determine the execution pattern, or a set of execution patterns, which will be possible to be followed Barros and Marques / J INFORM SYSTEMS ENG, 7 (1), 11681 / J INFORM SYSTEMS ENG , 7 (1), 11689 4 / 12 by the organizational transactions monitored, according to the current status of execution.  Dimension Reporting includes the features of all components and the objectives of all levels. The metrics of this dimension must assess whether the system can: report the results of the monitoring of transactions; notify the results of the verification of compliance; inform the results of estimation; and alert users of irregular situations in monitoring, compliance verification and estimation of negative results. Table 1 concisely summarizes this model, presenting quantitatively measurable metrics for each dimension. Table 1. Dimensions of the model and their metrics Dimensions Metrics Monitoring Real-time monitoring of operations Real-time identification of irregular operations Real-time verification of processing of required operations at all previous steps Real-time detection of lack of operations Real-time assessment of the state of execution of business transactions Compliance Recognition of execution patterns Ascertaining of fulfilling of rules Detection of potential errors Verification of compliance of existing policies Estimation Estimation of possible results Determination of possible execution patterns which are likely to be followed Reporting Real-time presentation of the executed operations which were monitored Real-time presentation of the compliance verification in transactions executions Real-time presentation of the risk estimated on determining possible execution patterns Real-time alert for irregular situations in monitoring, compliance verification and estimation of negative results Related Work An organization in the steel industry with 234 industrial and commercial units, present in 10 countries and with shares in New York Stock Exchange, was forced to implement measures of Continuous Assurance in accordance with SOX requirements. After these measures, all processes had to be documented and information started to be disclosed in time and with their integrity ensured and it held the participants responsible, inhibiting the occurrence of fraud (Hargadon and Fanelli, 2002). Siemens has made an effort in this area, and it has successfully experimented some aspects of Continuous Assurance, namely, to ensure the integrity of its ERP systems and their automated modules with audit functions. Siemens’s project proposes a methodology to monitor and evaluate the daily configuration of various settings of existing controls using CCM, because about 68% of its audit activities could be fully automated. Thus, an independent system was developed for interacting with read-only operations with the existing information systems and for reporting system and alarms in case of emergency (Alles et al., 2006). These implementations have ensured the effectiveness, efficiency and a timely character to the audit procedures (Alles et al., 2006, 2008). For example, the use of CCM at Siemens Financial Services enabled significant improvements in the average exception rate, through the testing and monitoring of performance analytics (e.g. input checks, validity checks, and compliance with regulations and internal policies). In some departments, the reduction in the average exception rate was over 20% in the first year of implementation. Also, and according to Vasarhelyi et al. (2010), one of the biggest banks in Brazil, with more than 1400 branches, has implemented measures for Continuous Assurance, proving the viability and benefits of these measures. This institution has a CDA, a system which daily analyses more than five million accounts and generates about six thousand alerts a month. This system aims to increase productivity with efficiency and quality and its mission is to assess the risks and controls automatically and continuously in order to identify exceptions, anomalies, trends and indicators of risk; advise about controls, risk assessment; and contribute to the corporate governance. These features include all products, processes and services which enable data extraction and analysis. The taken approaches are of detection (routines to detect possible errors), of deterrence (routines to inhibit inappropriate behavior and events), financial (routines to reduce or avoid losses), and of compliance (routines to ensure compliance with applicable laws, policies and standards) (Vasarhelyi et al., 2010). The audits carried out by sampling once a year are insufficient for the current business model. Thus, continuous auditing is the answer to greater effectiveness in combating fraud and errors and thus increasing investor confidence. It is important that transaction anomalies are found within organizations in real time and that their report is immediately issued (Boyda¸s and Hazar, 2021). Initially, continuous auditing software becomes a time-consuming process, as it requires a highly specialized service. However, the same modules can be used over and over again, reducing costs in the long run. Continuous auditing also allows remote access to company data, thus allowing auditors to carry out the audit outside the entity's location, thus reducing the audit cost and increasing time efficiency. The authors state that the concept of continuous audit is still not widely used by organizations, the implementation methods are still in the research phase and still have great difficulties in execution, however the continuity of research that has been verified and the development of information systems, will eliminate the difficulties of implementation and their use will be exponential (Boyda¸s and Hazar, 2021). A study (Ezzamouri and Hulstijn, 2018) in the public sector, more specifically in some municipalities in the Netherlands, shows the opportunities and needs to continuously monitor and evaluate financial data in the social area, as these services were decentralized from the government to the municipal level. According to the study, it was found that the technology applied to continuous monitoring easily detects errors and deviations, making it easier to carry out improvement processes. Marques et al. (2015; 2018) developed an innovative solution in order to implement Continuous Assurance services in Barros and Marques / J INFORM SYSTEMS ENG, 7 (1), 11681 / J INFORM SYSTEMS ENG , 7 (1), 11689 5 / 12 information systems applicable to any business process, regardless of its type, dimension, business area or even its information system support technology, supported by an ontological model at an abstraction level that guarantees that contextual independence. The results analysis from an implementation of this solution allowed to ensure the feasibility and the effective use of the solution. A study was carried out in Indonesia (Soedarsono et al., 2019), inquiring taxpayers from various Ministries, with the aim of researching the effects of the quality of information and management support, for the application of a continuous auditing and monitoring in the government sector. Continuous monitoring was implemented in real time to monitor the government's financial performance and concluded that the quality of information positively affects the application of continuous auditing and continuous monitoring, namely in the availability of real-time data for each technical area, the improving financing processes and increasing human resources skills. The authors also suggest the creation of a regulation for the transition from manual preventive procedures to automatic detection controls in order to increase the effect of continuous auditing and continuous monitoring to be reported to information users and management. At a renowned public university in the USA, a continuous monitoring system was implemented in the internal audit department, for shopping cards used by university employees (Tronto and Killingsworth, 2021). The usefulness of this implementation was to reduce the amount of documentation, and the costs associated with the analysis of this documentation, thus allowing the purchase of services and goods of reduced value to be more efficient. To reduce the risk of misuse of the card by employees, preventive actions were implemented in order to combat fraud. The project was successful, demonstrating that continuous monitoring plays a very important role in internal auditing, reducing the time required for checking transactions. Methodology Considering that the main objective of this study is to explore, from the internal auditor's perspective, the degree of importance and use of Continuous Assurance services. The following research questions were therefore raised:  RQ1: What are the most relevant Continuous Assurance services from the perspective of the internal auditor?  RQ2: What Continuous Assurance services are most used by the internal auditor?  RQ3: What factors influence the importance assigned by internal auditors to the Continuous Assurance service?  RQ4: What factors influence the use of the Continuous Assurance service by internal auditors?  RQ5: Is the degree of importance of the Continuous Assurance service, from the internal auditor's point of view, equivalent to the degree of use of the Continuous Assurance service? This is an exploratory study that uses a qualitative methodology. We opted for a questionnaire survey applied to professionals linked to the internal auditing function of companies of any size and of any sector of activity, between January and February 2019. The questionnaire was distributed by the social network LinkedIn. In this social network, private messages were sent to professionals who identify themselves in their profile as internal auditors in Portugal, asking them to respond to the questionnaire. In total, about 600 of these messages were sent. The questionnaire was divided into four parts: the first was to know the profile of the respondent (age, gender, academic qualifications, possession of an auditing certification, professional experience in auditing in number of years and function.), the second was intended to characterize the company where respondent works (size of the company in which the respondent works, number of internal auditors in the company), the third was to collect data on the degree of importance of the Continuous Assurance service and the fourth part about the use of the Continuous Assurance service. The third and fourth parts were supported by the dimensions and metrics included in the information systems model for evaluating information systems with Continuous Assurance services presented above. Regarding questions of importance and use of Continuous Assurance, a Likert Scale was used, consisting of qualitative variables with an ordinal scale of measurement in which the different categories of the scale are ordered in a graduated order, and the respondent can choose between five possible answers. Regarding the scale of response, it was used "not important", "slightly important", "moderately important", "important" and "very important" to the question of degree of importance. For the question of degree of use, it was used the options "not implemented", "rarely", "casually", "frequently" and "always". RESULTS AND DISCUSSION Sample Characterization The sample of this study consists of 91 valid answers to the questionnaire. The first part of the questionnaire, with respect to the characterization of the respondents, shows that most of the respondents, approximately 75.82%, are between 25 and 45 years of age (36.26% between 25 and 35 years and 39.56% between 35 and 45 years). About 15.38% are in the 46-55 age group, only 5.49% are under 25 years old and 3.30% of the sample are over 55 years old. The respondents are mostly male, about 73.63% of the answers, and the remaining respondents are female. Regarding academic qualifications, it is observed that most of the respondents, 49.45%, have an undergraduate degree. In addition, 25.27% have a postgraduate degree, 23.08% have a master's degree and only 2.20% got a PhD. The majority of respondents do not have any audit certification, corresponding to 64.84% of the answers. It should be noted that 21.98% of the sample are CIA (Certified Internal Auditor), and this percentage is still representative. There are also CISA (Certified Information Systems Auditor) and CRMA (Certified in Risk Management Assurance) corresponding to 3.30% Barros and Marques / J INFORM SYSTEMS ENG, 7 (1), 11681 / J INFORM SYSTEMS ENG , 7 (1), 11689 6 / 12 and 2.20% respectively, but without great impact on the total sample. The response regarding other certifications obtained 7.69% of the answers, indicating essentially internal certifications based on the activity/department of the organization in which the respondent is inserted. Regarding the professional experience of the respondents, we found that the those who have to less than 5 years of experience and between 11 to 20 years have similar frequency, with 28.57% and 27.47%, respectively. However, most of them, about 38.46%, have between 5 and 10 years of experience, and, finally, only 5.49% worked for more than 20 years. Furthermore, 71.43% of respondents perform auditing functions, 12.09% management functions and the other administrative, financial, accounting or information systems functions. Concerning the questions in the second part of the questionnaire which characterizes the companies where respondents work, it was found that the vast majority, around 79.12%, work in a large company, 16.48% work in a mediumsized enterprise, only 3.30% work in a small enterprise and only one respondent works in a microenterprise. The categories of micro, small and medium-sized enterprises is based on the European Commission Recommendation 2003/361/EC of 6 May 2003 concerning the definition of micro, small and medium-sized enterprises. With regard to the number of internal auditors working in the audit department where the respondent is inserted, the data show that 10.99% of respondents indicate that there is only one auditor in that department, 25.27% of respondents indicate that it is composed of 2 to 4 auditors, 24.18% indicate that it is composed of 5 to 10, and 39.56% of respondents say that there are more than 10 auditors working. This last finding is expected since the vast majority work in a large company. By relating some of the variables presented previously, it is possible to obtain a more detailed characterization of the sample. For example, Table 2 shows that 52.75% of the respondents working in a large company do not have any audit certification and only 16.48% are CIA. The remaining CIA work in mediumsized enterprises (3.30%), and in small and microenterprises (1.10% each). It should be noted that the only respondent who answered the questionnaire working in a microenterprise is CIA. We can also observe that almost 65% of the respondents, regardless of the size of the company where they work, have no audit certification. This percentage is similar when we analyze the number of professionals without certification by company size. Table 2. Certification of Respondents by Company Size No Certific CIA CISA CRMA Others Total Large Company 52.75% 16.48% 2.20% 1.10% 6.59% 79.12% Medium -Size Company 9.88% 3.30% 1.10% 1.10% 1.10% 16.48% Small Company 2.20% 1.10% 0.00% 0.00% 0.00% 3.30% Microenterprise 0.00% 1.10% 0.00% 0.00% 0.00% 1.10% Total 64.84% 21.98% 3.30% 2.20% 7.69% 100.00% Analyzing Table 3, we found that of the 79.12% participants who work in a large company, almost half have more than 10 internal auditors in their companies. It is also clear that 5.49% of respondents working in a large company have only one auditor. It should be also noted that the respondent working in the microenterprise indicated that there are between 2 and 4 auditors and that a respondent, working in a small company, specified that there are more than 10 auditors. Table 3. Number of Auditors by Company Size 1 2-4 5-9 10+ Total Large Company 5.49% 16.48% 20.88% 36.26% 79.12% Medium-Size Company 3.30% 7.69% 3.30% 2.20% 16.48% Small Company 2.20% 0.00% 0.00% 1.10% 3.30% Microenterprise 0.00% 1.10% 0.00% 0.00% 1.10% Total 10.99% 25.27% 24.18% 39.56% 100.00% Answers to Research Questions and Discussion RQ1: What are the most relevant Continuous Assurance services from the perspective of the internal auditor? To answer this question, we outline a ranking for the metrics of each dimension. The choice fell on the measure of location of central tendency: mode. Mode is the most frequent value of the responses made available to the respondent. It depends only on the frequency of observations and not on their value, and is not affected by extreme values, which makes it a very robust measure (Martinez and Ferreira, 2007). Table 4 shows the distribution of the answers by the various answer options; the mode being represented in bold for every metric. Barros and Marques / J INFORM SYSTEMS ENG, 7 (1), 11681 / J INFORM SYSTEMS ENG , 7 (1), 11689 7 / 12 Table 4. Distribution of responses regarding the importance assigned to each metric Dimensions Metrics Not Important Slightly Important Moderately Important Important Very Important Monitoring Real-time monitoring of operations 2.20% 6.59% 21.98% 45.05% 24.18% Real-time identification of irregular operations 1.10% 2.20% 7.69% 29.67% 59.34% Real-time verification of processing of required operations at all previous steps 2.20% 6.59% 19.78% 51.65% 19.78% Real-time detection of lack of operations 1.10% 2.20% 5.49% 39.56% 51.65% Real-time assessment of the state of execution of business transactions 2.20% 7.69% 20.88% 47.25% 21.98% Compliance Recognition of execution patterns 1.10% 1.10% 15.38% 47.25% 35.16% Ascertaining of fulfilling of rules 1.10% 6.59% 13.19% 46.15% 32.97% Verification of compliance of existing policies 0.00% 6.59% 9.89% 39.56% 43.96% Detection of potential errors 1.10% 4.40% 3.30% 45.05% 46.15% Estimation Estimation of possible results 1.10% 1.10% 3.30% 35.16% 59.34% Determination of possible execution patterns which are likely to be followed 1.10% 0.00% 5.49% 39.56% 53.85% Reporting Real-time presentation of the executed operations which were monitored 1.10% 6.59% 23.08% 42.86% 26.37% Real-time presentation of the compliance verification in transactions executions 1.10% 5.49% 8.79% 49.45% 35.16% Real-time alert for irregular situations in monitoring, compliance verification and estimation of negative results 2.20% 2.20% 9.89% 36.26% 49.45% Real-time presentation of the risk estimated on determining possible execution patterns 1.10% 3.30% 23.08% 42.86% 29.67% The "not important" option has achieved extremely low response rates, so this means that internal auditors are aware of the importance of Continuous Assurance services. Regarding the Monitoring dimension, the metrics “realtime identification of irregular operations” and “real-time detection of lack of operations” were considered the most important, having both obtained more than half of the respondents considering them very important. The importance attributed to this dimension by the internal auditors is very notorious. Regarding the Compliance dimension, it is important to note that the metric “ascertaining of fulfilling of rules” did not obtain any "not important" response, thus showing the concern of all auditors in complying with regulations and legislation, however, despite the mode being “very important”, this ranks second, with metric “detection of potential errors” being the most important. All metrics in the Estimation dimension were "very important" as mode, with a response frequency greater than 50%. In addition to the "real-time identification of irregular operations" metrics from the Monitoring dimension, we have the “estimation of possible results” metric, with a similar number of responses in the "very important" option (almost 60%). Of the 15 metrics considered in the study, these two are the most important in the perspective of internal auditors. We can also verify that the "real-time alert for irregular situations in monitoring, compliance verification and estimation of negative results” was considered the most important in the Reporting dimension, obtaining almost 50% of the responses. The response options which correspond to the highest importance (important and very important) were the most frequent, allowing to confirm that the overwhelming majority of respondents are aware of the benefits that the Continuous Assurance service provides in an organization and the importance may have in the execution of the audit work. The sum of responses in these two response options is always higher than 69% in each metric, with some of them reaching close to 95% of the answers. RQ2: What Continuous Assurance services are most used by the internal auditor? In the same way of the previous question, we used mode to analyze the degree of use of each metric under analysis. Table 5 shows the frequency of responses obtained in each response option by dimension and metric of the model. Similarly, the mode is represented in bold. Barros and Marques / J INFORM SYSTEMS ENG, 7 (1), 11681 / J INFORM SYSTEMS ENG , 7 (1), 11689 8 / 12 Table 5. Distribution of responses regarding the frequency of use of each metric Dimensions Metrics Not Implemeted Rarely Casually Frequently Always Monitoring Real-time monitoring of operations 19.78% 6.59% 18.68% 39.56% 15.38% Real-time identification of irregular operations 20.88% 4.40% 20.88% 34.07% 19.78% Real-time verification of processing of required operations at all previous steps 16.48% 9.89% 15.38% 40.66% 17.58% Real-time detection of lack of operations 20.88% 7.69% 15.38% 40.66% 15.38% Real-time assessment of the state of execution of business transactions 15.38% 9.89% 20.88% 36.26% 17.58% Compliance Recognition of execution patterns 13.19% 6.59% 24.18% 38.46% 17.58% Ascertaining of fulfilling of rules 3.30% 4.40% 14.29% 46.15% 31.87% Verification of compliance of existing policies 6.59% 1.10% 16.48% 39.56% 36.26% Detection of potential errors 7.69% 4.40% 13.19% 50.55% 24.18% Estimation Estimation of possible results 5.49% 8.79% 19.78% 40.66% 25.27% Determination of possible execution patterns which are likely to be followed 10.99% 7.69% 16.48% 47.25% 17.58% Reporting Real-time presentation of the executed operations which were monitored 20.88% 9.89% 18.68% 35.16% 15.38% Real-time presentation of the compliance verification in transactions executions 5.49% 7.69% 18.68% 36.26% 31.87% Real-time alert for irregular situations in monitoring, compliance verification and estimation of negative results 17.58% 4.40% 15.38% 38.46% 24.18% Real-time presentation of the risk estimated on determining possible execution patterns 9.89% 7.69% 14.29% 40.66% 27.47% As shown in the table above, the percentage of answers "not implemented” is more significant than the "not important" in the question about importance. Thus, we immediately perceive a gap between importance and implementation. Regardless of dimension, the most frequent answer is "frequently". In this question we no longer obtain the maximum scale "always" as the most frequent answer, and the answers were already much more distributed by the response options and not as concentrated in the two maximal scales as in the question about importance. Analyzing now the percentage of response obtained for the answer that corresponds to Mode, it is observed that in the dimension Monitoring, there are three metrics with higher frequency of use with identical percentages, namely “Real-time verification of processing of required operations at all previous steps”, “Real-time detection of lack of operations”, and “Realtime monitoring of operations”, both with about 40% responses. In the same way, it is evident that, in the dimension Compliance, the “detection of potential errors” has the highest percentage in the response option that correspond to Mode, obtaining 50.55% of answers. In the Estimation dimension, the two metrics have values not too far apart, but slightly below 50%. Finally, in the Reporting dimension, the most widely used metric is " Real-time presentation of the risk estimated on determining possible execution patterns" with almost 41%. In addition, we can further conclude that the metrics with the lowest implementation in companies are:  Real-time identification of irregular operations;  Real-time detection of lack of operations;  Real-time presentation of the executed operations which were monitored;  Real-time monitoring of operations;  Real-time alert for irregular situations in monitoring, compliance verification and estimation of negative results;  Real-time verification of processing of required operations at all previous steps. From the ranking of the least implemented metrics, we observe that those that compose the Monitoring dimension are the least implemented and used. This result is in line with what was found in the literature by Marques and Santos (2017a), which demonstrate by a bibliometric study that research on continuous monitoring is scarce and therefore may be one of the causes of the slow growth of research on continuous assurance, and consequently its implementation. Some real-time reporting services are also the least implemented because they are a consequence of the scarce implementation of real-time monitoring, since there will be no real-time reporting without continuous monitoring. RQ3: What factors influence the importance assigned by internal auditors to the Continuous Assurance service? In order to answer this question, the following hypotheses were put forward and the non-parametric Chi-Square independence test was applied, which allows us to determine whether two variables are related. All the characteristics of the respondent will be tested as independent variables. That is, the hypotheses below will be applied for every independent variable. The hypotheses considered for applying the Chi-Square test are: Barros and Marques / J INFORM SYSTEMS ENG, 7 (1), 11681 / J INFORM SYSTEMS ENG , 7 (1), 11689 9 / 12  H0: The importance assigned to the Continuous Assurance service is independent of the independent variable;  H1: The importance assigned to the Continuous Assurance service is dependent on the independent variable. For simplification of the sample: in the company size, since we only obtained 3.30% of the responses of small companies and 1.10% of microenterprises, which are not significant in the total sample, but nevertheless are valid answers, we considered these answers as being of medium-sized companies. In the same line of thinking, and regarding certification in auditing, as we only obtained 3.30% CISA and 2.20% CRMA, which are not significant in the total sample, however, are valid answers, we consider the answers in the category without CIA certification. In total we have 22% CIA and 78% of non-certified respondents. For a significance level of 5%, the decision rule depends on whether the value of Sig. is greater than 0.05. Table 6 shows the value of Sig. obtained from the Chi-Square test for each metric and independent variable. From this table, we observe that the value of Sig. is almost always greater than 0.05, so we do not reject H0. That is, the importance assigned to the Continuous Assurance service is, in general, independent of the analyzed variables. Table 6. Value of Sig. obtained from the Chi-Square test (independence of the perception of importance) Metrics Age Sex Habilitations Certification Experience Job Role Company Size No. of Auditors Dimension Monitoring Real-time monitoring of operations 0.672 0.634 0.941 0.865 0.314 0.355 0.197 0.028 Real-time identification of irregular operations 0.889 0.754 0.997 0.656 0.300 0.093 0.579 0.509 Real-time verification of processing of required operations at all previous steps 0.476 0.714 0.782 0.911 0.266 0.504 0.687 0.430 Real-time detection of lack of operations 0.690 0.827 0.795 0.968 0.182 0.110 0.720 0.204 Real-time assessment of the state of execution of business transactions 0.911 0.366 0.431 0.949 0.997 0.540 0.654 0.615 Dimension Compliance Recognition of execution patterns 0.742 0.063 0.676 0.785 0.829 0.190 0.816 0.303 Ascertaining of fulfilling of rules 0.991 0.817 0.669 0.904 0.432 0.273 0.775 0.078 Verification of compliance of existing policies 0.774 0.536 0.744 0.497 0.427 0.430 0.988 0.631 Detection of potential errors 0.293 0.328 0.960 0.605 0.269 0.191 0.958 0.111 Dimension Estimation Estimation of possible results 0.917 0.393 0.946 0.006 0.687 0.005 0.263 0.509 Determination of possible execution patterns which are likely to be followed 0.189 0.511 0.777 0.564 0.639 0.318 0.844 0.212 Dimension Reporting Real-time presentation of the executed operations which were monitored 0.992 0.366 0.965 0.972 0.275 0.061 0.371 0.852 Real-time presentation of the compliance verification in transactions executions 0.885 0.449 0.961 0.842 0.065 0.111 0.785 0.661 Real-time alert for irregular situations in monitoring, compliance verification and estimation of negative results 0.931 0.243 0.271 0.903 0.199 0.661 0.677 0.660 Real-time presentation of the risk estimated on determining possible execution patterns 0.808 0.636 0.963 0.991 0.212 0.227 0.607 0.567 However, we detected three metrics in which the exact opposite is true:  Real-time monitoring of operations is influenced by the number of auditors - the level of importance attributed to this metric is influenced by the number of auditors within the company, a greater attribution of importance is observed when the department is made up of several internal auditors.  Estimation of possible results is influenced by certification in internal auditing - the importance attributed to this metric is influenced by audit certification. All respondents who are CIA responded that this metric is "important" or "very important", thus certification influences the importance perceived.  Estimation of possible results is influenced by the auditor's role – the importance is also related by the auditor's function. It is observed that almost all internal auditors, considered this metric "important" or "very important". Therefore, when an auditor exclusively assumes the internal audit as its main function, the professional gives more importance to this metric. In these metrics, there is a dependence relation among the variables. An auditor with internal audit certification is more aware of the estimation of potential risks in the execution of his work. And the real-time monitoring of operations is performed more frequently and effectively in a department where works many internal auditors. We can confirm that regardless of the profile of the