Implementation of business intelligence in an IT organization: The concept of an evaluation model

Sitek, Tomasz; Litka, Michal Article Implementation of business intelligence in an IT organization: The concept of an evaluation model Foundations of Management Provided in Cooperation with: Faculty of Management, Warsaw University of Technology Suggested Citation: Sitek, Tomasz; Litka, Michal (2013) : Implementation of business intelligence in an IT organization: The concept of an evaluation model, Foundations of Management, ISSN 2300-5661, De Gruyter, Warsaw, Vol. 5, Iss. 3, pp. 61-74, https://doi.org/10.2478/fman-2014-0020 This Version is available at: https://hdl.handle.net/10419/184562 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. https://creativecommons.org/licenses/by-nc-nd/3.0 Foundations of Management, Vol. 5, No. 3 (2013), ISSN 2080-7279 DOI: 10.2478/fman-2014-0020 61 IMPLEMENTATION OF BUSINESS INTELLIGENCE IN AN IT ORGANIZATION – THE CONCEPT OF AN EVALUATION MODEL Tomasz SITEK*, Michał LITKA** Gdansk University of Technology, Gdańsk, Poland *e-mail: [email protected]a.pl **e-mail: [email protected] Abstract: This paper presents the issue of assessing the validity and effectiveness of implementing a Business Intelligence system in an IT Support Organization. This entity provides IT services to external clients involving, in particular, the storage and processing of large amounts of data. The vast amount of realized projects and also incidents reported in connection with those projects prevented effective decisions from being made without the support of dedicated technologies. The authors present the problems encountered by the studied entity and describe the tool that was selected to improve the situation. The aim of this study is to measure and describe the key processes in the organization on the basis of prepared aggregated measures, first prior to the implementation of the BI system and then a year after its implementation. The evaluation model developed by the authors allowed the assessment of the key aspects of the company's operation over 2 years. It thus helped decision makers to establish whether the decision to implement the Business Intelligence system was correct or not. Keywords: Business Intelligence, IT support organization, decision-making, evaluation model. 1 Decision-making issues in IT organizations Over the years, the amount of data needed by organizations for their proper functioning has grown considerably. The history of portable data storage devices is an excellent example of this. At the beginning of the 1990s, a floppy disk with <1.5 MB capacity was sufficient for most needs; at times, it was necessary to use several disks. A few years later, the CD appeared, which at that time had an unimaginable capacity of 700 MB. But this also proved to be too little – another leap forward in the form of DVDs increased the available portable storage capacity by more than six times. Today’s portable memory sticks are smaller than a matchbox, and yet allow a few (or a few dozen) times more data to be stored than a DVD. With the increase of available disk space, data management skills have been lost – why would anyone bother about three identical copies of the same file if hundreds of gigabytes of free space were still available? For home users, this is not such a big problem. However, what happens when the same approach is carried out by a company that uses hundreds of computers? The demand for available memory is growing rapidly and the cost of infrastructure is rising with it. It should also be noted that although data are collected for a specific purpose, obtaining the necessary information from it is often very difficult. When it comes to taking strategic decisions, it is then necessary to manually analyze multiple sets of data (files) and draw conclusions from them. Such a painstaking search rarely gives measurable results. In principle, the decision-making process is performed in several steps: defining the problem, examining the options, predicting the consequences and choosing the optimal variant [1]. The first one is often problematic, especially in the aforementioned case of data excess. It often transpires that the data are scattered. It can happen that the data from one branch is much older than the data from another branch, which makes taking a joint decision for the entire organization impossible. It may also be the case that the data from several sources is redundant, which leads to the situation where a manager is overwhelmed with data from among which it is hard to select the really important information. The organization can then quickly experience the so-called domino effect. Excess data complicate its analysis, and this less effective analysis affects the formation of available options to choose from. On the basis of incorrectly analyzed, incomplete or simply incorrect information, predicting the consequences of a choice is difficult, to say the least. In such circumstances, an option that appears to be optimal is not so in reality [2]. Paradoxically, in this day and age when a great deal of work has been computerized, decision-making is not easier than it was in the past. 62 Tomasz Sitek, Michał Litka In most organizations, including IT organizations, the decision-making process is accompanied by a number of problems [3]:  excess or insufficiency of data,  inability to interpret data and transform it into information,  inaccuracies of data (information),  dispersed data,  free access to data,  difficulties with the aggregation of data and information,  difficulties associated with reducing uncertainty. There are IT technologies that support the decisionmaking process [4]. This support involves solving or simplifying the problems that occur in the process. They allow faster data access, aggregation and analysis. A huge amount of dispersed information from different parts of the company can be aggregated in one place, while maintaining its readability and giving it an ordered structure. These technologies also allow for the analysis of the information and for the creation of reports, thanks to which the issue of uncertainty in the decision-making process is minimized. Thus, the risk of making a bad decision is reduced, and the probability of making the right one is increased. The tools we are referring to here are defined as Business Intelligence. The implementation of a Business Intelligence technology is a significant cost to the organization. The question is whether the cost can be justified in a quantitative manner? Is it possible to assess the effects of this decision after a system of this class is introduced? Our experience shows that this is a problem for many organizations. The assessment of the situation after the implementation of the system is usually of a qualitative nature and, as it is based on the subjective feelings of the users, can be radically different depending on their position, on the scope of their operations and on the changes which affected them directly [5]. Having obtained very detailed quantitative data collected in the organization prior to and after the implementation of a Business Intelligence system, the authors analyzed the data and established that it is the basis for the construction of an evaluation model to assess the validity of this implementation. This article thus presents a test organization, the indicators developed for measuring key aspects of the business and presents a method for the quantitative evaluation of the effectiveness of such an implementation. 2 Business Intelligence as a method for supporting the decision-making process Business Intelligence, also referred to as BI, is a term, which can be defined in many ways. This paper follows the definition developed by Gartner – a consulting firm specializing in the strategic use of technology [6]. Gartner defines BI as "is an umbrella term that includes the applications, infrastructure and tools, and best practices that enable access to and analysis of information to improve and optimize decisions and performance" [7]. A typical BI system architecture is composed of several elements. The heart of this system is formed by data warehouses, namely databases focused on dealing with analytical queries. Data are transferred to warehouses from several sources: systems such as ERP, CRM, SCM, Call Centers and others, with the help of ETL processes (ETL stands for Extract, Transform, Load) [8]. These processes integrate data from various systems into a single unified model, the so-called multidimensional model. Then, data from warehouses are often used in analytical query processing engines (OLAP, Online Analytical Processing), which allow for quick analyses at different levels and with different sets of questions, sometimes quite abstract ones. The user can then access the results of these analyses via business and analytical tools. These tools present information as a series of reports and analyses – very often this is achieved via the so-called managerial dashboards. Of course, these applications can provide information in other forms, such as spreadsheets or PDF documents. It is often possible to deliver reports to the mailboxes of selected users, whether as part of the normal course of monitoring, or in the event of an emergency. Typically, BI systems work on online platforms, allowing business users to access reports and analyses through a web browser. The aim is to facilitate access to information regardless of the location of the person who is trying to obtain it. Therefore, the BI system consists of data sources which, due to the presence of ETL processes, enter the data warehouses and go further to analytical and reporting applications. Schematically, the idea of BI can be illustrated as in Figure 1. Implementation of Business Intelligence in an IT Organization – the Concept of an Evaluation Model 63 Figure 1. A simplified diagram of the BI system The BI system helps in decision-making because it interferes in all stages of the process and simplifies them [9]. It stores data in one place (eliminating the problem of open access to data and its dispersion, as well as the existence of duplicates). It solves the problem of outdated information, enables a more accurate data analysis and data conversion into information. It is obvious that only a system that contains all the basic building blocks, listed in Figure 2, can actually support the decision-making process. Full and comprehensive BI systems are offered by corporations such as Microsoft, Oracle, IBM or SAP. The comprehensiveness of the systems offered by these companies is accompanied by a high price. Most small and mediumsized businesses have neither the resources nor the need for such expensive solutions. Nonetheless, it is possible to build a functional BI system, however small, out of components offered by smaller companies, to assist the decision-making process. Such a system, integrating the solution of several different suppliers, was used in the studied IT support organization. 3 The characteristics of the studied company and its main problems The studied entity is a company called Avena Technologies, operating in the IT and telecommunications industries. It is involved in the construction and administration of the urban network in Darłowo (Poland) and it provides services to over 3000 customers. The company provides IT outsourcing services: the installation and administration of network infrastructure, the implementation and administration of VoIP telephony and a helpdesk. In addition, Avena assembles and maintains a wireless Internet service in Gdansk and Gdynia using Hot Spots. Moreover, the company is currently working on an intelligent traffic monitoring system. The project involves the installation of cameras along the Tri-city ring road, the preparation of appropriate software, which allows constant monitoring of traffic on the roads, and in particular, events such as traffic jams, accidents and road works, in order to make this information available round the clock for all Internet users. Avena was founded 10 years ago, in 2003, and has since been developing dynamically. Currently, it has two offices in Poland, in Gdynia and Gdansk. The company provides external IT services to more than 30 companies in the Tri-city area. The majority of its employees in both locations work in the helpdesk department. Their remote technical support is innovative, and very few companies in Poland provide such a service, hence demand greatly exceeds supply (19 companies in Poland provide IT helpdesk services, none of which is located in the Tri-city) [10]. It was due to the significant workload in the helpdesk department that the company was forced to improve management. With thousands of service requests (SRs in short, this abbreviation will be used later in the paper), the company gradually lost the ability to control its own activities. It became impossible to manage customer data, monitor employee performance, analyze BISYSTEM DATASOURCES REPORTINGANDANALYTICALSERVICES DATAWAREHOUSE ETL INTERNETSCMCRMERPOTHERS 64 Tomasz Sitek, Michał Litka information and combine it in certain patterns. Moreover, the company had a problem with assessing the quantity of services and the management was not able to determine how much time was spent on customer service, how much it cost, and thus, whether cooperation with customers was remunerative. Since 2010, when the company started providing helpdesk services, the management's ability to make rational decisions decreased significantly. Due to the nature of the services and the large number of requests, numerous problems appeared preventing or at least hindering the decision-making process. They included:  excess of data in some areas (e.g. thousands of records related to SRs) and shortage of data in other areas (e.g. knowledge of the current state of the customer IT infrastructure),  difficult and slow access to data (e.g. a customer makes an inquiry about the possibility of installing a newer operating system on all of their computers – a manual check of hundreds of computers is very time consuming),  scattered data (over hundreds of computers among dozens of clients),  inaccuracies of data (without automatic updates on the state of the infrastructure, data must be obtained a new every time),  inability to monitor own activities (which employees provided services to a given client, how long it took, how to measure their performance),  problems with the aggregation of data and its transformation into specific information (e.g. due to clients using different hardware and software). Decision-making in such conditions was associated with working in a state of high uncertainty, regardless of whether the problem existed on the side of the client or the support organization. The management realized that the company is not able to function effectively without the support of appropriate IT systems. To cope with the above-mentioned problems, the company decided to implement a BI system. This was done, however, without the conviction that the decision was in fact the correct one. The management had no premise to justify the planned expenditure. Moreover, the changes that took place in the organization after the deployment of the system were not evaluated. This task became the target of our research. The next chapter presents a method, which the authors have developed to determine the validity of implementing BI in Avena. 4 Measuring the effects of implementing the BI system 4.1 Assumptions and key performance indicators Is it possible to measure the effects of implementing BI? For this purpose, it is necessary to create a number of evaluation indicators and then establish the changes in their values before and after the implementation of the BI system, preferably in similar periods of time. The scope of these indicators should cover the most part of the company's operation, not just one particular area. This is due to the fact that the BI system is meant to generally support decision-making in all aspects of business, and thus the indicators are designed to illustrate numerically as many of these aspects as possible. Although companies are different, several indicators may be used for most of them. Of course, they can be closely matched to the characteristics of the studied organization, and some may arise precisely tailored to the specific organization. Examples of indicators include [11]:  number of clients served/transactions executed/sales of goods, etc.,  number and productivity of employees,  actual working time of employees,  task initiation-to-completion time,  degree of customer satisfaction. It should be noted that some of these indicators can be easily quantified (e.g. the number of transactions), while others are qualitative (e.g. customer satisfaction). The authors, however, set themselves a goal to make a quantitative evaluation. Therefore, it was necessary to use certain simplifications to determine the value of indicators for the evaluation. Naturally, it was also necessary to collect relevant data for comparing the periods before and after the implementation of the BI system. This was obviously much easier to do in reference to the period "after implementation" because the BI system allows the necessary information to be obtained. To assess the validity of implementing BI at Avena, two periods were compared:  the period between April 1, 2011 and September 30,2011 (P1),  and a year later, namely the period between April 1, 2012 and September 30, 2012 (P2). Implementation of Business Intelligence in an IT Organization – the Concept of an Evaluation Model 65 Figure 2. Number of SRs in P1 and P2 To describe the business, the following four aspects were selected:  the number of SRs (Service Requests – understood as problems reported by users to an IT support organization), which require certain actions,  the number of administrators and their effectiveness,  mean time to repair (MTTR – the average time required to repair a damaged device; in this case, it is the estimated time in which the administrator is required to deal with a request from its opening to closure),  the degree of customer satisfaction. In the study, each of these aspects was described quantitatively and one or two indicators were developed for each aspect. It should be noted that the BI system was first launched in late March 2011, just before the start of P1. 4.2 Comparison of the number of service requests (SRs) Within 6 months of P2, the organization processed 4201 SRs in total, which makes an average of 700 requests per month. There were 130 working days between April 1 and September 30, 2012, which gives a little more than 32 requests per day AVG     32.32 Within P1, Avena processed fewer requests than in P2 – the number of completed SRs was 2691. There were 131 working days in period 1, which allows the calculation of the average daily number of requests AVG SR DAY  2691 131  20.54 This number of requests means that during P2, Avena handled far more SRs than the year before – making an increase of 56.11% 2 1 ∗ 100%  4201 2691 ∗ 100%  156.11% Of course, the number of requests, and thus – the average number of requests – it is not the only difference between the study periods. One quantitative value within the 6-month period is not yet conclusive. What is much more interesting is the distribution of the number of requests in given months, which is illustrated below in Figure 2. As the chart presents, there is a significant difference between the values obtained in the month of April. During the first month after the implementation of the BI system, the studied organization processed less than half the amount of SRs than a year later. In the following months, this difference began to diminish. In August, it was relatively small and amounted to only 16.35%. It is clearly visible that the company needed time to get used to the new system, although this certainly is not the only reason. Agent programs, used to operate the whole system and specifically for data collection, had to be installed on the computers of clients, which obviously took time. Similarly, the customers had to get used to the new request system. After a few months, the differences between P1 and P2 decreased significantly. 320 384 459 515 532 481 834 728 648 726 619 646 0 100 200 300 400 500 600 700 800 900 kwi maj czer lip sier wrz Okres1Okres2 Period1Period2 April May June July August September 66 Tomasz Sitek, Michał Litka Figure 3. Comparison of the number of SRs closed by administrators 4.3 Comparison of the number of administrators and their performance At the beginning of 2011, seven administrators were assigned to operate the SRs, which is one less than the following year. Five of the groups continue to work for the company. By April 2012, three new administrators were employed. Wherever possible, the number of SRs was compiled on the basis of both periods; in other cases, the amount of an administrator's SRs was shown in one period. The number of SRs closed by administrators in both periods is shown in Figure 3. The graph shows that in four out of five cases, the administrators who stayed in the company responded to more calls in 2012 than the year before. In one case, the situation is the reverse, but the difference is insignificant (only six SRs, which corresponds to about a day of work). Naturally, one period cannot be compared with the other only on the basis of the five employees who continue working, and a wider view is needed. For this purpose, the average activity time (the actual recorded working time) for all employees in both periods was measured. The value was obtained by dividing the total time spent on tasks (the BI system stores such data) by the number of man-hours available during these periods. The results are shown in Table 1. Table 1. Comparison of the average working times Period 1 (P1) Period 2 (P2) Administrator Task duration time (h:m) Average daily working time (h) Task duration time (h:m) Average daily working time (h) Difference P2-P1 Paweł S. 651 4.97 794:00 6.11 1.14 Michał S 678 5.18 728:00 5.60 0.42 Robert C. – – 414:45 3.19 3.19 Krystian M. – – 877:45 6.75 6.75 Jakub M. 569 4.34 749:30 5.77 1.43 Tomasz S. 472:30 3.61 312:45 2.41 -1.2 Michał D. 602 4.60 738:30 5.68 1.08 Małgorzata B. – – 479:15 3.69 3.69 Wojciech K. 341 2.60 – – -2.6 Mateusz R. 267:45 2.04 – – -2.04 3.9 4.9 Period1Period2 Implementation of Business Intelligence in an IT Organization – the Concept of an Evaluation Model 67 Table 2. Performance indicator ΔSR – Δt Admin. Avr. t. P1 (h) Avr. t. P2 (h) Δt SR P1 SR P2 ΔSR Performance Δ SR – Δt Paweł S. 4.97 6.11 22.94% 611 752 23.08% 0.14% Michał S. 5.18 5.60 8.11% 587 693 18.06% 9.95% Jakub M. 4.34 5.77 32.95% 478 472 -1.26% -34.20% Tomasz S. 3.61 2.41 -33.24% 217 422 94.47% 127.71% Michał D. 4.60 5.68 23.48% 228 389 70.61% 47.14% 10.85% 40.99% 30.15% By combining data on the average daily working time during in the two periods, it could be easily determined which employees were improving and which were not. It is also worth noting that in P1, two employees had a particularly low daily working time (a little over 2h per day). In effect, it was these employees who were dismissed. It can thus be concluded that employees spent more time solving problems in P2 than in P1. It is unsurprising, given that the number of SRs increased. Therefore, another table should be added to the analysis, one which compares several indicators simultaneously: the number of closed SRs, the average daily working time, the number of SRs per hour and the increase in these values in percentages. Such a comparison table is only relevant for those employees who worked in both periods. Table 2 illustrates the changes in employee performance in the two periods. First of all, it compiles the time needed to close the SR and shows whether the administrator worked more or less. The Δt indicator itself does not say, however, whether this is good or bad – after all, an employee may work less and be more efficient. To calculate such performance, the next three columns compare the changes in the number of problems solved (ΔSR). The last and most important column is the indicator of employee efficiency. It summarizes the difference in the time needed to complete a task and the change in the number of closed SRs. This helps to determine the growth in the administrator's performance. If the value in this column is 0, it means that the increase in the average working time is directly proportional to the increase in the number of SRs. If the value is positive, the administrator uses his time efficiently, if negative – less efficiently. The table shows that four-fifths of administrators increased their efficiency a year after the implementation of BI. 4.4 The comparison of mean time to repair (MTTR) The parameter of MTTR is the average time, which is needed to repair the device after failure [12]. This time is often specified in a contract and can determine whether a company dedicated to maintenance (in this case, the studied company meant to address issues related to IT) fulfills its obligations as expected (i.e. the SLA – Service Level Agreement). The same is true in this case – customers expect that their problems will be resolved within the specified time. If such time is not indicated by the SLA, the manager decides what the MTTR is for a particular task while allocating tasks. After the problem is resolved, the request must first be verified and then closed. The average closure time, verification time and MTTR in P1 were studied to establish whether the company met its obligations in the required time. In P2, in four out of six cases the MTTR was exceeded. Table 3 presents the comparison of these values with P1. As it transpires, in every month within P1, Avena managed to fit into the required task closure and verification time. In April 2011, the company had a significant time margin of over 1100 man-hours. The management realized that a lot of time was wasted and gradually reduced the MTTR. Similarly, clients came to the conclusion that the time they had allocated to such services was overestimated and they reduced it too. Nonetheless, in P1, the MTTR remained too high and the company had an excess of time. In P2, the situation was reversed: the company did not keep up with the work. If Avena signed an SLA in P1, it would have no problems with respecting the provisions of the agreement. In P2 it is impossible. 68 Tomasz Sitek, Michał Litka Table 3. MTTR compared with the closure time and the verification time of SRs in P1 Period 1 Period 2 Month MTTR (h) Closure time and verification time (h) Difference MTTR (h) Closure time and verification time (h) Difference April 1594.8 481.25 1113.55 1320.20 1010.05 310.15 May 1407.2 558.75 848.45 1161.10 1188.42 -27.32 June 1005.3 714.75 290.55 863.10 835.00 28.1 July 775.4 585.75 189.65 868.80 1144.25 -275.45 August 1394 702 692 756.60 1015.75 -259.15 September 1144.5 627 517.5 797.90 1093.25 -295.35 7321.2 3669.5 3651.7 5767.70 6286.72 -519.02 In a theoretically ideal situation, the MTTR should be equal to or greater than the times of closure and verification. Naturally, the smaller the difference between those times, the more effective the use of working time is. The time required for closure and verification should amount to about 90–95% of the MTTR, so that in the event of an emergency, there would be a certain time margin. Excluding such a time margin, the upper limit may be set at 100% (i.e. the time of closure and verification of SRs is exactly the same as the MTTR). Table 4 shows the value of the quotient of the closure and verification time and the MTTR. As shown in Table 4, in seven of the researched months the value of the closure time and verification time/MTTR is too low; in four months, it is too high, and only in one it is appropriate (June 2012). The Table is complemented by Figure 4. Table 4. Indicator of the closure and verification time/MTTR Period 1 Period 2 Month MTTR (h) Closure time and verification time (h) Closure time and verification time/MTTR MTTR (h) Closure time and verification time (h) Closure time and verification time/MTTR April 1594.8 481.25 30.18% 1320.20 1010.05 76.51% May 1407.2 558.75 39.71% 1161.10 1188.42 102.35% June 1005.3 714.75 71.10% 863.10 835.00 96.74% July 775.4 585.75 75.54% 868.80 1144.25 131.70% August 1394 702 50.36% 756.60 1015.75 134.25% September 1144.5 627 54.78% 797.90 1093.25 137.02% 7321.2 3669.5 53.61% 5767.70 6286.72 113.10%