User experience evaluation of academic information system application in android version

 Corresponding author: Novie Susanto Copyright © 2025 Author(s) retain the copyright of this article. This article is published under the terms of the Creative Commons Attribution License 4.0. User experience evaluation of academic information system application in android version Novie Susanto *, Elan Ayu Norsyarif and Nia Budi Puspitasari Department of Industrial Engineering, Faculty of Engineering, Diponegoro University. World Journal of Advanced Research and Reviews, 2025, 28(02), 376-383 Publication history: Received on 23 September 2025; revised on 01 November 2025; accepted on 03 November 2025 Article DOI: https://doi.org/10.30574/wjarr.2025.28.2.3692 Abstract The aim of this study is to assess the quality of the user experience of an academic information system application (called as SIAP) on Android using UEQ, to measure the efficiency and effectiveness of the application using Performance Measurement, to identify problem using Retrospective Think Aloud and to provide solutions for the improvement. Based on preliminary study, it is found that there are some complaints about the use of SIAP application in Android version. The complaints resume some problem including the less attractive color palette, absence of notification features, and absence of transcript features. The respondents were determined based on purposive sampling from active students and used the User Experience Questionnaire (UEQ) method, with neutral evaluation results and poor benchmark comparisons. Performance was measured using Success Rate and Time-Based Efficiency. In the Retrospective Think Aloud (RTA) method, the data produced is used as input of design prototype using Figma with the implementation of Schneiderman's Eight Golden Rules. Furthermore, prototype was tested again use UEQ and results a positive and neutral categories. The final result of the prototype shows a need to repair the interface and recommendations are established. Keywords: User Experience; Performance Measurement; Usability; Academic Information System; Application 1. Introduction Academic Information System SIAP application is an academic information system which integrated with Single Sign On (SSO) system in a university. It aims to give convenience for its users in academic administration activities, such as the process of accepting new students, making study plan card, filling in grades, and lecturer and student data management. SIAP application can be installed through Play Store on all over smartphone based on Android. With all the features contained in the SIAP application there were still complaints from the user side, especially in the bug section or error finding such as unable to login as well as various features that cannot be opened by the users. Apart from that, there are many features not yet available on the SIAP application on Android and only compatible for access on the website via desktop. One of the complaints that stands out is the User Interface (UI) and color palette which considered too rigid. Preliminary study by interviewing ten users of SIAP application as well as capturing application comments in Play Store was found some uncomfortable condition such as the less attractive color palette, absence of notification features, and absence of transcript features. From the results of open-ended interviews using semi-structured questions using an interview guide derived from the topic development, the largest percentage of deficiencies identified were the lack of application features, such as the addition of a Github feature to the application, improvements to the camera zoom and focus features for barcode scanning, the lack of a barcode scanning feature from the gallery, and the absence of a widget feature that can be added to the application dashboard. The purpose of these interviews was to identify problems World Journal of Advanced Research and Reviews, 2025, 28(02), 376-383 377 more openly by asking the interviewees for their opinions and ideas. Furthermore, other complaints from students included bugs in the application that made it difficult to log in, slow or no page loading, the inability to scan barcodes, and an overly rigid User Interface (UI) and color palette. These issues make the SIAP application on Android inefficient for launch and use by users, and improvements are still needed in future updates. Based on preliminary evaluation, it can be stated that the SIAP application on Android only has 39% of the total features on the SIAP website. However, there is a feature that is not available on the SIAP website and is only available on the SIAP application on Android, namely Absence History. Several things that make the SIAP application on Android still very lacking in terms of User Interface and User Experience. User Interface or commonly referred to as UI is everything that can interact with users to use a digital product or service. This includes everything from displays, pages, or visual elements such as buttons and icons on screens and touchscreens, keyboards, sound output, and even lights that function as a means of user interaction with devices in an application system (User Testing 2018; Griffin and Baston, 2014). User Experience itself or commonly referred to as UX is a user experience that encompasses all aspects of the interaction between users and a company, service, or product (Norman, 1990 and ISO, 1998) which is the result of UI improvements. Once there is something to interact with the user, their experience, whether positive, negative, or neutral, changes the user's feelings about the interaction. In the absence of these things, mobile device users will be faced with some content that does not appear optimally and confuses users in accessing and using existing features. From this statement, a survey can be conducted to conclude how good the user interface on the SIAP application is so that conclusions can be drawn that can be improved. Suggestions for improvement can be identified by assessing student perceptions of the SIAP application on Android. The method used is the UEQ (User Experience Questionnaire) which is an easy and efficient tool or questionnaire for measuring User Experience (UX) (Santoso et al., 2016 and Rauschenberger et al., 2013). The UEQ consists of 26 questions classified into six dimensions, namely attractiveness, clarity (perspicacity), efficiency (efficiency), accuracy (dependability), stimulation (stimulation), and novelty (novelty) (Schrepp, 2019). This perception assessment was examined on the SIAP application on Android and the SIAP application prototype which was later compared to see whether there were significant differences between the two objects. Next, observations were conducted on task completion by respondents using the performance measurement method which can be used to obtain quantitative data about respondent performance when using the application (Suwignjo et al., 2000). After completing the task, respondents also conducted a Retrospective Think Aloud (RTA) technique P (Nielsen, 2012; Galits, 2002; Ericson and Simon, 1993), which was used to obtain qualitative data derived from the verbal expressions expressed by respondents during the study, which was then processed into product recommendations. All of these methods yielded positive research results. This research will provide input for future efforts to improve the usability of the SIAP application information system. Previous research by Syahrina dan Kusumasari (2020) discusses process design of user interface from B2B Textile ECommerce application use Five Planes Framework and Eight Golden Rules. Result of this study was excluding user experience test after the redesign of the interface. By doing so, it is difficult to assess whether the redesign produce a better or worse design. This study aims to evaluate the SIAP application especially in UI/UX condition and recommend some features based on the evaluation result. 2. Material and methods This study focuses on the SIAP application for the Android operating system and its prototype, with 64 active students at University Y selected through a purposive sampling method. The research instruments used were informal interviews, questionnaires, and direct observation. The study began with a preliminary study to analyze the problems experienced by students when using the SIAP application on Android devices. Data was collected through the UEQ (User Experience Questionnaire), which can measure user experience (UX). In the User Experience Questionnaire, a benchmark dataset was conducted using questionnaire results with a larger sample size and from various types of respondents for various types of related products or services. This was intended to ensure that the User Experience Questionnaire has a universal assessment interval so that it can be used as a reference for various types of products or services being studied. Task observation using Performance Measurement and verbalization while performing tasks using Retrospective Think Aloud as the first objective of this study and recapitulating data on the time, speed, and accuracy of respondents' performance for Performance Measurement as well as recapitulating data on complaints, suggestions, and input from respondents while performing think aloud. Table 1 shows the task scenario for collecting observation data to test the effectiveness and efficiency of the application. World Journal of Advanced Research and Reviews, 2025, 28(02), 376-383 378 Table 1 Task Scenario Nr Task Scenario 1 Log in using SSO 2 View courses taken in Semester 2 via the Study Plan (IRS) menu 3 View Engineering Mechanics course grades in Semester 2 via the Study Result Card (KHS) menu 4 Show the schedule for September 5, 2022 5 View course absence history for the second week of September 6 View account information 7 Return to the home page 8 Log out of your SIAP account Afterward, UEQ data processing and validation and reliability tests were conducted to ensure that respondents' answers were consistent and valid using the Cronbach's Alpha Coefficient. Then, Performance Measurement data effectiveness was tested using the Success Rate and efficiency using Time-Based Efficiency. The following formulas were used in processing the Success Rate data: 𝑆𝑅=((𝑆+(𝑃𝑆 × 0.5)) 𝑇)×100% (1) Description • SR = Success Rate • S = Success • PS = Partial Success • T = Task In this study, the results of the success rate measurements were then interpreted based on the 1991 Ministry of Home Affairs Research and Development Reference Standards to determine the level of effectiveness, as shown in Table 2. Table 2 Standard of Effectivity Nr Effectivity Ratio Achievement Level 1 < 40% Very ineffective 2 40% - 59.99% Ineffective 3 60% - 79.99% Moderately effective 4 ≥ 80% Very effective Description: • TBE = Time-Based Efficiency • N = total number of tasks (goals) • R = number of users • nij = result of task i by user j. If the user successfully completes the task, Nij = 1; otherwise, Nij = 0; and tij = time spent by user j to complete task i (if the task is not successfully completed, the time is measured until the user stops the task). The measurement results are then interpreted using the time range in the time behavior indicator, which can be used to determine the user's time duration, as presented in Table 3. World Journal of Advanced Research and Reviews, 2025, 28(02), 376-383 379 Table 3 Time Interval on Time Behavior Indicator Nr Duration Qualification 1 1 minute – 5 minutes Very fast 2 6 minutes – 10 minutes Fast 3 11 minutes – 15 minutes Slow Furthermore, according to [9], the user verbalization process can enable observers to interpret problematic parts of the user interface. A summary of suggestions and feedback from the Retrospective Think Aloud is ranked as input for wireframe improvements. The data obtained from this think-aloud process is analyzable because it generates initial impressions from respondents, which are then summarized in the form of complaints and suggestions or input that can assist in developing wireframe improvements for the application. A prototype of the application improvement design will then be designed using the Figma platform, accessible on a mobile phone or PC using a link in a web browser, for further research. Next, value calculations and interpretation of the object evaluation results will be performed, and the user experience level of the SIAP application on Android will be calculated. From this data collection and processing, it can be concluded that wireframe improvement is the third objective of this research, which is aligned with the Eight Golden Rules principles (Shneidermann, 1986). 3. Result and discussion The graphical data shows a negative trend toward the SIAP Android application, based on the number of respondents' responses, as well as the data mode and distribution. The most significant negative trend is found in the assessment items: conventional/inventive, conservative/innovative, common/advanced, unsafe/safe, monotonous/creative, and troublesome/enjoyable. Table 4 shows a summary of the evaluation of the SIAP Android application from the preliminary study. The suggestions obtained were collected through brief online interviews, resulting in varied diction and wording, but suggestions with similar intent can be grouped into one. Table 4 Distribution of Opinions on the Disadvantages of the SIAP Application on Android Nr Information Classification Percentage A There are bugs in the login section Bug/Error 37.78% B There are bugs and errors in the loading screen C Cannot scan barcodes D No HER-Reg feature New Feature 53.33% E No Transcript feature F Addition of GitHub features* G Improved zoom and focus features for barcode scanning cameras H Attendance list feature is not displayed in a table, making it difficult to understand I Lack of internal browser J Transcript cannot be opened on mobile phones K Lack of barcode input feature L Lack of barcode scanning feature from gallery M Lack of widgets that can be added to the dashboard N Lack of search feature (courses, lecturers) O User Interface (UI) and color palette are too rigid UI/UX 8.89% Total 100% World Journal of Advanced Research and Reviews, 2025, 28(02), 376-383 380 The performance measurement method uses task scenarios created based on design constraints that will be improved upon from the existing application interface design. While respondents performed the task scenarios, researchers recorded and noted any errors they made while performing each task. Performance Measurement has two components: effectiveness and efficiency. The effectiveness section uses a Success Rate, which describes the respondents' success or failure in completing each task in the task scenario. Respondents were considered to have failed if they were unable to complete the assigned task due to difficulty. The results showed that all respondents successfully completed all tasks in the task scenario. This was possible because the SIAP application on Android was effective. The second component in the performance measurement section is efficiency. This section presents the calculated time required by respondents to complete each assigned task. This section is useful for determining how long respondents took to complete the task scenario. Table 5 summarizes the results of the Performance Measurement test for efficiency. Table 5 Recapitulation of Data Collection Results Completion Time Respondent Code T1 T2 T3 T4 T5 T6 T7 T8 Total Time (s) R1 62.88 22.87 33.97 13.52 6.4 3.66 1.67 3.22 148.19 R2 17.64 27.24 16.98 10.63 12.19 4.5 4.24 1.9 95.32 R3 11.1 20.96 16.63 9.65 12.43 6.99 5.45 5.25 88.46 R4 17.41 14.41 10.92 7.12 5.32 3.66 1.39 2.1 62.33 R5 15.37 15.54 14.74 10.13 8.74 5.26 1.99 3.32 75.09 R6 11.03 16.62 24.68 12.67 6.02 4.59 5.21 3.82 84.64 R7 10.12 18.46 25.08 8.79 7.78 7.65 1.88 2.93 82.69 Table 5 shows that all respondents completed all tasks quickly. The longest time required for a task was 62.88 seconds, which was the login process for the SIAP application on Android. Based on the measurements, all respondents completed the assigned task scenario with a 100% success rate (very effective). The average time required for student respondents to complete each task in the task scenario was 0.17 seconds per task, which can be categorized as a time behavior indicator with a Very Fast achievement level. The results of the evaluation of the UEQ-related research are shown in Table 6, which includes scale and benchmark evaluations from a total of 9,905 responses. Table 6 UEQ Evaluation Results of Existing SIAP Applications Scale Mean Result Comparison to Benchmark Attractiveness 0,150 Neutral Evaluation Bad Clarity 0,533 Neutral Evaluation Bad Efficiency 0,129 Neutral Evaluation Bad Accuracy -0,040 Neutral Evaluation Bad Stimulation 0,081 Neutral Evaluation Bad Novelty -0,324 Neutral Evaluation Bad Other research findings refer to the Retrospective Think Aloud, where users were asked to express anything on their minds, from their opinions about the SIAP app on Android to suggestions for improving the app. The following is a recapitulation of the Retrospective Think Aloud data expressed by respondents during observation data collection, as shown in Table 7. World Journal of Advanced Research and Reviews, 2025, 28(02), 376-383 381 Table 7 Recapitulation of Retrospective Think Aloud Data Nr Respondent Code Retrospective Think Aloud 1 R1 The IRS and KHS sections don't display the semester, only the academic year, making it less accessible than the semester itself. Widgets are added to the app, such as student schedules for the semester and pre-class alarms. 2 R2 The IRS and KHS sections don't display the semester, only the academic year, making it less accessible than the semester itself. The color palette is unattractive. 3 R3 The menu placement and symbols are unattractive, and the notification feature for replacement classes is missing. 4 R4 There's no transcript feature in PDF format. 5 R5 The absence history has the wrong status. The app features are incomplete compared to the website, the notification feature for replacement classes, and the color palette are unattractive. 6 R6 Boring, the color palette is unattractive, the app features are incomplete compared to the website, and the notification feature for class availability and the alarm feature before class start. 7 R7 The color palette is unattractive and the font and background color combinations are inconsistent. The app features are incomplete compared to the website, and there's no transcript feature in PDF format. It can be seen that the most frequently raised issues in the Think Aloud Retrospective were the lack of comprehensive features available in the SIAP Android application compared to the website, the unattractive color palette, and the interface not being designed according to the appropriate information hierarchy. Of all the issues raised by respondents in the Think Aloud Retrospective data collection in Table 8, the following is a consolidation with the evaluator, namely Shneiderman's 8 Golden Rules, used for the problem verification process and recommendations that will later be used as guidelines for implementing the SIAP Android application design development. This is expected to improve the application's performance in terms of both interface and functionality. Table 8 Think Aloud Retrospective Consolidation Recapitulation with Shneiderman's 8 Golden Rules for Existing SIAP Applications and proposed display improvements Shneiderman’s Principle Problem Recommendation Strive for consistency The color palette is unattractive, and the font and background color combinations are inappropriate. A more attractive color palette with an easy-to-read font and background color combination has been used. Reduce Short-Term Memory Load The menu placement and symbols are unattractive. Changes to several symbols to make them more representative. The IRS and KHS sections don't have the semester information listed, only the academic year, making it less easily understood than the semester. Addition of semester text to the IRS and KHS content. Penambahan Fitur There's no transcript feature in PDF format. Addition of a transcript feature to the homepage menu. There's no weekly semester schedule feature on the homepage. Addition of a semester schedule feature to the homepage. The app features are incomplete, unlike the website. Addition of academic achievement (GPA and SKSk), Her-Registration, and Transcript features to the prototype. World Journal of Advanced Research and Reviews, 2025, 28(02), 376-383 382 There's no replacement class notification feature. Addition of a replacement class notification feature. There's no notification feature for the class on that day. Addition of a notification feature for the class on that day. 4. Conclusion and further study The evaluation results using the UEQ method on the existing SIAP application on Android showed a neutral evaluation on the scale of attractiveness, clarity, efficiency, accuracy, stimulation, and novelty, meaning that users of this application felt that these aspects of the Existing SIAP application on Android were quite good but not perfect and there were several aspects that could be improved. The benchmark results showed poor quality (bad) with information in the range of the 25% worst results, which can be defined when compared with data from related research respondents collected by the UEQ Team. In the Performance Measurement method, the evaluation results showed that the level of effectiveness measurement using the Success Rate measurement produced a value of 100% because all respondents successfully completed the task scenario with a Very Effective achievement level. The next stage was the efficiency measurement processed using Time Based Efficiency with the results of the calculation of the efficiency level having an average value of 0.173 seconds/task, categorized as Very Fast. This can occur because students are already proficient in operating the application, so they get very good scores and are very fast in completing the task scenario. The results of the Retrospective Think Aloud method in this study were respondents' complaints after the Performance Measurement observations. The most frequent complaints concerned the application's unattractive color palette, with the font and background color combinations deemed inappropriate by respondents; the application's unattractive layout and use of symbols; and numerous incomplete features, such as the lack of a PDF transcript feature, a replacement class notification feature, incomplete application features compared to the website, etc. Based on the list of complaints, input, and recommendations provided by respondents, a design process for improvements to the SIAP application was carried out using Schneiderman's Eight Golden Rules as a design reference. The resulting prototype included several additional features, including an application that can be used for student HerReg, download temporary transcripts, and record attendance using the course attendance code. Future research could focus on creating an Android application prototype based on the recommendations and evaluating the UI and UX of the improvements. Compliance with ethical standards Disclosure of conflict of interest No conflict of interest to be disclosed. Statement of Informed Consent Informed consent was obtained from all individual participants included in the study References [1] Ericson, K.A and Simon, H.A. (1993). Protocol Analysis: Verbal Reports as Data. A Bradford Book, London: The MIT Press. [2] Galitz, W. (2002). The essential guide to user interface design: an Introduction to GUI design principles and techniques. Canada: John Wiley and Sons [3] Griffin, B., and Baston, L. (2014). Interface. Girton Colege University of Cambridge. Cambridge. [4] International Organization for Standardization. (1998). ISO 9241-11:1998 Ergonomic requirements for office work with visual display terminals (VDTs) — Part 11: Guidance on usability. Retrieved September 2nd, 2021 from ISO/TC 159/SC 4: https://www.iso.org/standard/16883.html [5] Nielsen, J. (2012). Thinking Aloud: The #1 Usability Tool. Retrieved on Februari 10th, 2022 from: https://www.nngroup.com/articles/thinking-aloud-the-1-usability-tool/ World Journal of Advanced Research and Reviews, 2025, 28(02), 376-383 383 [6] Norman, D. (1990). The psychopathology of everyday things. In D. Norman, The design on everyday things. New York: Bantam Doubleday Bell. [7] Rauschenberger, M., Schrepp, M., Perez-Cota, M., Olschner, S., and Thomaschewski, J. (2013). Efficient Measurement of the User Experience of Interactive Products. How to use the User Experience Qustionnaire (UEQ). International Journal of Interactive Multimedia and Artificial Intelligence 2: 39-45. DOI: 10.9781/ijimai.2013.215 [8] Santoso, H., Schrepp, M., Yugo Kartono Isal, R., Utomo, Y., and Priyogi, B. (2016). Measuring User Experience of the Student-Centered e-Learning Environment. The Journal of Educators Online-JEO, 13(1), 58-79. [9] Schrepp, M. (2019). User Experience Questionnaire Handbook. Retrieved online October 28th, 2025 from https://www.ueq-online.org/Material/Handbook.pdf [10] Shneiderman, B. (1986). Designing Menu Selection Systems. Journal of the American Society for Information Science. 37(2):57-70. [11] Suwignjo, P., Bititci, U.S and Carrie, A.S. (2000). Quantitative models for performance measurement system. International Journal of Production Economics. 64 (Issues 1-3), 1 March 2000, 231-241. [12] Syahrina,A. and Kusumasari, T. (2020). Designing User Experience and User Interface of a B2B Textile eCommerce using Five Planes Framework. International Jpurnal of Innovation in Enterprise System. 4(01): 44-55. [13] User Testing. (2018). UI vs. UX: What’s the difference between user interface and user experience? Retrieved on May 6th, 2021 from: https://www.usertesting.com/blog/ui-vsux#:~:text=%E2%80%9CUser%20experience%20(UX)%20is,a%20company's%20products%20and%20servi ces.andtext=User%20interface%20(UI)%20is%20the%20specific%20asset%20users%20interact%20with,suc h%20as%20colors%20and%20typography