Online Student Authentication and Proctoring System Based on Multimodal Biometrics Technology

Received April 27, 2021, accepted May 5, 2021, date of publication May 11, 2021, date of current version May 21, 2021. Digital Object Identifier 10.1109/ACCESS.2021.3079375 Online Student Authentication and Proctoring System Based on Multimodal Biometrics Technology MIKEL LABAYEN 1,3, RICARDO VEA 1, JULIÁN FLÓREZ2, (Member, IEEE), NAIARA AGINAKO 3, AND BASILIO SIERRA 3 1Smowltech, 20009 Donostia, Spain 2Vicomtech Research Center, 20009 Donostia, Spain 3Computer Sciences and Artificial Intelligence Department, University of the Basque Country, 20018 Donostia, Spain Corresponding author: Mikel Labayen ([email protected]) This work was supported by the Spanish Ministry of Sciences, Research and Universities (Ministerio de Ciencia, Innovación y Universidades (MCIU)/Agencia Estatal de Investigación (AEI)/Fondo Europeo de Desarrollo Regional (FEDER), Unión Europea (UE)) under Grant RTC-2016-5711-7. ABSTRACT Identity verification and proctoring of online students are one of the key challenges to online learning today. Especially for online certification and accreditation, the training organizations need to verify that the online students who completed the learning process and received the academic credits are those who registered for the courses. Furthermore, they need to ensure that these students complete all the activities of online training without cheating or inappropriate behaviours. The COVID-19 pandemic has accelerated (abruptly in certain cases) the migration and implementation of online education strategies and consequently the need for safe mechanisms to authenticate and proctor online students. Nowadays, there are several technologies with different grades of automation. In this paper, we deeply describe a specific solution based on the authentication of different biometric technologies and an automatic proctoring system (system workflow as well as AI algorithms), which incorporates features to solve the main concerns in the market: highly scalable, automatic, affordable, with few hardware and software requirements for the user, reliable and passive for the student. Finally, the technological performance test of the large scale system, the usabilityprivacy perception survey of the user and their results are discussed in this work. INDEX TERMS Biometric authentication, cloud computing, computer vision, data science applications in education, distance education and online learning, machine learning, security, computer vision. I. INTRODUCTION There is no doubt that online learning has been gaining popularity throughout the past years. This phenomenon is not surprising given that online learning allows education institutes to operate at a lower cost and with greater reachout to more students. Educational institutions are offering courses online to leverage the benefits of online learning. This is especially so since the advent of Massive Open Online Courses (MOOC). On the other hand, COVID-19 has been a challenge for traditional institutes offering face-toface teaching, and these institutions have had to migrate (in a very short period of time) to a fully online education model The associate editor coordinating the review of this manuscript and approving it for publication was Tony Thomas. forced by the pandemic situation. However, online learning implementation presents challenges. E-learning has a serious deficiency, which is the lack of efficient mechanisms that assure user authentication, in the system login as well as throughout the session. Especially for online certification and accreditation, the training organizations need to verify that the online learners who completed the learning process and received the academic credits are precisely those who registered for the courses. Inadequate methods of identity verification affect the reliability of credentials and certification earned online. Without certainty of the authenticity of the online learner’s identity, the aspiration towards fully online education is stymied and the evaluation of the knowledge and skills obtained by the online learner is unreliable. In order to prevent compromising the credibility of online accreditation, 72398 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ VOLUME 9, 2021 M. Labayen et al.: Online Student Authentication and Proctoring System validation must be carried out in a constant or continuous manner. At the same time, validation should be non-invasive and non-disruptive, and does not distract the learning process. Online proctoring, generally refers to proctors (humans) monitoring an exam over the internet through a webcam. It includes as well the processes, occurring at a distance, for authenticating the examinee as the person who should be taking the exam. Online proctoring was first introduced by Kryterion [1], [2] in 2006, marketing it as a technological solution in 2008. Since then, several other organizations have followed Kryterion’s lead creating more capable technologybased alternatives, which are gaining attention, such as online proctoring. Nowadays, there are commercial solutions in the market as well as research publications that try to solve this problem. Some of them only authenticate the identity, others monitor, some in real time, others record the sessions. Some cover only exams or specific activities. Some are totally human based solutions (non-scalable) or fully automatic ones (non-reliable). There are also a few scientific approaches which develop the idea of combining some of the cited functionalities. However, there is no comprehensive and reliable solution which combines multi-biometric continuous authentication with continuous visual and audio monitoring, with device activity monitoring and lock-down options and human supervision (only when required) to guarantee 100% reliable results. In this work we present a new system which gives commercial solutions to all that was needed. It is based on web applications which offer a continuous authentication identity service of online students through a constant biometric (face, voice, typing) recognition system (biometric traits cannot be lost, stolen, or recreated), as well as automatic continuous proctoring through automatic image and audio processing (device monitoring & lock-down and inappropriate behaviour detection) allowing online courses to gain value of what benefits both institutions and students. This solution is based on a high accuracy biometrics recognition and digital signal processing algorithms and it is complemented with human supervision for those situations in which the automatic algorithms are not able to determine reliable results. It can be used to continuously authenticate the learners, either throughout the entire learning process, or only at certain sensitive stages of e-learning. It is contactless and needs only a low level of user collaboration. In addition, the whole system is based on cloud computing technologies, which removes geographical and technological barriers for online learning providers. The article is organized as follows. Section II gives an overview of some relevant related works and highlights the main differences with our approach. Section III describes the whole system overview and workflow. Section IV contains a scientific-technical description of core modules. Section V presents system tests to measure the algorithms’ performance as well as a survey made for user experience evaluation. Section VI presents the results of the tests. Finally, section VII draws the conclusions and presents future works. II. RELATED WORK The ability to authenticate and monitor online users is becoming more important due to the increase of the internet world (e-learning, e-banking, e-gambling, e-government). Since first human based online proctoring systems, various fully or semi-automatic authentication and proctoring technologies based on biometric features have appeared in the last few years. Biometrics has proved itself to be one of the best methods for recognizing people based upon physiological or behavioural characteristics [3]. These technologies can be divided into two categories: those that are based on physical characteristics and those that are based on behaviour characteristics. The former includes face recognition, fingerprint scanners, iris scanners, vein matching, etc. The latter includes voice recognition, handwriting recognition, keystroke dynamics, etc. It is proved that no technology will provide the right answer on its own, but that the combination of different solutions will come up with the appropriate functionality depending on customer needs. In addition, most remote authentication proctoring technologies involve some level of human intervention for fully reliable service, thereby putting limitations on scale. These biometric technologies have been widely used for various purposes, and they have become more and more common in our daily lives. However, very few of them have been successfully adopted for online learning validation. A. COMMERCIAL SOLUTIONS Some initial approaches have been brought to market as commercial solutions. The following is an overview of these services: 1) Fully Live Online Proctoring: Students are on video and watched remotely by a live proctor. Live proctoring is a live online service for students taking exams online. After making an appointment, the students are taken to the online proctoring room where they will connect with a live proctor from one of the two online proctoring centres via their web cameras. The students connect their screen to the proctor. This allows the proctor to see their computer screen. The proctor asks them to show a photo ID and to answer a few questions about themselves in order to verify they are in fact the right student. During the exam, the proctor looks at the student directly through a webcam. It is a secure and complete solution for exam proctoring, but since it is a non-automatic solution, it cannot deal with continuous identification during all learning process. Furthermore, it needs a high speed internet channel to transmit video data, probably unaffordable for different parts of the world and it is not passive for students. Some commercial solutions in the market are ProctorU [4], Examity [5] and Software Secure - PSI [6]. 2) Recorded and Reviewed Proctoring: Sessions are recorded as the computer monitors students. A human can then review the video at any time afterward. VOLUME 9, 2021 72399 M. Labayen et al.: Online Student Authentication and Proctoring System In these systems, students use their own computer and a webcam to record assessment sessions, the student and the surrounding environment are recorded during the entire exam. Instructors can quickly review details of the assessment, and even watch the recorded video. Recorded proctoring has the same limitations as live proctoring. In addition, it is a passive system. However, nobody analyzes the videos, so teachers must watch all of them in order to detect undesirable behaviours and maintain the live proctoring advantages. Some commercial solutions in the market are Kryterion [1], ProctorExam [7], Respondus [8], Remote Proctor [9], ProctorCam [10], B virtual [11] and Learner verified [12]. 3) Fully Automated Solutions: The computer monitors students, it authenticates them and determines whether they are cheating. These are automatic and passive solutions. They just cover the beginnings of exams and work submission processes. However, users must be totally active in this kind of system (they must type a predefined paragraph and take an ID photo themselves). In addition, this kind of system does not cover all the learning process continuously. Some commercial solutions in the market are Proctorio [13], ProctorTrack [14], Comprobo [15], Sumadi [16], ProctorFree [17], HonorLock [18] and ExamSoft [19]. a) Authentication technologies: Recognition technologies are used to authenticate a student based on a prior examination of some physical feature. They are typically built upon a before/during/after analysis to verify that the same student who initially registered for the course was actually the same student who took the exam. Commonlyknown recognition technologies include facial, fingerprint, or voice recognition. In the last year, new biometric procedures such as keystroke dynamics (it recognizes typing patterns based on rhythm, pressure, and style) are gaining popularity. It is likely that recognition technologies will be most effective when used with some combination of other technologies available. b) Monitoring technologies: i) Webcams and microphones are one of the original technologies used to replace a live proctor and are present in most remote exam proctoring solutions on the market. They can record individual students when the camera is part of the computer, or groups when the camera is placed in a classroom. They can monitor the behaviour of the students, whether they are cheating, receiving help from other students, using mobile devices, books...Webcam/Microphone technologies often require significant storage capabilities so that video records can be reviewed if necessary. ii) Computer lockdowns are able to monitor the activity carried out by the student within their computer preventing them from ‘‘surfing the internet’’ while taking a test. This monitoring will be done only and exclusively when the student is doing an activity that can be evaluated. None of the cited commercial solutions provides a multibiometric authentication solution or continuous authentication/proctoring service (based on automatic analysis) through the whole learning course (not only exams). In addition, this work presents a completely new commercial approach to overcome barriers such as low-speed internet connection (using data samples, not continuous heavy video signals) or costly extra HW/SW requirements (using noninstallable and fully integrated in LMS web applications). B. SCIENTIFIC AND ACADEMIC APPROACHES 1) TECHNICAL WORKS Nowadays, although there are still some non-biometric based authentication approaches [20], the latest attempts for online student authentication automation tends to use biometric technologies; facial [21]–[26], fingerprints [27] or typing [28], [29]. On the other hand, some approaches try some combination of them, such as face and voice [30] or face, voice and typing [31], [32]. All the approaches are focused mainly on student authentication without providing proctoring service. It is through facial authentication complemented with other biometrics such as voice or typing recognition, that an opportunity appears in e-learning to verify the absence of frauds while the students do their activities on the platform. The main novel contribution of the work we present in this article includes a completely new combination workflow of three main biometrics providing a continuous and non-intrusive authentication service. It also adds new automatic and continuous proctoring features based on image and audio signal processing to the system. Furthermore, it integrates computer activity monitoring and lock-down possibility and, finally, it even complements the service with automatic alarms which trigger minimal human supervision, guaranteeing the reliability of results. Finally, the recent concern for safety and privacy has also provided recent research on this topic related to online proctoring [33]. 2) USER EXPERIENCE RELATED WORKS On the other hand, very few works completed the research about teachers and student user experience with this kind of authentication and proctoring approaches. One of them completed the research about the implementation of facial verification into education with a successful positive result [34]. The objective was to guarantee students authentication and to know exactly the amount of time that they spend in front of the computer reading or realizing their virtual activities. 72400 VOLUME 9, 2021 M. Labayen et al.: Online Student Authentication and Proctoring System TABLE 1. Commercial solutions vs SMOWL (solution described in this article). Service characteristics: 1-Authentication during whole exam or session; 2-Multi biometric authentication (at least 2 different); 3-Exam monitoring; 4-Continuous (full course) monitoring; 5-Dishonest behaviour detection; 6-Totally Passive and non-intrusive system; 7-Automatically analyzed results; 8-100% guaranteed and reliable results; 9-Personalised alarms; 10-Human real-time proctor; 11-Device monitoring. Technical features: 12-Scalable system; 13-Flexible access to students - no scheduled; 14-No extra SW/HW installation required for authentication and proctoring; 15-Works with low-speed connection; 16-Fully integrated in institution LMS; 17-Multi-Browser & device. Legal aspects: 18-EU-hosted solution; 19-GDPR compliance. XYes | XNo. In the same way, a facial authentication mechanism was also presented. This insured that the students are not impersonated to improve their marks in virtual tests [35]. III. SYSTEM OVERVIEW The system we present in this work aims to provide a practical cyber-security solution for both a) continuous online user identification (using biometric technology) and b) monitoring using automatic signal processing and a computer monitoring system. The authentication process is based on automatic authentication of facial images (captured by webcams), audio clips (captured by the microphone) and keystroke dynamics (captured by the keyboard), checking that it is the person that it really should be during the entire online interaction. The monitoring process is supported by webcams and microphones too, checking continuously that the student is not making any inappropriate behaviour (using forbidden devices and applications, receiving help. . .). It also locks down the computers (with a previous installation in the learner computer and consent) during exams or training sessions preventing the user from visiting web pages or other documents while performing the course. The system can be used for any online user authentication but it is specialized in the institutions that offer online courses TABLE 2. State-of-the-art solutions vs SMOWL (solution described in this article). Authentication method: 1-Face recognition; 2-Voice recognition; 3-Typing recognition; 4-Continuous authentication during whole session (not only at the beginning). Proctoring-Monitoring method: 5-Image processing; 6-Audio processing; 7-Screenshots capture; 8-Device information capture (active window, open processes, peripherals devices, copy/paste commands...). Proctoring-Device Lock-Down: 9-Device lock-down. Guarantee: 10-Human supervision to clarify doubts providing 100% guaranteed and reliable results. X-Yes | XNo. providing training and degree certification, including verified MOOCs and corporate training for employees. This system can help e-learning providers in their objective to be awarded credit by Quality Educational Agencies for their courses by seeking traceability of evidence of student authenticity and their behaviour. It can be used to track the continuous authentication of the student in all or in sensitive stages of VOLUME 9, 2021 72401 M. Labayen et al.: Online Student Authentication and Proctoring System FIGURE 1. Authentication and proctoring system set-up. FIGURE 2. Processing core description. e-learning. Figure 1shows general set-up of the system and Figure 2details the processing core description. The complete system workflow is embedded in cloud computing applications, and can be used anywhere, removing geographical and technological barriers. The general scheme of operation is as follows and is given in more detail in Figure 3: 1) The system is integrated into the virtual campus of the training centre (available for different LMS platforms). 2) The training centre sends a code (unique student identifier) with an image of the student to register in the system. According to system data privacy policy, the system works with images, audio clips...not identities, so it lacks connection with the student personal data such as name, age or address [36]. 3) The first time the student enters the virtual campus the system takes biometric samples (picture, short speech, predefined paragraph typing) which will help us create the tracking biometrical model. 4) Thereafter, whenever the student is connected to work, biometric samples will be taken randomly and continuously. This data is sent to servers in the cloud. The online management module stores and analyzes the data which is compared with the biometrical model that has been created previously for authentication purposes and analyzed to detect inappropriate behaviours. All storage, analysis and results report and alarm creation tasks are executed in online servers, making the integration, support and maintenance tasks for institutions easier and more transparent. During this period, the computer lockdown module can be activated for monitoring purposes. 5) The result leads to an individual user report that is updated constantly and to which the training centre has access. 72402 VOLUME 9, 2021 M. Labayen et al.: Online Student Authentication and Proctoring System FIGURE 3. System workflow. The key characteristics of the system are: 1) Continuous and not scheduled system. Proctoring and authentication processes are carried out throughout the entire session, not only when users log in. Furthermore, in the e-learning case, it can follow every session of the course, not only the assessments. It is very flexible. Service is given 24/7, anywhere. Previous schedule is not required. 2) Passive & non-intrusive system. The system offers a passive system for students when taking photos, audio clips or keystroke pattern. It does not need the collaboration of the student and it is contactless. For this reason, in the case of images, it properly works when the pose/appearance/complements/expressions of the students or the light conditions of the room are not controlled (in the wild), getting low-contrast images with partial occlusions due to wrong position or the appearance/compliments/expressions variations of the student. Regarding audio clips, the microphone only records when it detects some noise, nothing if the student is in silence. The clips are later analyzed and if voice is detected in the recording it is compared with the data gathered during registration of the student, to validate their identity, or to detect cheating when there are different voices in the recording. 3) Automatic and scalable: All capture, verification, data management and monitoring report modules are carried out with cloud computing technology as services in the cloud. Photos and patterns are taken automatically and randomly and compared with the biometric model made during registration. This scalable automatic setup makes it possible to bring this solution to overcrowded scenarios such as MOOCs. 4) Few requirements for the end user. Cloud-based (SaaS) automatic solution. Needed Hardware - Software (HW/SW): basic webcam, microphone, keyboard and any updated browser. Final users do not have to install anything. This system works over any device, platform, OS and browsers with no installation needed. 5) Automatic analyzed results. 100% guaranteed results with custom alarms. If automatic validation cannot be confirmed (if the pictures or audio clips do not compile with the quality needed to allow the system to automatically validate the student), a manual checking by staff will be set to certify the results 100%. 6) Fully integrated in customer LMS. It can be integrated in any Learning management system (LMS) using a general API but it has a specific plugin for Moodle, Moodlerooms, Blackboard, OpenedX, Canvas, etc. (most used LMS). 7) Secure. Data is transmitted under secure internet protocol and stored in safe cloud servers. 8) Private. The user’s identity remains protected because we only handle data that are not linked to identities but to user codes provided by the online entity. A. DATA CAPTURE AND STORAGE MODULE This module captures data from the student webcam, microphone and keyboard. The core of this application has been developed using the latest HTML5 standard implementation in web browsers. The application is downloaded into the student’s terminal and executed without any installation needed. Whenever the user is connected to the course, quiz or specific exercise into LMS, pictures, audio clips and keystroke dynamics samples will be taken randomly and continuously with predefined mean periodicity. This data is sent VOLUME 9, 2021 72403 M. Labayen et al.: Online Student Authentication and Proctoring System to servers in the cloud, through a SSL encrypted channel, with the user identification code. The system online management module stores and analyzes the images. B. AUTHENTICATION MODULE Once all data is stored in cloud servers, it is compared with the biometrical model, linked to student’s identification code, which has been created at registration time and has been updated with recent positive data. The result is stored in the system database. The system recognition and training algorithms are developed using the latest algorithms in artificial intelligence (explained in Section IV) which are improving constantly their recognition precision and robustness facing light, position and student appearance (physical changes and complements such as hat, glasses...) change problems, noise in audio clips and variability in typing samples. The authentication result is a combination of each biometric authentication module result (face, voice and typing). C. PROCTORING AND COMPUTER LOCK-DOWN MODULES During monitoring sessions, the captured image and audio clips (which have been used for authentication purposes) are processed with different techniques in order to detect inappropriate behaviour of students during e-learning activities. For this reason, the system is able to detect if the student is receiving help (by phone, help from presential friend...) or is checking forbidden documentation (books, other devices connected to the internet...). All these actions can be strictly forbidden in some face-to-face learning activities according to the institution code of honour. In addition, attempts to cheat are detected and reported if any student tries to trick the system, such as mounting a photograph in front of the camera or replacing the image of the ID card with someone else’s. Attempts to insert another image or video signal into the camera are also detected. On the other hand, the system contains a computer lockdown module. During all the online session, a computer lockdown module (Section IV) will monitor the computer of the student detecting connected peripherals, active windows, computer information (HW/SW), executing programs or processes, browsing history/webs and copy-paste commands. All the information captured in each session is stored in the database. D. HUMAN VERIFICATION MODULE As part of the quality warranty, a random data and results auditory must be set. This task will test try the quality assurance mechanism definition and implementation with a huge number of students connected at the same time. It will be based on a random data cross-verification (same images, voice and keystroke patterns validated by different persons) of images, voice and keystroke samples captured during the session with registered data. Besides, when the quality of the photos or audio does not reach the threshold needed, a human verification is made by trained staff delivering a 100% reliable verification of the student. E. REPRESENTATION MODULE OF THE RESULTS Final results are presented by the data representation module. It creates graphic charts and tables on demand, 24h/365d, as a dynamic web page. The final reports can be downloaded or printed in different formats. In addition, the data representation module also generates automated alarms when some predefined prohibited behaviour happens. IV. AUTHENTICATION AND PROCTORING MODULES IMPLEMENTATION As explained in the previous sections, the system presented in this work contains artificial intelligence-based modules for user authentication as well as computer lockdown technologies for device monitoring. In this section the scientific algorithm behind authentication modules and technology and functionalities of the computer monitoring are explained and referenced in depth. A. FACE DETECTION AND RECOGNITION This system includes a facial detection and recognition module through a biometric model created using registration time face pictures. The module output results are clustered in five groups determining: a) If there is someone in front of the webcam or not, b) How many people (if any) are in front of the webcam, c) If one of these people is the person who should be in front of the screen, d) when only one person is in the image, whether this person is the person it should be, e) If the person who it should be is not involved in any inappropriate behaviour (book or electronic device use). Some examples are shown in Figure 4. There are different approaches for face detection in the literature [37]. However, few of them are robust enough when dealing with variation in pose and lighting of captured images (remember that pictures are taken without student attention and randomly). The facial detection procedure presented in this work is based on the FaceBoxes methodology [38]. This methodology is known for being the most common ‘‘Deep Learning’’ based technique whose optimal deployment is based on use of GPUs. This methodology obtains better results in the Face Detection Data Set and Benchmark (FDDB) benchmark (Jain and Learned-Miller, 2010) than other methodologies tested in the development process of this module. The image processing and authentication processes takes [39] as the base reference method for the extraction and normalization of facial texture. This algorithm contains the following subtasks: (1) face detection, (2) face characteristic points detection in the facial region and (3) deformable parametric 3D facial model adjustment based on the detected points. However, the requirement of system passiveness makes it necessary to have continuous improvements in the detection and authentication algorithm to deal with high 72404 VOLUME 9, 2021 M. Labayen et al.: Online Student Authentication and Proctoring System FIGURE 4. Authentication and proctoring system captured and analyzed image examples. variability of input images. Starting in this reference work, a series of improvements have been added: 1) Pose and expressions correction: A new method, called as M3L (Multilevel, Multi-modal, Multi-task Learning) [40], is used to improve efficiency in face points and other facial attributes detection (gestures of the face and eyes). M3L addresses the problem of extracting all these facial and ocular data through a hierarchy of neural networks using existing correlations between the data. Furthermore, a new multi-level deformable 3D facial model adjustment distributes the deformation error in an equitable way, distinguishing three stages with different levels of priority in estimation of (from greater to minor): (1) pose, (2) interpersonal deformations (user-specific facial shape) and (3) intra-personal deformations (deformations due to facial expressions). 2) Aspect normalization, feature selection and classification: The extraction of biometric features through a deep neural network [41] has been improved training a database with 10M of images of 100K individuals with great variability of appearances and facial shapes, lighting, facial expressions, accessories and poses (Guo et al., 2016). 3) Normalization of the lighting: The procedure of normalization of the lighting has been carried out with a hierarchical method in which the facial region as a whole as well as specific and normalized regions of the face are analyzed. This normalization is performed using the Contrast Limited Adaptive Histogram Equalization (CLAHE) algorithm [42], which equalizes the image locally, highlighting the contrasts, applied to each RGB color channel. 4) Robustness against partial occlusions: Occlusions detection is based on the MobileNet-SSD neural network [2], [43]. Combining this person detector and the face detector, the system increases its robustness in detection when (at least) partial face occlusion is occurring. This people detector (body) is more robust than the face detector in these cases. Therefore, if a person is detected, but not a face, it is more likely that this face is at least partially occluded. In this case, the face detection alarm is considered. Additionally, the methodology proposed in [44] has been implemented and adapted to the framework of the needs of the project to handle the occluded normalized facial images. The facial detection returns more partially occluded facial cuts than desirable ones. Normally these occlusions are given either by the user’s hands in front of the face or because the camera is only pointing to the tophalf of the face. This occlusion negatively influences the later stages of facial point detection and biometric vector extraction. This system includes a facial image synthesis from Generative Adversarial Network [45], which fills the occluded part with close facial features obtained from the trained model. In this way, the negative impact of occlusion can be reduced. B. VOICE DETECTION AND RECOGNITION This module implements a continuous voice detection and authentication algorithm. The developments are based on the Kaldi tool [46] and the implementation of the method of [47]. Both include tools for the development of the biometric model, the vector representation of each speaker’s characteristics. The algorithm works on four tasks: 1) Analysis, interpretation and normalization of audio by VoIP: Since the data used in VoIP (technology in which this system is based on) use the G.711 codec with a 64 kbps bit rate, which implies a loss of important information in order to compress the audio signal, all training data from the available acoustic databases are transformed into this encoding and format. In this way, the training and evaluation audio matches were obtained in the different frequency VOLUME 9, 2021 72405 M. Labayen et al.: Online Student Authentication and Proctoring System ranges. Signal pre-processing is integrated to discard that acoustic segments that do not contain speech (silence, music or noise). The final version of the VAD vocal activity detection module has been developed using GMM Gaussian mixture models and processing functions proposed in the Kaldi code tool. A total of 3 model training level were performed. The difference between each of them is based on the transformation of training data for greater robustness versus the high acoustic variability of the application scenario. 2) Background and speaker modelling: The speaker modelling is based on d-vectors or speaker embeddings using deep neural networks. This solution offers better performance in terms of robustness and accuracy. The implementation follows the solution presented by Google in 2018 [47]. In this approach, a recurrent neural network based on LSTM cells is generated. It receives an acoustic characteristic of a specific audio (Mel filter bank) as input and returns its d-vector. Once the training is finished, the neural network can be used to generate d-vectors from the acoustic characteristics of the speaker. Then, a centroid is generated, which is considered as the speaker’s biometric footprint. 3) Patterns comparison: For a verification or identification process, given a vector of acoustic characteristics and its associated d-vector, they are compared with the centroids of each of the speakers in a new similarity matrix. 4) Speaker segmentation on streaming audio: This diarization system employs d-vectors or speaker embeddings and an agglutination model based on recurrent neural networks [38]. This approach aims to overcome the traditional agglutination approach problems, which work with the sentences individually and independently, it being difficult to benefit from the information provided by large amounts of labelled data. This system is based on the work presented by [48]. An independent text announcer recognition network is used to extract d-vectors or speaker embeddings from 240 millisecond windows and a 50% overlap. A vocal activity detector based on Gaussian models is used to eliminate speechless parts and split the signal into segments less than 400 milliseconds. These segments are converted to d-vectors and included in the RNN network based diarization system. C. TYPING RECOGNITION Keystroke dynamics are an effective behavioural biometric, which captures the habitual patterns or rhythms an individual exhibits while typing on a keyboard. According to neurophysiological analysis [49], these typing styles are idiosyncratic, in the same way as handwriting or signatures, due to their similar governing neuronal mechanisms. For this reason, they can be used to authenticate an individual. The system presented in this work applies keystroke dynamics in dynamic text, that is, the analysis occurs for any text that is typed by the user and continuously. Keystroke dynamics in static text requires less effort to be implemented and it also reached lower error rates in the literature [50]. However, a dynamic text analysis [51] is necessary to keep final student passiveness in the authentication process without bothering them by asking them to type a predefined paragraph (usually not related to the e-learning activities in progress). This approach considers the fact that the keystroke dynamics of one person may vary in different psychoemotional states. For example, researches noticed [52] that tired people usually type more slowly and make more mistakes, for this reason, every typed sample is stored to make the recognition model more robust. Two distinctive processes are involved in the keystroke dynamics analysis module: 1) Feature extraction: The extracted features (detailed timing information [53]) are time differences between the instants in which: a) DT: A key is pressed and released. b) PR: A key is pressed and the next key is released. c) FT: A key is released and the next is pressed. d) PP: A key is pressed and the next key is pressed. e) RR: A key is released and the next key is released. Based on different analysis carried out in develop and test cycles, DT (dwell time) and FT (flight time) features are considered the most relevant ones and they are weighted accordingly. In addition, a number of typing mistakes (number of presses of such keys such as ‘‘Delete’’ and ‘‘Backspace’’) are calculated separately as auxiliary parameter. 2) Classification of the extracted features: This module employs the CNN+RNN model [54] to learn a more complete personal keystroke input mode to carry out continuous authentication. The sequence length of 30 keystroke data (best performance) is vectorized and then divided into fixed-length keystroke feature sequences in order to enable keystroke sequences to be input into the RNN networks. The fact that the input data is pre-processed by CNN (extract a higher-level keystroke feature) improves the performance of the network model. D. COMPUTER MONITORING The needs of online proctoring have evolved. In recent times, the market not only seeks to identify students, but also to verify that they are not performing any type of cheating or behaviour that is not allowed with the device on which students perform the activity. In other words, one of the greatest changes is without any doubt the desire to monitor the activity within the device of the students who are doing evaluable activities. The objective of this development is to obtain an application which is able to monitor the activity carried out by the student within their computer. This monitoring will be done only and exclusively when the student is doing an activity that 72406 VOLUME 9, 2021