Control Mouse Cursor by Head Movement: Development and Implementation

Applied Medical Informatics Original Research Vol. 40, No. 3-4 /2018, pp: 39-44 39 [ Control Mouse Cursor by Head Movement: Development and Implementation Hussein Ali ALHAMZAWI* Faculty of Informatics, University of Debrecen, Kassai Street, no. 26, 4028 Debrecen, Hungary. E-mail: [email protected]; * Corresponding author: Tel.: +36304390299 Received: February 7, 2018 /Accepted: August 29, 2018/ Published online: September 5, 2018 Abstract As there is an excellent advancement in the technology in recent years, there has been much improvement in various fields of computing such as Human Computer Interface (HCI), Computer Vision and Perceptual User Interface (PUI). The input to the computers has sensed information about the physical properties of the user, places, or things. For example, a computer mouse operates by motion imparted by the user’s hand. Many active researchers have been working on finding an alternative solution to control computer mouse like by using a finger, eye, hand/palm, marked gloves, etc. These all techniques may not be suitable to physically disabled people. The system of control the mouse cursor by the movement of the head of a person who is positioned in front of the monitor of a microcomputer is presented in this paper. At the top of a monitor is a color video camera with the standard National Television System Committee (NTSC) output that captures user images. These images are processed using digital image processing techniques, and the final objective is to proportionally relate the user's head movements (identified by a blue circle painted on a tassel) with the mouse cursor. The captured of the colored images are 320 pixels by 240 wide by high. Preliminary results have shown efficiency in controlling the position of the cursor on the computer screen. Keywords: Quadriplegic; Segmentation; Binarization; Digital Image Processing (PDI) techniques Introduction An estimated two million people living with spinal cord injuries (SCI) and every 49 minutes another suffers new injury [1]. Of this total, 48.7% are the quadriplegic. To facilitate the accomplishment of their daily activities and to provide a more productive and happy life, at home, at school or work, it is urgent to create new utensils, devices, and equipment for these users. Moreover, today the computer a means of leisure, study, work and communication, new tools have been created to improve their accessibility [2] and make it more efficient in the new tasks that arise. Researchers have tried in the field of robotics and human-computer interaction to control mouse movements using video devices. Almost all researchers use different methods to make the event clickable. Erdem et al. [3] use finger tracking to control mouse movement. The mouse button over the implementation of the screen that has been defined so that a click event when the user's hands passed through the area. Zhou, Hailing, Lijun Xie, and Xuliang Fang [4] used fingertips to control the mouse pointer and click. This method depends on the intensity of the image and is required to hold the mouse pointer on the desired point for a short period. Lien, Chu-Feng, Used another method of clicking. He used the motion of the thumb (from thumbs-up position to fist) to mark a clicking event thumb and the movement of the hand to make a particular hand sign move to move mouse pointer [5]. In this work, user header movements are set in the mouse input. The webcam is set to take pictures continuously. In the user's head, there must be a blue colored segment so it would Hussein A. ALHAMZAWI 40 Appl Med Inform 40(3-4) September-December/2018 be visible for the camera when taking a picture . This color of the pixel image is detected, and the pixel position is set in the mouse input. Depending on the size of the image taken by the camera the pixel position in the image will not have a correspondence with screen resolution. Therefore various scaling techniques are used. HeadMouse® [6], from Origin Instruments Corp., has as principal component a 94×56×13 mm USB device which contains an optical sensor and Infrared LEPs. Some Disposable reflect will be attached to the head of the user; an infrared optical sensor will be detected this marks as a dots. A mouse pointer movement on the computer is coming from the proportionate translation of the head movements. On the top of the computer screen, the device can be installed. Where a “perfect rule thumb is to place the HeadMouse so that the eye level is near, and look directly at the point. From this, a maximum range of motion will provide for the user. From the Origin Instruments, the following extensions can be performed using Mouse click: • On a headset, a sip/puff switch is mounted • Dwell clicking with the software Dragger By using third-party products (e.g., switches), a mouse clicks can also be performed. The windows software Dragger 95$, the sip/puff switch costs 295$ and the main component, the Head Mouse Extreme costs 995$. The disadvantage of the HeadMouse are the need of extra equipment and the prices of this extra-equipments that are expensive. Smart-Nav®, of NaturalPoint Inc., uses LEDs positioned on the front of the user; the light emitted is acquired by a camera that is on the monitor, and an algorithm calculates the position of the cursor [7]. The disadvantage of this method is:  Erratic tracking, cursor "jumps" around the screen.  In some case Software loads and runs and green indicator light blinks, but the cursor does not move.  In some cases, Unable to proceed past driver signing during hardware install.  In some case, All or some of the Hot Keys will not work. With the impossibility of using the upper limbs, quadriplegic people are deprived of the use of personal computers. The primary objective of this project was to create a control system for mouse cursor by moving the head of a user positioned in front of a computer to assist these patients in using computers and help him/her in his/her routine life. Methodology A control system for mouse cursor has been designed by moving the head of a user positioned in front of a computer running Windows 95 (or higher) operating system. On the monitor, a color video camera (standard NTSC) is placed that captures images of the head of the user and, using digital image processing (DIP) techniques, converts them into information necessary for the mouse cursor to move in the requested direction. In the developed design, a sports-type tape is placed on the user's head (Headband). In the center of this tape, there is a blue painted circle, which is the reference point that the camcorder should look for when capturing an image. This search for the centroid is performed using PDI techniques. Once the coordinate is located, the mouse cursor is moved to its new position, which is proportional to the centroid displacement of the circle in the captured image. Figure 1 illustrates the implemented system. Any movement of the head, horizontal or vertical, moves the circle fixed on the tester. As the camcorder is receiving images from the user, this shift will be reflected in the images, causing the centroid of the circle to change coordinates. This change will proportionately affect the position of the mouse cursor on the computer screen. The simple mouse click is emulated when the user sits at least n seconds without moving the head and double-clicking when n + 1 seconds is left. The developed application was assembled in C++ programming language. A color video camera WAT 202B from Watec Inc. was used. Moreover, a video signal capture board, CPH050 frame grabber from Tview Inc., running on an IBM / PC with Pentium IV processor, 1,700 GHz, and 256 Mbytes of RAM. The captured images were 320 pixels by 240 (wide by height), mounted on the RGB color model. Control Mouse Cursor by Head Movement: Development and Implementation [ Appl Med Inform 40(3-4) September-December/2018 41 Figure 1. The system to control the mouse by movement of the head. The blue circle placed on the user is the reference of the images captured by the video camera on the monitor. DIP techniques determine its centroid in the image proportionally moves the mouse cursor on the screen to the desired position. The algorithm depicted in Figure (2) was implemented. Figure 2. Flow chart of the system. For the detection of head movements of the user, a webcam was used that will work as a sensor. The head movements of the user are captured at a fixed frame ratio and resolution which is obtained by the hardware of the camera. By changing the HSV value the frame ratio and resolution can be changed if required. These operations are identified on the captured images with the Open CV. The captured video is divided into individual image frames based on the frames per second of the camera, and individual image frames are processed. Colored images are formed by the combination of n information channels (n=3) [8]. The scanned images always the RGB, where R represents the red channel, G the green channel, and B the blue channel. However, the RGB type is not best suited for segmentation and for this reason, conversion from RGB to HSI is performed, where H represents the hue of the color, S is saturation, and I is intensity (brightness). The pixel matrix value is given in degrees, where 0° means red, 60° yellow, 120° green, and so on. To find regions of the image with pixels of the blue color, we only need to look for hues that have values close to 240º. A new image is then generated, where blue pixels are marked as white and the others as black pixels. From a binarized image where white pixels are treated as objects and black pixels as background, the technique proposed by Hough [9] and Parker [10], now called the Hough Transform, is recognized as being one of the most effective in locating existing geometric shapes in images [11]. Circular geometric shapes can be identified by Equation (1). Acquiring the real-time video from the webcam convert video in to the image Blue color segmentation Location of the circle and mouse cursors movement and event Hussein A. ALHAMZAWI 42 Appl Med Inform 40(3-4) September-December/2018 𝑝 = 𝑘√k2− c , for k2− c ≥ 0 (1) where p = distance, θ = the polar coordinate of the centroid of the circle relative to the position (0, 0) of the image, upper left corner,k = xcos(θ) + ysin(θ) and c = x2+y2-r2, r = the radius of the circle. The (x,y) coordinate refers to any white pixels in the image. The variable r represents the radius of the circle to be located which must vary in a predetermined interval, according to the dimensions of the image. The distance p and the angle θ represent the polar coordinate of the centroid of the circle relative to the position (0, 0) of the image, upper left corner. The angle should also vary within a specific range. The result of this transform is a 3D array, where one dimension is represented by r, another by θ and the last by p. In this array, the coordinate elements (r, θ, p) that have higher values are identified as circles of radius r, and the polar coordinate of the centroid (θ, p). Processing is terminated when converting the coordinate of the polar representation to rectangular [12,13]. The circle localization algorithm by the Hough Transform had the following input parameters: 10 ≤ r ≤ 25 pixels and 15 ≤ θ ≤ 75 degrees. The developed system was tested only in the laboratory on people without any motor problems and respectively with spinal cord injury. Results and Discussion The reflection of the head movement on the mouse cursor on the screen is depicted in Figure 3. For the click of the mouse, n = 2 s was adopted. The processing time of all system steps was less than 67ms, with a frame capture rate of 15/s. Figure 3. The reflection of the head movement on the mouse cursor on the screen. A dx difference in the centroid is related to aαdx offset at the cursor, and a difference dy causes a βdy offset. The constants α and β determine the sensitivity of the system. The right and left images are reversed because the camcorder captures mirror images. The proposed system captures the image through a webcam so, if the web camera cannot capture the image accurately, then the operation cannot perform. In some cases due to the presence of other colors in the background, an error can occur affecting the threshold values and other parameters of the system. Furthermore, the system might run slow due to the high resolution of the camera [14]. The user should always sit nearer to the desktop, because of external hardware mouse and resolution of the camera. If the mouse is damage in any situation, then cost will increase. The results of the evaluations performed are presented in Table 1 and 2. The tests were conducted on three videos from tow volunteers the first one was healthy, the second was spinal cord. Although the accuracy of the system is outstanding in controlled environments, we find many drawbacks and limitations when user behavior and the environment is unpredictable:  High error ratio on fast motion animation. Control Mouse Cursor by Head Movement: Development and Implementation [ Appl Med Inform 40(3-4) September-December/2018 43  If environmental lights are changing, false segmentation of blue color is resulting in poor fatality classification for click events. Table 1. Accuracy of blue circle detection # Total frames # Detected frames Accuracy (%) Video1 892 801 88.72 Video2 750 698 91.23 Video3 960 932 96.22 Average Accuracy (%) 95 Table 2. Accuracy for click event detection #Total frames #Click event frames #Detected frames Accuracy (%) Video1 889 62 51 80.36 Video2 755 44 40 87.13 Video3 953 92 81 87.1 Average Accuracy (%) 84.86 The main advantages of the proposed system are as follow:  Help people who have damage in their arms to use the computer.  The Virtual Keyboard is not restricted to the QWERTY touch-typing paradigm; adjustments can be made to the software to fit other touch-typing paradigms as well.  No drivers are necessary; it can use a Plug and Play device.  Portability.  Cost-effectiveness. Harada et al. [14], and respectively Sporka et al. [16] reported the used of voice control of mouse pointer but their approaches are valid just when the used have voice. Our proposed method addressed to those subjects who did not have any voice control. Functionalities of the presented control mouse cursor system are to detect and track the object, then get the position of the head and convert this head’s movement to cursor coordinates. Detection and tracking the blue circle in headband using the low-resolution camera with the cheap equipment and easy algorithm to get right and almost precisely result in that help the people that have unable to move them upper limb to using the computer as the healthy people is presented in this paper. We introduced a new method in our paper for controlling a mouse cursor by using a blue segmentation placed on the user’s forehead. Two methods of cursor control, velocity-control, and position-control. We also compared along with desktop Mouse. The mean throughputs for the desktop mouse cursor were 4.42 bps, while throughputs for the velocity control modes and position were 1.07 bps and 1.61 bps respectively. Noise in tick disclosure was a determination for the position-control process. However, in order to speed control is less conjectural, performance mode control speed is worse than the position control process. Improved movement for the position-control would be the best way to deal with noise detection mark, to improved accuracy and decreasing the jitter. There is to approach to deal with this problem: For a more effective way to detect signs and determine the direction of the mark, and Get a better smoothing algorithm that reduces the noise after the orientation has been calculated. Conflict of Interest The author declares that he have no conflict of interest. Hussein A. ALHAMZAWI 44 Appl Med Inform 40(3-4) September-December/2018 References 1. NSCIA. About Spinal Cord Injury. 2003. [online]. Available from: www.spinalcord.org (Accessed 12/10/2017). 2. The Freedom to Connect! © 2012, North Dakota Assistive. [online] Available from: http://ndipat.org/computer-access (Accessed 14/08/2018) 3. Erdem A, Erdem E, Yardimci E, Atalay Y, Cetin AE. Computer vision based mouse. 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing, Orlando, FL, 2002, pp. IV-4178IV-4178. doi: 10.1109/ICASSP.2002.5745637. 4. Zhou H, Lijun X, Xuliang F. Visual mouse: Sift detection and PCA recognition. 2007 International Conference on Computational Intelligence and Security Workshops (CISW 2007), Heilongjiang, 2007, pp. 263-266. doi: 10.1109/CISW.2007.4425487. 5. Lien C-F. Portable Vision-Based HCI–A Realtime Hand Mouse System on Handheld Devices. [online] Available from: https://www.csie.ntu.edu.tw/~p93007/projects/vision/vision_hci_p93922007.pdf (Accessed 12/10/2017) 6. HeadMouse® Nano Wireless Head Controlled Mouse [online]. Available from: http://www.orin.com/access/headmouse/ (Accessed 14/08/2018) 7. Smart-Nav® [online]. Available from: https://www.naturalpoint.com/smartnav/ (Accessed 14/08/2018) 8. Gonzalez RC, Woods RE. Digital Image Processing. Boston, Addison-Wesley Publishing, 1993. 9. Hough PVC. Method and means for recognizing complex patterns.U.S. Patent 3,069,654, Dec. 18, 1962. 10. Parker JR. Algorithms for Image Processing and Computer Vision. New Jersey, Wiley Computer Publishing, 1997. 11. Bradski G, Kaehler A. Learning OpenCV: Computer vision with the OpenCV library. O'Reilly Media, Inc., USA, 2008. 12. Bloch I, Henri M. Fuzzy mathematical morphologies: a comparative study. Pattern Recognition 1995;28(9):1341-1387. 13. Homma K, Takenaka E. An image processing method for feature extraction of space-occupying lesions. J Nucl Med. 1985;26(12):1472-1477. 14. Betke M, Gips J, Fleming P. The camera mouse: Visual tracking of body features to provide computer access for people with severe disabilities. EEE Transactions on Neural Systems and Rehabilitation Engineering 2002;10(1):1-10. doi: 10.1109/TNSRE.2002.1021581 15. Harada S, Landay JA, Malkin J, Li X, Bilmes JA. The vocal joystick:: evaluation of voice-based cursor control techniques. Proceedings of the 8th international ACM SIGACCESS conference on Computers and accessibility, 2006, pp. 197-204. 16. Sporka AJ, Kurniawan SH, Slavík P. Acoustic control of mouse pointer. Universal Access in the Information Society 2006;4(3):237-245.