Biological data tracing and pattern recognition in real-time

Biological Data Tracing and Pattern Recognition in Real-Time W.D. Scherz1, L.M. Soria Morillo2, and R. Seepold1 1 Computer Science Department, University of Applied Sciencies, Konstanz, Germany [email protected] 2 Computer Languages and Systems Dept., University of Seville, Seville, Spain Abstract— Stress is a recognized as a predominant disease with growing costs of treatment. The approach presented here is aimed to detect stress using a light weighted, mobile, cheap and easy to use system. The result sho ws that stress can be detected even in case a person’s natural bio vital data is out of the main range. The system enables storage of measured data, while maintaining communication channels of online and postprocessing. Keywords— ECG, stress detection, e-health, bio signals, ubiquitous computing I. INTRODUCTION Stress is considered as negative sensation and organisations like the World Health Organisation (WHO) distinguish stress as a predominant disease [1] due to its continued presence in modern life. Some well-known consequences of high and perdurable stress are failure to give adequate respond to physical, mental and emotional demands ([2], [3], [4]). Stress also has consequences for a modern society: long-term high stress levels lead to many diseases like burnouts or cardiac infarcts ([5], [6]) The result of high levels of stress in mod-ern society is the fact that the amount of people that will face limitations is increasing and this leads to the growth of the treating and healing costs for people suffering from long-term stress. Assuming that many countries have an aging popula-tion, stress and the consequences of stress will have a nega-tive influence on the health of people while the total amount of budget spent on the consequences of stress will strongly in-crease. Stress may have positive effects, for example, it helps us to handle dangerous situations. Stress helps the body to concentrate forces, for example to be able to run away. Short time stress is a natural response of the body and it is a mech-anism that allows people and animals to react fast and ef-fectively in dangerous situations. Stress releases biological mechanisms, which reorganise body priorities and functions, and it enables us to reach the maximum performance when Fig. 1 Symptoms and physical response of stress there is imminent danger. This is called the ’fight or flight response’ [7]. In contrast to that, nowadays stress is caused by constant exposure to high demands and pressure in daily life, that can be both mental and physical [8], e.g. constant desertions demand or constant time pressure. Some of the symptoms of overabundance of stress are fatigue, sleep problems, etc. [9]. In order to prepare the experiments within this research work, stress need to be provoked and it need to be sure that a person is under stress, There are special methods like Trier test [10] or the Strop test [11]. In Figure 1 some of the mechanism and symptoms of stress are shown. In case of a threat, the body it prepared to flee or to confront the thread. The brain when releasing hormones like cortisol and adrenaline is supporting this preparation process. As a consequence, the functionality of systems that are not necessary for imminent surviving is reduced, like genitourinary system, digestion, hearing, peripheral view, etc. At the same time the functionality of systems that are essentials for a successful flight or fight are increased. E.g. the heart rate is increased what increases the flow of blood in the body and as consequence the muscles get more nutrients and oxygen what enables them to work better. In other words, when we realise that we are in dangerous the brain analyses the situation and decides if we flee or fight. After that the brain releases hormones like cortisol and adrenaline and different systems that are not relevant for fleeing or fighting are shout down and important systems are busted. Being ’under stress’ is often used as a colloquial term. It reflects more a person’s subjective perception of the current situation than an objective evaluation. This subjective interpretation of stress is very it is not noticing it or there is even ’no time’ to think about it. In contrast to that, a different person may feel to be ’under stress’ in case there is no objective indicator but the individual perception. Some people show clear symptoms of stress while others do not. Stress is very often underestimated and it is complex to detect it in an objective way. This may explain why it is one main reason for underestimating the objective stress level and consequences [12]. A desirable case would be only to detecting stress but trying to prevent it. This would require a system to be in place monitoring individual behaviour and evaluating in what moment a person runs the risk to become stressed. In other words, the ultimate goal is not to detect the existence of stress but to avoid the status of being ’under stress’; always assuming the objective stress. A person should receive a warning when being close to the threshold and a person should be supported not passing this threshold. The purpose of this work is to make a first step to detecting emotion patterns with focus into stress. For this, a model to detect a stress pattern has been developed. The approach applied in this work uses the electrical characteristics of the heart (ECG). The system consists of a hardware and a software platform both capable of hosting various algorithms and sensors for bio vital parameter measurement. In order to be flexible for future change, the system provides basic connectivity to a body area network and telemetric support for professional online analysis so that the user is continuously informed about the current status. II. STATE OF THE ART A quite common setup is that the results of stress measurement is not directly reported to a person but the analysis is done offline and without any direct feedback to the person under monitoring. Monitoring of the stress indicators is often used only for capturing the physical characteristic of an indicator but without correlating these parameters to bio vital data tracing like for example the heart rate. Thanks to miniaturization a shift is observed from professional and certified systems into a grey zone of semi-/non-professional reporting and recommendation systems that are not directly involved in professional medical systems. Generally and independently from the area of expertise, form and cost factors the methods can be divided into three approach categories: - Approaches that do not use additional sensors - Approaches that require laboratory equipment and environment - Approaches that require external sensors The first group covers approaches that do not require sensors. These approaches analyse small differences in behaviour that occur between being stressed and not being stressed. Examples for this are ([13], [14]) in which the ways of typing while being stressed were examined and monitored. The disadvantages are the complexity and difficulty in adapting this kind of approaches to different environments. This kind of approaches are not human centred and mostly are context based. The second group covers tests that were realised in controlled laboratory environments. The stress detection provided in these cases is very accurate and precise. Usually hormones like cortisol and adrenaline that are released in saliva and blood are used for the determining of stress [15]. These approaches are limited by the lack of mobility and lack of real time detection. These approaches can be considered as invasive methods, which are expensive due to the necessary laboratory equipment. The third group is characterized by the use of external sensors that are able to capture biological data like in [16], where the stress is measured while driving. In this scenario, a driver is monitored with an electrocardiogram (ECG) and an electromyogram that records the electrical activity of muscles (EMG), a sensor to measure skin conductivity (SC), a breathing sensor and a video camera that observes the driver. The major drawback of this approach is the limitation in the degree of movement and in this case the missing online analysis of the data collected. Furthermore, the driver does not receive immediate feedback. The approach that we have developed uses a self-designed low cost ECG that is compact, wearable, non-invasive and it is also real time capable allowing usage in different contexts. The analysed data can be represented directly to the user via a simple user interface, while raw data is buffered and/or saved locally or remotely for further processing by professionals. The buffering capability feature is important, as mobile solutions do not offer permanent connectivity (e.g. in metro). This paper is also based on our previous studies and models for stress measurement ([2], [4]). III. SYSTEM ARCHITECTURE According to the previous studies mentioned above, we developed a light weighted and low cost system, which is independent from the availability of a smartphone as communication entity or dependent on smartphone sensors. This approach focuses on the ECG signals because we would like to examine how accurate stress can be detected when capturing only one bio vital parameter. Of course it is clear that with the help of more parameters, a more precise detection is enabled but this is beyond the scope of this paper. However, the system is open to capture more parameters and it is able personalized and that means a person may be under stress but Fig. 2 System architecture for collecting, pre-processing and visualizing of biological data to cooperate with other external sensors like smartphone sensors. For this first prototype, we used the smartphone as the communication platform providing connectivity because it is not considered as a violation of the mentioned one-parameter measurement; a mobile communication feature can be easily added at a later stage. The architecture for the system is shown in Figure 2. Three parts form the system: the ECG with its sensors providing the capability of continues recording, the microprocessor for processing the signal and the visualisation device providing feedback to the user (a smartphone or similar device for providing connectivity). The ECG module computes the obtained signal from the electrodes. For this we use the traditional placement of three electrodes. The ECG module generates the signal and outputs it as an analogue voltage (∼0V to 4,9V). This analogue voltage is passed to the microcontroller that provides an analogue to digital converter: The microcontroller not only digitalizes the signal but it performs some pre-processing and filtering. From the obtained data, the heart rate (HR) and the RR interval are calculated. The RR interval is defined as the interval between two R peaks as shown in Figure 3. The data that is from in the RR interval and the heart rate is later used for determining stress. A prototyping board whit a display or a smartphone can be used for interacting whit the user. Using the display (of smartphone or prototyping board) the user can be informed about his current status. As used in previous prototypes, a traffic light interface is shown to the user. This is easy to understand and easy to check: red light indicates high stress and green light low stress. This simple interface can be part of an embedded board or part of an app for the smartphone. Besides the simple interface, all raw data obtained from the sensors are sent to a local buffer for storing the data. This should to be done in the prototyping board because a constant connection to a smartphone cannot be guaranteed. Fig. 3 Definition of an RR interval in a QRS segment IV. METHODS The used method to detect stress is based on ECG signal. As mentioned before there is a variety of bio vital signals that can be captured but in our case the ECG signal is the easiest to capture and has the advantage that it is resistant against many external influences and thus guaranteeing always-available signal. Based on the system architecture, the prototype supports direct access and processing of sensor data in real time. Usually the ECG signal is used for diagnostic purposes [17] but it is also unique enough for identification of persons [18]. As shown in Figure 3, the RR interval equation 1 and the heart rate is used to calculate the heart rate variability (HRV). The HRV is later used for determining stress. RRinterval =Ri−Ri−1(1) We calculate the HRV by examining the relations between two heartbeats that is strongly correlated with the respiration sinus. In [19] the influence of the breathing sinus on the heart rate is described. We can assume that the HRV stays constant when a person is not stressed (constant to the respiration) and when a person is stressed the value changes stronger and stops behaving regularly. The values in Figure 3 are normalised in y-axis over a range between 0 to 350 mV equation 2 and the x-axis shows the time in ms equation 3. 0≤Y≥350 (2) x≥0(3) For optimisation and reducing of the resources needed for the calculation, the detection of the R peaks for the RR intervals equation 4 a threshold of 250 mV was defined. Rj>250mV (4) A second criterion to be fulfilled by the R value is that the R has to be a maxima according to equation 5. Riis a list of RR maxima. (Rj−1<Rj)(Rj>Rj+1)−→ Ri=Rj(5) If both conditions equations (4, 5) are fulfilled, we have successfully detected the R-peak equation 5. Only if we have Fig. 4 Correlation plot of RR intervals detected two consecutive R peaks, we can calculate the RR interval equation 1 by measuring the time difference between two peaks. The next step is to correlate the RR and the HRV. If we visualise the data obtained in a two-dimensional plot, we obtain a correlation plot, e.g. Figure 4. The x-axis and y-axis of the plot are defined following the equations 6 and 7. X=RRi,RRi+1,···,RRn(6) Y=RRi+1,RRi+2,···,RRn+1(7) Figure 4 shows a high concentration in the centre of the plot (800 msec). In case the values do not spread widely, it indicates a lower stress level. If the values are widely spread, it indicates that the person is under stress. The variability can be expressed as a product of the derivations SD1and SD2 defined in equation 8. SD1,2=var(x1)→x1=(xn±xn+1)/√2(8) The current variance of the RR interval is calculated using the standard derivation like formalized in equation 9. σ =(1/n−1 n ∑ i=0 ((xi−x)2))1/2(9) According to the system architecture described in the previous section the prototype has been implemented according to Figure 2. The system has been realised on the top of an Arduino Uno R31prototyping board with a self-made ECG component stacked on the top of the board. The architecture and design of the ECG module will be discussed in another paper. The ECG electrodes are connected to the board and an additional board has been added to provide a slot for a micro SD memory card, for storing and buffering proposes. The memory card may be integrated in future directly into the 1http://arduino.cc Fig. 5 Left (yellow) with high HR and no stress, right (blue) with low HR but stress ECG board and that will be part of a redesign. For the measurement experiment series, volunteer candidates have been selected, which are students aged between 23 and 28 years. None of them where smokers or alcoholics. We also assume that none of the candidates suffered from cardiac problems or mental anomalies nor used a pacemaker. For inducing stress, the Trier Social Stress Test [10] method was used because this method can be easily established in our laboratory environment. As second method used for inducing stress was a driving simulator. The Trier Social Stress Test was divided in three segments: anticipation period, presentation period and cool down period. Each had a duration of 5 minutes. As an example, during the test a volunteer had to prepare and make a small presentation on a random topic. With the help of the driving simulator two mechanisms were used to induce stress: A ’point system’ designed as a reward system in which the candidate has to perform fast and complex driving manoeuvres. The level of difficulty increases over time, so that the volunteer has to drive faster and perform more complex manoeuvres to order to receive more points. While the complexity and speed increases, also the risk increase because in case of a failure (e.g. accident) the driver will lose all earned points. The data considered in this paper have been derived from these two experiences. In the following, the results of two volunteers with special characteristics are discussed. The first one has per se a lower heart rate but is being under constant stress. The second volunteer has higher heart rate but is not under stress. Both volunteers where analysed for the same duration of time. The data of both volunteers is visualised in Figure 5. The right dataset clearly shows that the values in blue are wider spread than the values of the volunteer visualised in yellow (left). The spreading of the blue values is caused by the stress (see Figure 1). Stress influences the heart rate and as a consequence, this results in a variation between two heart beats i.e. the RR interval becomes b igger. The mean values of the blue volunteer are between 0.7 and 0.9 sec for the RR interval (heart rate interval is between 66.7 bpm and 85.7 bpm). The main values of the yellow volunteer are between 0.45 sec and 0.6 sec (130 bpm and 85 bpm). This means that the yellow volunteer has a higher heart rate but he is less stressed than the blue one. V. DISCUSSION Some of the weakness of the developed system is the energy sources sensibility that can induce interference and artifacts in the signal, also the sensibility of strong movements for example during sports can generate artifacts in the signal. These problems can possibly be solved by adding more filtering (software and hardware filtering) or by replacing the ECG module for another sensor that does not have these weaknesses. An alternative sensor for the ECG approach like for example a pulse oximeter is currently under evaluation. A second point that is important to mention is that possibility to verify the results and identify false positives. VI. CONCLUSIONS AND FUTURE WORKS This work describes a system architecture and an algorithm based on the HRV to recognize stress. The system is light weighted, mobile, cheap and easy to use. It can be integrated into a mobile embedded device (like sketched out in the prototype) or in a smartphone (app). Due to the embedded storing capacity, the system can also work stand-alone. The current version of the prototype can be connected to systems like PC and smartphone. The implementation of all hardware and software components was developed under the premises to have a small footprint and support easy porting to different small platforms like e.g. an Intel Edison2.The results of the experiment show that stress can be detected even in cases when people have naturally low or high heart rate. Future challenges of the system have to take into account the influence on the signal caused by abrupt and strong movements like in the case of sport activities. Furthermore, the movement of the electrodes during the measurement can generate interferences and artifacts in the signal. A possible solution for this problem could be software filtering and reconstruction of the signal or exchanging the current measurement technique of a sensor without generating interferences. Next steps planned for this work are to replace the micro controller, to use a more powerful micro controller board in order to shift the processing functionality to the place where 2https://software.intel.com/en-us/iot/hardware/edison the data is captured, add more filtering fluxionality, and finally, to overcome lacks in sensor signals during sport activities. Also a redesign of the ECG module must be considered to tune it for low-power and being more compact to be wearable. Another important task is the evaluation and collection of data in a longer time interval. Currently the examination interval is 15 minutes long and should compare between stress and no stress. This has to be expanded to 24 hours and detect the main spots of stress while the day. Although in this paper a preliminary work has been presented, a comparison with other devices is currently under evaluation. Among other commercial wearables, our approach is being faced with Emvio, Olive, Being and Embrace. Results of this testing will be published as soon as the results are obtained. DECLARATION OF CONFLICTS The authors declare that they have no conflict of interest. REFERENCES 1. Andrade Laura, Caraveo-Anduaga JJ, Berglund Patricia, et al. Crossnational comparisons of the prevalences and correlates of mental disorders 2013. 2. Fernandez Jose M, Augusto Juan Carlos, Seepold Ralf, Madrid Natividad Mart´ınez. A Sensor Technology Survey for a Stress-Aware Trading Process Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on. 2012;42:809–824. 3. Fern´andez Javier Mart´ınez, Augusto Juan Carlos, Seepold Ralf, Madrid Natividad Mart´ınez. Why Traders Need Ambient Intelligence in Ambient Intelligence and Future Trends-International Symposium on Ambient Intelligence (ISAmI 2010):229–236Springer 2010. 4. Fern´andez Javier Mart´ınez, Augusto Juan Carlos, Trombino Giuseppe, Seepold Ralf, Madrid Natividad Martinez. Self-Aware Trader: A New Approach to Safer Trading. J. UCS. 2013;19:2292–2319. 5. Orth-Gomer Kristina, Wamala Sarah P, Horsten Myriam, Schenck- Gustafsson Karin, Schneiderman Neil, Mittleman Murray A. Marital stress worsens prognosis in women with coronary heart disease: The Stockholm Female Coronary Risk Study Jama. 2000;284:3008–3014. 6. Mei Yu, Thompson Melissa D, Cohen Richard A, Tong XiaoYong. Autophagy and oxidative stress in cardiovascular diseases Biochimica et Biophysica Acta (BBA)-Molecular Basis of Disease. 2015;1852:243– 251. 7. Jansen Arthur SP, Van Nguyen Xay, Karpitskiy Vladimir, Mettenleiter Thomas C, Loewy Arthur D. Central command neurons of the sympathetic nervous system: basis of the fight-or-flight response Science. 1995;270:644–646. 8. Kidd Tara, Carvalho Livia A, Steptoe Andrew. The relationship between cortisol responses to laboratory stress and cortisol profiles in daily life Biological psychology. 2014;99:34–40. 9. ˚ Akerstedt Torbj¨orn, Axelsson John, Lekander Mats, Orsini Nicola, Kecklund G¨oran. Do sleep, stress, and illness explain daily variations in fatigue? A prospective study Journal of psychosomatic research. 2014;76:280–285. 10. Kirschbaum Clemens, Pirke K-M, Hellhammer Dirk H. The Trier Social Stress Test–a tool for investigating psychobiological stress responses in a laboratory setting Neuropsychobiology. 1993;28:76–81. 11. Stroop J Ridley. Studies of interference in serial verbal reactions. Journal of experimental psychology. 1935;18:643. 12. Madrid N Mart´ınez, Fern´andez J Mart´ınez, Seepold R, Augusto JC. Sensors for Ambient Assisted Living (AAL) and Smart Homes 13. Gunawardhane Suranga DW, De Silva Pasan M, Kulathunga Dayan SB, Arunatileka Shiromi MKD. Non invasive human stress detection using key stroke dynamics and pattern variations in Advances in ICT for Emerging Regions (ICTer), 2013 International Conference on:240– 247IEEE 2013. 14. Vizer Lisa M, Zhou Lina, Sears Andrew. Automated stress detection using keystroke and linguistic features: An exploratory study International Journal of Human-Computer Studies. 2009;67:870–886. 15. Hellhammer Juliane, Schubert Melanie. The physiological response to Trier Social Stress Test relates to subjective measures of stress during but not before or after the test Psychoneuroendocrinology. 2012;37:119–124. 16. Healey Jennifer, Picard Rosalind W, others . Detecting stress during real-world driving tasks using physiological sensors Intelligent Transportation Systems, IEEE Transactions on. 2005;6:156–166. 17. Dubin Dale. Rapid Interpretation of EKG’s. USA: Cover Publishing Company, 1996 1996. 18. Israel Steven A, Irvine John M, Cheng Andrew, Wiederhold Mark D, Wiederhold Brenda K. ECG to identify individuals Pattern recognition. 2005;38:133–142. 19. Hirsch Judith Ann, Bishop Beverly. Respiratory sinus arrhythmia in humans: how breathing pattern modulates heart rate American Journal of Physiology-Heart and Circulatory Physiology. 1981;241:H620–H629.