Fire detection with a frame-less vision sensor working in the NIR band

VII International Conference on Forest Fire Research D. X. Viegas (Ed.), 2014 1 Fire detection with a frame-less vision sensor working in the NIR band Juan A. Leñero-Bardalloa, Jorge Fernández-Bernia , Ricardo Carmona-Galána, Philipp Häfligerb, Ángel Rodríguez-Vázqueza aInstitute of Microelectronics of Seville (CSIC – Universidad de Sevilla), Avda. Américo Vespucio s/n Seville (Spain), Contact: juanle@imse‐cnm.csic.es bDepartment of Informatics, University of Oslo, Oslo, Norway, PO Box 1080, Blindern, Contact: [email protected]io.no Keywords: Flame monitoring, event-based vision sensors, fire detection, NIR, image processing. 1. Abstract This paper draws the attention of the community about the capabilities of an emerging generation of bio-inspired vision sensors to be used in fire detection systems. Their principle of operation will be described. Moreover experimental results showing the performance of an eventbased vision sensor will be provided. The sensor was intended to monitor flames activity without using optic filters. In this article, we will also extend this preliminary work and explore how its outputs can be processed to detect fire in the environment. 2. Introduction Infrared cameras can easily detect fire and hot spots. Unfortunately, they are expensive, difficult to handle, and fragile (Briz et al. 2003). For this reason, CMOS cameras are still preferred for some applications where it is not strictly necessary operating in the whole infrared band, (Cheon et al. 2009, Naoult et al. 2007, Bendiscio et al. 1998, Fernandez-Berni et al. 2012). Silicon can detect Near Infrared Radiation (NIR) within the band [700 1,100] nm. This property can be exploited to monitor flame and fire activity. Flames have a very characteristic oscillatory behavior. They flicker with certain frequency components that depend on their nature. The study of these frequency components is interesting for industrial processes and applications. Traditionally CMOS cameras with NIR filters have been employed (Yan et al. 2006). As an alternative to this traditional approach, recently we proposed an bio-inspired event-based system to monitor flame activity without using optic filters (Leñero- Bardallo et al. 2013). The idea behind this method was to compare the photocurrents of stacked photodiodes at different depths. The top one was more sensitive to shorter wavelengths and the bottom one was more sensitive to longer wavelengths, having a very good sensitivity within the NIR band. Just comparing the difference between their photocurrents, we were able to determine if the incident radiation was within the NIR band without optic filters. Moreover, we could process with a real-time algorithm the temporal variations of the NIR levels in the visual scene provoked by flames and compute their frequency components values. This paper studies how the prior work to monitor flame activity could be extended to detect fire in the environment and emit an alarm in case. We will show that the previously implemented algorithm outputs can be processed to detect fire with a low computational load. Up VII International Conference on Forest Fire Research D. X. Viegas (Ed.), 2014 2 to date, all the forest fire detection systems with cameras use frame-based vision sensors. With this contribution, we also want to draw the attention of the community about this new generation of bio-inspired vision sensors. This document is organized as follows: In Section 3, we explain briefly the fundamentals of bioinspired vision sensors and we discuss their feasibility for fire detection; Section 4 explains what kind of radiation can be sensed with CMOS sensors; Section 5 describes our sensor and algorithm to process its outputs and detect radiation within the NIR band; Section 6 provides experimental results when our system is employed to monitor flame activity; Section 7 explores how the NIR separation algorithm can be modified to emit an alarm if there is fire in the environment; finally, Section 8 draws some conclusions. 3. Bio-inspired Event-based Vision Sensors Conventional vision sensors are frame-based systems (also known as imagers). A frame is a 2D-dimensional matrix that contains information about the visual scene (typically light intensity or color). Such sensors always provide a continuous output data flow (frames are transmitted with a continuous periodicity) that can be very redundant if the visual scene does not change. If we compare their performance with the human retina, we can state that they perform worse in the majority of uncontrolled situations and environments: their dynamic range is much lower, their sensitivity to light is also lower, and their power consumption is much higher. For these reasons, some authors considered the idea of developing bio-inspired vision sensors (also called retinas, (Mahowald 1994, Leñero-Bardallo et al. 2010, Leñero-Bardallo et al. 2011). These sensors try to mimic the interactions of the cells of the human retina. The retina operates in a very different way than conventional vision sensors. Its cells process the visual information before transmitting it to the brain. Thus, redundant information is not sent to the optical nerve. Retinal cells can compute the spatio-temporal contrast that contains almost all the relevant information about the visual scene. This information is transmitted continuously. There is a massive parallel architecture and cells can send anytime spikes to the brain. Therefore biological systems are inherently faster than classic imagers which bandwidth is limited by the frame rate. The main drawback of artificial event-based sensors is their low pixel fill factor. Their pixels have more transistors than classic frame-based pixels with three or four transistors. Extra circuitry per pixel is required to process the visual information and to implement and asynchronous pixel communication (Häfliger 2000). However, such sensors are good candidates for applications like fire detection or surveillance, where high image quality is not mandatory, and speed and low power consumption are preferred. They have a great potential to be nodes of wireless sensor networks. In such networks it is mandatory to minimize the amount of data transmitted from the sensors to a central node which processes the information. 4. Fire Detection in the NIR Band A CMOS camera without filters detects radiation beyond the visual spectrum, i.e. Ultra Violet ([250, 390] nm), visible ([390, 750] nm), and NIR ([750, 1100] nm). Hot spots, fire or flames can be detected with a CMOS sensor with a NIR filter (Naoult et al. 2007)0, 0(Li et al. 2010) (hot spots need to be above 350Cº). Smoke detection is possible too when employing NIR sources (e.g. NIR LEDs). In our particular case, we developed a system (Leñero-Bardallo et al. 2013) with an event-based sensor to monitor the temporal variations of the visual scene provoked by flames. Flames have VII International Conference on Forest Fire Research D. X. Viegas (Ed.), 2014 3 frequency components within the interval [0, 250] Hz and their flickering can easily be tracked with an asynchronous sensor. As a continuation of this preliminary work, we propose to develop new real-time algorithms, (Leñero-Bardallo et al. Flame-2013), to detect fire in indoors and outdoors environments. According to our experimental results, flames provoke very fast and periodic variations of NIR levels in the visual scene that can be easy to identify from other NIR sources like sunlight. Such periodic variations of the NIR levels are easy to detect when they appear, just computing the standard deviation of previously recorded NIR levels and normalizing it. 5. Methodology We employ an event-based vision sensor to detect the NIR variations of the visual scene (Leñero-Bardallo et al. Fire 2013). The sensors pixels are made up of three stacked photodiodes at different depths. Each one is connected to a circuitry that sends pulses out the chip, with a frequency proportional to its local photocurrent (see Figure 1). The top one is more sensitive to shorter wavelengths. The bottom one is more sensitive to longer wavelengths and to the radiation within the NIR band. This property is based on the light penetration depth. Shorter wavelengths photons have more energy and on average, they travel less time before they generate electronhole pairs. Longer wavelengths have less energy and they travel more time across the silicon before generating electron-hole pairs. Figure 1 –Experimental setup: vision sensor and data logger. Photodiodes cross-section. Each one is connected to circuitry (channel) that generates pulses with a frequency proportional to its inputs photocurrents. VII International Conference on Forest Fire Research D. X. Viegas (Ed.), 2014 4 Figure 2 – Integrate-and-fire neuron. This circuitry is replicated three times within each pixel. Depending on the input is connected to the photodiode cathode/anode, we use this circuit or a complementary version of it, with p-MOS transistors instead n-MOS transistors and viceversa. Note that the photocurrents at the inputs of the integrate-and-fire neurons do not correspond with each photodiode current. For this reason it is necessary to process the events outputs to estimate the photocurrent of each photodiode. Unfortunately, we do not know the photodiodes quantum efficiencies, their doping profiles, and the PN junction depths (such information was not disclosed by the foundry). For this reason, it is impossible to know the exact photocurrent values. We denote the photocurrent estimations after processing the integrate-and- fire outputs as '},,{ RGGBB I . However, to represent the color information of each pixel, we only need to know the relative values of each photocurrent. For this reason, it is possible to represent RGB images after processing the channels outputs. Every time that a pixel integrate-and-fire neuron spikes, its frequency * f is updated. Periodically, the estimation of the photocurrents is computed according to the following equations: B BthBB Be VCf I_ '  (1) G BBGBthGBGB Ge eIVCf I ' _ '  (2) G BGGBthRGRG Re eIVCf I ' _ '  (3) Where },,{ RGGBB C are the photodiodes capacitances, },,_{ RGGBBth V are the voltage increments at the integrate-and-fire neurons capacitances to elicit one spike, and }.,{ RGGBB e are parameters that represent the photodiodes quantum efficiencies. VII International Conference on Forest Fire Research D. X. Viegas (Ed.), 2014 5 Figure 3 –Experimental setup: vision sensor and data logger. 6. Experimental Results Figure 3 shows the experimental setup. We designed a custom board for the vision sensor. That board is connected to a data logger that sends the output data to a PC. We programmed a dedicated Java interface 0 (jAER open source) to compute the values of '},,{ RGGBB I and show real-time color images. Main system specifications are summarized on Table I. Table I –System Specifications Array Size 22x22 Technology STM 90nm Power Supply 2.5V Chip Size 1mm x 1mm Pixel Size m x m Fill Factor 28% Power Consumption 0.03mA @ 10Keps Dynamic Range >60dB 6.1. Detection of NIR levels A real-time NIR extraction algorithm was proposed, (Leñero-Bardallo et al. Flame 2013). It operates with photocurrents estimations ' G I and ' R I . The NIR levels are proportional to the photocurrents ratio after a certain number of events are received: ' ' ' R G RI I I NIR  (4) VII International Conference on Forest Fire Research D. X. Viegas (Ed.), 2014 6 Its performance is depicted in Figure 4. Within the NIR band, the value of ' R I is larger than ' G I . To take into account the light intensity, we multiply ' ' G R I I by ' R I . Once we have estimated the NIR levels of each pixel, we can use this information to either create intensity images proportional to the NIR levels that each pixel detects or to estimate global NIR levels variations of the visual scene. This can be done computing the FFT of the global NIR levels within a time interval. The pixel updating frequency is in the order of KHz and the main NIR frequency components of a flame are below 100Hz. Hence, the system is fast enough to track flames flickering. Figure 4 –Algorithm response on infrared detection mode. The sensor was stimulated with a monochromator emitting light at different wavelengths. Figure 5 –System Diagram for fire detection. Figure 5 shows a diagram of the experimental setup operating with flames. A gas lighter emitting flames was placed in front of the sensor at a distance of 0.5m. Figure 6 displays the temporal variations of the NIR levels of the visual scene with and without flames. With flames the NIR levels change with a very characteristic periodicity due to the flames flickering. Without flames, the NIR levels are much more stable. Figure 7 shows the spectra of temporal NIR variations with flames in indoors and outdoors environments. Results are quite similar to the ones reported by other authors with other methods with optic filters (Yan et al. 2006). Sunlight VII International Conference on Forest Fire Research D. X. Viegas (Ed.), 2014 7 provokes an increase of the average NIR level of the scene and makes more challenging the frequency analysis. A custom Java interface (jAER open source) was programmed to show real-time NIR images as it is depicted in Figure 8. The interface can display images and implements the NIR algorithm previously explained. Figure 9 shows a snapshot of the NIR levels emitted by a flame. The different regions of the flame can be distinguished. The root and the central region of the flame have less energy than the periphery where the flame emits more energy in the NIR band. Figure 6 – Real-time temporal variations of the NIR levels of the visual scene in different environments with and without flames. Figure 7 – Frequency components emitted by a flame in indoors and outdoors environments. Outdoors the average NIR level is higher due to sunlight. VII International Conference on Forest Fire Research D. X. Viegas (Ed.), 2014 8 Figure 8 – Computer interface showing real-time NIR levels emitted by one flame. Figure 9 – Snapshot showing NIR levels when the sensor was exposed to a flame. It is possible to distinguish the different regions of the flame according to (Yan et al. 2006). 7. New approach to detect fire So far an algorithm has been successfully employed to monitor NIR variations of the visual scene and to compute the flame frequency components: see Figure 6 and Figure 7. Once we have computed the global NIR levels in the scene, we can compute periodically the standard deviation of the previously recorded NIR levels and normalize it.          1 0 N k NIR knNIR N nA  (5) VII International Conference on Forest Fire Research D. X. Viegas (Ed.), 2014 9 The parameter   nA represents the variations of the NIR levels within the visual scene. It does not depend on illumination. If its value exceeds a certain limit, we can assume that there are flames in the scene. Depending on how many previously recorded samples are considered to compute the average NIR levels (see Figure 10), the value of   nA with and without flames will be different. Thus there is a trade-off between algorithm execution speed, the algorithm robustness to false alarms, and the number of events that have to be received to detect fire. Figure 10 illustrates the values of the normalized standard deviation computing the average NIR levels of the scene with different number of global NIR levels previously computed. Without flames,   nA values are always lower than 5%. With flames, such value is higher two or three times higher. To prevent generate fire alarms, a value of   %10nA seems to be practical according to our measurements. The algorithm only requires to store N previously estimated NIR levels and to update the value of   nA periodically. To reduce the computational cost detecting fire   nA could be updated every second, for instance. Figure 10 – Normalized standard deviation computation. With flames, the value is always higher than without flames. The normalized value does not depend on illumination conditions. 8. Conclusions In this paper we have presented event-based sensors as an alternative to classic framebased vision sensors for fire detection and surveillance systems. One particular example of a system with a bio-inspired camera that can be used to monitor flame activity is shown. The sensor outputs can be processed real-time to obtain NIR levels from the visual scene. Furthermore, we have extended this prior work define a real-time algorithm that detects fire in the environment analysing the transient variations of the NIR levels of the scene. Our goal is to create a portable system that emits an alarm if fire is detected in the visual scene. Experimental results with flames are provided. Table I summarizes the main system features.