Image recognition-based access security

Universidad Complutense de Madrid Facultad de Informática Final Year Project Computer Science Engineering Academic Year 2022 / 2023 Image recognition-based access security / Seguridad de acceso con reconocimiento de imágenes Student Pablo Castellanos García Advisor Juan Carlos Fabero Jiménez Contents 1 Introduction 1 1.1 Motivation ........................................ 1 1.2 Project goals ...................................... 2 1.3 Document structure .................................. 4 2 State of the Art 5 2.1 Similar projects ..................................... 5 2.1.1 Homemade IoT projects ............................ 5 2.1.2 Commercial products .............................. 7 2.2 Tools & technology ................................... 9 2.2.1 ESP32-CAM .................................. 9 2.2.2 Run-time operating system .......................... 13 2.2.3 Telegram bot API ............................... 15 3 System architecture & design 19 3.1 Features ......................................... 19 3.2 System architecture ................................... 22 4 Implementation 25 4.1 ESP32-CAM Server .................................. 25 4.1.1 esp32-cam-security.ino ........................... 25 4.1.2 app_httpd.cpp ................................. 26 4.1.3 bot_api.cpp .................................. 32 4.2 Telegram bot ...................................... 33 4.2.1 Bot.java .................................... 33 4.2.2 UDPServer.java ................................ 37 4.2.3 Other files .................................... 38 4.3 Implementation problems ............................... 39 iii 4.3.1 Library errors .................................. 39 4.3.2 Multithreaded server .............................. 40 4.3.3 Telegram bot in ESP32 ............................ 41 5 Conclusions 43 5.1 Results .......................................... 43 5.2 Limitations ....................................... 44 5.3 Further improvements ................................. 44 Abstract The Internet of Things (IoT) and Artificial Intelligence (AI) are two of the most important topics in Computer Science today, and two of the most commercially successful. The main focus of this project is to use a combination of the two—what is known as Artificial Intelligence of Things, or AIoT—to design and build a security system that, through the use of facial recognition, is able to a) grant or deny access to a hypothetical location autonomously; b) send alerts to users about events like detected or recognized faces; and c) function as a security camera. The system will be implemented using an ESP32-CAM microcontroller, and the notifications to users will be sent via a Telegram bot. Resumen El Internet de las Cosas (IoT, por sus siglas en inglés) y la Inteligencia Artificial (IA) son dos de los ámbitos más importantes en la Informática de hoy en día, los dos con gran éxito comercial. El objetivo principal de este proyecto es usar una combinación de ambos—la llamada Inteligencia Artificial de las Cosas o AIoT—para diseñar y construir un sistema de seguridad que, haciendo uso de tecnología de reconocimiento facial, sea capaz de a) conceder o denegar acceso a una localización hipotética de forma autónoma; b) enviar alertas a usuarios sobre eventos como una cara detectada o reconocida; y c) hacer de cámara de seguridad. El sistema será implementado usando un microcontrolador ESP32-CAM y las notificaciones a los usuarios se harán a través de un bot de Telegram. Keywords Internet of Things, IoT, Artificial Intelligence, AI, facial recognition, security system, ESP32, Telegram Palabras clave Internet de las Cosas, IoT, Inteligencia Artificial, IA, reconocimiento facial, sistema de seguridad, ESP32, Telegram Chapter 1 Introduction The focus of this project is on designing and building a security system based on facial recognition, which will be able to grant or deny access to a hypothetical location (a private home, for instance) and send alerts to the users. This chapter contains some motivation for why this is an interesting project in today’s world and lists the main goals to be accomplished. Finally, it contains a brief explanation of the contents of this document and how it is structured. 1.1 Motivation Thanks to the mass production of inexpensive, small and energy-efficient computer chips, in the last few decades it has become a common practice to integrate them in most everyday objects (things)—anything from a TV set to a lightbulb or a washing machine can now be considered a ‘smart’ device. On top of that, given the faster and faster network connections that are available to more and more people, it is only natural to take advantage of this high number of ‘intelligent’ devices by connecting them among themselves—sometimes, but not necessarily, through the Internet—and getting them to collaborate, in what has come to be known as the Internet of Things (IoT). This kind of technology has become pervasive in today’s world and all of the top technological companies have noticed its potential. Acording to Amazon, “the Internet of Things has a wideranging impact on human life and work. It allows machines to do more heavy lifting, take over tedious tasks and make life more healthy, productive, and comfortable.” 1To mention just a few examples, in 2014 Amazon announced its Amazon Echo smart speakers that connect with smart home appliances and respond to voice commands, and two years later Google followed with 1https://aws.amazon.com/what-is/iot/ 1 Chapter 1. Introduction their Google Nest, which allows users to “control their smart home” using only their voice. More recently, gadgets such as smart doorbells and smart security cameras have gained popularity. “In this hyperconnected world, digital systems can record, monitor, and adjust each interaction between connected things. The physical world meets the digital world—and they cooperate.” 2 However, not everything is as ideal when it comes to the Internet of Things as it may seem at first sight. After all, many of these companies have a reputation for not taking user privacy so seriously as many would like, and some are concerned [1] about giving them access to microphones and cameras inside our homes. Other problem that this approach entails is the fact that these devices too often rely on the manufacturers’ servers, so when—inevitably—there is an outage [2], they become useless. Perhaps the only technology that has gained more traction in mainstream media than IoT in the recent years is Artificial Intelligence (AI), whose primary goal is to solve complex problems automatically, relieving people from tedious or repetitive jobs that are nonetheless important to do. Typical examples of fields in which AI has been successfully applied include automatic recommender systems—such as the infamous “algorithms” employed by most streaming services or social media platforms to decide what content to show its users—and image recognition systems—which can be used to detect forest fires from satellite images [3] or tumors from MRI scans [4], to name a few. Even with all its potential for good, Artificial Intelligence—most notably face recognition systems—has also been used for not-so-noble ends. For instance, it plays a big role in the mass surveillance program [5] by the Chinese government and has even been used to supply evidence in court with desastrous consequences [6] when the system makes a mistake. Artificial Intelligence and the Internet of Things are without a doubt very interesting tools and some of the most relevant advances made in the last decades, but they can also be used against the interests of the user, as the evidence shows. This is why this project will focus on using these two technologies to build a working proof of concept of an access security system in which the end user is in full control and is as independent as possible from third party servers, which could potentially render the whole system useless, and which will attempt to minimize the negative consequences that could arise from the use of a facial recognition system, which is inherently inexact. 1.2 Project goals The primary goal of this project is to build a working prototype for an access security system using an ESP32 microcontroller. To achieve this, it will use image recognition technology to 2https://www.oracle.com/internet-of-things/what-is-iot/ 2 Project goals 1.2 detect human faces on frames and facial recognition to register the faces of authorized people and assign a unique identifier to each of them. After a successful face detection, its main function will be to perform face recognition on every detected face and: a) “Grant” access to users matching one of the registered faces, in a way that will aim to be as flexible as possible. b) “Deny” access to users not matching any of the registered faces, optionally sending an alert signalling that an intruder has tried to access the system. On top of this core functionality, the system will also: •Offer an interface to facilitate the addition and removal of registered faces. •Offer an interface via an HTTP server that will allow for the transmission of static images or live video feed (optionally including a mark around each detected face) and for configuration of certain parameters of the system. •Allow users to subscribe to a notification list in order to receive alerts whenever an intruder—or any user—is detected. –For improved security, the system will optionally have a whitelist of privileged users that are allowed to subscribe to the list. •For greater ease of use for the end user, these notifications will be sent directly to their smartphones or computers by a Telegram bot, which will also simplify the configuration of the system parameters. As for the learning goals, during this project I will: •Get familiar with developing software targeted at a microcontroller. •Get familiar with working with various APIs3. •Design the software architecture for a new project. •Apply, in completely practical settings, many of the tools taught in the Computer Science degree—mainly interacting directly with the Operating System via system calls and libraries—and building the basic infrastructure for web servers. 3Application Public Interfaces 3 Chapter 2. State of the Art To upload code to the ESP32 MCU (microcontroller unit), it needs to be in flashing state, which is determined by the GPIO 0 pin: the flashing state is active whenever GPIO 0 is connected to the ground (GND) pin. If this is the case, the code can be uploaded to the MCU from a computer through a UART10 interface. To make this task easier, there exist ‘programmer’ shields that attach to the GPIO pins and allow to easily flash code onto the MCU through a micro-USB port and on-board buttons to make the GPIO 0 –GND connection. Once the code is flashed on the MCU, this shield also is useful for powering the device with a USB cable rather than using the GPIO pins directly. For this project I used the ESP32-CAM-MB programmer shield, which has a CH340G chip to enable the USB – UART connection. (a) ESP32-CAM microcontroller (b) ESP32-CAM-MB programmer shield Figure 2.3: ESP32-CAM and programmer shield Once the MCU is in flashing state and connected to a computer, there are various methods for actually uploading the code, the most relevant being the Espressif IoT Development Framework (ESP-IDF) and the Arduino IDE11, which is what will be used in the project. This means that most of the code for the project will be written in C++. The camera that has been used in this project is the OV2640 model, which attaches directly to the ESP32-CAM via a FPC cable (see figure 2.3a). It has UXGA12 frame size (1600 ×1200 pixels), a frame buffer of 8 MB and a signal-to-noise ratio of 40 dB. Even though it is only capable of transferring UXGA images at a rate of 15 frames per second, the maximum framerate goes up to 30 fps for SVGA (800 ×600 pixels) and to 60 fps for CIF (400 ×296). The whole system occupies a very small space and is very economical: after assembly, the ESP32-CAM itself, the programmer and the OV2640 camera fit in a 39 ×27 ×23 mm box and cost under e6 when bought off the Internet. 10Universal Asynchronous Receiver-Transmitter 11Integrated Development Environment 12Ultra Extended Graphics Array 10 Tools & technology 2.2 Libraries Espressif provides some tools to make working with their SoCs as simple as possible, from networking basics to thread management. The official libraries designed for use with the Arduino IDE include all the software that is needed to develop applications for the ESP32 microcontrollers. They are hosted in a GitHub repository [14], which contains as well several examples that demonstrate how to work with the libraries and a .json file with all the necessary, up-to-date information about the various supported ESP32 boards to be imported in the Arduino IDE. More specifically, the version used is release 2.0.x, which introduces important fixes with respect to the 1.0.x releases. Another library developed by Espressif is ESP-DL [15] (“a library for high-performance deep learning resources dedicated to ESP32”), which offers an interface for face detection and recognition through its Model Zoo APIs and will be crucial for the development of the project. This library is included in [14] as a Git submodule, so that the Arduino IDE manages the installation automatically. The main code for the project is somewhat based on the ‘CameraWebServer’ example [16] from the same GitHub repository, as it showcases a few of the core functionalities of the system; in fact, this example shows up frequently in the projects mentioned in section 2.1.1. To be more specific, it offers the following: •Taking and displaying still images. •Streaming and displaying live video feed. •Enabling and disabling face detection / recognition. •Registering (“enrolling”) new faces to the system. •Displaying the results of running face detection / recognition on a still picture or a frame from the video feed. •Changing the camera settings (e.g. white balance, resolution, exposure, contrast, left/right and up/down flipping...). Because of this, the CameraWebServer example is a good foundation on which to build the system, but it does have a few shortcomings. First of all, the user interface is confusing because it offers many options at once, most of which will likely never be used by the end user, and some elements of the interface suggest a different function than the one they offer (for instance, the ‘Enroll Face’ button works as a toggle, and the ‘LED Intensity’ slider does not seem to have any effect), see figure 2.4. The user interface should be as simple as possible while maintaining the usability of the system. More importantly, it is lacking essential features of a security system, as it only displays what is being captured by the camera in real time, while the security system developed in this project should be able to keep some record of relevant events (such as a face detected), send notifications 11 Chapter 2. State of the Art Figure 2.4: CameraWebServer web interface to users, or have some kind of effect on the real world (unlocking a door, for instance). Another missing feature is the ability to access the system from outside the local network (e.g. for remote home surveillance or working as a smart doorbell). Below is a brief summary of the libraries used in the CameraWebServer example and which will be used in the project as well. In this example, the face detection is actually carried out by the infer method in the HumanFaceDetectMSR01 and HumanFaceDetectMNP01 classes. The first is simpler and faster, and is therefore used when speed is a priority or to obtain a first approximation. If precision is preferred over speed, the second can be used on the return value of the first to get better results. Both methods receive the input image as an unsigned integer pointer and two types are supported for each of them: •uint8_t* is used when the input image is encoded using RGB888. •uint16_t* is used when the input image is encoded using RGB565. The picture taken by the camera is not encoded in any of these formats; to perform the various conversions, there are several functions in the libraries (img_converters.h). The face recognition is performed by the FaceRecognition112V1S8 class. There is also a FaceRecognition112V1S16 class available, but it is not used in the CameraWebServer example because it has some issues: it produces “very large firmware”, is “very slow”, and “reboots when streaming” [16,app_httpd.cpp:56]. It should be noted that face recognition takes a long time (∼15 seconds per frame) on the ESP32 and ESP32-S2 chips even using the S8 class and is disabled by default [16,app_httpd.cpp:30] for all chips except for the ESP32-S3. The web server itself is created using the esp_http_server library, so that only the ‘handler’ 12 Tools & technology 2.2 functions need to be implemented. These are functions that take one HTTP request as input (in the form of a httpd_req_t variable) and implement all the necessary logic to process the request and send back the response. The most relevant functions from this library are the following: •httpd_resp_send and httpd_resp_send_chunk to send the response message. •httpd_resp_send_404,httpd_resp_send_408 and httpd_resp_send_500 to reply with the corresponding error codes (404: Not Found, 408: Request Timeout, 500: Internal Server Error [17]). •httpd_req_get_url_query_str and httpd_req_get_url_query_len to get the query string from the URL13 and its length. •httpd_query_key_value to parse a query from a URL and obtain the (key, value) pair. •httpd_resp_set_type and httpd_resp_set_hdr to set the response type and header, respectively. •httpd_register_uri_handler to add a URI14 handler (represented by a httpd_uri_t variable) to the web handle (httpd_handle_t variable). •httpd_start and httpd_stop to start and stop an HTTP server, respectively. The underlying implementation of the TPC/IP protocol suite used in the ESP32 is esp-lwip [18], a fork of the lwIP15 stack with some patches specific for the ESP-IDF framework. The BSD Sockets API [19] is currently the only officially supported API (and the one used in the project), but the Netconn API is also enabled. 2.2.2 Run-time operating system The operating system used in the ESP32 SoC is ESP-IDF FreeRTOS [20], which is a dual-core, SMP16-capable RTOS developed in part by Espressif. It is built upon the original FreeRTOS [21] (version 10.4.3), a real-time operating system designed specifically for microcontrollers. Below are outlined the features of FreeRTOS [22] and the ESP-IDF version [20] that will be most relevant in the project. The main difference between a real-time operating system and a non-real-time one is in their scheduling policies: as real-time / embedded systems are designed to respond to real-world 13Uniform Resource Locator 14Uniform Resource Identifier 15https://savannah.nongnu.org/projects/lwip/ 16Symmetric Multiprocessing: where a single instance of FreeRTOS manages tasks across multiple processors 13 Chapter 2. State of the Art events which usually need a fast response time, a real-time scheduling policy has to ensure that the highest priority task that is able to run is the task that is given processor time (it may be necessary to share the processing time “fairly” between tasks of equal priority, if there is more than one ready to run). When there is no other task ready to run, the idle task — which was created by the RTOS itself — is given all the processing time. Its only functions are to free the memory of tasks that have already been deleted and to run the ‘idle task hook’, so it is important to ensure that it is able to run in the case that tasks are deleted or the idle task hook is needed; however, it “can legitimately be starved of microcontroller time under all other conditions” 17. In FreeRTOS, the logic for a task is implemented in a void function that takes one void* argument, and the task itself is created via the xTaskCreate or xTaskCreateStatic functions (which also set the task priority and stack size) and deleted via the vTaskDelete function. At a given time, each task is in one of four possible states: running,ready (able to run but not currently executing because a task with equal or higher priority is running), blocked (waiting for an event, such as a timer expiring or a resource becoming available; normally includes a ‘timeout’ period after which the task will be unblocked even if the event has not ocurred yet) and suspended (similar to ‘blocked’ but without a timout; tasks can only enter or leave this state when explicitly requested through the vTaskSuspend and xTaskResume functions). In a single-core chip, only one task can be in the ‘running’ state at a given time. The FreeRTOS scheduler ensures that the ‘running’ task is the one with the highest priority among all the ‘ready’ tasks; if there are more than one task with the highest priority, the scheduler uses a round-robin policy. If a task enters the ‘ready’ state and its priority is higher than that of the currently running task, the latter is preempted by the scheduler to give the higher-priority task processor time. For inter-task communication, FreeRTOS provides the usual primitives (semaphores and mutexes), as well as queues for sending data from one task to another and its own notifications (which only use one internal notification array for each task and no other object; this makes them faster than the other options but they also require more memory if many different notification types are needed). It also offers support for the POSIX threads API (for both tasks and synchronization primitives). Listed below are the most relevant modifications introduced by Espressif into FreeRTOS in order to support dual-core SMP. Tasks can be pinned to a certain core (it is said that the task has an affinity for that core) when they are first created, using the xTaskCreatePinnedToCore or xTaskCreateStaticPinnedToCore functions. If a task is not pinned to any particular core, it is allowed to switch cores during the course of its execution. 17https://www.freertos.org/RTOS-idle-task.html 14 Tools & technology 2.2 For the most part, the two cores behave in the same way in ESP-IDF FreeRTOS, (tasks can be pinned to any of the two and unpinned tasks can run equally on both of them) but there are some exceptions when it comes to the internal workings: only CPU0 is responsible for incrementing the scheduler’s tick count and for unblocking any ‘blocked’ tasks whose timer has run out, while both cores are responsible for checking whether a context switch is required. As only CPU0 is in charge of keeping time, it is important not to prevent it from incrementing the tick counter so as not to disrupt the scheduler’s time. ESP-IDF also provides a Task Watchdog Timer (TWDT) [23] for monitoring particular tasks and making sure that they are not starved of processor time. By default, only the idle tasks are registered to be monitored, but any task can subscribe to being watched by using the esp_task_wdt_add function. Every monitored task must regularly call the esp_task_wdt_reset function to reset the watchdog timer (feed the watchdog); if the timer is allowed to expire, an exception is raised with message ‘task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:’ indicating that the monitored task has not been given processing time since the timer was last reset. 2.2.3 Telegram bot API Telegram is a popular free instant messaging application with a large set of features and official clients for the main mobile platforms (Android, iPhone and iPad) and desktop operating systems (Windows, macOS and Linux), as well as browser-based clients. One of the distinguishing features of Telegram with respect to other messaging apps is its support for bots, special Telegram accounts that users can interact with to make queries or receive notifications directly to their device — anything from searching in Wikipedia18 to creating PDF documents using L A T EX19 or looking for public transport schedules20;“the possibilities for bots are endless — from simple scripts to complex web apps. Whether you’re a beginner or professional programmer, you can create personalized tools with the help of the Bot Platform.” [24]. All this is accesible via the Telegram Bot API [25], an HTTP-based interface to allow developers to build bots for Telegram. To create a Telegram bot, the first step is to obtain a unique token for it using the BotFather bot21: after choosing a username, BotFather replies with the token that will be used to access the HTTP API. Once the bot is already created, BotFather can also be used to change the bot’s description, name or profile photo. 18https://t.me/wiki 19https://t.me/InLaTeXbot 20https://t.me/emtmadbot 21https://t.me/botfather 15 Chapter 2. State of the Art Figure 2.5: Creating a new bot with BotFather The bot API supports all standard functions of Telegram accesible to users (sending messages, photos, polls, etc.) but it also allows developers to create custom keyboards (inline keyboards) to get quicker input from users without the need to fill the chat with messages, or to define a “menu button” with information about the supported commands: •Inline keyboards are implemented as sets of “buttons” which, when pressed, cause a callback query to be sent back to the bot. These queries can contain different attributes, but the most important in this project is the data field, which contains a string set previously by the developer. Telegram clients display a progress bar after a button has been pressed, so each callback query must be answered to acknowledge that it has been processed. •The menu button is essentially just a list of the available commands with a short descriptive message to explain their function. To actually implement the code for the bot there are several third-party libraries, some of them featured in the official Telegram website22, to connect with the Telegram API. The chosen library for this project is TelegramBots [26] for Java, which is essentially a one-to-one mapping from the Telegram API [25], but a special mention must be made of the Universal-ArduinoTelegram-Bot library, which is “designed to be used with multiple Arduino architectures” [27], including ESP boards. In TelegramBots, all objects in the Telegram Bot API are implemented as Java classes. The most relevant to the project are listed below: •TelegramLongPollingMode is an abstract class to be extended by concrete bots, which must provide their own tokens. The received messages are handled by the onUpdateReceived method, which receives an object of type Update. •Update objects may contain at most one of several different parameters depending on the type of the update. The only types that are of interest are Message and CallbackQuery. •SendMessage objects are constructed with the recipient’s ID and the message text. To actually send a message, a SendMessage object needs to be passed to the execute method. 22https://core.telegram.org/bots/samples 16 Tools & technology 2.2 •SendPhoto is similarly constructed with a user ID and an InputStream which represents the picture to be sent. To send the picture, a SendPhoto object needs to be passed to the execute method. To perform the action represented by any of these objects, the execute method must be called. 17 Chapter 3 System architecture & design This chapter contains an in-depth exposition of the features of the system and an explanation of way the project is organized. 3.1 Features All the features of the designed system revolve around the ability to take still pictures or live video at any time and to perform face detection and recognition on them. Specifically, users can use the system to: •Take a still picture at any time, immediately receiving the captured image, and automatically running face detection and recognition on it if they are enabled. A box is shown around any detected face and, if face recognition was executed, a text is written on the picture with the face ID, in the case that a known face has been recognized, or a message indicating that an unknown face was detected otherwise. The color of the box indicates the result of the facial recognition: if a known face is detected, the box is drawn in green; if an unknown face is detected, the box is red; and if only face detection was run, but not recognition, the box is drawn in yellow. •Register new ‘known’ or ‘authorized’ faces by enabling the ‘enroll new faces’ option and either taking a picture or having the face appear in a video frame. When a new face is enrolled it is stored so that, even if power is lost, the authorized faces are preserved. This option takes precedence over facial recognition, meaning that if an enrolled face is present in the frame but the ‘enroll new faces’ option is turned on, the face will be enrolled again with a new face ID. 19 Chapter 4. Implementation Then, the camera is initialized by setting the configuration in a camera_config_t variable and calling the esp_camera_init() function; if the camera fails to be initialized correctly, the function returns with an error code and the program terminates. The main settings that can be changed are the following: •pixel_format, the format in which the pictures are taken. It is set to PIXFORMAT_JPEG because this is the format that is used for sending the pictures and the video frames, which means that no format conversion is needed just for the image transmission. •frame_size, which is initially set to FRAMESIZE_UXGA, the highest available value. This is set here because it is the esp_camera_init() function that is responsible for allocating the frame buffers, so they need to be big enough to hold any possible frame size. However, taking pictures with such a high resolution is quite slow (recall from 2.2.1 that the maximum framerate the camera can handle with the UXGA frame size is 15 fps), so the frame size is set to a lower value just after initializing the camera: in the lines sensor_t *s=esp_camera_sensor_get(); s->set_framesize(s, FRAMESIZE_CIF); the camera is configured to use the CIF frame size for all new pictures taken until the frame size is changed again. •jpeg_quality, initially set to 10, which works well for situations with good lighting. After starting the camera, the WiFi library is initialized using the SSID and password provided in wifi_credentials.h and the program waits until a connection is established. Lastly, the web server itself is started by calling the start_server(). In addition to setup(), Arduino files also need a loop() function, which is called continually once setup() has returned; however, since all the logic for the system is implemented in tasks defined in app_httpd.cpp, this loop function does not need to do anything. Instead of being left empty (which would unnecessarily consume CPU cicles), it only contains a call to delay(10000) to sleep for 10 seconds. Besides setup() and loop(), this file also contains the functions switch_flash(bool) and switch_led(bool) which respectively turn the camera flash and the pilot LED on or off by setting the corresponding pins to high or low voltage. 4.1.2 app_httpd.cpp This file handles all the image captures and the web servers, which arguably makes it the most important part of the whole system. The main function here is start_server, which is called 26 ESP32-CAM Server 4.1 from setup() and is in charge of initializing the alerts system, registering the different URIs with the corresponding HTTP handles and loading the previously registered faces from a dedicated partition. URI handlers The URI handlers are registered through the httpd_register_uri_handler function, which receives as arguments the HTTP handle (httpd_handle_t) and a variable of type httpd_uri_t representing the URI; in these variables, the field method is always set to HTTP_GET while user_ctx is always NULL, so only the fields uri and handler are of interest here (see the definition of the type httpd_uri_t below from [14, file esp_http_server.h]). Note that the handler functions return an esp_err_t value indicating a possible error; if this value is not ESP_OK (meaning that any kind of error has occured), the underlying socket is closed. typedef struct httpd_uri { const char *uri; /*!< The URI to handle */ httpd_method_t method; /*!< Method supported by the URI */ /** * Handler to call for supported request method. This must * return ESP_OK, or else the underlying socket will be closed. */ esp_err_t (*handler)(httpd_req_t *r); /** * Pointer to user context data which will be available to handler */ void *user_ctx; } httpd_uri_t; The string uri is one of the URIs discussed in 3.2 (/,/status,/control,/capture or /stream), while the handler field points to functions that manage each of these URIs: •index_handler only checks that the camera is available and replies with the main page— from file index.html—if it is; if the camera sensor is not found, it replies with a 500 (internal server error) status code. For efficiency reasons, index.html is sent as a gzip-compressed file, which is stored in the code as an array of bytes. To make it easier to work with this file, a Perl script (html_gz.pl) 27 Chapter 4. Implementation has been created to perform the two-way conversions: –“decompression” (CLI option -d): extracts the textual representation of the bytes from the input file and writes them into the output file before running gzip. –“compression” (CLI option -c): runs gzip and writes the textual representation of the resulting bytes to a file. •status_handler sends a JSON response containing the current system settings; it is useful to update the web interface at the beginning with the actual configuration the system is using. •cmd_handler manages the /control URI, which is used as an interface to change the system settings. It expects that requests to /control contain a query with two key-value pairs: (var,variable) and (val,value). The query string is extracted by a helper function called parse_get, which is essentially a wrapper around the library function httpd_req_get_url_query_str but also reserves enough dynamic memory to contain the query string and returns an error code if a query was not found in the URI. The query string is then parsed to obtain the two values using the httpd_query_key_value function and the system configuration is set accordingly. The possible values for variable are: –flash, to change the state of the flash LED. –detect,recognize and enroll, to enable or disable face detection, recognition or enrollment, respectively. –send-alerts, to enable or disable the ‘send alerts’ mode. –whitelist-enable, to change whether to use the whitelist or not. –whitelist_add and whitelist_remove, to add or remove users to the whitelist, respectively. In both of these cases, value is interpreted as a string containing the username to be added or removed. –framesize, to set the frame size for the next pictures taken. value should be an integer in the range 0–13, in which case the frame size is set according to table 4.1. –quality, to set the JPEG quality that will be used for the next pictures. value should be an integer. –photos, to change whether to send photos to subscribed users or not. This is not expected to be changed directly by users (it is not exposed through the web interface nor through the bot interface), but is used by the bot when users subscribe or unsubscribe to receive photos: because photos are several kilobytes in size, they should not be sent to the bot when there are no users interested in receiving them. 28 ESP32-CAM Server 4.1 Value Frame size 0 (96 ×96) 1 QQVGA (160 ×120) 2 QCIF (176 ×144) 3 HQVGA (240 ×176) 4240 ×240 5 QVGA (320 ×240) 6 CIF (400 ×296) Value Frame size 7 HVGA (480 ×320) 8 VGA (640 ×480) 9 SVGA (800 ×600) 10 XGA (1024 ×768) 11 HD (1280 ×720) 12 SXGA (1280 ×1024) 13 UXGA (1600 ×1200) Figure 4.1: Internal representation of frame sizes [14, file sensor.h] –cooldown, to enable or disable the cooldown timer between successive face detections when the video is active. –cooldown-time, for setting the length of the cooldown timer. In this case, value is an integer representing the number of second for the timer. –running, to turn off the video. This is only useful when value is 0 (false) and the video is actually running; if this is not the case, the action has no effect (see stream_handler below). Unless stated otherwise, all variables receive an integer value that is interpreted as a boolean. When variable is not one of the mentioned strings, or there is any error when changing the camera settings, the handler replies with a internal server error status code; otherwise, it sends an empty response and exits. It was previously noted that some of the settings are dependent on one another, so some care must be taken when actually changing them. The function enforces the following restrictions: –When face detection is disabled, so are facial recognition and enrolling. –When facial recognition is enabled, so is face detection. When it is disabled, so is enrolling. –When enrolling is enabled, so are face detection and recognition. These three actions serve to preserve the following invariant in the system configuration: enrolling ON ⇒recognition ON ⇒detection ON 29 Chapter 4. Implementation –By design, face detection is never run on images with a higher frame size than CIF because it would take longer to process the frame and because the resulting image would be too large to send to the bot. Because of this, the frame size is never set higher than CIF if face detection is on, and face detection is not enabled if the frame size is too high. •capture_handler takes a photo and performs all the necessary actions depending on the system configuration. In the first place, a JPG picture is taken via the esp_camera_fb_get function; if the function fails to get an image, the handler sends an internal server error response and returns with an error code. There is no need to do any additional processing to the picture if face detection is disabled: in this case, the photo is sent in the response message and the function returns. Otherwise, at least face detection needs to be performed, which needs the picture to be encoded using one of PIXFORMAT_RGB888 or PIXFORMAT_RGB565; only the former is used in the system. The JPG picture is converted to RGB888 using the fmt2rgb888 function and then detection is run on the result: if a face is detected, alerts are sent and recognition/enrolling is performed depending on the settings; otherwise, the photo is sent and the function returns immediately. The steps taken after the detection has ended are summed up in the pseudo-code below: if (detected) { if (send_alerts) notify that a face has been detected; if (is_enrolling) enroll; if (recognition_enabled) { run facial recognition and get face ID; if (send_alerts) notify that face with ID has been recognized; } draw face boxes; convert back to JPG; 30 ESP32-CAM Server 4.1 respond to the request with updated photo; if (send_alerts && send_photos) send the photo to the bot to broadcast to the subscribed users; }else { respond to the request with photo; } If any error occurs when performing the image conversions or allocating dynamic memory, the handler responds with an internal server error status code and returns an error code indicating failure. Also, an information message is printed to the serial monitor with the size of the final image and the recognized face ID, when applicable. •stream_handler manages the live video, whether it is actually streamed or not. Its implementation is very similar to that of capture_handler but wrapped inside a while(running) loop. It only breaks out of the loop when running is set to false (via control_handler) or when there is an error performing the image conversions or memory allocations. However, there are some differences with respect with capture_handler: –The frames are only sent back to the client when neither detection nor recognition is run, i.e. when face enrolling is disabled and the alerts are turned off. This means that, even though registering new faces is possible in live video, users will not be able to see the image that was used for the enrollment. –If face detection is run and the alert cooldown timer is enabled, the system sleeps for alert_cooldown_time seconds after the frame has been processed. As the vTaskDelay() function takes its input in number of ticks [20], the cooldown time in seconds needs to be converted to ticks dividing by the constant portTICK_RATE_MS; Each of these functions is set as the handler for the corresponding URI (index_uri,status_uri,cmd_uri,capture_uri and stream_uri). Of these, the first four are registered with the ‘main’ server, represented by the camera_httpd handle, while the last is registered with a separate handle for the stream, stream_httpd. Helper functions The system relies on the following auxiliary functions: •rgb_print and rgb_printf are used to write messages in color on the captures images. They rely on the fb_gfx_print library function. 31 Chapter 4. Implementation •draw_face_boxes draws the color-coded boxes around each detected face using the fb_gfx_drawFastHLine and fb_gfx_drawFastVLine library functions. •enroll_faces uses the enroll_id method from the FaceRecognition112V1S8 class [15] to register new faces into the system—storing them into flash memory—provided that the number of registered faces is less than the constant FACE_ID_SAVE_NUMBER (which is set to 7). To signal that the system is busy enrolling a new face, the pilot LED is turned on before the call to enroll_id (by calling switch_led(true)) and it is turned off again after it has finished (switch_led(false)). •run_face_recognition has a similar structure to enroll_faces, using the recognize method instead. If the face ID is non-negative, meaning that the face has been recognized as a ‘known’ face, a message is written in green on the picture via rgb_printf indicating both the ID and the similarity to the ‘original’ picture; if it is negative, meaning that it has not been recognized as a known face, the message “Intruder Alert!” is written in red. Two-stage detection The system implements the face detection in two stages bacause it gives much better results for use with facial recognition (recall from 2.2.1 that there are two classes available to perform face detection). This is done by first running the infer method in the HumanFaceDetectMSR01 class to obtain a list of candidates, and then using these candidates to obtain the final results with HumanFaceDetectMNP01: std::list<dl::detect::result_t> &candidates = s1.infer((uint8_t *)out_buf, {(int)out_height, (int)out_width, 3}); std::list<dl::detect::result_t> &results = s2.infer((uint8_t *)out_buf, {(int)out_height, (int)out_width, 3}, candidates); However, it is still possible to disable two-stage face detection during compile time by setting the TWO_STAGE preprocessor macro to 0, for a faster but less precise face detection. 4.1.3 bot_api.cpp This file contains the necessary functions to send information to the Telegram bot using a BSD socket [19]. The socket descriptor and struct sockaddr are stored as global variables inside the file, and are initialized by the init_alerts function using the address and port defined in bot_config.h. The implemented functions are: 32 Telegram bot 4.2 •send_alert(const char* msg, size_t len), a generic function to send a message to the bot which is used by the rest of the functions but is not exposed outside of the file. •whitelist_enable(bool b), to tell the bot whether or not it should respect the whitelist. •whitelist_add(const char* username), to add a user to the whitelist. •whitelist_remove(const char* username), to remove a user from the whitelist. •face_recognized(int face_id), to send a text notification about a recognized face, including its ID. •face_detected(), to send a text notification about a detected face. •face_recognized_photo(const char* photo, size_t len), to send a photo with a recognized face. •face_detected_photo(const char* photo, size_t len), to send a photo with a detected face. These functions are explained in more detail in subsection 4.2.2. 4.2 Telegram bot The Telegram bot has been implemented in Java using the TelegramBots library [26]. The entry point is in Main.java, whose only method main is in charge of creating a new object of the class Bot—which will be responsible for interacting with the user and the Telegram servers—and starting the UDP server—which will receive the messages from the ESP32-CAM. 4.2.1 Bot.java This file defines the Bot class, the most important piece in this part of the project. To be able to connect automatically with the Telegram API, it needs to extend either of the abstract classes TelegramLongPollingBot or TelegramWebhookBot—in this case, the former has been chosen. The class Bot has seven private attributes: •HashSet<String>whitelist, a set of strings containing the names of authorized users. •HashMap<String, HashSet<Long>> users, a map containing the information about subscribed users as subscriptionList 7→ users subscribed to subscriptionList 33 Chapter 4. Implementation •boolean whitelistEnabled = true. •boolean addingUserToWhitelist = false. •boolean removingUserFromWhitelist = false. •boolean settingCooldownTimer = false. •boolean settingQuality = false. There is only one constructor, which receives the bot token given as a string and does the following: •Call super(botToken); •Initialize whitelist with the default whitelisted users (as defined in BotConfig). •Initialize users by creating new empty subscription lists at keys detected,registered, intruder and photos. •Set the list of supported commands to provide auto-completion and a menu button inside the chat. The important inherited methods from TelegramLongPollingBot that need to be overriden are getBotUsername and onUpdateReceived—getBotUsername only returns the bot username that is defined in BotConfig.java, while onUpdateReceived—the method that implements most of the logic—is explained next. onUpdateReceived method This method receives a single object of the class Update; even though Telegram updates may be of several different types, the implemented bot only handles Message or a CallbackQuery (which are checked by the hasMessage() and hasCallbackQuery() methods): •Message updates: these are received when the bot receives a message from a user, most often in the form of a command; in this case, the message text, the username and the user ID are extracted from the message and a SendMessage object is constructed to respond to the user. In the first place, the method checks whether the user is authorized to send a message (either the whitelist is disabled, or the user is in the whitelist). If this is not the case, the response text informs the user that he or she is unauthorized and exits; otherwise, the request is processed. 34 Telegram bot 4.2 The commands are handled by comparing the message text with the command names (/start,/help,/subscribe,/unsubscribe,/get_photo,/config,/whitelist_add or /whitelist_remove). In most cases, to reply to the user it is only necessary to set the response text and sometimes a “reply markup” as well (in the form of an inline keyboard); this is done by calling the setText and setReplyMarkup methods in the SendMessage object with the appropiate strings or keyboards (see subsection 4.2.3), which are constructed in the CustomStrings.java and CustomKeyboards.java files. The commands that are handled differently are these: –/get_photo, which needs to contact the ESP32-CAM server to get a photo and forward it to the user. This is done by creating a new InputStream from the /capture URI and using it to create a SendPhoto object. –/whitelist_add, which in addition to setting the response text and reply markup, also sets the addingUserToWhitelist attribute to true. –/whitelist_remove, analogous to /whitelist_add but setting to true the attribute removingUserFromWhitelist. Before checking whether the message received is a command, the bot checks if either of the variables addingUserToWhitelist or removingUserFromWhitelist is set to true from a previous /whitelist_add or /whitelist_remove command; if this is the case, the message text is interpreted as a username to be added or removed to the whitelist: it is first checked that the message text is indeed a valid username and, if it is, it is added to or removed from whitelist and the corresponding boolean variable is reset to false. If the text is not a valid username, a message is sent to the user informing about the error. Next, the attributes settingCooldownTimer and settingQuality are checked. In these cases, the text contains the integer value that must be sent to the ESP32 server via the /control URI. Because the string may not contain a valid integer, the bot needs to be prepared to deal with an internal server error, which is done by a catch block that handles an IOException. If a message is received when all of these booleans are false and the text does not match any of the supported commands, the response text informs the user that the text is not a valid command and exits. •CallbackQuery updates: these are received when the user clicks on a button in an inline keyboard; in this case, the query data is extracted and handled according to these rules: 35 Chapter 4. Implementation worked correctly with images hosted on other servers, but it always failed when trying to send a photo from the server itself: using an address such as http://<local_address>/capture or http://127.0.0.1/capture it returned the following JSON: {"ok": false, "error_code": 400, "description": "Bad Request: wrong file identifier/HTTP URL specified"} Then, I tried to implement a function to send the captured frame directly by writing the bytes to a TCP client; this way, there were errors in the memory allocation: Connect to api.telegram.org [ 6055][E][ssl_client.cpp:37] _handle_error(): [start_ssl_client():264]: \ (-32512) SSL - Memory allocation failed [ 6056][E][WiFiClientSecure.cpp:144] connect(): start_ssl_client: -32512 Connected to api.telegram.org failed. Connected to api.telegram.org failed. [ 6243][E][ssl_client.cpp:37] _handle_error(): [start_ssl_client():273]: \ (-10368) X509 - Allocation of memory failed [ 6244][E][WiFiClientSecure.cpp:144] connect(): start_ssl_client: -10368 These errors were fixed when setting the stack size for the task even higher (around 24 KiB were needed), but the problem still was not fixed because then the server failed to establish a stable connection: just having the bot on (even if it was idle), the video livestream stopped randomly or failed to start in the first place, [ 31693][E][app_httpd.cpp:726] stream_handler(): Send frame failed [ 41196][E][ssl_client.cpp:129] start_ssl_client(): socket error on fd 54, \ errno: 113, "Software caused connection abort" [ 41197][E][WiFiClientSecure.cpp:144] connect(): start_ssl_client: -1 In this case, I also failed to find any more information about how to solve the problem, so I decided to implement the bot separately. 42 Chapter 5 Conclusions This final chapter presents the conclusions that have been reached after the implementation of the proposed system, including the results that have been obtained in relation with the original project goals as well as the limitations in the final system. 5.1 Results The security system that has been implemented during the course of this project meets all of the goals stated in section 1.2: •It performs facial recognition on frames taken by the camera—either as static pictures or as video frames—and is able to tell registered users apart from intruders, which enables the system to perform different actions depending on the outcome of the recognition. •It allows the user to easily register new faces as “known” faces. •It offers a web interface to access the camera feed and change the system configuration. •It is capable of sending alerts to any authorized user that has subscribed to any subset of the available notification lists. •It allows authorized users to change the system configuration remotely through a Telegram bot. Moreover, the system offers similar functionalities to those of many commercial products (see subsection 2.1.2) with two main advantages: it does it at a fraction of the cost, as all of the needed hardware can be bought for less than e6 from the Internet, and it gives users full access to the implementation, which has three important consequences: 43 Chapter 5. Conclusions a) Users are completely in control of their data and they can be certain that their pictures are not sent to the servers of a company they do not trust: the only external service used is Telegram, and it can be easily switched off or substituted for other ways of responding to recognized faces. b) Users are able to implement new features or disable those they do not want. c) The system is not dependent on a third party in a substantial way, so it is unlikely that the system will stop working from an outside cause. 5.2 Limitations The implemented system can be used in practice in some situations, but it has some limitations that are important to keep in mind: 1. Facial recognition is very slow on this combination of hardware and recognition software (it takes around 15 seconds). Apart from being potentially frustrating to use, it poses a security problem as it is relatively easy to perform a Denial of Service (DoS) attack on the system: by presenting any face to the camera when facial recognition is enabled, attackers can get a 15 second period during which the system is essentially frozen and is not capturing any new frames. For this reason, the system could not be reliably used as a security camera with facial recognition enabled. This could be fixed by upgrading the hardware to a ESP32-S3 microcontroller (recall from 2.2.1 that facial recognition is faster on this chip). 2. The system does not offer a way to see the live video from outside of the local network. 3. The web interface is potentially confusing to use if the settings are changed from a different device, as the HTML page is not updated dynamically when the server configuration is changed. 5.3 Further improvements Below are listed some features that have not been implemented in the final system, but could be useful to have: •Make the video feed available from outside the local network. Ideally, this would need a more sophisticated solution than port forwarding for security reasons—only authorized users should be able to watch the stream. For this, either an authentication mechanism 44 Further improvements 5.3 should be implemented, or the system should rely on external services to provide the security layer. •Register new faces into the system remotely by sending a photo to the Telegram bot and running the enrolling with that photo. This would need a more versatile form of communication from the bot to the server than what is currently implemented to be able to transmit the picture from the bot to the ESP32 microcontroller. •Delete registered faces from the system. As noted in 4.1, all new enrolled faces are stored permanently in flash memory, and there is not currently an easy way to remove them. The library provides a function to delete a face by ID, int delete_id(int id, bool update_flash =false); but this feature has not been implemented in the system because it is not considered particularly useful. 45 Bibliography [1] Sean Hollister. Read Google’s new Nest privacy promise and tell me if you’re swayed.url: https://www.theverge.com/2019/5/7/18536184/google-nest-privacy-commitment (visited on 13/03/2023). [2] Isabella Steger. How Amazon outage left smart homes not so smart after all.url:https: //www.mercurynews.com/2021/12/08/how-amazon-outage-left-smart-homes-notso-smart-after-all (visited on 13/03/2023). [3] Rachel Metz. How AI is helping spot wildfires faster.url:https://edition.cnn.com/ 2019/12/05/tech/ai-wildfires/index.html (visited on 13/03/2023). [4] Nadia Jaber. Can Artificial Intelligence Help See Cancer in New, and Better, Ways? url: https://www.cancer.gov/news-events/cancer-currents-blog/2022/artificialintelligence-cancer-imaging (visited on 13/03/2023). [5] Isabelle Qian et al. Four Takeaways From a Times Investigation Into China’s Expanding Surveillance State.url:https://www.nytimes.com/2022/06/21/world/asia/chinasurveillance-investigation.html (visited on 14/03/2023). [6] Tate Ryan-Mosley. The new lawsuit that shows facial recognition is officially a civil rights issue.url:https : / / www . technologyreview . com / 2021 / 04 / 14 / 1022676 / robert - williams-facial-recognition-lawsuit-aclu-detroit-police/ (visited on 14/03/2023). [7] M. Schwartz. Wireless Security Camera with the Arduino Yún.url:https://learn. adafruit.com/wireless-security-camera-arduino-yun/ (visited on 20/03/2023). [8] Tom Nardi. DIY ESP32 Video Doorbell Locks Out Big Brother.url:https://hackaday. com/2020/12/22/diy-esp32-video-doorbell-locks-out-big-brother/ (visited on 20/03/2023). [9] Ashish Choudhary. ESP32-CAM Face Recognition Door Lock System.url:https : / / circuitdigest . com / microcontroller - projects / esp32 - cam - face - recognition - door-lock-system (visited on 20/03/2023). 47 BIBLIOGRAPHY [10] Google. Nest Cam (Battery) - Google Store.url:https : / / store . google . com / us / product/nest_cam_battery (visited on 20/03/2023). [11] Chloe Matthews. Best Smart Doorbells of 2022: Home Security for Any Budget.url: https://www.gearhungry.com/best-smart-doorbells/ (visited on 20/03/2023). [12] Jennifer Oksien. The best video doorbell for 2023: if you like it should you put a Ring on it? url:https://www.techradar.com/best/bestvideodoorbells (visited on 20/03/2023). [13] Ai Thinker Technology. ESP32-CAM camera development board.url:https://docs.aithinker.com/en/esp32-cam (visited on 16/04/2023). [14] Espressif Systems. Arduino core for the ESP32.url:https://github.com/espressif/ arduino-esp32/ (visited on 22/12/2022). [15] Espressif Systems. Espressif deep-learning library for AIoT applications.url:https:// github.com/espressif/esp-dl (visited on 17/04/2023). [16] Espressif Systems. CameraWebServer GitHub repository.url:https : / / github . com / espressif/arduino-esp32/tree/23f653ad8d64e95231166e133845228b3db17e9d/libraries/ ESP32/examples/Camera/CameraWebServer (visited on 22/12/2022). [17] Internet Engineering Task Force. RFC 9110 - HTTP Semantics §15. Status Codes.url: https://httpwg.org/specs/rfc9110.html#status.codes (visited on 06/05/2023). [18] Espressif Systems. Fork of lwIP with ESP-IDF specific patches.url:https://github. com/espressif/esp-lwip (visited on 19/04/2023). [19] The FreeBSD Project. Chapter 7. Sockets | FreeBSD Documentation Portal.url:https:// docs.freebsd.org/en/books/developers-handbook/sockets/ (visited on 19/04/2023). [20] Espressif Systems. FreeRTOS (ESP-IDF) - ESP32.url:https://docs.espressif.com/ projects/espidf/en/latest/esp32/apireference/system/freertos_idf.html (visited on 19/04/2023). [21] FreeRTOS. Market leading RTOS (Real Time Operating System) for embedded systems with Internet of Things extensions.url:https://www.freertos.org/ (visited on 19/04/2023). [22] FreeRTOS. FreeRTOS Features - FreeRTOS.url:https://www.freertos.org/features. html (visited on 19/04/2023). [23] Espressif Systems. Watchdogs - ESP32.url:https://docs.espressif.com/projects/ esp-idf/en/latest/esp32/api-reference/system/wdts.html#task-watchdog-timertwdt (visited on 21/04/2023). 48 BIBLIOGRAPHY [24] Telegram Messenger Inc. Bots: An introduction for developers.url:https : / / core . telegram.org/bots (visited on 21/04/2023). [25] Telegram Messenger Inc. Telegram Bot API.url:https://core.telegram.org/bots/api (visited on 21/04/2023). [26] Rubén Bermúdez. Telegram Bot Java Library.url:https://github.com/rubenlagus/ TelegramBots (visited on 21/04/2023). [27] Brian Lough. Universal Telegram Bot Library.url:https://github.com/witnessmenow/ Universal-Arduino-Telegram-Bot (visited on 21/04/2023). [28] Pablo Castellanos. ESP32-CAM Security.url:https://gitlab.com/pacastega/esp32cam-security (visited on 07/05/2023). [29] Pablo Castellanos. ESP32-CAM Security Bot.url:https://gitlab.com/pacastega/ esp32-cam-security-bot (visited on 07/05/2023). 49