ASAP: adaptive transmission scheme for online processing of event-based algorithms

Autonomous Robots https://doi.org/10.1007/s10514-022-10051-y ASAP: adaptive transmission scheme for online processing of event-based algorithms R. Tapia1·J. R. Martínez-de Dios1·A. Gómez Eguíluz1·A. Ollero1 Received: 1 June 2021 / Accepted: 24 May 2022 © The Author(s) 2022 Abstract Online event-based perception techniques on board robots navigating in complex, unstructured, and dynamic environments can suffer unpredictable changes in the incoming event rates and their processing times, which can cause computational overflow or loss of responsiveness. This paper presents ASAP: a novel event handling framework that dynamically adapts the transmission of events to the processing algorithm, keeping the system responsiveness and preventing overflows. ASAP is composed of two adaptive mechanisms. The first one prevents event processing overflows by discarding an adaptive percentage of the incoming events. The second mechanism dynamically adapts the size of the event packages to reduce the delay between event generation and processing. ASAP has guaranteed convergence and is flexible to the processing algorithm. It has been validated on board a quadrotor and an ornithopter robot in challenging conditions. Keywords Event camera ·Event-based vision ·Robot perception 1 Introduction In the last years event cameras have attracted increasing interest in the robotics community. Event cameras are neuromorphic sensors that capture asynchronous illumination changes at pixel level with µs resolution. They are insensitive to motion blur and have a wide dynamic range, providing This work was performed within GRIFFIN ERC Advanced Grant (Action 788247) (griffin-erc-advanced-grant.eu) funded by the European Research Council. Partial funding was obtained from the Spanish Project ROBMIND (PDC2021-121524-I00) and from Plan Estatal de Investigación Científica y Técnica y de Innovación of the Ministerio de Universidades del Gobierno de España (FPU19/04692). BR. Tapia [email protected] http://raultapia.com J. R. Martínez-de Dios [email protected] A. Gómez Eguíluz [email protected] A. Ollero [email protected] 1GRVC Robotics Laboratory, University of Seville, 41092 Seville, Spain Fig. 1 Picture from ASAP validation experiments on board a DJI Flamewheel F450 and the GRIFFIN E-Flap ornithopter high robustness to lighting conditions. A good number of successful event processing techniques have been proposed evidencing their capabilities, see e.g. (Gallego et al., 2020). Most existing event handling frameworks deliver the events generated by the camera to the event processing algorithm in event packages—i.e., structures in which several events are buffered to reduce the transmission cost. Packaging using a fixed number of events—i.e., fixed package size—or a fixed triggering rate are the most common approaches. Event delivery has a critical impact on the pro- 123 Autonomous Robots cessing algorithm performance. For instance, in cases with low event generation rates, using large packages can generate unnecessary delay, whereas short packages would enable better exploiting the asynchronous nature of event cameras. On the other hand, in cases with high event generation rates, using short packages would require devoting significant time to transmit and receive the events, increasing the risk of processing bottlenecks and saturation. When using event cameras on board robots in complex environments, changes in lighting conditions, type of scene, or robot motion can induce strong unpredictable changes in the event rate and in the processing algorithm efficiency. Static delivery is configured for the expected event rate and algorithm processing time, and cannot adapt to these changes, often resulting in processing responsiveness loss or computational overflow. This paper presents ASAP, an event handling framework that dynamically adapts event delivery to enhance processing responsiveness and prevent computational overflows. It absorbs the variations in the event rates and in the algorithm event processing time through two adaptive feedback-based mechanisms that run simultaneously and have a synergistic behaviour in which they help one another in the adaptation. Any event-based algorithm can be fed with ASAP requiring only minor implementation modifications. The proposed event handling scheme is flexible to the processing algorithm, its computational complexity, and its implementation, and also offers the possibility of packaging with fixed time windows and fixed number of events. The main contributions of this paper are: 1. an adaptive event handling scheme that enables responsive event processing with no overflows; 2. its experimental validation and evaluation on board a quadrotor and an ornithopter robot, see Fig. 1; 3. ASAP code1to contribute to the robotics, computer vision, and neuromorphic sensing communities. This work is inspired by our previous workshop paper (Tapia et al., 2020). The main novelties are: (a) deep improvements in both adaptive mechanisms to take into account the temporal cost fed back from the event processing algorithm; (b) improvement in the event packaging mechanism, which can now be tuned using closed analytical expressions; (c) mathematical demonstration of its convergence; and (d) new and detailed experimental validation including the implementation and evaluation in two aerial robots in challenging conditions. The structure of the paper is as follows. The main related work is summarized in Sect. 2. The proposed scheme and its two mechanisms are presented in Sects. 3,4, and 5, respectively. The convergence demonstration is described in Sect. 6. 1https://raultapia.com/repositories/asap. The experimental evaluation and validation are described in Sect. 7. Section 8summarizes the conclusions and future work. 2 Related work Existing event-based methods can be classified into: event images,event-by-event, and hybrid algorithms. The most widely used approach in robotics relies on processing event images, created by accumulating the events received. A method for detection and tracking using a Micro Aerial Vehicle (MAV) was developed in (Mitrokhin et al., 2018). It compensated for the global motion of the robot using a model of the affine transformation between two consecutive event images. Work (Falanga et al., 2020) presented a reactive obstacle avoidance method for quadrotors using event images from a stereo set-up. All these works provided valid solutions to some existing and novel problems. However, processing event images does not fully exploit the asynchronous behaviour of event cameras. Event-by-event methods process events one by one without grouping, exploiting the asynchronous nature of the event cameras. A number of solutions have been developed for feature detection (Vasco et al., 2016) (Mueggler et al., 2017)(Li et al., 2019), tracking (Alzugaray & Chli, 2018), and clustering (Barranco et al., 2018), among others. Although these methods are designed to process events one-by-one, few of them have focused on mobile robot applications, where computational restrictions can be relevant. An event-by-event pose tracking method for Unmanned Aerial Vehicles (UAVs) was developed in (Mueggler et al., 2014). The method was validated during high-speed maneuvers and reached accurate pose estimations. An event-based surveillance system using UAVs was presented in (Rodríguez-Gómez et al., 2020). The authors proposed a fully event-by-event scheme that combines feature detection, tracking, and clustering. Although event-by-event processing overcomes some of the limitations of event images, events are noisy and processing one-by-one results in high computational cost. Hybrid approaches combine the advantages of event-by-event and event image processing. A hybrid motion detector method was implemented in (Vasco et al., 2017) by learning the motion model of corner features when no object was present. Theworkin(GómezEguíluzetal.,2020) presented a bioinspired approach to time-to-contact maneuvers for multirotors. Although most of the above processing methods provide valid solutions to robotic perception, the online implementation of event-based vision systems on real robots requires mechanisms capable of efficiently managing the computational resources. However, few works have addressed the efficient delivery of events between event cameras and event 123 Autonomous Robots processing algorithms. Existing event handling frameworks can be classified according to the manner in which they feed the event processing algorithms. The use of event packages is the most widespread trend. The event-based framework included into the YARP (Yet Another Robot Platform) architecture (Glover et al., 2018b)ortheRPGROS DVS driver—used, for instance, in (Mueggler et al., 2014), (Lichtsteiner et al., 2008) or (Brandli et al., 2014)—rely on the use of event packages to feed the processing algorithms. Packages can be built up by accumulating a fixed number of events or by accumulating events during a fixed period of time. Although this kind of event handling prevents the algorithm’s communication overhead to some extent, they are not efficient in cases when the processing of one event package has been completed before the next package arrives. On the other hand, event-by-event transmission is an efficient solution when algorithms can process events faster than they are received. The work in (Marcireau et al., 2020) presented a framework for event-by-event handling that consists of three modules: an I/O library (called Sepia), a computation toolbox (Tarsier ), and a visualization tool (Chameleon). Their framework suppresses the use of buffers between the modules and limits the use of event buffers to the reception of events from the camera. Although event-by-event frameworks provide a lower communication delay than solutions based on event packages, its use is restricted to algorithms that complete the processing of an event before the reception of the following. Otherwise, high event rates can cause bottlenecks and, eventually, saturation. Online event processing on board real robots requires effective management of the computational load. Some works have started to explore how to adapt the incoming event stream from the event camera. Since (Berner et al., 2007), removing events that cannot be delivered in time has been a solution adopted by the event camera logic interface. Other works such as (Rodríguez-Gómez et al., 2020) or (Tapia et al., 2020) used the idea of randomly discarding a percentage of the incoming events to reduce the number of events to be processed. A feedback control scheme was presented in (Delbruck et al., 2021) for the adaptive setting of the internal parameters of the DVS sensor. Their method regulates the event rate by controlling the threshold and refractory period changing the internal bias currents, and also regulates noise using bandwidth control. Regarding adaptive packaging, the work in (Glover et al., 2018a) adjusts event package size by finding the error between the event rate produced by the camera and the event rate that can be processed. The method proposed in this paper focuses on event delivery under strong event rate changes such as those that can be found in aerial robotics in complex and dynamic environments with low-resource onboard processing capacity. It dynamically adapts event delivery by using two closedloop mechanisms with synergistic behaviours that help one another in the adaptation. To the best of the authors’ knowledge, it is the first event adaptive handling framework validated on board aerial robots performing aggressive maneuvers. 3 General description Consider an event camera, which events are delivered for processing at an event processing algorithm. Each event is represented as eiand includes the timestamp at which it was generated, the event pixel coordinates, and the event polarity. Events are transmitted from the pixel array to the output bus by using Address Event Representation (AER). In many applications, the event rates registered by the event camera may suffer unpredictable changes. For instance, that is the case in many robotics applications in which ground or aerial robots equipped with onboard event cameras navigate in complex and dynamic environments. Although static event packages, using either a fixed number of events or a fixed triggering rate, is the most widely approach, it can cause processing saturation and loss of responsiveness in cases with strong event rate changes. If the event generation rate is lower than expected, static packaging can cause unnecessary delays between event generation and processing. If it is higher, it can cause processing bottlenecks and overflows, since the algorithm may receive a new event package before completing the processing of the previous one. Two metrics are interesting to analyze the transmission of an event package pk:thedelivery time τkand the package building time πk.First,τkis defined as the difference between the time at which pkis sent to the processing algorithm and the time the processing of pkstarts –including a possible package buffer. πkis defined as the difference between the timestamps of the oldest and newest event in pk. In addition, the term latency could be applied to different time intervals regarding generation, packaging, buffering, and processing. As a convention, this work defines the latency λkof pkas the time between the generation of the oldest event in pkand the start of the processing of pk, i.e. πk+τk. The objective of ASAP is to dynamically adapt the transmission of events from the camera to the processing algorithm to keep its responsiveness and prevent computational overflow by feeding the algorithm with a new package Fig. 2 General scheme of ASAP 123 Autonomous Robots Table 1 Summary of the notation used Variable Description eii-th event of the incoming event stream γeii-th event of the event stream after γ-filter riIncoming event rate at the time when eiarrives γriγ-filtered event rate at the time when eiarrives pkk-th event package generated by ASAP skNumber of events in the event package pk tkTime required by the algorithm to process pk τkTemporal difference between pkis sent to the algorithm and processing of pkstarts πkTemporal difference between the timestamps of the oldest and newest event in pk λkLatency of pk, computed as τk+πk as soon as possible, i.e. as soon as it has finished the processing of the previous package. The scheme of ASAP is shown in Fig. 2. It includes two closed-loop adaptive mechanisms, both use tk−1, the time required by the algorithm to process the events from the last package pk−1. The first mechanism, based on the adaptive packaging module, enhances the algorithm responsiveness by dynamically adapting the size of the event packages.Itusestk−1to select sk, the size of the next event package. Adapting the event package size can be insufficient for preventing processing overflows in case of high event rates. The second mechanism, based on module γfilter, prevents overflows by randomly removing an adaptive percentage of events that will not be packaged or processed. In previous work (Rodríguez-Gómez et al., 2020) we noticed that in cases with high event rates, removing up to a maximum percentage of events had a low impact on the algorithm performance. γ-filter measures the current event generation rate and uses it to dynamically select the percentage of events to be removed. Event removal percentage is minimum at low event rates and increases with the event rate. Both closed-loop mechanisms are executed simultaneously and interact exhibiting a joint synergistic behaviour that enables fast adaptation. For instance, if tk−1has increased, the adaptive packaging module reacts by increasing sk. This will increase the latency λk, losing responsiveness. Simultaneously, due to the higher tk−1,γ-filter reacts by increasing the event removal percentage, reducing λkand improving responsiveness. Also, in the opposite case, if tk−1 has decreased, adaptive packaging reduces sk, and γ-filter reduces the event removal percentage. The notation of the main variables used in ASAP is shown in Table 1. 4 Adaptive event packaging This mechanism adapts the size of the event packages delivered to the processing algorithm to reduce the delay between event generation and processing. It uses tk−1,the time required by the algorithm to process the events in the last package pk−1, to select sk, the number of events to be included in the next package pk. Package pkis created by buffering events until the size of pkis ≥sk. Then, pkis delivered for processing. There is a dependence between the number of events in the package and the time required to process the packet. On the one hand, smaller packages will require lower time to be processed. On the other hand, the processing of events delivered in smaller packages is less efficient due to the time required to transmit and receive the events. Adaptive packaging selects skaccording to tk−1using the following generic function: sk=A(tk−1)+B,(1) where A,B∈Renable adjusting the ranges of skand tk−1, namely [smin,smax]and [tmin,tmax].Aand Bshould be adjusted such that smin =A(tmin)+Band smax = A(tmax)+B. Hence, Aand Bshould satisfy: A=smax −smin (tmax)−(tmin),(2) B=smax −A(tmax). (3) is a continuous function that relates tk−1and sk. ASAP can use any function that satisfies the following conditions: 1. Every value of tk−1∈[tmin,tmax]must have an associated value of sk. Hence, should be defined ∀tk−1∈ [tmin,tmax]. 2. Larger package sizes should be assigned to higher processing times. Hence, should be monotonically increasing, i.e. ∂(tk−1) ∂tk−1>0,∀tk−1∈[tmin,tmax]. 3. Event packaging should have ranges with different sensitivity of skagainst variations in tk−1. Hence, should 123 Autonomous Robots have at least one inflection point tflex, i.e. ∃tflex : ∂2(tk−1) ∂t2 k−1|tflex =0, such that: (a) For tk−1<tflex, the sensitivity of to changes in tk−1should decrease with tk−1to prevent small variations in tk−1from causing large changes in sk.The sensitivity of to changes in tk−1should be lower the closer to tmin. (b) For tk−1>tflex, the sensitivity of should increase with tk−1to prevent small changes in tk−1from causing no change in sk. The sensitivity of should be higher the closer to tmax. Additionally, we want to operate satisfactorily with event processing algorithms with wide differences (several orders of magnitude) in tk−1. Hence, we used log(tk−1) instead of tk−1, where log represents the natural logarithm. The suggested in the ASAP code is: (tk−1)=arctan(κ log(tk−1)). (4) This function fulfills the above conditions, guarantees the existence of one inflection point, and includes only one parameter κthat adjusts the curvature of to set the desired inflection point. Given κ, the inflection point tflex can be determined by a closed expression: tflex =exp −√κ2−1 κ−1.(5) |κ|<1 provides unfeasible solutions in Eq. (5). For |κ|> 1 there is a second inflection point tflex 2in when taking the two solutions of the square root. It can be noticed that tflex 2>tflex and that ∂2(tk−1) ∂t2 k−1≈0, ∀tk−1∈[tflex 2,tmax]. Hence, tflex 2has negligible influence on the sensitivity of . The value of κmodifies the shape of . Figure 3shows the resulting curves sk=A(tk−1)+Bwhen using different values of κ∈[1,11]and computing Aand Bwith Eqs. (2) and (3)usingtmin =10−6s, tmax =1s,smin =1, and smax =104. These values are the roughly-selected upper and lower bounds for tk−1and skthat we observed in a wide variety of problems and event processing algorithms, and are proposed as default values in the ASAP code. Although they have been validated for all the problems tested, they can be fine-tuned if desired, see Sect. 7. The value of κdetermines the position of tflex and hence, the point of operation of dynamic packaging. The default value for κin the ASAP code was roughly selected as κ=5, the center of the range shown in Fig. 3. This value provided satisfactory results in all the problems that were tested, some of which are shown in Section 7. Additionally, Sect. 7briefly presents how to adapt κto specific conditions or applications. Fig. 3 Values of function sk=A(tk−1)+Bfor κ∈[1,11]taking tmin =10−6s, tmax =1s,smin =1, and smax =104. For these values, A≈6.395 ·103and B=104 To reduce the time to compute Eq. (1), e.g., for interest in low-resource computers such as those used in Sects. 7.3 and 7.4,was approximated by its Taylor series expansion around an operating point a: (tk−1)≈ N  j=0 cj(tk−1−a)j,(6) where cjis the j-th Taylor coefficient of and Nis the order of the Taylor series expansion. Coefficients cjare tabulated for logarithmically-spaced values of a. For instance, using N=3 obtains a computational cost saving of 68.9% w.r.t. obtaining directly and an approximation error of 1.07 · 10−6, whereas N=5 involves a computational cost saving of 60.41% and an approximation error of 2.0247 ·10−9. 5-filter Adaptive packaging is insufficient for preventing algorithm processing overflow in case of high event rates. γ-filter reduces the number of events to be processed by randomly removing an adaptive percentage of the input events. It receives as input the event stream captured by the camera and outputs a filtered stream with a lower or equal number of events. Discarding events in cases with high event rates is not uncommon and has been proposed in several works. In our system, the event removal percentage is dynamically adapted depending on the current event rate and the value of tk−1. Additionally, event removal should be performed efficiently to reduce the delay. We adopted random event removal. We performed preliminary experiments (Sect. 7.2) in which different types of event processing algorithms were fed with event streams resulting from filtering the input stream with different event removal percentages. We confirmed that with high event rates, the algorithm performance degradation was very low with event removal percentages of up to 80%, and that removal was more effective (lower algorithm degradation and higher processing saving) with high event rates. 123 Autonomous Robots The operation of γ-filter depends on γi∈R:0≤γi≤1. For each received event ei, a random number ρi∈R: 0≤ρi≤1 is generated according to a uniform distribution U(0,1).Ifρi<γ i,eiis removed. Otherwise, eiis provided as output. With γi=0, all incoming events are removed; with γi=1, all are given as output. γiis selected proportionally to the incoming event rate. Let eibe an incoming event from the camera. Let ri∈R+be the current event rate when eiis received, measured by counting the events received during a sliding time window. The adaptation law makes γilinearly dependent on ri, which requires setting the maximum and minimum values of ri, namely rmax iand rmin irespectively. Using static values for rmax iand rmin iwould constrain adaptation to event rate changes. Hence, they are updated for each incoming event eias follows: rmax i=αrmax i−1if ri≤rmax i−1 riif ri>rmax i−1 ,rmin i=⎧ ⎨ ⎩ 1 αrmin i−1if ri≥rmin i−1 riif ri<rmin i−1 (7) If riis lower than rmin i−1, the minimum value rmin iis updated to ri. Also, if riis higher than rmax i−1,rmax iis updated to ri.Otherwise, rmin iand rmax iare computed from rmin i−1and rmax i−1using a forgetting factor α∈R:0<α≤1. In addition, using a linear law to adapt γialso requires establishing the upper and lower bounds for γi. The lower bound γmin reflects the maximum event removal percentage, which, following the preliminary experiments, was taken as γmin =0.2. Using a fixed value for the minimum event removal percentage establishes a static relation between riand γi. However, adapting it depending on the computational cost of the event algorithm is very interesting since it enables synergies between event packaging and γ-filter. The minimum event removal percentage ˆγkis updated for each tk−1as follows: ˆγk=γmax −tk−1−tmin tmax −tmin (γ max −γmin), (8) where tk−1,tmin, and tmax are as defined in Sect. 4.γmax is a fixed parameter that depends on the application, acts as the upper bound of ˆγk, and reflects the minimum event removal percentage. The value of ˆγkdepends on tk−1but always lies within [γmin,γmax]. The synergistic behaviour between both mechanisms can be clearly noticed. If the event algorithm becomes slower, tk−1will tend to tmin. On one hand, adaptive packaging will increase the package size. On the other hand, according to Eq. (8), ˆγkwill tend to γmin, which will help select a lower value for γkhence, avoiding the processing of a higher percentage of events, helping reduce the algorithm processing time. The opposite effect can be observed if tk−1tends to tmax. Hence, using tk−1to adapt ˆγkprovides synergistic effects in which γ-filter helps event packaging in reducing the algorithm response time. Finally, the value of γiis computed for each incoming event eias follows: γi=ˆγk−ri−rmin i rmax i−rmin i (ˆγk−γmin), (9) γiadapts depending on riand on tk−1. This dual mechanism provides fast adaptation to changes in the event rate and in the algorithm response time. We selected the random discard approach in γ-filter due to its extremely low computational burden. However, other filtering techniques can be used. For instance, spatio-temporal consistency filters could be used to discard events generated repeatedly in the same pixel or removing noisy events generated sporadically or in isolated regions on the image. Event filtering must be applied on each incoming event ei, therefore, computationally complex filters may generate delays in cases with high event rates. Although the event random removal approach has been taken as default, different filters are provided in the ASAP code and the addition of new ones is straightforward. 6 ASAP adaptive packaging convergence Adaptive packaging selects a suitable package size skfor each processing temporal cost tk−1. At the same time, the event algorithm processes the package of size skin time tk. It is necessary to demonstrate that adaptive packaging converges to feasible values of skand tk−1.Also,γ-filter does not disturb the relation between skand tk−1, since it only affects the number of events that are packaged, regardless of how they are packaged. The convergence of adaptive packaging can be demonstrated as follows. First, assume that tk=g(sk), where g:R+→R+models the time required by the algorithm to process the events in pk. Function gis assumed positive and strictly increasing in all its domain, i.e. g(sk)>0 and ∂g(sk) ∂sk>0, ∀sk>0. These two conditions are fulfilled in practice since the algorithm takes some time to process the package, and the higher number of events in the package, the longer it takes to process them. tkis used to select the size of the next package sk+1.Letsk+1=f(tk),f:R+→R+be the function used to select sk. Using Eqs. (1) and (4), the f functionusedinASAPis: sk+1=f(tk)=Aarctan (κ log(tk)) +B.(10) Using Eqs. (2), (3), and (5), it can be deduced that A∈R+, B∈R, and κ≥1. It can be also noticed that fis strictly 123 Autonomous Robots Fig. 4 Package size skfrom numerical solution of sk=A(tk−1)+B and tk=β0+β1skfor β0,β 1∈(0,0.1]susingsmin =1, tmin =10−6 s, smax =103,andtmax =0.1s increasing ∀tk>0: ∂f(tk) ∂tk=Aκ tk(κ2log2(tk)+1).(11) Assuming that s0>0, it can be proved that the sequence of values of skis either strictly increasing (sk+1>sk), strictly decreasing (sk+1<sk), or constant (sk+1=sk), ∀k>0. First, as ∂g(sk) ∂sk>0, ∀sk>0, the larger the input sk,the larger the output tk. Second, from Eq. (11), the larger the input tk, the larger the output sk+1. Consequently, if a>b, f(g(a)) > f(g(b)) and vice versa. Therefore: 1. Constant: Assuming sk+1=sk, the package size at k+2 is sk+2=f(g(sk+1)) =f(g(sk)) =sk+1. Hence, by induction, sk+n=sk,∀k>0, ∀n>k. 2. Strictly increasing: Assuming sk+1>sk, the package size at k+2issk+2=f(g(sk+1)) > f(g(sk)) =sk+1. By induction, sk+n>sk,∀k>0, ∀n>k. 3. Strictly decreasing: Assuming sk+1<sk, the package size at k+2issk+2=f(g(sk+1)) < f(g(sk)) =sk+1. Hence, sk+n<sk,∀k>0, ∀n>k. Thus, it can be also proved that skconverges when k−→ ∞in any of the three cases. Case Constant, all the values in the sequence are equal, so sk=s0∀k. Cases Strictly increasing and Strictly decreasing,as fis bounded −1 2Aπ+B<f(tk)<1 2Aπ+B,∀tk>0, skhas to converge, i.e., limk→∞ sk∈R.Otherwise,ifskdiverged, then |limk→∞ sk|=∞. As an example, convergence is analyzed for tk=g(sk)=β0+β1sk, where β0∈R+ is the time required by the algorithm to perform operations that do not depend on the number of events and β1∈R+ is the time required for processing each event in the package. Figure 4presents the values to which skconverges for β0,β 1∈(0,0.1]susingsmin =1, tmin =10−6s, smax =103, and tmax =0.1s. 7 Experimental results ASAP is configured by only setting the bounds of γi,tk−1, and sk, and the value of κ. We suggest reasonable values (default in the ASAP code) that cover a wide variety of applications and event processing algorithms, see Table 2. These values exhibited the expected performance in the wide variety of tested applications and conditions, some of which are presented in this section. Manually setting these parameters enables fine-tuning ASAP for a specific problem. In cases with low event rates and efficient event processing algorithms that operate with packet sizes near to 1, it is interesting to select values of smax and tmax lower than the default to provide ASAP with higher sensitivity when selecting skwith values close to 1. In these cases it is interesting to use higher values of κthat set lower values of tflex, the point of operation of event packaging. Conversely, with high event rates and high time-consuming event algorithms it is interesting to set higher values of smax and tmax than the default, and lower values of κto force ASAP to use larger packages as tk−1becomes larger. ASAP can operate with any event processing algorithm. The only requirement is that the algorithm should feed back the time it required to process each received package. Such modification is minor and was easily performed in all the different algorithms we tested. ASAP was implemented in C++11 under ROS Melodic Morenia. It uses the libcaer library for low-level event acquisition from the camera, but it can be integrated with other widely-used acquisition drivers (e.g. jAER). First, both adaptive mechanisms are analyzed in Sects. 7.1 and 7.2. In Sects. 7.3 and 7.4, ASAP is validated and evaluated on board a quadrotor and a flapping-wing robot. In the analyses we use the delivery time τkand building time πk presented in Sect. 3. Table 2 ASAP parameters and their default values Parameter Description Default value γmin Lower bound for γk0.2 γmax Upper bound for γk1 tmin Lower bound for tk−110−6s tmax Upper bound for tk−10.1s smin Minimum package size 1 event smax Maximum package size 1000 events κfactor see Eq. (4)5 123 Autonomous Robots 7.1 Evaluation of adaptive packaging First, we analyze the isolated effect of adaptive packaging. A DAVIS346 event camera was placed in front of a spinning wheel (see Fig. 5-top-left) that rotated at a constant angular velocity providing an approximately constant event rate of ∼70 events/ms. Figure 6shows its behaviour when there are sudden changes in tk−1. Before ∼0.75 s an efficient event processing algorithm with computational cost O(1)was used, and adaptive packaging selected sk≈1. At ∼0.75 s, the computational cost of the algorithm was increased to O(25).It can be seen in Fig. 6b that skadapted with no overshoot and rapidly reached the steady state. We define νas the number of iterations required by tk−1to reach an steady-state error Fig. 5 Experimental setups for validation of packaging (top-left), γfilter (top-right), and full ASAP scheme (bottom) (a) (b) (c) Fig. 6 Values of tk−1(a), sk(b), and τk(c) resulting in an experiment in which the algorithm computational cost changed at ∼0.75 s from O(1)to O(25) Fig. 7 Mean package sizes skfor event processing algorithms with different computational complexity <±1% w.r.t. the mean steady-state value. In this experiment, ASAP adapted in only ν=5 iterations. The resulting package delivery times τkshowninFig.6c were always lower than 3 ms and had similar values in both algorithms, meaning that there were not computational overflows despite the high differences in tk−1between both algorithms. In case of computational overflow, the packages would be buffered and τk would increase over time. Algorithms with different values of tk−1will result in different sk.Fig.7shows the mean values of skwhen using event algorithms with linear, quadratic, and cubic complexity. Figure 8shows the performance of adaptive packaging to changes in the computational cost of event algorithms with linear O(n)(left), quadratic O(n2)(center), and cubic O(n3) (right) complexity. In each case, nis changed every ∼2s following the sequence 10, 50, 200, 500, 200, 50, and 10. The figure shows the evolution in the values of tk−1(Fig. 8a), sk (Fig. 8b), τk(Fig. 8c), and πk(Fig. 8d). Adaptive packaging successfully adapted to every change in a low number of iterations ν, shown in Table 3. Note that the more computationally expensive the algorithm, the higher the value of skand hence πk. In all cases, τkremained lower than 3 ms despite the high difference in sk, meaning that there were no increasing waiting buffers and packages were processed as soon as they arrived. The values of λkare not shown. As λk=τk+πk,latency can be obtained from Fig. 8c and d. The values of τkhad high similarity for all packages, hence, the main changes in λkwere caused by the different values of πk. This behaviour means that ASAP increases latency only in cases where the algorithm requires it, and not due to increasing waiting buffers. The value of πkdepends on the performance of ASAP (sk) and the environment (ri). The influence of ASAP in πkcan be easily noticed in this experiment since the values of riare approximately constant along this experiment. This influence is not that clear in the experiments below since rihas wide variations, and for clarity πkis not shown. The objective of ASAP is to keep τksteady regardless of the environment and the algorithm processing time. If τkremains approximately constant, it means that the packages were processed as soon 123 Autonomous Robots Fig. 8 Values of: tk−1(a), sk(b), τk(c), and πk(d) when changing the algorithm computational cost with linear (left), quadratic (center), and cubic (right) complexity. Every ∼2snchanged following the sequence 10, 50, 200, 500, 200, 50, 10 Table 3 Number of iterations νrequired by tk−1to reach an steady-state error <±1% in the tests shown in Fig. 8 n Number of iterations O(n) O(n 2)O(n 3) 10 →50 5 7 4 50 →200 6 3 4 200 →500 8 4 2 500 →200 5 3 5 200 →50 4 2 4 50 →10 3 2 3 as they arrive. Hence, τkis the most critical metric in the performance of our method and its analysis is the main focus in the experiments below. 7.2 Evaluation of -filter Next,the isolatedeffectof γ-filterisanalyzed. The DAVIS346 was placed in front of a UAV propeller attached to a brushless motor (see Fig. 5-top-right). The motor angular velocity was varied between 450 rpm (providing ∼500 event/ms in average) and 4300 rpm (∼8000 event/ms). First, we analyze its performance with changes only in the input event rate ri, and then, changes simultaneously in riand tk−1. Figure 9shows the operation of γ-filter when the motor accelerated from 450 to 4300 rpm and then decelerated to 450 rpm. The value of tk−1was kept constant. γ-filter dynamically adapted γito ri, see Fig. 9-bottom. Figure 9-top shows Fig. 9 Operation of γ-filter when the motor accelerated until ∼20 s, and decelerates until ∼40 s: top) input and output event rates from γ-filter; bottom) resulting values of γi the event rate in orange and the resulting event rate output from γ-filter in yellow. Figure 10 shows an experiment in which the motor angular velocity was increased from 450 rpm to 4300 rpm and, simultaneously, tk−1had sinusoidal oscillations between 10−6and 0.1 s, see Fig. 10-top. The values of ˆγkadapted to tk−1 through Eq. (8), and the oscillations in tk−1induced oscillations between γmin =0.2 and γmax =1, see Fig. 10-center. Also, using Eq. (9), γidecreases the higher the value of ri and varied between γmin and ˆγkdepending on ri. This effect can be noticed in Fig. 10-bottom. A very low percentage of events are removed with low values of riand tk−1,e.g.at the start of the experiment. A higher percentage of events are removed with higher values of tk−1and ri,e.g.at18s. 123