EdgeDRNN: Recurrent Neural Network Accelerator for Edge Inference

EdgeDRNN: Recurrent Neural Network Accelerator for Edge Inference Chang Gao ,Student Member, IEEE, Antonio Rios-Navarro ,Member, IEEE,XiChen , Student Member, IEEE, Shih-Chii Liu ,Senior Member, IEEE, and Tobi Delbruck ,Fellow, IEEE Abstract— Low-latency, low-power portable recurrent neural network (RNN) accelerators offer powerful inference capabilities for real-time applications such as IoT, robotics, and humanmachine interaction. We propose a lightweight Gated Recurrent Unit (GRU)-based RNN accelerator called EdgeDRNN that is optimized for low-latency edge RNN inference with batch size of 1. EdgeDRNN adopts the spiking neural network inspired delta network algorithm to exploit temporal sparsity in RNNs. Weights are stored in inexpensive DRAM which enables EdgeDRNN to compute large multi-layer RNNs on the most inexpensive FPGA. The sparse updates reduce DRAM weight memory access by a factor of up to 10x and the delta can be varied dynamically to trade-off between latency and accuracy. EdgeDRNN updates a 5 million parameter 2-layer GRU-RNN in about 0.5 ms. It achieves latency comparable with a 92 W Nvidia 1080 GPU. It outperforms NVIDIA Jetson Nano, Jetson TX2 and Intel Neural Compute Stick 2 in latency by 5X. For a batch size of 1, EdgeDRNN achieves a mean effective throughput of 20.2GOp/s and a wall plug power efficiency that is over 4X higher than the commercial edge AI platforms. Index Terms— Edge computing, FPGA, embedded system, deep learning, RNN, GRU, delta network. I. INTRODUCTION DEEP neural networks (DNNs) have been widely applied to solve various practical problems with state-of-the- art performance. Recurrent neural networks (RNNs) which are a subset architecture of DNNs, are particularly useful in applications involving time series inputs, such as speech recognition [1], [2] and dynamical system control [3], [4]. In contrast to Convolutional Neural Networks (CNNs) which use filter kernels, RNNs are fully-connected networks: They take a 1D vector as input and produce a vector of output. The feature vectors generated by CNNs can be fed into an RNN for further processing. In this way, RNNs can connect Manuscript received June 30, 2020; revised September 15, 2020 and October 31, 2020; accepted November 18, 2020. Date of publication November 24, 2020; date of current version December 11, 2020. This work was supported in part by the Samsung Advanced Institute of Technology, in part by the Swiss National Science Foundation under Grant 200021_172553, and in part by the SNSF Sinergia WeCare under Grant CRSII5177255. This article was recommended by Guest Editor F. Conti. (Corresponding author: Chang Gao.) Chang Gao, Xi Chen, Shih-Chii Liu, and Tobi Delbruck are with the Institute of Neuroinformatics, University of Zürich and ETH Zürich, 8057 Zürich, Switzerland (e-mail: [email protected]; [email protected]; [email protected]; [email protected].ch). Antonio Rios-Navarro is with the Robotics and Technology of Computers Laboratory, Universidad de Sevilla, 41012 Seville, Spain (e-mail: [email protected]). Fig. 1. A: EdgeDRNN accelerator concept. B: Example target application from [7]. the high dimensional input features over time, which is useful for complex sequential classification or regression tasks. Gated RNNs modify a “vanilla” RNN to add nonlinear operations to the units that allow them to memorize and gate their output. Long Short-Term Memory units (LSTM) [5] and Gated- Recurrent Units (GRU) [6] are used to overcome the vanishing gradient problem frequently encountered during vanilla RNN training with backpropagation through time (BPTT), where the sequential operations of the RNN are unrolled to compute the weight updates based on output error. By using BPTT with labeled training data, GRU and LSTM RNNs can be trained to high accuracy for tasks involving time series such as continuous speech recognition. Edge computing devices that embed some intelligence implemented through trained DNNs are gaining interest in recent years. An advantage of edge computing is that computations are done locally on end-user devices to reduce latency and protect privacy [8]. Most literature reports on the use of CNNs for edge devices. There is less reported on devices that use RNNs particularly on embedded low-latency, high energyefficient platforms that use FPGAs. RNNs also have larger memory footprints and memory access of the fully-connected weight matrices dominates power consumption. RNNs are usually computed on the cloud which introduces large and variable latency, thereby making it hard to guarantee real-time performance for edge applications such as human-computer interaction devices and mobile, robotic applications. Optimization methods have been applied to RNNs for embedded hardware implementations (e.g. weight pruning in ESE [9], and structure pruning in BBS [10]). We also previously reported the DeltaRNN accelerator [11] that uses the delta network algorithm [12]. Our first DeltaRNN implementation [11] stored the large weight matrices in FPGA block RAM and thus needed expensive FPGA boards with greater than 15W power consumption. However, this work only focused on pushing the limit of high batch-1 throughput without considering the memory and power constraints of extreme edge devices. Typical chips for edge applications, such as microcontrollers and small-footprint FPGA, only have a few hundreds of kilobytes (kB) of on-chip SRAM, but large RNNs usually have megabytes (MB) of parameters, making them difficult to be buffered on-chip even after compression. In this case, storing RNN parameters in off-chip memory such as flash or DDR memory is an inevitable choice for edge devices. Therefore, reported hardware RNN implementations [9], [10] cannot be easily scaled down for edge platforms. This work is on EdgeDRNN, a hardware accelerator for RNN computation on the edge [13]. Because of our interest in real-time edge applications, our focus is on supporting lowlatency batch-1 inference of large RNNs for real-time performance on low-cost and low power but very constrained edge platforms. We show that our implementation can run largescale multi-layer RNNs using a small number of processing elements with the large weight matrices stored on off-chip DRAM. Besides accelerating RNN inference, it leaves most cycles of the CPU in the system-on-chip (SoC) available for other tasks, such as feature extraction and I/O management. EdgeDRNN can be implemented on a small-footprint FPGA with 19X less logic cells and 15X less on-chip memory compared to the one used in DeltaRNN [11]. Thus, EdgeDRNN is suitable for embedded system applications such as robotics (Fig. 1B). Moreover, in previous work using the current delta network algorithm, a global threshold is applied on both the inputs and hidden unit activations of every layer of the network in sparsifying the activation vector. In this work, we looked at how different threshold values on the inputs and activations of hidden units affect the trade-off between the accuracy of the network on a regression task and the sparsity levels of the change in the activation vectors. A higher sparsity level implies reduced weight memory access and reduced computes. This article makes the following contributions: 1) We describe a flexible, low-cost, high throughput edge FPGA RNN accelerator that uses a spiking neural network-inspired Delta RNN principle to provide state- of-art latency and power efficiency for a wide range of gated RNN network sizes with batch size of 1. 2) We report the first study of a delta network that uses different delta thresholds for the input and activations of the hidden units. On our tested regression task, this modification increases temporal sparsity of hidden delta states by 16% compared to using a global threshold. 3) We compare the usability and throughput performance of two different EdgeDRNN implementation on the SoC FPGA: Bare-metal and embedded Linux. The latter enables faster development and we report the correct FPGA memory bus port configuration that minimizes the performance loss due to CPU contention for the memory controller. 4) We report benchmark latency and throughput numbers of RNN inference on state-of-the-art commercial chips for edge applications. To our best knowledge, these numbers have never been reported before. The rest of this article is organized as follows. Section II describes the background of gated recurrent unit (GRU) based RNN and the algorithm of delta network based GRU-RNN, which is called DeltaGRU . Section III describes the architectural design of the accelerator and its implementation on MiniZed. Section IV discusses experimental results including the experiments using different delta thresholds for the network. Section V compares the proposed accelerator with prior work and commercial edge devices. Section VI concludes the paper. II. BACKGROUND A. DNN Hardware Optimization Methods Various methods have been proposed to reduce the dominant RNN matrix-vector (MxV) operations. Static approximation methods (i.e. constructed during training) include quantization, arithmetic,andweight pruning. 1) Quantization: Quantizing floating-point weights or activations to fixed-point numbers with shorter bit width reduces memory footprint of networks and make it possible to use fixed-point MAC units instead of expensive floating-point MAC units [9], [14]–[16]. Chip area can be further reduced by replacing conventional fixed-point multipliers by look-up table based [17] or multiplexer [18] based multipliers on low bit precision networks with 2-4 bit weights. By including quantization during training (e.g. by using an approach like dual-copy rounding [19]) it is possible to reduce weight precision to 8 bits without accuracy loss. 2) Weight Pruning: Pruning removes unimportant neuron connections that results in sparse weight matrices [20]. Sparse matrices can be encoded into a sparse matrix format such as the Compressed Sparse Column (CSC)and Compressed Sparse Row (CSR). With an accelerator that can decode the sparse matrix format on-chip, the sparse matrix-vector (SpMV) multiplication can be accelerated by executing multiply-and-accumulate (MAC) operations only on nonzero weights. This approach was adopted by the Efficient Speech Recognition Engine (ESE) [9]. Because unstructured pruning results in computation that is hard to balance across processing elements, structured pruning methods have also been proposed to improve load balancing during the SpMV computation [10], [21]. This approach was used by the LSTM FPGA accelerator using Bank Balanced Sparsity (BBS) [10]. It is also used by the custom digital IC of [21] where it is called Hierarchical Coarse Grain Sparsity (HCGS). Structured pruning is a popular approach for improving RNN hardware performance; both BBS and HCGS use it to increase effective MAC efficiency, but large increases in efficiency result in significantly worse inference accuracy [21]. For example, a 16X compression increases the error rate by a factor of about 1.2X. It allows static (compile-time) optimization, but training is fairly complicated since exploration of the additional structure hyperparameter values is needed to find optimum values that are matched to the particular hardware. 3) Arithmetic: Bit-serial NN accelerators such as [22], [23] utilize a flexible bit-serial MAC to support various precision of network parameters and are smaller in area compared to conventional fixed-point MAC units. However, since a bit-serial MAC units requires more cycles to finish a multiplication between high bit precision operands, more bit-serial MAC units are required to achieve higher throughput than using conventional MAC units and larger adder trees are need for accumulating partial sums. Thus, the average speedup using this method is only around 2X but it comes with extra overhead area. The C-LSTM accelerator used Toeplitz-like weight matrices in the form of blocked circulant matrices to reduce RNN memory requirements [24] since multiple rows in each circulant matrix block can be generated from a single vector. The method also enables the use of Fast Fourier Transform (FFT) to reduce the MxV cost from O(n2)to O(nlog(n)) [25]. However, forcing weight matrices to be blocked circulant is coarsegrained and leads to higher accuracy degradation compared to weight pruning [9], [10]. Moreover, the method leads to hardware overhead of computing the FFTs of activations and weights. 4) Temporal Sparsity: The delta network algorithm [12] capitalizes on the temporal sparsity of activation state vectors in a network. Setting a finite threshold that is greater than zero has the effect of zeroing-out below-threshold elements of the activation vector, which results in sparse delta vectors. Since zero activations have no downstream influence, these MACs can be skipped. It means that entire columns of the weight matrix can be skipped. Thus delta networks marry the temporal sparsity of spiking networks with the synchronous update and analog state transmission of conventional deep networks. Combining these principles provides the benefits of sparse computing and efficient communication of precise analog information with reduced and predictable memory access of inexpensive DRAM which is crucial for storing the weights. A set of studies [7], [12], [13] showed in a variety of networks that by applying the delta principle during training, the accuracy loss is minimal even with a 5-10X improvement of RNN throughput and latency. For example, [12] used a 4-layer 320 units per layer GRU RNN for continuous speech recognition on the Wall Street Journal dataset. The Word Error Rate (WER) increased by only a factor of 1.08X but with a reduced memory access of 6.2X. B. Gated Recurrent Unit The update equations for a GRU layer of Hneurons and I-dimensional input, which are as follows: rt=σ(Wxrxt+Whrht−1+br) ut=σ(Wxuxt+Whuht−1+bu) ct=tanh (Wxcxt+rt(Whcht−1)+bc) ht=(1−ut)ct+utht−1(1) where r,u,c∈RHare the reset gate,theupdate gate and the cell state respectively. Wx∈RH×I,Wh∈RH×Hare weight matrices and b∈RHare bias vectors. The σvariable denotes the logistic sigmoid function. Each GRU input is the vector xtand its output is the vector ht. Fig. 2A illustrates the update of the normal GRU reset gate as a flow diagram. C. DeltaGRU The delta network method is applied to the GRU-RNN architecture; we call this DeltaGRU. Assume an input vector sequence X={xt|t=1,2,...,T}with sequence length of T, we first declare the following variables: ˆxi,t=xi,t,|xi,t−ˆxi,t−1|≥x ˆxi,t−1,|xi,t−ˆxi,t−1|< x ˆ hj,t=hj,t,|hj,t−ˆ hj,t−1|≥h ˆ hj,t−1,|hj,t−ˆ hj,t−1|< h xi,t=xi,t−ˆxi,t−1,|xi,t−ˆxi,t−1|≥x 0,|xi,t−ˆxi,t−1|< x hj,t=hj,t−ˆ hj,t−1,|hj,t−ˆ hj,t−1|≥h 0,|hj,t−ˆ hj,t−1|< h (2) where ˆxi,tis the i-th element of input state memory vectors ˆxt in timestep t.ˆ hj,tis the j-th element of hidden state memory vectors ˆ htin timestep t.xi,tis the i-th element of delta input state vectors xt.hj,tis the j-th element of delta hidden state vectors ht.xand hare respectively the delta thresholds of inputs and hidden state for each layer. In the initial timestep (t=1), ˆxi,0,hi,0,ˆ hi,−1are all initialized to zeros. The update equations for the DeltaGRU are now: Mr,t=Wxrxt+Whrht−1+Mr,t−1 Mu,t=Wxuxt+Whuht−1+Mu,t−1 Mxc,t=Wxcxt+Mxc,t−1 Mhc,t=Whcht−1+Mhc,t−1 rt=σ(Mr,t) ut=σ(Mu,t) ct=tanh(Mxc,t+rtMhc,t) ht=(1−ut)ct+utht−1(3) where Mr,t=0=br,Mu,t=0=bu,Mxc,t=0=bc,Mhc,t=0=0 are delta memory vectors and M ∈RH. Variables σand  indicate the sigmoid function and element-wise multiplication of vectors respectively. Fig. 2. Computation flow of the reset gate in gated recurrent units (GRUs). A: Normal GRU reset gate update. B: A DeltaGRU reset gate update. See Eqs.1-3. Fig. 2B illustrates these operations for the DeltaGRU reset gate. The input vector xtand the hidden state vector ht−1are respectively replaced by the delta input state vector xtand the delta hidden state vector ht−1. Values of the previous state memory vectors ˆxt−1,ˆ ht−2are updated using Eq. 2 to generate new state memory vectors ˆxt,ˆ ht−1. The previous delta memory vector Mr,t−1holds the previous step’s partial sum-product and the resulting new delta memory vector Mr,t is stored. Otherwise the operations are the same as for the original GRU reset gate, as shown in Fig. 2A. The other gates have similar flow diagrams. The state and delta memories are 1D vectors and can be easily fit into on-chip SRAM buffers. D. Temporal Sparsity The temporal sparsity of a DeltaGRU network of Llayers with an input sequence length of Tis defined as the fraction of zeros in the xand hvectors, signified by xand hrespectively. The effective temporal sparsity Eff is the weighted average of xand haccording to the number of network parameters they correspond to. The definition of temporal sparsity is given by Eq. 4: x=1 L·T·I T  t=1 n1 x,t+1 (L−1)·T·H L  l=2 T  t=1 nl x,t h=1 L·T·H L  l=1 T  t=1 nl h,t Eff =3HI+3H2(L−1)·x+3H2L·h 3HI+3H2(L−1)+3H2L =(I+H(L−1))·x+HL·h I+H(L−1)+HL (4) where nl x,tand nl h,tare the number of zero elements respectively in the delta vectors xand hin layer lat timestep t. Because operations on biases are negligible, they are ignored in Eq. 4. By skipping zero elements in delta vectors, whole columns of matrix-vector MAC operations can be skipped. If the delta network is properly trained (by including the delta operation), [7], [12], [26] showed that the number of operations can be reduced by 5X to 100X with negligible loss of accuracy, depending on the temporal evolution of the states of the input and hidden units. Fig. 3. EdgeDRNN accelerator architecture. E. Datasets Two datasets are used in this article: the TIDIGITS [27] dataset for the classification task and the SensorsGas dataset for the regression task. The TIDIGITS speech dataset has more than 25k digit sequences spoken by over 300 men, women, and children. The entire training and test sets are used in our experiments. The SensorsGas dataset consists of recordings of metal-oxide sensors in response to various concentrations of carbon monoxide gas over 14 days [28], [29]. This dataset was used in [30] to evaluate the network performance of a gated RNN in predicting the concentration of carbon monoxide. The dataset used here comes from the 70/30 Split variant, that is, 70% of the sequences are randomly selected to form the training set, while the remaining sequences form the test set. III. EDGEDRNN ACCELERATOR A. Overview Due to limited weight reuse, it is difficult to compute RNNs efficiently for real-time applications that usually work best with a batch size of 1. Therefore, a big challenge of RNN inference on the edge is the scarce off-chip memory bandwidth available on portable platforms, and the limited amount of on-chip block RAM on small FPGAs. EdgeDRNN uses cheap off-chip DRAM for weight storage and reduces memory bandwidth by exploiting temporal sparsity in RNN updates. Fig. 3 shows the architecture of the EdgeDRNN accelerator. The main modules consist of the Delta Unit for encoding delta Fig. 4. Flow chart of the Delta Unit. vectors and generating weight column pointers (pcol); the Processing Element (PE) Array for matrix-sparse vector multiplications; the (CTRL) control module which contains finite state machines (FSMs) and encodes instructions to control the AXI Datamover. Other modules include the configuration module (CFG) composed of configuration registers; the output buffer (OBUF) for buffering and redirecting outputs back to the Delta Unit and the W-FIFO for buffering weights. B. Delta Unit & CTRL The Delta Unit stores state memory for delta state vector encoding in a block random access memory (BRAM1). The FSM addresses the BRAM according to the valid signal of input state vectors xtand ht, one of which, is selected to be processed as stat a time depending on the FSM state. The Delta Unit encodes one element of a delta state vector st in each clock cycle after the valid signal asserted until the whole vector is processed. The vector sizes are provided by the config signal from the CFG module. Delta state vector elements that are greater than or equal to threshold xor h; and their corresponding physical weight column address pointer (pcol) are respectively dispatched to the D-FIFO and CTRL. The corresponding state element stis written into the BRAM to update the state memory. Otherwise, elements are discarded without being written into the D-FIFO. By using only 1 Delta unit, the latency in clock cycles for the Delta Unit to process a vector is exactly the length of that vector. It is possible to reduce the latency by searching for nonzero elements in Nsubsections of a vector simultaneously. It can be realized by using NDelta Unit blocks in parallel to fill at most one nonzero value into the D-FIFO on every clock cycle. Assuming that nonzero elements are uniformly distributed in a delta state vector and using N Delta Unit blocks running in parallel, the latency τDU in clock cycles to process a whole vector is τDU ≈max  D N·d,D·(1−)(5) where Dis the length of the vector, dis the length of the subsection of the vector or the look-ahead window side of the Delta Unit; and is the temporal sparsity defined in Eq. 4. 1BRAM is the standard SRAM memory block on FPGAs; on Xilinx Zynq FPGAs a single BRAM block has 18-bit words and a capacity of 18kb. Fig. 5. Architecture of the EdgeDRNN processing element (PE). Although τDU can be hidden under τm, the latency of computing MxV, τDU becomes a bottleneck of total latency when τDU >τ m, which could happen when an accelerator uses a large number of MAC units to compute small networks. However, in this work, we aim to run large network inference with a small number of MAC units for edge applications, making τDU τm; thus, η=1 is used in EdgeDRNN. The MAC utilization results shown in Section IV.D prove that this choice did not lead to latency bottleneck. The CTRL module contains FSMs that control the PE array. This module generates 80-bit instructions for controlling the Xilinx AXI Datamover IP [31] to fetch RNN parameters. The instruction contains pcol and the burst length calculated from the dimensions of the network stored in configuration registers. C. Processing Element Array Two-dimensional arithmetic unit arrays such as systolic arrays are difficult to be fully utilized in portable edge devices due to scarce on-chip memory resources, the low external memory bandwidth of the system and the limited weight reuse nature of RNNs. In order to fully utilize every PE, a vector PE array is used in EdgeDRNN. Fig. 5 show the internal structure of a PE. The PE has a 16-bit multiplier MUL and two adders, 32- bit ADD0 and 16-bit ADD1. Multiplexers are placed before operands of MUL so that the PE can be reused for both MxV and vector dot products. The multiplexer below ADD0 selects between ’0’ and the BRAM data. ’0’ is chosen when an initialization of BRAM is needed as shown in Fig. 5. ADD1 is responsible for element-wise vector additions. All units are parameterized in the System Verilog RTL and configurable at compile-time to support any fixed-point precision within their designed bit width. The PE supports tanh and sigmoid functions by using look-up tables (LUTs). The input bit width of LUTs is fixed to 16 bits while the output bit width can be set anywhere between 5 (Q1.4) to 9 (Q1.8) bits. Fig. 6 shows the sparse MxV flow. The weight matrices of the GRU-RNN are concatenated following the arrangement shown on the right half of the figure. Biases are appended to the concatenated weight matrix as the first column and an element 1 is appended to each input state vector xtas Fig. 6. Flow of sparse matrix-vector multiplication in EdgeDRNN and the arrangement of GRU weights in the concatenated weight matrix. the first element. The PE array multiplies only nonzero delta state elements with corresponding valid columns. Products are accumulated in the Accumulation Memory (ACC Mem)to compute delta memory vectors Mr,Mu,Mic,Mhc. Products involving br,Wir,Whr are accumulated to Mr;bu,Wiu,Whu to Mu;bc,Wic to Mic;Whc to Mhc. According to the delta update scheme defined by Eq. 2, the appended 1 in the delta state vector xtbecomes 0 after the initial timestep, which means that biases br,bu,bcare only accumulated to the ACC Mem by once and will be skipped by the Delta Unit after the initial timestep. The calculation of activation htafter the MxV is also done by the PE array and stages of this process are shown in Fig. 7. The PE array fetches the delta memory vectors from the ACC Mem to calculate htin 8 pipeline stages. Paths without any operator in any stage are buffered for 1 clock cycle using flipflops. During execution of the activation htgeneration, stages S0∼S2 are executed simultaneously with S5∼S7 to reuse the arithmetic units using time-division multiplexing. Finally, assuming that the DRAM interface can deliver WDRAM bits per RNN clock cycle for weight fetch, the optimum number Kof PEs in the array is determined by the weight precision bit width WWeight. The definition of Kand corresponding theoretical peak throughput, νPeak,isdefined below: K=WDRAM/WWeight, νPeak =2·fpl ·K(6) where fpl is the clock frequency of the programmable logic. For example, the FPGA used in this article has a 64-bit DRAM interface, so, with 16-bit weights, K=8 is optimal. D. Implementation on MiniZed Fig. 8 shows the latest implementation of EdgeDRNN on the $89 MiniZed development board [32] which has a Zynq- 7007S SoC. EdgeDRNN is implemented in the programmable logic (PL). The SoC also has a programmable CPU which is in a module called the Processing System (PS). Data is TABLE I RESOURCE UTILIZATION OF MINIZED USING 5-BIT (Q1.4) LUT transferred between PS and PL through high performance (HP) slave ports while control signals generated by the PS is transferred through general purpose (GP) master ports. The hard IP block, AXI Datamover, is controlled by the PS to fetch weights to the PL from DDR3L memory. Another hard IP block, AXI DMA is controlled by the PS to transfer inputs and outputs of the accelerator. Compared to our previous work [13], we reduced resource utilization by replacing the AXI SmartConnect IP with the AXI Interconnect IP while preserving the same throughput and latency. To further reduce on-chip power, we used the power optimization strategy during implementation in Xilinx Vivado and lower the ARM CPU clock frequency from 667 MHz to 400 MHz for the baremetal version. The peak DRAM read memory bandwidth is 1 GB/s at the 125 MHz clock frequency (64-bits×125 MHz/8-bits/byte). EdgeDRNN can be configured to support 1, 2, 4, 8, 16-bit fixed-point weights and 16-bit fixed-point activations. In this article, EdgeDRNN is configured to support 16-bit activation and 8-bit weights. To fully exploit this HP port bandwidth, we implement K=8 PEs following Eq. 6. Adding more PEs would only leave them idle since weight fetches are limited by the DRAM bandwidth. The AXI-Lite General Purpose (GP)masterportisused for the single-core ARM Cortex-A9 CPU to control the AXIDMA and to write the configuration to the accelerator. Configurations include physical start address of the concatenated weights, delta thresholds, and network dimensions. The PL is driven by single clock domain of 125MHz generated by the PS. Table I shows the resource utilization solely for EdgeDRNN (with 5-bit (Q1.4) LUTs) and for the whole PL after synthesis and implementation. BRAMs are used to store previous state memory in the Delta Units and the accumulation memory in PEs and FIFOs. 8 DSPs are used for the MAC units in the 8 PEs while the remaining DSP in CTRL produces weight column addresses. The most consumed resources are LUTs (72%). This entry-level XC7Z007S FPGA has only 14.4k LUTs. By comparison, the top level XC7Z100 has 19X more LUTs and 11X more BRAM. E. Petalinux OS Integration Xilinx’s Zynq chips are hosted on heterogeneous embedded platforms with a variety of peripherals and communication interfaces. To work with this type of system there are two workflows, bare-metal and embedded OS. The bare-metal workflow is similar to the workflow of conventional microcontrollers. Bare-metal has a set of libraries that establish a very thin software layer over all the hardware resources available in the system and that helps a little during the elaboration of the software that will be deployed in Fig. 7. Stages of GRU activation pipeline in the PE Array. the system; however, detailed knowledge of the hardware is still necessary to ensure correct functionality. The resulting software runs on the PS processor making use of all its computing power since it is the only software running on the core. Bare-metal allows a more dedicated use of the system resources to achieve high performance execution but it offers little flexibility and versatility. The second option is to use an embedded Linux OS provided by Xilinx called PetaLinux. This OS establishes several software layers over the system hardware that simplifies its use and the development of applications that make use of the system’s peripherals like USB, Bluetooth, and Wi-Fi. The Linux system is a preemptive multitasking operating system that can make application development much faster. Since running Linux slightly slows down inference (Sec. IV), users can decide to pay the throughput price of using Linux for faster development time and easier maintenance. For EdgeDRNN, we implemented both systems to meet our various application requirements. IV. EXPERIMENTAL RESULTS We previously developed two EdgeDRNN system-level demonstrations: continuous spoken digit recognition [26] and real-time control of a powered leg prosthetic [7]. Here we report the results of new experiments to measure accuracy, throughput, and power efficiency on the spoken digit task and a new regression task on gas concentration estimation. We also report measurements of embedded Linux implementation of EdgeDRNN. A. Experimental Setup: Training We evaluate the accuracy of DeltaGRU and the hardware performance of EdgeDRNN using this DeltaGRU network on both a classification task using the TIDIGITS [27] dataset and on a regression task using the SensorsGas [30] dataset. 1) Classification: For the classification task, we trained 6 different DeltaGRU network sizes and compared their WER on the TIDIGITS audio digit dataset, evaluated using the greedy decoder. Inputs to the networks consist of 40-dimensional log filter bank features extracted from audio sampled at 20 kHz using a frame size of 25ms and frame stride of 10 ms. We use the Connectionist Temporal Classification (CTC) loss [33] Fig. 8. Top-level diagram of the EdgeDRNN implementation on the MiniZed development board. to handle variable input sequence lengths. The DeltaGRU networks were trained for 50 epochs using a learning rate of 3e-4 and batch size of 32. Following a similar procedure in [26], a continuous spoken digit recognition demo is built using EdgeDRNN to prove the system functionality.2 2) Regression: For the SensorsGas regression task, the input dimension of the network is 14 corresponding to data from the 14 sensors. We adopt a 2-step pretrain and retrain scheme we developed for [7]: 1) We pretrain a cuDNN GRU model on the training set for 100 epochs. The learning rate is 5e-4 and the batch size of 64. 2) We load these parameters into a DeltaGRU network with same size as the cuDNN GRU and retrain for another 10 epochs with learning rate of 3e-3 and batch size of 256. In this step we optimize the deltas for the visible and hidden units. Because the cuDNN GRU model is highly optimized for NVIDIA GPUs, the pretrain step helps to train the network to achieve high accuracy with 5X less time. All networks are trained using the Adam optimizer and quantization-aware training using quantization scheme similar to [19]. To improve accuracy, we use nonlinear functions with the same input and output bit precision as the LUT in the forward phase of the training. In the backward phase, the gradient of the nonlinear function is calculated using 2https://www.youtube.com/watch?v=XyN-jh5yiMI Fig. 9. Mean effective throughput and word error rate evaluated on the TIDIGITS test set as a function of the delta threshold used in both training and inference of a 2L-768H-DeltaGRU network. x=h(is shown as Q8.8 integer values corresponding to 0∼0.5 floating point threshold). the original nonlinear functions in FP32 precision. Training was done with PyTorch 1.2.0 on NVIDIA GPUs running CUDA10 and cuDNN7.6. B. Experimental Setup: Network Implementation After the quantized DeltaGRU is trained for a particular task, a Python script converts the PyTorch network modules into C/C++ hardware network header files. These files contain the network parameters and configuration register values for EdgeDRNN. By including the header files, bare-metal or PetaLinux applications are compiled using the standard cross compiler. The resulting system image is transferred to the QSPI flash (bare-metal) or eMMC storage (PetaLinux) on the MiniZed. In each timestep of the RNN, a feature vector is transferred from the PS to the accelerator using the AXI DMA. For measuring the performance of the accelerator, features are calculated offline on a computer and stored in a header file. For using the accelerator in real-world applications, features such as log filter bank and spike count features for audio, are calculated online by the ARM core in the PS. A flag connected to a PS hardware register is raised at the end of each timestep. C. Accuracy & Throughput 1) Classification: Fig. 9 shows the EdgeDRNN throughput and WER on the TIDIGITS test set versus the used in training and testing of a 2L-768H-DeltaGRU network. is the same for both xand h. With K=8 PEs and PL frequency fpl =125MHz, EdgeDRNN has a theoretical peak throughput of 2Kfpl =2 GOp/s. At =0, there is still a speedup of about 2X from natural sparsity of the delta vectors. Higher leads to better effective throughput, but with gradually increasing WER. The optimal point is at =64 (0.25), just before a dramatic increase of WER, where EdgeDRNN achieves an effective throughput around 20.2 GOp/s with 1.3% WER. WER and throughput of smaller tested networks are shown in Table II. The 5-bit (Q1.4) LUT was used for this task and did not lead to accuracy loss compared to the network running on CPU with FP32 nonlinear functions. 2) Regression: In this regression task, we evaluate the impact of using different delta thresholds for xand hon the accuracy results of a 2L-256H-DeltaGRU model evaluated Fig. 10. RMSE (smaller is better) and R2(larger is better) versus xand hof the 2L-256H-DeltaGRU model evaluated on the SensorsGas test set. Fig. 11. Temporal sparsity xand hversus xand hof the 2L-256H-DeltaGRU model evaluated on the SensorsGas test set. on the SensorsGas testset. Fig. 10 and Fig. 11 show respectively the regression accuracy and temporal sparsity versus x and h. The pretrained 2L-256H-GRU network without using a delta threshold, achieves a root-mean-square error (RMSE) of 0.995 and coefficient of determination (R2) of 0.976. This accuracy was achieved using 5-bit (Q1.4) LUTs, which gave the lowest RMSE out of all other LUT bit precision values. Similar to the results for the classification task, Fig. 10 shows that the accuracy degrades when larger delta thresholds are used. Fig. 11 shows that the sparsity levels of xand h are heavily influenced by their corresponding delta thresholds. The accuracy degrades faster with increasing xfor a fixed hthan with increasing hfor a fixed x.xhas a minor TABLE II WORD ERROR RATE,LATENCY AND THROUGHPUT OF EDGEDRNN RUNNING WITH BARE-METAL ON DELTAGRU NETWORKS TRAINED WITH =64, β=1E-5 TABLE III LATENCY AND THROUGHPUT OF EDGEDRNN RUNNING WITH PETALINUX ON DELTAGRU NETWORKS TRAINED WITH =64, β=1E-5 impact on hand vice versa. The results from this regression task indicate that propagating changes more often in input states is more important than propagating changes in hidden states. By exploiting this phenomenon, we get the optimal point (x,y)=(4,8), where the RMSE and R2are 1.078 and 0.972 respectively. With x=59.7% and h=69.2%, the latency of the optimal model is 206 μs. In comparison, Jetson TX2 runs a 4.8X smaller 1L-200H-GRU network in 271μs [30]. D. Theoretical & Measured Performance Eq. 7 gives the estimated mean effective throughput νEff of EdgeDRNN running a DeltaGRU layer: νEff =Op τm+τa ≈23HI+3H2(L−1)+3H2L (3HI+3H2(L−1))(1−x)+3H2L(1−h) Kfpl +3H Kfpl (7) where Op is the number of operations in a DeltaGRU layer per timestep, τmthe latency of MxV, τathe latency of remaining operations to produce the activation, and the other variables are defined as in Eqs. 4 and 6. Table II compares the Eq. 7 predictions with benchmark results of different DeltaGRU network sizes running on Edge- DRNN. Estimated results calculated from Eq. 7 are close to measured results and the maximum relative error between them is smaller than 7.1%. Thus Eq. 7 can be used to estimate EdgeDRNN performance for a particular RNN network size. On average, EdgeDRNN can run all tested networks under 0.54 ms latency corresponding to 20.2 GOp/s effective throughput for the 2L-768H-DeltaGRU. E. Performance in PetaLinux For performance measurements on the PetaLinux-based system, we implemented an application that performs the same operations as the software implemented for bare-metal but use the AXI DMA driver included in the OS. Table III shows the latency and performance results for the 6 networks used in this work. The minimum and mean latency numbers in the PetaLinux version are respectively up to 3.4% and 11.3% higher than the numbers obtained for the bare-metal version. Because the minimum PetaLinux latency is nearly the same as the bare-metal latency, the big difference in maximum latency numbers between the PetaLinux and the bare-metal version is due to CPU contention for other tasks running on the PS that lock the single PS DDR controller. EdgeDRNN fetches weights from HP ports (Fig. 8) that are routed through the PS DDR controller. (The FPGA’s ACP interface should not be used to access DRAM memory under PetaLinux because it is connected directly to the L2 cache on the ARM core where the OS runs. This configuration creates conflicts and the performance of the system is seriously compromised.) Under PetaLinux, the HP interface should be used to connect any module placed on the PL that requires direct access to the DRAM memory. To understand the impact of CPU load and CPU DRAM access on the RNN inference time, we wrote a small program that loops over a memory array and is designed to trigger L2 cache misses. We used two different memory array sizes to study the effect of cache misses since the large memory array causes more L2 cache misses. Table IV shows that the impact on RNN latency is minor: a small network takes about 50% longer to run with either memory array size, and a large RNN isonlysloweddownby10%. The RNN inference time varies between 50 us to 0.5ms across the different network sizes. During this inference time, the PS is free for other tasks (e.g. computing features) and only needs to check if the RNN update is finished when these tasks are completed. F. Po w e r M e a su re m e n t Table V shows the power breakdown of the MiniZed system. The total power is measured by a USB power meter; the PS, PL and static power is estimated by the Xilinx Power Analyzer. The whole system burns at most 2.3W but the EdgeDRNN