Open-source GEMM hardware kernels generator: toward numerically-tailored computations

Open-Source GEMM Hardware Kernels Generator: Toward Numerically-Tailored Computations Louis Ledoux∗†, Marc Casas∗† ∗Barcelona Supercomputing Center, Barcelona, Spain †Universitat Polit` ecnica de Catalunya, Barcelona, Spain E-mail: {louis.ledoux, marc.casas}@bsc.es Keywords—GEMMs, matrix-matrix-multiply, full stack framework, automated pipeline, flopoco, OpenCAPI, OpenBLAS, High Performance Computing, approximate/trans/extended precision. I. EXTENDED ABSTRACT Many scientific computing problems can be reduced to Matrix-Matrix Multiplications (MMM), making the General Matrix Multiply (GEMM) kernels in the Basic Linear Algebra Subroutine (BLAS) of interest to the high-performance computing community. However, these workloads have a wide range of numerical requirements. Ill-conditioned linear systems require high-precision arithmetic to ensure correct and reproducible results [1]. In contrast, emerging workloads such as deep neural networks, which can have millions up to billions of parameters, have shown resilience to arithmetic tinkering [2] and precision lowering [3]. General purpose arithmetic units and computer formats such as the IEEE754 standard naturally underperform in this vaste land of scenarios. We propose the generation of numerically tailored circuits where the necessary and sufficient internal precision is generated to target the computations requirements in terms of numerical quality while improving the energy cost. A. Open Source SW/HW co-designed framework for numerically tailored MMMs As depicted by Fig. 1, our framework is composed of two distinct phases, the prior Hardware generation flow and the runtime execution flow. Because MMMs are basically made of arbitrary long dot products, we design a custom Fused Dot Product (FDP) operator that is agnostic to the computer format and supports posit, IEEE754, and bfloat16 variations, while never rounding between two accumulations. The intermediate precision of the fixed-point accumulator used in the dot-product is a key aspect of this work, and is xcvu3p-ffvc1517-2 OCXL kernel module DMA Engine libocxl OpenCAPI link ; DMA transfer @ ~20GB/s I/O control; pointer passing; start dma; register pooling A priori Hardware generation flow Runtime execution flow address translation liboc-accel kernel & HOST HW OS & userspace malloc() cast() gemm() OpenBLAS user app pytorch numpy POWER9 HOST Shared DDR4 AlphaDATA 9v3 Functional specification 1 3 4 6 5 2 oc-accel OpenCAPI Transceivers <5,-30,2> Systolic Array 32x31 bfloat16 Performance specification flopoco EOB Signy Signx Iy.Fy Scaley Scalex isNaNy isNaNx unsigned multiplier unsigned adder ωS 11 1 1 1 1 1 0 sign extension RCA S C RCA S C RCA S C isNaNoA CEOBQ CSA radix-2k ωQ 1 k-1 FTZ 0 1 0 1 0 1 ωF+1 Ix.Fx 2ωF+2 ωS+1 2ωF+2 shi val gen part select ωShi 1 1 barrel shi pad LSBMSB OVF 1 too_big too_small config.txt hwgen.py N,M LSB,MSB,OVF Arithmetic Chip Freq Systolic Array Generation buses wrapping VUP.cpp FDP.cpp PE PE PE PE PE PE PE PE PE A2S3 A2S3 A2S3 L2AL2AL2A columns B rows A SOB EOB columns C A2S3 A2S3 A2S3 Legend: HSSD data input data (input format) intermediate data (S3 format) control output data (output format) FF_delay LUT_delay C8_delay Adder model A B C D Fig. 1. Overview of the 2 phases framework. Left is Runtime execution flow and right is Hardware generation flow. 27,7× 5× 15,1× 5,6× Fig. 2. Sea Surface Height computation comparing IEEE-754 double-, quadpecision FMAs and a 91-bit FDP wrt numerical quality and power consumption. configurable through the length of the scratchpad delimited by the parameters MSB (Most Significant Bit) and LSB (Least Significant Bit). We leverage the automated pipeline feature of flopoco [4] which is an effective tool for efficiently exploring the wide range of functional specifications along with performance specifications to produce MMM kernels with the necessary basic elements (LUTs, FFs, Carry chains, DSPs) for a targeted (chip, frequency)couple (see Fig. 1B ). The essence of this work is to make intermediate precision tweakings from the hardware accessible to high-end software code as transparent as possible. We achieve that by taking into account that many HPC codes rely on BLAS libraries to perform MMM operations. Such libraries receive the function call to perform a GEMM and dispatch adequately to the underlying hardware at their disposal. B. HPC workloads results We experiment with two families of real HPC workloads with contrasting numerical requirements, namely Artificial Intelligence (AI) and Sea Surface Height (SSH), whose respective results can be observed in Fig. 3 and Fig. 2. Fig. 3. Top1 Accuracy vs validation dataset inference Energy cost for various combinations of datasets,models,computer formats, and accumulators. For the SSH computation, the results obtained with 64-bit and 128-bit FPUs exhibit decreasing reproducibility as the vector size increases. In contrast, our 91-bit ⟨ovf : 30, msb : 30, lsb : 30⟩FDP maintains reproducibility for all vector sizes without deviation. Our proposed FDP consistently exhibits 52 correct bits, which is at least 5×and 27.7×more than quad-precision and double-precision. Our measurements on VU3P-2 FPGA at 200MHz show that the units power consumption are 0.266, 0.549, and 0.491 watts for double-precision FMA, quad-precision FMA, and the 91-bit FDP, respectively. For all evaluated sizes, the 91-bit FDP yields at least 5.6×and 15.1×more correct bits for the same wattage as quad-precision and double-precision FMAs, respectively. For AI workloads, we employ Pytorch as a base framework and link it to our modified OpenBLAS. We use popular neural network models such as ResNet18, ResNet34, ResNet50, DenseNet121, DenseNet161, DenseNet169, and VGG11 with batch normalization, and evaluate them on the CIFAR-10 and ImageNet datasets. To measure power consumption and accuracy, we use the BrainFloat16 and IEEE-754 32-bit formats for our computations with a large variety of accumulators varying their OV F ,MSB, and LSB parameters. Fig. 3 shows the relationship between power consumption and accuracy for different accumulator and arithmetic combinations. For example, if 84% Top1 accuracy is satisfying for Imagenet with Resnet50, the most suited arithmetic/accumulator combination is IEEE-754 32bit/⟨ovf : 9, msb : 6, lsb :−20⟩represented by a light purple hexagon as all other markers are either on the right or below. C. Conclusion Overall, our work highlights the importance of numerically tailored accumulators for reproducibility in scientific computing applications. Our results provide valuable insights into the trade-offs between power consumption and accuracy, and we believe that our results have the potential to inform the design of future AI and scientific computing systems, and we encourage other researchers to explore the possibilities of low precision accumulators using our open-source framework. II. ACKNOWLEDGMENT Marc Casas is supported by Grant RYC-2017-23269 funded by MCIN/AEI/ 10.13039/501100011033 and by “ESF Investing in your future”. REFERENCES [1] D. Bailey and J. M. Borwein, “High-Precision Computation and Mathematical Physics,” in Proceedings of XII Advanced Computing and Analysis Techniques in Physics Research — PoS(ACAT08). Erice, Italy: Sissa Medialab, Oct. 2009, p. 014. [Online]. Available: https://pos.sissa.it/070/014 [2] J. Johnson, “Rethinking floating point for deep learning,” arXiv:1811.01721 [cs], Nov. 2018, arXiv: 1811.01721. [Online]. Available: http://arxiv.org/abs/1811.01721 [3] M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, and Y. Bengio, “Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or - 1,” arXiv:1602.02830 [cs], Feb. 2016, arXiv: 1602.02830. [Online]. Available: http://arxiv.org/abs/1602.02830 [4] M. Istoan and F. de Dinechin, “Automating the pipeline of arithmetic datapaths,” in Design, Automation & Test in Europe Conference & Exhibition (DATE 2017), Lausanne, Switzerland, Mar. 2017. [Online]. Available: https://hal.inria.fr/hal-01373937 Louis Ledoux received his BSc degree in 2016 in Computer Science from Universit´ e de Rennes1, France. The following years, he pursued his MSc degree in parallel with an Engineer diploma from ´ Ecole Sup´ erieure d’Ing´ enieurs de Rennes (ESIR). He concluded in 2018 his studies in Rennes with a position of Hardware Engineer at b<>com, a national research laboratory. This position allowed him to experiment with the first FPGAs in the cloud and their virtualizations. Since 2018, he has been a PhD candidate at the Computer Architecture departments of Barcelona Supercomputing Center (BSC) and Universitat Polit` ecnica de Catalunya (UPC), Spain.