Fuzzing Guided by Bayesian Program Analysis (Paper Artifact)
Full text
Fuzzing Guided by Bayesian Program Analysis (Paper Artifact) All scripts and source code of this artifact are released under the MIT License. List of Claims We provide complete reproduction steps for the following illustrations in the paper: Figure 9 Figure 10 Figure 11 Figure 12 The complete table, including Table 3 and Table 4, presents the median TTE (time-to-exposure) of each fuzzer over 10 runs on 24 real-world programs We also provide detailed information for each newly discovered vulnerability that has not yet been assigned a CVE, which we have submitted to the developers and present in Table 5 and Table 6. Download, Installation, and Sanity-testing Due to the efficiency requirements of fuzzing, our artifact currently only supports running on Linux x86-64/AMD. We provide a Docker image that you can pull and rename using the following command: docker pull jjleo/bayzzer-artifact:latest docker tag jjleo/bayzzer-artifact:latest bayzzer-artifact:latest You also need to install the following Python packages: pip3 install numpy matplotlib scipy Please download our scripts archive bayzzer-artifact-scripts.tar.gz from Zenodo and extract it: tar -xzvf bayzzer-artifact-scripts.tar.gz
Please note that all scripts must be executed from within the bayzzer-artifact-scripts directory. You need to execute the following commands to ensure that the fuzzer runs properly: echo core | sudo tee /proc/sys/kernel/core_pattern echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor Next, enter the script folder and use the following command to test whether fuzzing can run normally in the container (this process will use CPU cores 0-3 and the entire runtime is expected to take no more than 1 hour): cd bayzzer-artifact-scripts python3 run.py all all 10 1 4 sudo chown -R $USER:$USER output If the program runs without errors and non-empty fuzzer_stats files exist in all output/runtime_1/out/<benchmark>/<fuzzer>/default/ folders, then the test is running normally. Next, run the following command to ensure that the vulnerability reproduction scripts can run normally: python3 copy_results.py output results python3 copy_overhead.py output results/overhead python3 reproduce.py results 1 The program will enter an interactive Docker container. Enter the following command: /reproduce/start.sh exit This will exit the interactive Docker container. Enter the following command: sudo chown -R $USER:$USER results If the above programs run without errors and the content of results/log/0/gif2tga.san is similar to the following (it won't be exactly the same, but there should not be a large number of <null>):
{ "bayzzer": { "1488": [ "heap-buffer-overflow", "WritePixel->DecodeGifImg->LoadGif" ], "12567": [ "heap-buffer-overflow", "WritePixels->DecodeGifImg->LoadGif" ], "12888": [ "heap-buffer-overflow", "GifIndexToTrueColor->WritePixels->DecodeGifImg" ], "12977": [ "stack-overflow", "DecodeGifImg->LoadGif->main" ] }, "prospector": { "1474": [ "heap-buffer-overflow", "WritePixel->DecodeGifImg->LoadGif" ], "12514": [ "heap-buffer-overflow", "WritePixels->DecodeGifImg->LoadGif" ], "12834": [ "heap-buffer-overflow", "GifIndexToTrueColor->WritePixels->DecodeGifImg" ], "12921": [ "stack-overflow", "DecodeGifImg->LoadGif->main" ] }, "ffapp": { "1386": [ "heap-buffer-overflow", "WritePixel->DecodeGifImg->LoadGif" ], "12278": [ "heap-buffer-overflow", "WritePixels->DecodeGifImg->LoadGif" ],
"12599": [ "heap-buffer-overflow", "GifIndexToTrueColor->WritePixels->DecodeGifImg" ], "12686": [ "stack-overflow", "DecodeGifImg->LoadGif->main" ] }, "funfuzz": { "1686": [ "heap-buffer-overflow", "WritePixel->DecodeGifImg->LoadGif" ], "13000": [ "heap-buffer-overflow", "WritePixels->DecodeGifImg->LoadGif" ], "13446": [ "heap-buffer-overflow", "GifIndexToTrueColor->WritePixels->DecodeGifImg" ], "13553": [ "stack-overflow", "DecodeGifImg->LoadGif->main" ] }, "aflpp": { "1663": [ "heap-buffer-overflow", "WritePixel->DecodeGifImg->LoadGif" ], "12989": [ "heap-buffer-overflow", "WritePixels->DecodeGifImg->LoadGif" ], "13457": [ "heap-buffer-overflow", "GifIndexToTrueColor->WritePixels->DecodeGifImg" ], "13588": [ "stack-overflow", "DecodeGifImg->LoadGif->main" ] }, "ablation": {
"1468": [ "heap-buffer-overflow", "WritePixel->DecodeGifImg->LoadGif" ], "12522": [ "heap-buffer-overflow", "WritePixels->DecodeGifImg->LoadGif" ], "12853": [ "heap-buffer-overflow", "GifIndexToTrueColor->WritePixels->DecodeGifImg" ], "12945": [ "stack-overflow", "DecodeGifImg->LoadGif->main" ] } } Then it indicates that the vulnerability reproduction scripts can run normally. Finally, run the following command to ensure that the illustrations generation scripts can run normally: python3 RQ1.py results-origin/log python3 RQ2.py results-origin/overhead python3 RQ3.py results-origin/log If the program runs without errors (there may be font missing issues, which is normal) and successfully generates figure_9.pdf, figure_10.pdf, figure_11.pdf, figure_12.pdf, and complete_table_3_4.csv in the bayzzer-artifact-scripts directory with the same data as in the paper, then it indicates that the illustrations generation scripts can run normally. Evaluation Instructions First, make sure you have executed the following commands to ensure that the fuzzer can run properly: echo core | sudo tee /proc/sys/kernel/core_pattern echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor Please note that all scripts must be executed from within the bayzzer-artifact-scripts directory.
Our experiments set a time limit of 60 hours for each fuzzing run and repeated it 10 times. If you need to reproduce all experimental results, you can run: python3 run.py all all 216000 10 <CPU> sudo chown -R $USER:$USER output Here, <CPU> indicates the number of CPU cores to be used for parallel experiments (from core 0 to core <CPU>-1). Reproducing all experiments as reported in the paper will require approximately 9.86 CPU-years of computation. If you do not have access to sufficient computational resources to reproduce all experiments within the artifact evaluation timeframe, please let us know; we can provide access to a server with 256 CPU cores to help with reproduction. Note: Our method demonstrates improved performance over longer evaluation periods (particularly after 40 hours). Reducing the experimental time limit may impact the observed results and may not fully reflect the effectiveness of our approach. If you are unable to reproduce the full results due to resource limitations, we encourage you to contact us for assistance or to discuss alternatives. Next, run the following command: python3 copy_results.py output results python3 copy_overhead.py output results/overhead python3 reproduce.py results 10 The program will enter an interactive Docker container. Enter the following command: /reproduce/start.sh exit This will exit the interactive Docker container. Enter the following command: sudo chown -R $USER:$USER results Finally, use the following command to generate illustrations: python3 RQ1.py results/log python3 RQ2.py results/overhead python3 RQ3.py results/log This script will generate figure_9.pdf, figure_10.pdf, figure_11.pdf, figure_12.pdf, and complete_table_3_4.csv in the bayzzer-artifact-scripts directory.
We provide the original experimental data in results-origin. You can use the following command to generate illustrations based on the original data: python3 RQ1.py results-origin/log python3 RQ2.py results-origin/overhead python3 RQ3.py results-origin/log Finally, we provide specific information on each newly discovered vulnerability that has not yet been assigned a CVE in Table 5 and Table 6. Vulnerabilities that have already been assigned CVEs can be searched directly online. Here, we only present the Bug Only vulnerabilities. Program Vulnerabilities draco https://github.com/google/draco/issues/1100 https://github.com/google/draco/issues/1102 https://github.com/google/draco/issues/1103 https://github.com/google/draco/issues/1105 bloaty https://github.com/google/bloaty/issues/398 libyaml https://github.com/yaml/libyaml/issues/312 librdkafka https://github.com/confluentinc/librdkafka/issues/5091 https://github.com/confluentinc/librdkafka/issues/5092 Additional Artifact Description We will first provide a detailed introduction to the architectural organization and usage documentation of the scripts in the artifact, then introduce the architectural organization of the Docker image and how to modify it to test new programs. Scripts We first introduce the usage of each script in bayzzer-artifact-scripts:
├── analysis.py # used for reproducing crashes ├── asan.crash.json # used for reproducing crashes ├── bug_id.csv # used for classifying crashes ├── copy_overhead.py # used for copying overhead logs ├── copy_results.py # used for copying fuzzing results ├── reproduce.py # used for reproducing crashes ├── results-origin # original data from the paper ├── RQ1.py # used for generating illustrations in RQ1 ├── RQ2.py # used for generating illustrations in RQ2 ├── RQ3.py # used for generating illustrations in RQ3 ├── run.py # used for running fuzzing ├── runtime # corpus and scripts for running fuzzing └── venn.py # used for generating Venn graphs Next, we will explain the core scripts used in the reproduction process. Please note that all scripts must be executed from within the bayzzer-artifact-scripts directory. Reproducing Fuzzing Process Use the following command to reproduce the fuzzing process in the experiment: python3 run.py <fuzzer> <benchmark> <time> <repeat> <CPU> where <fuzzer> can be one of: Bayzzer, Prospector, FishFuzz, FunFuzz, AFL++, Ablation, or all to run all fuzzers sequentially. <benchmark> can be one of exiv2, tiffsplit, mp3gain, pdftotext, flvmeta, tcpdump, cflow, lame, jhead, mujs, wav2swf, objdump, sqlite3, gdk-pixbuf-pixdata, MP4Box, lou_checktable, nasm, gif2tga, tcpprep, tic, mp42aac, nm-new, imginfo, jq or all to run all benchmarks sequentially. <time> indicates the time limit (in seconds), <repeat> indicates the number of repetitions, and <CPU> indicates the number of CPU cores to be used for parallel experiments (from core 0 to core <CPU>-1). Each run is assigned a dedicated Docker container bound to a specific CPU core. After the run completes, the results will be saved in the output folder under the bayzzer-artifact-scripts directory. The results from the x-th run (with x starting from 1) of <fuzzer> on <benchmark> will be stored in output/runtime_<x>/out/<benchmark>/<fuzzer>/default. Since the Docker container uses the root user, you need to use the following command to restore permissions: sudo chown -R $USER:$USER output
Copy Crashing Inputs Use the following command to copy crashing inputs during fuzzing process: python3 copy_results.py <output> <results> Here, <output> and <results> denote the fuzzing output directory and the directory to which the target is copied, respectively. The results from the i-th run (with i starting from 0) of <fuzzer> on <benchmark> will be copied to <results>/<fuzzer>/<benchmark>/<i>/crashes. Copy Overhead Logs Use the following command to copy overhead logs during fuzzing process of Bayzzer: python3 copy_overhead.py <output> <results> Here, <output> and <results> denote the fuzzing output directory and the directory to which the target is copied, respectively. The results from the i-th run (with i starting from 0) of Bayzzer on <benchmark> will be copied to <results>/<benchmark>/<i>.log. Reproducing Crashing Inputs Use the following command to reproduce crashing inputs during fuzzing process: python3 reproduce.py <results> <repeat> Here, <results> and <repeat> denote the copied crashing inputs and the number of repetitions, respectively. Please note that <results> must be located in the bayzzer-artifact-scripts directory. This script will create a Docker container for reproducing the crashing inputs. The program will enter this interactive container, where you can input the following command to start the reproduction process and exit the container once reproduction is complete: /reproduce/start.sh exit The reproduction logs will be stored in <results>/log. The results from the x-th run (with x starting from 0) of each fuzzer on <benchmark> are stored in <results>/log/<x>/<benchmark>.san.