FAIL: FuzzJsonUnmarshal (0.04s) — FAIL: FuzzJsonUnmarshal (0.00s) fuzz_test.go:26: json: cannot unmarshal string into Go value of type map[string]string

FAIL exit status 1

This is the expected error in case the input to `encoding/json.Unmarshal()` is valid json but cannot be parsed, so let’s ignore that:

```golang
package testfuzzing

import (
        "encoding/json"
        "strings"
        "testing"
)

func FuzzJsonUnmarshal(f *testing.F) {
        f.Add([]byte(`{"foo":"bar"}`))
        f.Fuzz(func(t *testing.T, data []byte) {
                if !json.Valid(data) {
                        t.Skip()
                }
                m := make(map[string]string)
                err := json.Unmarshal(data, &m)
                if err != nil {
                        if !strings.Contains(err.Error(), "into Go value of type") {
                                t.Fatal(err)
                        }
                }
        })
}

Now when we run it again, we see that it runs without throwing a fatal error:

fuzz: elapsed: 0s, gathering baseline coverage: 0/434 completed
fuzz: elapsed: 0s, gathering baseline coverage: 434/434 completed, now fuzzing with 8 workers
fuzz: elapsed: 3s, execs: 363435 (121114/sec), new interesting: 0 (total: 434)
fuzz: elapsed: 6s, execs: 813593 (150060/sec), new interesting: 0 (total: 434)
fuzz: elapsed: 9s, execs: 1270779 (152423/sec), new interesting: 1 (total: 435)
fuzz: elapsed: 12s, execs: 1696551 (141835/sec), new interesting: 1 (total: 435)
fuzz: elapsed: 15s, execs: 2115315 (139603/sec), new interesting: 1 (total: 435)
fuzz: elapsed: 18s, execs: 2621051 (168623/sec), new interesting: 1 (total: 435)
fuzz: elapsed: 21s, execs: 3060560 (146528/sec), new interesting: 1 (total: 435)
fuzz: elapsed: 24s, execs: 3446854 (128750/sec), new interesting: 1 (total: 435)
fuzz: elapsed: 27s, execs: 3842040 (131743/sec), new interesting: 2 (total: 436)
fuzz: elapsed: 30s, execs: 4246125 (134683/sec), new interesting: 2 (total: 436)

Bugs to find

Fuzzing in Golang can be useful to find both coding issues and logical errors. Both of these types of bugs can be reliability issues and security issues dependending on the context and prerequisites for triggering the bug. In general, with coding issues we think about panics which include:

• Index out of range.
• Slice bounds out of bounds.
• Nil-dereference
• Out of memory
• Interface conversion

Logic bugs require the developers to formalize high-level assumptions about the code; If the code does not conform to these assumptions, then it has a logical bug. This can be both easy and hard to do, and often, developers will do this without thinking much about it. Some systems have clear assumptions. Take for example an authorization system which involves verifying that a username/password combination is correct. The code should only return “true” if that is the case, and if an untrusted user can make the code return “true” when passing a username/password combination that is incorrect, then the system is likely to have a logical bug.

Fuzzing has found bugs from both categories in production-level open source projects, and some of these bugs have had security implications.

Structured Go Fuzzing

Often, when fuzzing, we want to fuzz an API or method that takes as input a struct rather than a primitive, and our job is to transform the primitive data into a struct. This can be trivial if the struct is small. Consider the following fuzzer:

package main

import (
        "testing"
)

type User struct {
        Name string
        Password string
}

func FuzzUserAuth(f *testing.F) {
        f.Fuzz(func(t *testing.T, name, password string) {
                user := &User{
                        Name: name,
                        Password: password,
                }
                // test authentication with the user:
                authenticate(user)
        })
}

This is simple because we can pass the values directly to the struct fields when creating &User{}. This is a rare example, and often structs - especially in cloud-native applications - contain much more information that makes it tedious to manually specify every field. For that purpose you can use the go-fuzz-headers library with your fuzzers. go-fuzz-headers has an API called GenerateStruct() which does the heavy lifting of adding values to a struct based on the input from the fuzzing engine.

Let’s demonstrate that with a more complex struct. Say we want to create and randomize the User struct again, but now it is a bit more complex:

type User struct {
        Name string
        Password string
        Family []*User
        Education *Education
}

type School struct {
        Name string
        Address string
        City string
}

type SchoolAttendance struct {
        School *School
        YearStart int
        YearEnd int
}

type Education struct {
        Schools []*SchoolAttendance
        CurrentlyStudying bool
}

go-fuzz-headers.GenerateStruct can do the heavy lifting for us with just a few lines of code:

package main

import (
        "testing"
        fuzz "github.com/AdamKorcz/go-fuzz-headers-1"
)

type User struct {
        Name string
        Password string
        Family []*User
        Education *Education
}

type School struct {
        Name string
        Address string
        City string
}

type SchoolAttendance struct {
        School *School
        YearStart int
        YearEnd int
}

type Education struct {
        Schools []*SchoolAttendance
        CurrentlyStudying bool
}

func FuzzUserAuth(f *testing.F) {
        f.Fuzz(func(t *testing.T, data []byte) {
                user := &User{}
                c := fuzz.NewConsumer(data)
                c.GenerateStruct(user)
        })
}

The fuzzer may take a few minutes to generate meaningful Users, but it will then start to transform the data parameter into structs like this one:

{
  "Name": "0",
  "Password": "000000000000000000000000000000000000000000000000",
  "Family": [
        {
            "Name": "000",
            "Password": "0000",
            "Family": [],
            "Education": {
                "Schools": [],
                "CurrentlyStudying": false
            }
        },
        {
            "Name": "00--0--000",
            "Password": "0=-=000",
            "Family": [],
            "Education": {
                "Schools": [
                    {
                        "Name": "AS{P",
                        "Address": "NBAJHSD",
                        "City": "00",
                    }
                ],
                "CurrentlyStudying": false
            }
        }
    ],
    "Education": {
        "Schools": [
            {
            "Name": "BBBB",
            "Address": "BBB",
            "City": "CCC",
            }
        ],
        "CurrentlyStudying": false
    }
}

Python fuzzing

This section introduces fuzzing for Python projects.

Atheris fuzzing engine

The most popular fuzzing engine for Python is the Atheris fuzzer developed by Google https://github.com/google/atheris . This fuzzer is built on top of libFuzzer, and, therefore, has a lot of similarities to libFuzzer. For example, the common set of command line arguments to libFuzzer also applies to Atheris.

Example: python hello-world fuzzing

The following example illustrates an initial set up of Atheris and a complete run for finding a bug.

import sys
import atheris

@atheris.instrument_func
def fuzz_hello_world(str1):
    if len(str1) < 5:
        return
    if str1[0] != 'H':
        return
    if str1[1] != 'I':
        return
    if str1[2] != 'T':
        return
    if str1[3] != '!':
        return
    if str1[4] != '!':
        return


    raise Exception("Fuzz success")

def TestOneInput(fuzz_bytes):
    fdp = atheris.FuzzedDataProvider(fuzz_bytes)
    s1 = fdp.ConsumeUnicodeNoSurrogates(10)
    fuzz_hello_world(s1)

def main():
    atheris.Setup(sys.argv, TestOneInput)
    atheris.Fuzz()

if __name__ == "__main__":
    main()

In order to install Atheris we can simply use pip as Atheris is published as a pypi package here . Once installed, we have all that is ready to build and run Python fuzzers.

$ python3 -m virtualenv .venv
$ . .venv/bin/activate
$ (.venv) python3 -m pip install atheris
$ (.venv) python3 ./helloworld.py 
INFO: Using built-in libfuzzer
WARNING: Failed to find function "__sanitizer_acquire_crash_state".
WARNING: Failed to find function "__sanitizer_print_stack_trace".
WARNING: Failed to find function "__sanitizer_set_death_callback".
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 810532610
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: A corpus is not provided, starting from an empty corpus
#2      INITED cov: 2 ft: 2 corp: 1/1b exec/s: 0 rss: 36Mb
#211    NEW    cov: 4 ft: 4 corp: 2/7b lim: 6 exec/s: 0 rss: 36Mb L: 6/6 MS: 4 ChangeBit-ShuffleBytes-InsertByte-InsertRepeatedBytes-
#2750   NEW    cov: 6 ft: 6 corp: 3/13b lim: 29 exec/s: 0 rss: 36Mb L: 6/6 MS: 4 CopyPart-ShuffleBytes-CopyPart-ChangeBinInt-
#3672   NEW    cov: 8 ft: 8 corp: 4/19b lim: 38 exec/s: 0 rss: 36Mb L: 6/6 MS: 2 ChangeByte-ChangeBinInt-
#21188  NEW    cov: 10 ft: 10 corp: 5/26b lim: 205 exec/s: 0 rss: 36Mb L: 7/7 MS: 1 InsertByte-
#22169  REDUCE cov: 10 ft: 10 corp: 5/25b lim: 212 exec/s: 0 rss: 36Mb L: 6/6 MS: 1 EraseBytes-
#24377  REDUCE cov: 12 ft: 12 corp: 6/32b lim: 233 exec/s: 0 rss: 36Mb L: 7/7 MS: 3 ChangeBit-CopyPart-InsertByte-
#26743  REDUCE cov: 12 ft: 12 corp: 6/31b lim: 254 exec/s: 0 rss: 36Mb L: 6/6 MS: 1 EraseBytes-

 === Uncaught Python exception: ===
Exception: Fuzz success
Traceback (most recent call last):
  File "/home/dav/atheris-example/./helloworld.py", line 25, in TestOneInput
    fuzz_hello_world(s1)
  File "/home/dav/atheris-example/./helloworld.py", line 18, in fuzz_hello_world
Exception: Fuzz success

==164529== ERROR: libFuzzer: fuzz target exited
SUMMARY: libFuzzer: fuzz target exited
MS: 3 CopyPart-ChangeByte-CopyPart-; base unit: 86c0d496ba0c76e9c196190433f7b393973deaa4
0x59,0x48,0x49,0xd4,0x21,0x21,
YHI\324!!
artifact_prefix='./'; Test unit written to ./crash-042ee5ba3acacff40f24201d69df2931ff92a0c7
Base64: WUhJ1CEh

n this case, we can see from the stack trace that Atheris ran into an issue that was an uncaught exception. Furthermore, we see the exception message itself which is the one from our Python module “Fuzz success”. It took Atheris a second or two to find the exception, meaning it generated a string that has the first five characters equals to HIT!!.

Example: pyyaml fuzzing

Consider the popular Python library PyYAML https://pypi.org/project/PyYAML used for parsing and serialising YAML files. This is an ideal target for fuzzing in that parsers and serializers are traditionally good targets for code exploration techniques. Following the official pyyaml documentation we can see the load function is used for serialising, and this function accepts two parameters: the stream used for serialisation (which can be a buffer of bytes) and a Loader object, which is a type defined in pyyaml itself. This is an ideal target for fuzzing since the binary buffer is well correlated with what the fuzz engine itself provides. In order to fuzz this code we would write our fuzzer as follows:

import sys
import atheris

with atheris.instrument_imports():
  import yaml


@atheris.instrument_func
def TestOneInput(input_bytes):
  try:
    context = yaml.load(input_bytes, Loader=yaml.FullLoader)
  except yaml.YAMLError:
    return


def main():
  atheris.Setup(sys.argv, TestOneInput)
  atheris.Fuzz()

if __name__ == "__main__":
  main()

We can dissect the source code as follows:

with atheris.instrument_imports():
  import yaml

Is used to import the target library yaml in a manner where the fuzzing engine Atheris instruments the underlying Python code, indicated by the with atheris.instrument_imports(). The effect of importing yaml in this manner is coverage-feedback instrumentation will be added to the yaml module. Effectively, this makes the fuzzing much more efficient than if we were to not import yaml in the same way.

The code at the bottom of the fuzz harness initiates the fuzzing:

def main():
  atheris.Setup(sys.argv, TestOneInput)
  atheris.Fuzz()
if __name__ == "__main__":
  main()

atheris.Setup specifies the fuzz harness entrypoint, which in this case is TestOneInput and then atheris.Fuzz() launches the fuzzing. atheris.Fuzz() will never return, rather, it launches the regular libFuzzer fuzzing engine behind the scenes which will run the infinite fuzzing loop.

The fuzzing entrypoint itself is the TestOneInput function:

@atheris.instrument_func
def TestOneInput(input_bytes):
  try:
    context = yaml.load(input_bytes, Loader=yaml.FullLoader)
  except yaml.YAMLError:
    return

The input_bytes argument corresponds to the raw buffer provided by the fuzzing engine. In this case it’s a bytes type. The fuzzer passes this object directly to the yaml.load function, together with a second argument specific to yaml.load.

The harness itself wraps the call to yaml.load in a try-except where the exception caught is yaml.YAMLError, which is an exception thrown by the yaml library in the event an error happens in the serialization logic. The idea behind this is that the bugs we are looking to find are those that exhibit odd behaviour, and throwing a controlled exception is considered within the scope of the library. However, if the function can throw an exception that has not been specified, then that can be a problem.

Structured python fuzzing

The Atheris Python provides a helper module for extracting higher level data types that are related to fuzzing. This comes from the FuzzedDataProvider class and the documentation is here . We will not go in detail with the module but refer to the documentation instead, but rather give a short introduction to it and highlight why it’s important.

To initialise the FuzzedDataProvider class you provide it a list of bytes, which in most cases is just the buffer provided by the fuzzing engine:

fdp = atheris.FuzzedDataProvider(bytes)

We can now use fdp to create higher level types seeded by the fuzz data, e.g. in order to extract a Unicode string that has no surrogate pairs we can simply:

unicode_string = fdp.ConsumeUnicodeNoSurrogates(10)

This produces a string with random characters of length 10, and if we want to generate a string of random characters and random length, we can use an additional function from FuzzedDataProvider:

string_length = fdp.ConsumeIntInRange(1, 100000)
unicode_string = fdp.ConsumeUnicodeNoSurrogates(string_length)

In this case, unicode_string will be a string of length 1-100000 with arbitrary characters.

The FuzzedDataProvider class has around 20 functions that each offer generation of some type of data. They are often used in combination with each other to construct a more complex way of interacting with the target application, but are all deterministic so they ensure reproducibility.

A fuzzer for the Python package markdown-it-py and that uses FuzzedDataProvider is available here :

import sys
import atheris
from markdown_it import MarkdownIt


def TestOneInput(data):
    fdp = atheris.FuzzedDataProvider(data)
    md = MarkdownIt()
    raw_markdown = fdp.ConsumeUnicodeNoSurrogates(sys.maxsize)
    md.parse(raw_markdown)
    md.render(raw_markdown)


def main():
    atheris.instrument_all()
    atheris.Setup(sys.argv, TestOneInput)
    atheris.Fuzz()


if __name__ == "__main__":
    main()

The fuzzer uses ConsumeUnicodeNoSurrogates to construct a string that should be acceptable to MarkdownIt.parse and MarkdownIt.render in a manner where no exception is thrown.

Bugs to find

In Python fuzzing the general bug that is being found by fuzzing is uncaught exceptions. Another common type of bug to find from Python fuzzing includes memory safety bugs in native fuzzing modules. It’s a perfectly viable case to build native modules with sanitizers, including bug-finding sanitizers and code coverage feedback sanitizers, and then fuzz the native module by way of Python. Atheris holds a list, albeit bit outdated, of trophies found from fuzzing by way of Python: https://github.com/google/atheris/blob/master/hall_of_fame.md .

\newpage

OSS-Fuzz: continuous Open Source Fuzzing

In this section we will go into details with the open source fuzzing service OSS-Fuzz. This is a cornerstone in open source fuzzing in that it manages fuzzing for more than a thousand critical open source packages. Many of the CNCF projects that use fuzzing also run on OSS-Fuzz and in general CNCF projects are encouraged to submit an OSS-Fuzz integration. However, OSS-Fuzz itself has a learning curve, and, when first integrating into OSS-Fuzz it can be difficult to understand what to do and what to expect from it. This is in part because some pieces of OSS-Fuzz are only available to the user once their project has integrated into OSS-Fuzz and fuzzers are running continuously. This section aims to highlight in detail OSS-Fuzz from a user experience perspective and will be going through a complete end-to-end OSS-Fuzz integration.

Introduction

OSS-Fuzz itself comes in the form of a GitHub repository available at https://github.com/google/oss-fuzz . The projects folder of this repository holds the list of projects integrated into OSS-Fuzz and at the time of writing there are more than 1100 projects integrated.

In order to have your open source project running on OSS-Fuzz you need to add a folder in the projects directory with the name of your project. Each project folder is organized into three primary files:

• Dockerfile: specifies the container that will be used for building your fuzzers.
• build.sh: a script for building the fuzzers of your project. This will be run several times with various environment variables specifying which compiler to use.
• project.yaml: holds metadata for your project, including emails that will be receiving bug reports and access to the OSS-Fuzz dashboards.

As such, the high-level process of integrating an open source project with OSS-Fuzz is to:

1. Develop a set of fuzzers in your repository.
2. Create the relevant build.sh, Dockerfile and project.yaml and submit these to https://github.com/google/oss-fuzz in a pull request.
3. If the pull request is accepted, OSS-Fuzz will use the Dockerfile and build.sh to build your fuzzers and run them continuously in a daily cycle, and will notify the emails listed in project.yaml when a bug is found, and if set also make a public GitHub issue with notification of the finding.

OSS-Fuzz provides a lot of features other than building and running fuzzers. Specifically, it provides features such as:

• Generating code coverage reports
• Bug triaging capabilities
• Extensive build and runtime features, including running fuzzers with various different fuzzing engines
• Introspection capabilities to make it easier to spot new areas of code to fuzz
• Visual Studio Code extension to make development easier.

In addition to providing project-level features for continuous fuzzing, OSS-Fuzz publishes a macro-level overview of its open source fuzzing on https://introspector.oss-fuzz.com . This website shows various high level statistics, as well as offering various search functionalities such as searching about whether a function is analyzed https://introspector.oss-fuzz.com/function-search . For example, if we wanted to search if parse_hostname_login from the CURL application was being fuzzed we can simply search https://introspector.oss-fuzz.com/function-search?q=parse_hostname_login and we get:

In this case, we can see that the function is indeed covered in the Curl project and has 88.23% code coverage. Furthermore, clicking the name of the functions gives us more specific details, such as which fuzzer covers it, and, ultimately a link to the code coverage report where this function is covered: https://storage.googleapis.com/oss-fuzz-coverage/curl/reports/20230804/linux/src/curl/lib/urlapi.c.html#L418

OSS-Fuzz example: end-to-end walkthrough

The OSS-Fuzz project is well-documented with extensive information on how to navigate the project in the document here , and this section will not attempt to address the information laid out in the doc. Instead this section will go through a practical end-to-end OSS-Fuzz and will touch upon topics that we often receive as feedback from CNCF project maintainers. The project that will be used is a toy project that has been through the whole process and is indeed running on OSS-Fuzz and in particular, this section will based on a practical example go in detail with:

1. How to set up an initial OSS-Fuzz project from scratch.
2. Receive bug reports from OSS-Fuzz, including details that are normally private to maintainers.
3. Fixing a bug report and going through the steps of how OSS-Fuzz handles this.
4. Using the tools provided by OSS-Fuzz to visualize how much code coverage our fuzzers have and identify where there is missing code coverage.

Preparing target library

The project that we will integrate is oss-fuzz-example . It is a small C library with a single header file char_lib.h that exposes the functions:

int count_lowercase_letters(char *input);
int parse_complex_format(char *input);
int parse_complex_format_second(char *input);

The project has two fuzzers fuzz_complex_parser.c:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdint.h>

#include "char_lib.h"

int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
    char *ns = malloc(size+1);
    memcpy(ns, data, size);
    ns[size] = '\0';
    
    count_lowercase_letters(ns);
    
    free(ns);
    return 0;
}

and fuzz_char_lib.c

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdint.h>

#include "char_lib.h"

int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
    char *ns = malloc(size+1);
    memcpy(ns, data, size);
    ns[size] = '\0';
    
    parse_complex_format(ns);
    
    free(ns);
    return 0;
}

Each of the fuzzers create a null-terminated string that is passed to two of the functions exposed by char_lib.h. The implementation of count_lowercase_letters and parse_complex_format are some sample parsing routines that have various checks on the provided buffers. The details of their implementation are not important at this stage, the only thing that is important to verify is whether the fuzzers are using the APIs in a correct manner. Given the simplicity of the library we can assume the functions simply accept a null-terminated string and require no setup/teardown. As such, the fuzzers are providing data within the scope of what is considered an acceptable input that the library should be able to handle without fault.

At this stage we have the source code ready for building fuzzers against our library, and the next stage is to ensure we can build them. In this case we can build the fuzzers with a few commands:

git clone https://github.com/AdaLogics/oss-fuzz-example
cd oss-fuzz-example
clang -fsanitize=fuzzer-no-link,address -c char_lib.c -o char_lib.o
clang -fsanitize=fuzzer-no-link,address -c fuzz_char_lib.c -o fuzz_char_lib.o
clang -fsanitize=fuzzer,address fuzz_char_lib.o char_lib.o -o fuzz_char_lib

The above commands will build the fuzz_char_lib.c fuzzer against the library and compiled with address sanitizer as well. The final binary fuzz_char_lib is runnable in the expected fuzzing manner. We can perform the same set of operations with fuzz_complex_parser.c to build this respective fuzzer.

At this stage we have the necessary components of setting up an OSS-Fuzz integration, namely a target library as well as a set of fuzzers for this library that we know how to build and run.

Initial OSS-Fuzz set up

In order to create an OSS-Fuzz integration we must do two things:

1. Create the relevant OSS-Fuzz artifacts.
2. Make a pull request on OSS-Fuzz for the OSS-Fuzz maintainers to review.

The two primary artifacts we need to make are Dockerfile and build.sh. The build.sh will build our code in a way that allows OSS-Fuzz to use relevant sanitizers and compilers, and the Dockerfile will download all the relevant repositories and packages needed for the building.

The Dockerfile we use is as follows:

FROM gcr.io/oss-fuzz-base/base-builder
# Install the packages needed for building the project
RUN apt-get update && apt-get install -y make autoconf automake libtool

# Clone the relevant project
RUN git clone --depth 1 https://github.com/AdaLogics/oss-fuzz-example oss-fuzz-example
WORKDIR oss-fuzz-example
COPY build.sh $SRC/

The Dockerfile inherits from the gcr.io/oss-fuzz-base/base-builder image, which is an image provided by OSS-Fuzz. This image contains relevant compilers, helper tools and more to make the fuzzing process smooth. The Dockerfile also makes a clone of our oss-fuzz-example repository and copies that build.sh (which we are about to create) into the container.

The build.sh we will be using is as follows:

# Build project using the OSS-Fuzz environment flags.
$CC $CFLAGS -c char_lib.c -o char_lib.o

# Build the fuzzers. We must ensure we link with $CXX to support Centipede.
# Fuzzers must be placed in $OUT/
$CC $CFLAGS -c fuzz_char_lib.c -o fuzz_char_lib.o
$CXX $CXXFLAGS $LIB_FUZZING_ENGINE fuzz_char_lib.o char_lib.o -o $OUT/fuzz_char_lib

$CC $CFLAGS -c fuzz_complex_parser.c -o fuzz_complex_parser.o
$CXX $CXXFLAGS $LIB_FUZZING_ENGINE fuzz_complex_parser.o char_lib.o -o $OUT/fuzz_complex_parser

The build files simply build the char_lib.c file of the library as well as the fuzzers. The most important pieces of the build.sh file which is different to a non-OSS-Fuzz build are:

• Use of CC, CFLAGS, CXXFLAGS and LIB_FUZZING_ENGINE for compiling and linking.
  • CC flags represent the C compiler used by OSS-Fuzz. In general, this is clang although for some fuzzing engines it’s a different compiler.
  • CXX flags represent the C++ compiler used by OSS-Fuzz. In general this is clang++ although for some fuzzing engines it’s a different compiler.
  • CFLAGS and CXXFLAGS hold the command line flag that OSS-Fuzz uses to instrument the code for fuzzing purposes. This includes, e.g. -fsanitize=fuzzer-nolink and -fsanitize=address.
  • LIB_FUZZING_ENGINE is a flag needed during the linking stage of building a fuzzer. This environment variable includes, e.g. the clang command line flag -fsanitize=fuzzer.
• Playing the final binaries in the folder defined by the OUT environment variable.

This is a rather simple build script and quite verbose. The important thing to notice is that we need to use certain environment variables provided by OSS-Fuzz to control the compilation and linking process. This can become more complicated for certain build systems, and will often require a deeper integration with relevant build files.

The process for creating build.sh and Dockerfile differs for which language the project is targeted. OSS-Fuzz has guides on how to do this in the guide for setting up a new project here.

Finally, we need to create a project.yaml file that holds metadata and contact information to the people involved in the OSS-Fuzz integration. The contact information is also used for authentication purposes when accessing fuzzing details provided by OSS-Fuzz, such as detailed bug reports. In the case of our example, the project.yaml looks as follows:

homepage: "https://github.com/AdaLogics/oss-fuzz-example"
language: c
primary_contact: "david@adalogics.com"
main_repo: "https://github.com/AdaLogics/oss-fuzz-example"
file_github_issue: true
# Add emails here for any contacts that should receive emails with bugs found.
auto_ccs:
- adam@adalogics.com

These three files will then constitute our OSS-Fuzz integration and the next step is to make a pull request on the OSS-Fuzz repository with our artifacts in the folder projects/oss-fuzz-example. However, before we make a pull request we will test the set up using OSS-Fuzz-provided helper scripts, specifically infra/helper.py. This script is the central helper for running OSS-Fuzz tasks locally.

First, we test the the set up actually builds from the root of the OSS-Fuzz repository:

$ python3 infra/helper.py build_fuzzers oss-fuzz-example

INFO:__main__:Running: docker build -t gcr.io/oss-fuzz/oss-fuzz-example --file /home/oss-fuzz/projects/oss-fuzz-example/Dockerfile ...
...
...
---------------------------------------------------------------
Compiling libFuzzer to /usr/lib/libFuzzingEngine.a...  done.
---------------------------------------------------------------
CC=clang
CXX=clang++
CFLAGS=-O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link
CXXFLAGS=-O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link -stdlib=libc++
RUSTFLAGS=--cfg fuzzing -Zsanitizer=address -Cdebuginfo=1 -Cforce-frame-pointers
---------------------------------------------------------------
+ clang -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link -c char_lib.c -o char_lib.o
+ clang -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link -c fuzz_char_lib.c -o fuzz_char_lib.o
+ clang++ -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link -stdlib=libc++ -fsanitize=fuzzer fuzz_char_lib.o char_lib.o -o /out/fuzz_char_lib
+ clang -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link -c fuzz_complex_parser.c -o fuzz_complex_parser.o
+ clang++ -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link -stdlib=libc++ -fsanitize=fuzzer fuzz_complex_parser.o char_lib.o -o /out/fuzz_complex_parser

We can see the build commands of our build.sh script and verify the commands run with proper sanitizer flags. The artifacts from our build is now located in build/out/oss-fuzz-example:

$ ls build/out/oss-fuzz-example/
fuzz_char_lib  fuzz_complex_parser  llvm-symbolizer

Finally, we can verify that our fuzzers are runnable using the helper.py:

$ python3 infra/helper.py run_fuzzer oss-fuzz-example fuzz_char_lib
/out/fuzz_char_lib -rss_limit_mb=2560 -timeout=25 /tmp/fuzz_char_lib_corpus < /dev/null
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 4203223377
INFO: Loaded 1 modules   (59 inline 8-bit counters): 59 [0x5eaf80, 0x5eafbb), 
INFO: Loaded 1 PC tables (59 PCs): 59 [0x5a7b40,0x5a7ef0), 
INFO:        0 files found in /tmp/fuzz_char_lib_corpus
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: A corpus is not provided, starting from an empty corpus
#2      INITED cov: 6 ft: 7 corp: 1/1b exec/s: 0 rss: 29Mb
#4      NEW    cov: 7 ft: 10 corp: 2/3b lim: 4 exec/s: 0 rss: 29Mb L: 2/2 MS: 2 ShuffleBytes-InsertByte-
#10     NEW    cov: 7 ft: 13 corp: 3/6b lim: 4 exec/s: 0 rss: 30Mb L: 3/3 MS: 1 CrossOver-
#19     NEW    cov: 8 ft: 14 corp: 4/9b lim: 4 exec/s: 0 rss: 30Mb L: 3/3 MS: 4 InsertByte-ChangeBit-CopyPart-CMP- DE: "\000\000"-
#32     NEW    cov: 8 ft: 17 corp: 5/13b lim: 4 exec/s: 0 rss: 30Mb L: 4/4 MS: 3 ShuffleBytes-CopyPart-InsertByte-
#46     NEW    cov: 9 ft: 18 corp: 6/17b lim: 4 exec/s: 0 rss: 30Mb L: 4/4 MS: 4 ShuffleBytes-ChangeBit-ChangeBit-InsertByte-
#58     NEW    cov: 9 ft: 19 corp: 7/21b lim: 4 exec/s: 0 rss: 30Mb L: 4/4 MS: 2 CrossOver-CopyPart-
#64     REDUCE cov: 9 ft: 19 corp: 7/20b lim: 4 exec/s: 0 rss: 30Mb L: 2/4 MS: 1 EraseBytes-
#143    NEW    cov: 10 ft: 20 corp: 8/24b lim: 4 exec/s: 0 rss: 30Mb L: 4/4 MS: 4 ChangeByte-InsertByte-ChangeByte-InsertByte-
#154    REDUCE cov: 10 ft: 20 corp: 8/22b lim: 4 exec/s: 0 rss: 30Mb L: 2/4 MS: 1 EraseBytes-
#161    REDUCE cov: 10 ft: 20 corp: 8/21b lim: 4 exec/s: 0 rss: 30Mb L: 1/4 MS: 2 PersAutoDict-EraseBytes- DE: "\000\000"-
#164    NEW    cov: 10 ft: 21 corp: 9/25b lim: 4 exec/s: 0 rss: 30Mb L: 4/4 MS: 3 ChangeBinInt-CrossOver-CrossOver-
#175    NEW    cov: 10 ft: 22 corp: 10/29b lim: 4 exec/s: 0 rss: 30Mb L: 4/4 MS: 1 CopyPart-
#219    REDUCE cov: 10 ft: 22 corp: 10/28b lim: 4 exec/s: 0 rss: 30Mb L: 3/4 MS: 4 ShuffleBytes-ChangeBit-EraseBytes-CopyPart-
#223    REDUCE cov: 10 ft: 22 corp: 10/27b lim: 4 exec/s: 0 rss: 30Mb L: 2/4 MS: 4 CrossOver-CopyPart-ShuffleBytes-CrossOver-
#395    REDUCE cov: 10 ft: 22 corp: 10/26b lim: 4 exec/s: 0 rss: 30Mb L: 1/4 MS: 2 ShuffleBytes-EraseBytes-

The run was successful and we can see the usual fuzzer output showing it is indeed exploring code. Finally, we can run a check provided by OSS-Fuzz that verifies if the set up passes the build checks OSS-Fuzz requires:

$ python3 infra/helper.py check_build oss-fuzz-example
...
INFO: performing bad build checks for /tmp/not-out/tmppjsl7bmd/fuzz_char_lib
INFO: performing bad build checks for /tmp/not-out/tmppjsl7bmd/fuzz_complex_parser
INFO:__main__:Check build passed.

Submitting the project for integration

At this point we are ready to proceed with a pull request on OSS-Fuzz with our set up. In general, when making a pull request for OSS-Fuzz with an initial integration of a project, the OSS-Fuzz reviewers will seek information about the importance of a project. This is primarily to ensure that bugs found will be fixed and also have impact, and also because OSS-Fuzz itself offers some lighter versions of fuzzing infrastructure that can be used in the CI without OSS-Fuzz integration, such as ClusterFuzzLite . To simulate the process of an initial integration we made a pull request displaying this here: https://github.com/google/oss-fuzz/pull/10807 .

The pull request with our initial integration was accepted and merged in https://github.com/google/oss-fuzz/pull/10807 . In the following sections we will go through commonly happens after a project has integration, such as when issues are found.

Receiving bug reports

In our sample library we injected synthetic bugs and OSS-Fuzz found and reported this. OSS-Fuzz has two primary ways of notifying users that a bug was found: through GitHub issues and by way of email.

In the project.yaml we enabled issue reporting via GitHub, specifically file_github_issue: true. The consequence of this is that issues found by OSS-Fuzz will also be reported by an OSS-Fuzz bot using GitHub issue and the issue for our example project is found here: https://github.com/AdaLogics/oss-fuzz-example/issues/2

In addition to the GitHub issue we also received an email notification at the same time, with the exact same content as in the GitHub issue. This email was sent out to all emails listed in the project.yaml. The content of the text is scarce and to extract more insights we need to follow the links in the description to bug reports. There are two links to further details about the issue, one for https://bugs.chromium.org/ and one for https://oss-fuzz.com/ . The bug report on https://oss-fuzz.com/ has the most details and will always remain only visible to the emails listed in project.yaml and the details listed on https://bugs.chromium.org/ has slightly more information about the bug report than the GitHub issue and this report will remain private until the bug disclosure deadlines has passed, which is 90 days, or until the issue is fixed.

Viewing detailed bug reports

The detailed bug report by OSS-Fuzz gives a lot of information for locating the issue as well as reproducing it. The overview page of the detailed issue looks as follows:

The overview gives us insight into specific high-level parts of the bug:

• The bug was found by the fuzz_complex_parser fuzzer
• The bug was found using Address Sanitizer (ASAN)
• The bug is a heap overflow
• The bug reliably reproduces
• Link to a minimized test case, which we can use for reproducing the bug.

The detailed bug report then continues with further information, specifically including the stack trace:

The stack trace gives us the exact location where the heap overflow occur, and even provides us with a link in the GitHub source that we can use for easy tracing: https://github.com/AdaLogics/oss-fuzz-example/blob/c5d2a988f0f66ff3a20519d9b389aeb563305485/char_lib.c#L58

Reproducing and fixing a bug

The step after having identified and analyzed a bug from a high-level perspective is to fix the issue. OSS-Fuzz will daily triage which bugs are reproducible, and if a bug is no longer reproducible will declare it as fixed.

The first step in fixing the bug is to reproduce it. Once we have reproduced it we will come up with a fix and test the issue no longer reproduces against our fix. Once that is achieved then we will push our fix.

The detailed bug report from OSS-Fuzz gives us a minimized testcase:

This testcase is specifically a file that we can give as input when running our fuzzer, and the fuzzer will then use that as its seed. As such, reproducing issues found by a fuzzer is simply running a single iteration of the fuzzer using a specific buffer as input.

To reproduce the issue we download the minimized testcase provided by OSS-Fuzz which in this case is a file called clusterfuzz-testcase-minimized-fuzz_complex_parser-4884946693783552. We place this file in our root OSS-Fuzz folder and then run the commands:

$ python3 infra/helper.py build_fuzzers oss-fuzz-example
...
$ python3 infra/helper.py reproduce oss-fuzz-example fuzz_complex_parser ./clusterfuzz-testcase-minimized-fuzz_complex_parser-4884946693783552
...
/out/fuzz_complex_parser: Running 1 inputs 100 time(s) each.
Running: /testcase
=================================================================
==13==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60b0000002c2 at pc 0x00000056cb1d bp 0x7fff49b80f40 sp 0x7fff49b80f38
READ of size 1 at 0x60b0000002c2 thread T0
SCARINESS: 12 (1-byte-read-heap-buffer-overflow)
    #0 0x56cb1c in get_the_right_character /src/oss-fuzz-example/char_lib.c:58:14
    #1 0x56cb1c in read_key_figures /src/oss-fuzz-example/char_lib.c:72:12
    #2 0x56cb1c in parse_complex_format /src/oss-fuzz-example/char_lib.c:102:16
    #3 0x56c3df in LLVMFuzzerTestOneInput /src/oss-fuzz-example/fuzz_complex_parser.c:27:3
    #4 0x43ddb3 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:611:15
    #5 0x429512 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:324:6
    #6 0x42edbc in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:860:9
    #7 0x4582f2 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10
    #8 0x7f91067ef082 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24082) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee)
    #9 0x41f6dd in _start (/out/fuzz_complex_parser+0x41f6dd)

We reproduced the issue successfully! First, we built the project from new and then we used the reproduce command from infra/helper.py to perform the reproduction task.

Now, the next stage is to fix the issue at hand. To do this in a way where we can verify our fix before pushing our fix to the main repository we need to have OSS-Fuzz use a local version of our oss-fuzz-example project. There are various ways we can do this, OSS-Fuzz supports mounting a local folder to a project folder inside the container. Instead of mounting, we could also make a temporary OSS-Fuzz set up where, instead of having the OSS-Fuzz Dockerfile clone our repository from GItHub, we could simply have it copy a local folder into the container. In this example we will go for the latter.

To get our local copy of oss-fuzz-example into the folder we change the following line of the Dockerfile:

RUN git clone --depth 1 https://github.com/AdaLogics/oss-fuzz-example oss-fuzz-example

To

COPY oss-fuzz-example oss-fuzz-example

Then, we place a local version of our oss-fuzz-example repository inside the projects/oss-fuzz-example folder. To test our set up works we run the build_fuzzers and reproduce commands again

We now have the artifacts available from OSS-Fuzz to fix our crash and it’s up to us to do the necessary analysis for fixing it. After having done some analysis we realise the issue is because the following line :

return read_key_figures(input, length);

Expects the second argument of the function to be half the length, and the entire length. So we change this argument in our local oss-fuzz-example report to:

return read_key_figures(input, length/2);

We then run the same commands for building our project and reproducing the issue:

$ python3 infra/helper.py build_fuzzers oss-fuzz-example
...
+ clang -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link -c char_lib.c -o char_lib.o
+ clang -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link -c fuzz_char_lib.c -o fuzz_char_lib.o
+ clang++ -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link -stdlib=libc++ -fsanitize=fuzzer fuzz_char_lib.o char_lib.o -o /out/fuzz_char_lib
+ clang -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link -c fuzz_complex_parser.c -o fuzz_complex_parser.o
+ clang++ -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link -stdlib=libc++ -fsanitize=fuzzer fuzz_complex_parser.o char_lib.o -o /out/fuzz_complex_parser

$ python3 infra/helper.py reproduce oss-fuzz-example fuzz_complex_parser ./clusterfuzz-testcase-minimized-fuzz_complex_parser-4884946693783552
...
/out/fuzz_complex_parser: Running 1 inputs 100 time(s) each.
Running: /testcase
Executed /testcase in 0 ms
***
*** NOTE: fuzzing was not performed, you have only
***       executed the target code on a fixed set of inputs.
***

In this instance we observe that the crash is no longer happening when we run reproduce and we, therefore, declare that the issue is fixed and for good measure we can also run the fuzzer for a couple of minutes to verify that we did not introduce a new crash. We then proceed to submit our patch upstream. At this point, we will wait up to 24 hours for OSS-Fuzz to verify that our bug has been fixed.

Having waited the 24 hours we receive an email as well as GitHub notification with information that the bug is fixed:

Furthermore, following the link to the monorail report also shows the bug report has now been marked as fixed, as well as opened to the public. That is, the detailed report and the reproducer sample has not been made available to the public, however, the overview report has been made publicly available, as visible here .

Viewing code coverage

The next stage in a continuous fuzzing setup is making sure we can use introspection tools provided by OSS-Fuzz to assess how well our fuzzing is going. We are particularly interested in extracting how much code is covered by our fuzzers and, in a more pragmatic sense, identify the code that is not yet covered by our fuzzers and adjust so we can adjust our set up to cover these areas.

OSS-Fuzz makes coverage reports accessible at https://oss-fuzz.com with the same login as you used in your project.yaml. In general, on https://oss-fuzz.com a myriad of information is accessible, also including bug reports and other private data. However, OSS-Fuzz makes all coverage reports publicly available by way of https://introspector.oss-fuzz.com and we can find the one for our sample project here .

We both get access to an overview of the project:

https://introspector.oss-fuzz.com\n" src="imgs/Open-source-fuzz-introspection-overview.png">

And we also get access to statistics showing a historical progression and how coverage has evolved over time.

https://introspector.oss-fuzz.com\n" src="imgs/Historical-progession-of-example-project.png">

Interestingly, we can see that our code coverage percentage actually fell after we introduced the fix.

Following the link to the code coverage report we observe an overview of the files involved and how much code coverage are in them:

https://introspector.oss-fuzz.com\n" src="imgs/Code-coverage-of-example-project.png">

The missing coverage is within the char_lib.c file itself and following the link to the file we can identify the exact functions that are missing coverage:

https://introspector.oss-fuzz.com\n" src="imgs/Source-level-code-coverage.png">

It is worth noting that in this example the whole process of integrating into OSS-Fuzz, finding issues and observing limitations in the coverage was drastically minimized relative to the efforts it takes for a real-world project. As such, this example should be considered a reference for minimum-viable example and when integrating a mature CNCF project it will likely take much more efforts.

The OSS-Fuzz team responds to questions and answers about the infrastructure on the GitHub issue tracker . As such, it is advised to use this while running into problems during the integrations. Furthermore, OSS-Fuzz provides specific documentation on how to get started with integrating a new project .

\newpage

CNCF fuzzing

In this chapter we will introduce how fuzzing is used by CNCF projects and where to find further references that may be relevant to your project. There is a significant variety, from a code-perspective, amongst the projects in the CNCF landscape. As such, projects will often apply fuzzing in ways that are tailored their projects, which requires hands-on efforts often by security researchers or maintainers of the relevant projects.

CNCF fuzzing resources

CNCF maintains the CNCF-fuzzing repository which contains a myriad of information regarding fuzzing CNCF projects. This includes a placeholder for fuzzing source code that projects can use either for a complete OSS-Fuzz fuzzing integration or as a placeholder while developing a more mature fuzzing set up.

In addition to holding resources for guiding on how to fuzz, the repository also holds a lot of fuzzer-related source code for many CNCF projects. This can be used either to study how existing projects use fuzzing or to add additional fuzzing capabilities to a given project.

CNCF fuzzing audits

CNCF regularly performs fuzzing audits of projects in the CNCF landscape. The goal of these audits is to assist projects in setting up continuous fuzzing efforts by way of OSS-Fuzz and collaborate with maintainers on how to adopt fuzzing. The output of the audits is most often:

1. A set of fuzzers developed for the CNCF project.
2. An OSS-Fuzz integration ensuring the fuzzers continue to run post-audit.
3. A report that outlines the efforts, including bugs found during the process, fuzzers developed and more.

The audits are done in collaboration with the maintainers and can take everything from a few weeks to several months depending on the scope and maintainer availability.

The reports from the fuzzing audits are available in several places. First, all audit reports (including both security and fuzzing audits) can be viewed on the CNCF landscape:

It is also commonplace for security audits to involve some kind of fuzzing, thus, looking at the security-audit reports will often provide deatils of how a given CNCF project has applied fuzzing.

Sample CNCF projects using fuzzing

There are many projects in the CNCF landscape that have adopted fuzzing. Some of these projects have applied fuzzing for several years whereas for some it’s a more recent effort.

Envoy integrated into OSS-Fuzz in early 2018 , meaning it has fuzzed been for almost six years at the time of writing. Furthermore, inspecting the introspector.oss-fuzz.com profile we can observe that Envoy has a total of 63 fuzzers running continuously:

In terms of source code, the Envoy project has its fuzzers spread across the codebase . However, they have a central directory here containing multiple helper utilities for writing fuzzers. The Envoy fuzzing set up is an example of a project with a lot of person-hours devoted to it, and is a representation of a mature fuzzing set up. It is worth in this context to refer to Harvey Tuch’s (Envoy maintainer) from the 2022 CNCF fuzzing review on the impact of fuzzing on Envoy: Fuzzing is foundational to Envoy’s security and reliability posture – we have invested heavily in developing and improving dozens of fuzzers across the data and control plane. We have realized the benefits via proactive discovery of CVEs and many non-security related improvements that harden the reliability of Envoy. Fuzzing is not a write-once exercise for Envoy, with continual monitoring of the performance and effectiveness of fuzzers.

The significant efforts put in place by the Envoy team has also paid off. Following the bug tracker of OSS-Fuzz there are more than 1200 issues reported to the Envoy team by OSS-Fuzz. This includes, however, many issues that may be false positives or were issues in testing or build infrastructure.

Istio integrated into OSS-Fuzz in late 2020 and has been running continuously since then. From the profile we can observe they currently have 69 fuzzers running on OSS-Fuzz, and from the historical tracking we can see there hasn’t been new fuzzers added in the last 4 months. A blog post from early 2022 by security researchers and Istio maintainers covers some of the findings from the Istio fuzzers, including how fuzzing was used to find high severity CVE-2022-23635 . In contrast to Envoy, Istio maintainers a lot of its fuzzers in one location in the Istio source code repository.

Fluent-bit has been integrated into OSS-Fuzz since April 2020 and the introspection profile shows Fluent-Bit has 28 fuzzers currently running, having added a handful of fuzzers in recent months. Similarly to Istio, Fluent-Bit maintains all fuzzers in a single folder making it easy for third-parties to go through the relevant code.

\newpage

Conclusion

Fuzzing is a proven technique for finding security and reliability issues in software. It can be time-consuming to integrate fuzzing into a codebase, and even more so of maintaining the fuzzing set up afterwards. Fuzzing is a complex task to control and it can be difficult to assess where to start and how to proceed.

In this handbook we have provided introductions and examples to overcome the initial barrier of setting up fuzzing for a given open source project. We have provided an introduction to the concepts behind fuzzing, introductions on how to apply fuzzing in a variety of languages, how to set up a continuous fuzzing workflow using mature open source fuzzing frameworks as well as given reference points to how CNCF projects use fuzzing.