By David Korczynski and Adam Korczynski November 7, 2023

Community post by David Korczynski and Adam Korczynski of Ada Logics

Fuzzing is a central role in today’s software development practices for ensuring secure and reliable software. CNCF is committed to promoting the use of fuzzing as part of the development practices of CNCF projects and in this blog post we will highlight some of the updates of 2023.


We recommend going over previous fuzzing blog posts that provide useful data and metric on how CNCF is helping apply fuzzing:

NB: The CNCF fuzzing team will be present at a booth in the projects pavilion for Kubecon NA, please come visit if you’re curious about fuzzing.

Introducing the CNCF Fuzzing Handbook

A central role in assisting developers to apply fuzzing is developing reference materials that can be easily consumed and describe clearly how to apply fuzzing on a given software package. Improving this has been a large part of our efforts in recent months, and to this end we are happy to announce that we have created a fuzzing handbook that introduces fuzzing in a manner that uses fully open source frameworks for fuzzing.

The fuzzing handbook is accessible in the TAG-Security repository here and is a comprehensive hands-on guide to fuzzing with more than 60 pages of fuzzing related materials. The handbook first provides an introduction to the concepts and core techniques behind fuzzing, and then proceeds to have a pragmatic coverage of fuzzing including many code examples. The goal is to provide a practical intuition for fuzzing such that the reader can apply it to an arbitrary software package. The handbook also has an extensive documentation on how to use the open source fuzzing framework OSS-Fuzz to establish a continuous fuzzing setup. The handbook introduces fuzzing for  C/C++/Golang and Python, and we have plans for introducing Rust and Java fuzzing as well.

The handbook exists as a live document in that it is expected to be extended and adjusted regularly according to the needs of fuzzing users. The handbook focuses on general concepts and techniques, and we also recommend visiting the dedicated CNCF repository for fuzzing resources, which holds many source code samples used for fuzzing large CNCF projects as well as links to relevant fuzzing audit reports.

We welcome feedback, limitations and suggestions for the handbook, and are happy to add content that may be relevant for CNCF projects.

Three recently completed fuzzing audits

In addition to the handbook, the CNCF continues to support CNCF projects with fuzzing integrations and development. In the past few months three CNCF projects completed dedicated fuzzing audits: Dapr, Knative and Kyverno. See the respective full fuzzing audit reports in the following links:

Below we provide a summary for each of the reports:

Dapr


Dapr’s fuzzing audit resulted in a mature fuzzing suite of 39 fuzzers running continuously on OSS-Fuzz. The fuzzers found 3 issues during the audit itself which were all fixed. Interestingly, Dapr’s security audit which was completed after its audit added new fuzzers on top of the fuzzing suite, which was enabled by the continuous setup engineered during the fuzzing audit. The auditors of Daprs security audit wrote a fuzzer for a third-party dependency of Dapr which found a high-severity CVE in the dependency (CVE-2023-37475). The CVE did not impact Dapr in a high-severity manner. Daprs fuzzing efforts demonstrated how a dedicated fuzzing audit enabled a subsequent audit – that involved thorough threat modelling – to add specific fuzzers targeting exposed code paths identified during the threat modelling.

Kyverno

Kyverno’s fuzzing audit added a complete continuous fuzzing suite with 15 fuzzers. The fuzzers target high-level entry points, specifically Kyverno’s admission handlers. These handlers are exposed to untrusted input in the form of admission requests from the cluster. The incoming requests are already authenticated when they reach Kyvernos handlers, however, they process requests from both low-privilege users as well as high-privilege users. As such, the fuzzers test a known attack surface of Kyverno. The fuzzers need time to explore Kyverno, given that they target high-level APIs, and the element of continuity is important to Kyverno’s fuzzing suite. Kyvernos fuzzers found three issues during the fuzzing audit.

Knative

Knative’s fuzzing audit focused on the core Knative subprojects: Eventing, Serving og Pkg. The auditors built Knatives continuous fuzzing suite around OSS-Fuzz and added fuzzers for complex data handling as well as for Knative’s custom resource types. The complex data paths were identified through manual review. Knative already had fuzzers in their source tree to test for bugs in its custom resource types. The auditors rewrote these fuzzers to use the coverage feedback from the fuzzing instrumentation more efficiently. All fuzzers written during the audit were added to Knatives continuous fuzzing suite and are still running months after the completion of its audit.

CNCF Fuzzing extensions in 2023

The three audits come after a busy year of fuzzing CNCF projects. Earlier this year, four projects completed their dedicated fuzzing audits: Cilium, containerd, Crossplane and Helm. At the time of the audits, Containerd and Helm had both graduated, and Cilium has graduated some time after completing its fuzzing audit. containerd’s fuzzing audit found a vulnerability (CVE-2023-25153) that could allow an unauthenticated user to exhaust memory of the machine with a malicious OCI image. The fuzzer that found this issue was able to craft an OCI image in such a way that it could exhaust memory and cause full denial of service of containerd. Helms fuzzing audit also resulted in uncovering multiple vulnerabilities with multiple CVEs assigned (CVE-2022-23524, CVE-2022-23525, CVE-2022-23526, CVE-2022-36055). 

You can read the fuzzing reports of Cilium, containerd, Crossplane and Helm here:

We are excited to continue supporting the CNCF landscape with fuzzing efforts and encourage feedback and fuzzing requests from CNCF project maintainers.