By Adith Sudhakar, VMWare; Mohit Arora, Dell; and Souheil Moghnie, NortonLifeLock
SAFECode’s fuzzing team is back to answer the question most asked by readers: “How to integrate fuzzing into the SDLC?”. If you are just finding us for the first time, be sure to take a look and check out the rest of our Focus on Fuzzing series.
Introduction
In our last blog post, we asked readers to share their challenges with fuzzing and areas that they’d like us to shed some light on in this blog series. Based on reader feedback, the challenge most identified was how to practically integrate fuzzing into the software development lifecycle (SDLC). Many SAFECode member companies have faced this same obstacle and have volunteered to share their experiences with you. We’ve captured some of their insight in this blog post that will focus on how fuzzing fits within your SDLC, when to start and when to stop fuzzing, how often to perform fuzzing, and how to practically integrate it into your fast-paced CI/CD pipeline.
When should I fuzz?
A pertinent question might be when is the right time to fuzz? Should I fuzz during development, after I release, at the beginning of a sprint, or at the end of the release cycle? The short answer is “all of the above.” You should fuzz at a minimum after a feature or a set of features has been completed from a development perspective. This practical approach applies to both existing products and new product releases.
As we have mentioned before, fuzzing is complementary to traditional testing, as it uncovers many security vulnerabilities and defects. Fuzzing should therefore be considered a validation time activity within the SDLC. Even within Microsoft’s Security Development Lifecycle (SDL) fuzzing is listed under the verification phase of the SDL.
Another question that we get all the time: Now that you’ve started fuzzing – when are you done? Technically speaking, there is no “done” when it comes to fuzzing. This gives rise to the question of when to stop fuzzing from a release planning perspective. With “Coverage Guided Fuzzing” you could technically stop when there is sufficient coverage of your software. At a minimum, you want to make sure all untrusted interfaces and untrusted inputs have been sufficiently fuzzed prior to a release.
The simple approach of fuzzing after feature-complete, typically within the validation phase, applies to the traditional waterfall or spiral development methodology. However, what about fuzzing in an Agile or a DevOps development model? We will discuss this in the next section.
How often should I fuzz?
Ideally, fuzzing should be part of your software release and conducted on an ongoing basis on all new features and interfaces being developed. Fuzzing should begin from the early stages of the SDLC, continue through the pre-release cycle, and can also be incorporated into the post-release process for certain products.
In the previous section, we discussed when to fuzz, which leads us to the question of how often to fuzz, especially in the Agile development model and within the CI/CD framework. When the release cadence is done in an iterative fashion and does not have a clear start and end of development, we must consider fuzzing in an iterative manner as well. When SDL activities are usually mapped into an Agile development model, they fall into three categories. First, there are activities that need to happen every Agile sprint, such as static code analysis. Next, we have activities that may only need to happen once in a project. Finally, we have activities that don’t need to happen every sprint but should be completed across the life of a project on a regular cadence. The latter category is where fuzzing should be conducted in an Agile model. Fuzzing should be prioritized with other security activities that need to happen on a regular cadence and completed on a regular cadence that makes sense for your project.
Other considerations that may impact how often you fuzz include cost, time, and competing priorities. There are investments in tools and infrastructure when it comes to fuzzing. There are also additional resources needed to plan, configure, and maintain fuzzing for a project. These need to be part of the project plan and budget. Overall, the additional cost and resources may conflict with other business priorities such as time to market and return on investment. However, fuzzing is an integral part of your development process, and as such, it needs to be properly planned and fully integrated into your CI/CD as we will explain later. Additionally, proper metrics and maturity levels may be used to enhance the efficacy of your fuzzing approach.
Because of the additional investment in resources and infrastructure and the skills needed, fuzzing is usually mapped into security maturity models to allow organizations to grow into it. This is a good path to getting fuzzing on your organization’s roadmap and allowing time to plan and fund fuzzing as the organization matures.
In any scenario, you will come to the question of what should I fuzz first? Our next section goes deeper into this question.
What should I fuzz?
Fuzzing does not have to be a time-bound or release-bound activity and consequently, there is often a question of how to prioritize what needs to be fuzzed first. In the next section, we will shed light on when to do continuous fuzzing, but first, let’s explain how to prioritize our fuzzing targets.
First, consider your Threat Model when selecting where to focus your fuzzing. Priority should be given to all security-sensitive areas, especially when the data flow is crossing a trust boundary. These areas should be carefully fuzzed. Other important factors include public API’s, input processing, data parsing, loading points (e.g., configuration files, dynamic libraries…), communication channels, etc.
It is very important to remember that input into your software does not necessarily have to be the traditional user input, or a UI, or a command line. Input vectors can be the file/object that we’re processing, or the database we’re parsing, or the settings that we’re loading, or the values that we’re reading (e.g., from the Windows registry), or anything that we’re processing in some way, especially if an adversary can supply or tamper with it.
Note that Fuzzing helps ensure that our software is resilient to adversarial attacks as well as to normal corruption of data that we’re processing. In all cases, we need to make sure that our system fails gracefully instead of choking on a corrupted or a fuzzed input.
In previous Fuzzing Blog posts we talked about Coverage Guided Fuzzing that could help you cover 100% of your code, but if not practical, at least make sure that the sensitive areas listed above are sufficiently covered.
Finally, make sure to include error handling, and uncommon paths in your fuzzing efforts since these are usually least covered in a conventional testing cycle.
How does fuzzing fit within the CI/CD pipeline?
Fuzzing should be incorporated into your existing CI/CD pipeline that builds, tests, and deploys your software. All fuzzers discussed in our previous blogs can be automated, and given the nature of fuzzing, automation will help uncover bugs in a systematic way.
Adding fuzz test ‘steps’ in your CI/CD pipeline that run in parallel to other tests, such as unit tests, will help ensure that any bugs introduced in that changeset are caught. This fuzzing step is not a gating factor for the pipeline since fuzzing is meant to run continuously and is not suitable to be time-boxed. Please note that even though fuzzing should not be a gated step within the CI/CD pipeline, it may be necessary to address some issues promptly.
For instance, fuzzers such as libFuzzer stop when a bug is found and cannot continue until the bug is fixed. In these cases, it is important to run the fuzzer in a continuous fashion so that, as soon as the bug is fixed, the fuzzer can continue to run and traverse paths in the code that were previously unexplored.
When this method is used in conjunction with fuzzing frameworks such as clusterfuzz, the framework can automatically ‘close’ a ticket that it created when it finds that a changeset has fixed the issue.
An example implementation of running a fuzzer in this way is OSS-Fuzz. In OSS-Fuzz, a Jenkins job continuously syncs from the git repo, and uploads new versions of the target to a storage bucket. Additionally, a core part of OSS-Fuzz is Clusterfuzz, which monitors this bucket for new versions, and when it finds one, it downloads it and starts fuzzing the new version (see our previous blog for details). As stated earlier, when a bug is found, an issue is automatically created in the issue tracker, and the developers are notified by email.
The approach illustrated above has been practically adopted by some SAFECode members.
Coming Up Next
We hope this blog post helps shed some light on how some SAFECode members have successfully integrated fuzzing into their secure development lifecycles. Our working group is currently working to shed some light on another often asked about an aspect of fuzzing – when and how to use differential fuzzing. Stay tuned!