(Re-) Introducing Diligence Fuzzing
With hacks happening more frequently in web3, improving the security of contract code is more critical than ever. This is why we’re excited to announce the wide release of Diligence Fuzzing— a greybox fuzzer for analyzing contracts written in Solidity.
Running (manual) tests for smart contracts can be difficult and time-consuming, and you rarely have the assurance that a test suite covers all possible scenarios. Diligence Fuzzing makes testing your contracts easier by automating the hard parts, such as generating test cases.
It also uses advanced coverage-guided analysis to intelligently select inputs that cover more parts of your codebase. This way, you can achieve a higher coverage in much less time and detect hidden vulnerabilities that could affect your smart contract’s security guarantees.
The rest of the post explains the magic behind fuzzing and how it can enable you to build more secure contracts. Importantly, we’ll highlight some benefits of integrating Diligence Fuzzing into your web3 project’s secure development lifecycle (SDLC).
What is Diligence Fuzzing?
Fuzzing (or fuzz testing) is an automated testing process for exposing bugs and vulnerabilities in software programs. Specifically, fuzzing involves sending random input data to an application to assess its security guarantees. For example, a fuzzer might be useful for understanding how a program handles errors or responds to unpredictable inputs.
A smart contract fuzzer like Diligence Fuzzing invokes a contract’s functions with randomly generated data to detect possible runtime errors. This process relies (in part) on creating a formal specification that describes the expected behavior (properties) of a contract. Using the specification as a guide, Diligence Fuzzing will try to generate transaction sequences capable of violating your assumptions about a contract’s behaviour.
Not all smart contract fuzzers are created equal, however. Different fuzzing tools will provide different results depending on the approach to program analysis and generation of inputs.
“Blackbox fuzzers” typically operate without any knowledge of the target program, such as internal behavior or code structure. Rather, a blackbox fuzzer focuses on generating test data by mutating valid inputs provided by the developer.
Since test-case generation is random, blackbox fuzzing produces a high number of inputs quickly—but these inputs may not necessarily exercise enough paths in the code to discover vulnerabilities. Foundry and DappTools provide this type of fuzzing to test functions with multiple variations of a given parameter value.
Diligence Fuzzing is a “greybox fuzzer” that learns details about the target program’s behavior and structure. For example, the fuzzer tracks what paths your contract’s execution follows at runtime and leverages that information to move execution towards new paths.
This informs how the fuzzer generates new input values during testing. Specifically, Diligence Fuzzing will try to mutate those inputs whose execution exercises more parts of a codebase. Compared to blackbox fuzzing techniques, greybox fuzzing achieves higher code coverage in a shorter amount of time and increases your chances of detecting deeper bugs.
With that crash course of fuzzers out of the way, it’s time to answer a pressing question: “Why should I bother with fuzzing when other methods of assessing code correctness exist?” The next section explains how fuzzing stacks up to traditional methods of testing contracts and the advantages of using tools like Diligence Fuzzing.
The benefits of fuzzing smart contracts
The main use case for fuzzers in general is automating the generation of test cases. This may not seem like a big deal (“I can just write test cases, no?”) until you realize two things:
- There’s a limit on how many test cases you can create, which limits the number of user interactions you can analyze
- There’s no limit on how many transactions can invoke your contract once it’s deployed (ie. user interactions are potentially limitless)
What if you miss a particular user scenario during manual testing—one that results in a bug or vulnerability? This could mean an early debut for your DeFi protocol on the Rekt leaderboard.
“Simply because something is not seen does not mean it is not there. Even the most observant can make this mistake. One must always be alert.” — Miles Teg (Hunters of Dune)
Fuzzers can automatically generate multiple inputs for tests based on a zero or small number of seeded values. For instance, Diligence Fuzzing can efficiently iterate through thousands of test cases and explore all potential interactions with users and malicious actors. This reduces the risk of some unforeseen scenario putting your code at risk and reduces the effort at testing business logic correctness in smart contracts.
Fuzzing is also prized for its ability to test invariants. An invariant is a property critical to the safety of a code—that is, an assumption about the contract’s state or execution that must always hold. For example, an invariant on user balances in an ERC20 contract prevents arithmetic underflow/overflow issues.
Manual testing is quite limited for testing invariants because test cases only capture a small subset of user interactions and may miss potential edge cases. Conversely, contract fuzzers generate a broad class of transaction inputs and efficiently cover multiple user scenarios. This makes it easier to detect violations of invariants that could threaten a protocol’s safety and reliability.
The OGs get it. (Source)
Given that smart contracts rely on user-provided inputs to execute, fuzzing contract functions is a no-brainer. It’s easy to assume a contract will always receive correct inputs, but a user could provide invalid or unexpected data (by design or accident).
If your dapp lacks proper error-handling mechanisms, these edge cases can potentially disrupt its functionality—or worse, compromise security. Fortunately, a smart contract fuzzer like Diligence Fuzzing can detect vulnerabilities arising from improper input validation. This is based on a simple process described previously:
- Feed the target smart contract with randomly generated (and possibly malformed) inputs
- Monitor the program for unexpected code execution, crashes, and other behaviors that deviate from the specification
While a well-executed fuzzing campaign can help developers detect concealed bugs, not every fuzzer offers the same performance. This is why we’ve dedicated the next part of this article to explain why choosing Diligence Fuzzing is your best shot at improving smart contract security.
Why you should use Diligence Fuzzing testing
Here are some benefits of using Diligence Fuzzing to analyze smart contracts:
1. Catch smart contract bugs early
Conventional fuzzers may take a naive approach of generating random inputs that execute a handful of code paths and leave other parts untouched. While this is an improvement on manual testing, it rarely guarantees that your contract operates correctly. Diligence Fuzzing, however, uses the following techniques to discover vulnerabilities that could violate invariants in your contract:
Coverage-guided analysis
As a greybox fuzzer, Diligence Fuzzing collects code coverage information before selecting inputs to mutate. This way, it generates input values that exercise more paths in a codebase and, consequently, increase the odds of catching concealed bugs.
Input prediction
Input prediction is a novel mechanism (described in the Harvey paper) that maximizes the capacity of greybox fuzzing to generate inputs most likely to trigger assertion violations. This feature separates Diligence Fuzzing from traditional greybox fuzzers—like AFL—and speeds up the process of catching bugs in smart contract systems.
2. Save time on smart contract development and testing
Efficient fuzzing-as-a-service (FaaS) model
You may be thinking: “Why should I pay for a Diligence Fuzzing subscription when I can use other contract fuzzers for free?” One answer: Efficiency. Using an open-source contract fuzzer like Echidna costs nothing, but you’ll need to run your own hardware or cloud service before testing.
In comparison, Diligence Fuzzing is a fully-featured software-as-as-service (SaaS) tool that operates a CLI-to-server model. This reduces the steps necessary to configure your testing environment, saving you valuable time and effort.
Using the Diligence fuzzing-as-a-service tool further opens up opportunities for seamless continuous integration, continuous development, and continuous testing. We’re hoping to provide direct integrations with GitHub in the future, so you can automatically run fuzz tests whenever you commit changes to your Solidity source files.
Incremental fuzzing
Another way Diligence Fuzzing improves your productivity is through incremental fuzzing. This feature allows you to save a corpus of fuzzed inputs that exercise interesting behaviors and increase path coverage. Future fuzzing runs can use this corpus as a starting point for mutation operations, eliminating the need to start from scratch next time you need to fuzz a contract.
Testing insights
Diligence Fuzzing provides actionable reports you can use to quickly debug your smart contracts and fix any issues identified during runtime testing. Here are some of the insights you can access after each fuzzing campaign:
- Number of issues found
- Location of each issue
- Total coverage
- Residual risk
Example of metrics available with Diligence Fuzzing (Pro Mode)
4. Complement manual audits
Fuzz testing doesn’t eliminate the need for a manual audit, but it can improve your relationships with auditors. For example, describing what you expect your contract to do using Scribble’s human-readable syntax makes it easier for auditors to reason about the correctness of your code.
More importantly, you can think of fuzzing as bridging the gap between lightweight testing and manual audits. Diligence Fuzzing doesn’t replace auditors, but provides an additional security layer by exploring those edge cases an auditor might miss while going over your codebase.
You’re probably thinking: “All of this sounds awesome! How can I start using Diligence Fuzzing now?”
Great! We’re also excited for you to start running fuzz tests and catching sneaky bugs with Diligence Fuzzing. But first, you’ll need to understand how using the service works. We’ve put together the following high-level description of the Diligence Fuzzing workflow for this purpose:
How Diligence Fuzzing works for testing in three simple steps
ConsenSys Diligence offers Fuzzing as a Service (FaaS) for blockchain developers, enterprises, and organizations. The Diligence Fuzzing service is an easy-to-use solution for bootstrapping smart contract security and follows a three-part process:
Step #1: Create smart contract specifications using Scribble
Using the Scribble language, you can specify your smart contract’s properties and verify these using property-based analysis methods. Properties can cover conditions necessary for the security of a smart contract (e.g., invariants on contract balances). The output of this process is an instrumented version of your code that can be used for runtime testing in the fuzzing client.
Step #2: Submit your contract for fuzzing
Active Diligence Fuzzing subscribers can submit (instrumented) Solidity code for analysis via the fuzzing CLI. From this point onwards, everything is pretty much automatic. You can sit back and look at a dashboard that dynamically updates as the fuzzing engine hammers away at your code looking for security vulnerabilities.
Step #3: Fix identified vulnerabilities
Diligence Fuzzing provides a report showing violated assertions and identifies edge cases that you may miss during manual testing. You can now fix any bugs or vulnerabilities, ensuring that your user’s funds are never at risk!
Start fuzzing your way to secure smart contracts
After a successful private beta program, Diligence Fuzzing is now open to the public. If you’re a new user, we recommend reading Fuzzing ERC20 Contracts With Diligence Fuzzing as well as the Fuzzing docs and Scribble docs to get up to speed with fuzzing smart contracts.
Diligence Fuzzing—like any automated testing tool—doesn’t replace manual auditing. But using it to find smart contract bugs provides an extra layer of security and allows developers to ship secure code much faster.
Are you a developer or enterprise that needs custom service level agreements (SLAs), information about pricing, or have questions about Fuzzing? Contact the team—we’re here to help. Even better, you can start your journey towards building secure smart contracts today by signing up for Fuzzing as a Service.