Diligence Fuzzing Now Supports Foundry Projects

Announcing Diligence Fuzzing support for all Foundry developers to ensure easy and efficient smart contract security.  

In our post on charting the web3 security landscape, we explained the importance of rigorously testing smart contracts before launching publicly. We also highlighted the benefits of using automated techniques—such as fuzzing—to detect software vulnerabilities at scale and improve security for blockchain applications. 

In this context, we’re pleased to announce the newest feature available to users of Diligence Fuzzing: (open beta) support for fuzzing Foundry projects. The latest upgrade to Diligence Fuzzing achieves key goals, such as reducing onboarding friction for builders new to the Fuzzing-as-a-Service (FaaS) platform and improving UX for existing Foundry users. Most importantly, it strengthens our commitment towards creating a safe and secure web3—one where million-dollar hacks and exploits are the exception, not the norm. 

This blog provides more information on the new release (including its benefits) and explain how new and existing users can take advantage of this feature. Plus, we’ll share some exciting news regarding changes to pricing plans for users (spoiler: Diligence Fuzzing-as-a-Service now has a free plan). 

Recap: What is Diligence Fuzzing? 

Diligence Fuzzing (aka Fuzzing-as-a-Service) is an automated analysis tool for detecting vulnerabilities in Solidity smart contracts. Inspired by coverage-guided fuzzing approaches used in state-of-the-art fuzzers like AFL and LibFuzzer, Diligence Fuzzing offers increased code coverage and faster vulnerability detection compared to existing fuzzing tools used for analyzing smart contracts. In fact, recent benchmarks show Diligence Fuzzing outperforms competitors (in overall code coverage, time-to-coverage, and no. of bugs found) by significant margins. 

After running a closed beta program to gain feedback from early users, we announced the general availability of Diligence Fuzzing in April and have continued refining the FaaS product to better serve users. For example, we adjusted pricing plans—you asked, we listened!—to make Diligence Fuzzing more accessible. Now, developers and project teams can enjoy more features on a free subscription and benefit from an expanded range of paid subscription options—with plans tailored to each project/developer’s unique needs and requirements. 

We (highly) recommend reading the post introducing Diligence Fuzzing for details on the science behind fuzzing and the how and why of integrating Fuzzing-as-a-Service into your web3 project’s secure development lifecycle (SDLC). Having (re-)introduced Diligence Fuzzing, we’re ready to delve into the details of our newest and most exciting feature yet: support for fuzzing Foundry projects.

Why is Diligence Fuzzing adding support for Foundry projects? 

Launched in 2021, Foundry’s development framework has rapidly become popular for its speed, efficiency, and modularity. Even better, Foundry is built with security in mind and ships with a built-in fuzzer for running property-based tests.These factors, coupled with surging demand for Foundry (the 2022 Solidity Developer Survey reveals a 28.4% YoY increase in usage among Solidity developers), made support for Foundry projects in Diligence Fuzzing a no-brainer. 

“Lots of developers are getting Foundry-pilled, so why not?” is slightly less accurate, but you get the idea.

In particular, it represents an opportunity to extend the benefits of Diligence Fuzzing to a new class of users (developers already writing property tests in Foundry) and improve flexibility for builders using the FaaS platform for smart contract testing. For context on the latter, Diligence Fuzzing now supports testing smart contracts from five of the most popular Ethereum development frameworks (Truffle, Hardhat, Brownie, DappTools, and the newest addition, Foundry). 

But, before going into the specifics of the new feature, it is important to address a question that some have asked in the past: “How is fuzzing with Foundry’s `forge test` different from fuzzing with Harvey (the engine powering Diligence Fuzzing)?” You’ll learn more about the differences between Foundry’s Forge and Diligence Fuzzing in subsequent sections, but we’ll do a quick comparison between both tools. That way, you have a useful heuristic to guide your decision:

1. Cost: Foundry is open-access and comes with a built-in fuzzer to analyze smart contracts. Diligence Fuzzing-as-a-Service is now free as well; previously it was only accessible via a paid subscription, but with the new “Explorer” plan builders can access Diligence Fuzzing for free to test the platform or upgrade to a paid plan to unlock more features. This way, developers can add the most powerful fuzzer in the market to their web3 security tool box—to complement measures like bug bounty programs—at no cost. In summary: Foundry’s fuzzer is free; Harvey (the fuzzer used in Diligence Fuzzing) is also free and performs better (more on this later). 

2. Features: Similar to other development frameworks for smart contracts (Truffle is a great example), Foundry provides a comprehensive set of functionalities for building, compiling, debugging, and deploying smart contracts. Hence, you can complete the end-to-end process of building a new dapp—including testing contracts—without leaving Foundry’s development environment (at least in theory).

Diligence Fuzzing is a standalone tool for analyzing smart contracts; however, it can now be used seamlessly with Foundry (you can even use Forge and Diligence Fuzzing interchangeably). Moreover, using Diligence Fuzzing means you get access to better performance and professional, in-depth campaign reports on a dedicated web UI (improving analysis and mitigation of discovered vulnerabilities). 

We can summarize this point of comparison thus: Foundry does a lot of things (including fuzzing); Diligence Fuzzing does one thing (fuzzing) and, crucially, does it well. This also provides an opportunity to answer the original question (running fuzz tests in Foundry vs. fuzzing with Harvey) more concisely:

• To find interesting bugs with a fuzzer you need some properties/specifications that the fuzzer should check. 
• Developers using Foundry for property and invariant tests have already spent time to create such properties. 
• Diligence Fuzzing allows developers to check those properties with a specialized, mature tool.

The last point highlights one crucial reason to start fuzzing your Foundry tests with Diligence Fuzzing: better performance. In the next section, we’ll provide even more reasons to integrate Diligence Fuzzing-as-a-Service into your Foundry development process.  

Three core benefits of Diligence Fuzzing for Foundry projects

Performance 

Submitting Foundry projects to Diligence Fuzzing for security analysis yields an immediate (and important) benefit: a boost in performance from using a fuzzer that implements state-of-the-art techniques—such as coverage-guided fuzzing and input prediction—to make fuzzing campaigns more effective. What do we mean by this?

Foundry’s Forge and Harvey (used in Diligence Fuzzing) are typically described as tools for “property-based testing” or “property-based analysis”.  Property-based testing analyzes software code by checking that certain properties of its output or execution are fulfilled for a broad range of inputs. A property testing tool generates test data automatically and, having found an input that violates a security property, reduces or “shrinks” it to the minimal test case required to reproduce the bug.

Property-based testing is often used to complement traditional unit testing—that is, testing an application using inputs chosen by the developer or tester. Traditional unit tests can test for known edge cases, and use simple inputs or specific inputs known to uncover bugs in previous testing. In comparison, a property testing tool will search for more complex inputs that violate security-related assertions and reveal subtle bugs in a smart contract. 

The difference between property-based fuzzers like Foundry’s Forge and Harvey boils down to how the generation of inputs is carried out. Forge uses “blackbox fuzzing” techniques which, as the name suggests, treats the fuzzing target (that is, one or more smart contracts) as a black box. Here, the fuzzer has no knowledge of the program’s internal structure and randomly chooses transaction inputs to generate/mutate during fuzzing runs. 

Blackbox fuzzers are simple to use and can generate a large number of test cases quickly. The latter increases the possibility of catching bugs that unit tests (involving limited inputs and fewer test cases) might miss; for example, this could be a withdrawal function in an escrow smart contract that works well for small amounts, but reverts for high amounts. 

But because inputs are generated “blindly”, blackbox fuzzing can run into the quantity-over-quality problem. For example, inputs generated by a blackbox fuzzer (the type used by Foundry’s Forge) may fail to touch certain parts of a codebase or simply exercise the same execution path on each run. This reduces the fuzzer’s capacity to generate/mutate inputs that trigger interesting behaviors or expose deep vulnerabilities. 

In graybox fuzzing, the target program is instrumented to enable collection of information about the program’s execution at runtime. Specifically, graybox fuzzers like Harvey use lightweight instrumentation to track which parts of a codebase are covered and identify inputs that exercise certain execution paths. Information about code coverage is then used to inform generation of test data, with inputs that exercise new code paths during execution saved to a “corpus” and used to seed the creation of new inputs. 

This approach—described as “coverage-guided fuzzing”—enables Harvey to intelligently generate inputs whose execution increases the amount of code that’s covered during testing. Naturally, this increases the fuzzer’s chances of finding bugs and assertion violations; after all, “you don’t find bugs in code you don’t cover.” It also adds some fresh context to results from our recent experiment where Harvey (the engine behind Diligence Fuzzing) found 25% more property violations than Foundry (and in a fraction of the time). 

The fundamental law of bug finding is No Check = No Bug. If the tool can’t check a system, file, code path, or given property, then it won’t find bugs in it. Assuming a reasonable tool, the first order bound on bug counts is just how much code can be shoved through the tool. Ten times more code is 10 times more bugs. — Al Bessey et al. (Using Static Analysis to Find Bugs in the Real World)

Harvey also performs “stateful fuzzing”—that is, generating sequences of transactions, instead of a single transaction, to uncover edge cases arising from bugs in protocol logic. Thus, Harvey may find bugs in existing (stateless) fuzz tests by considering multiple transactions instead of the single transaction that Foundry’s fuzzer would generate. Particularly, stateful fuzzing is crucial for detecting “stateful bugs” (no pun intended) that typically occur during complex state transitions and require more than one transaction to trigger violations of a security property or invariant(s) in one or more smart contracts. 

To add more context: most existing fuzzers pick a fixed sequence length and generate random transactions without considering previous transactions. Harvey instead curates a corpus of transaction sequences that increase coverage and mutates them—for instance, by inserting new transactions or mutating existing ones—to test deeper interactions between functions in a smart contract (the length of the sequence expands organically and is not explicitly bounded). Stateful fuzzing is admittedly a complex topic, but you can read our post on fuzzing with multiple transactions for more context around this feature and its value in finding vulnerabilities. 

We should also note that Foundry offers stateful fuzzing via invariant testing. The key difference is that Harvey performs stateful fuzzing out-of-the-box and requires no additional setup from a developer. Stateful fuzzing with Forge requires writing and configuring specific invariant tests, adding extra overhead to the testing process (which can and often does discourage developers from using the feature, even if it’s a powerful technique for exposing incorrect assumptions about a protocol’s business logic).  

Ease of use 

Integrating an external fuzzing tool into the Foundry development process can be tough, especially if the fuzzer in question requires self-hosted infrastructure or complex configurations. However, using Diligence Fuzzing to analyze Foundry contracts requires minimal configuration from the user’s side. That, as you can imagine, didn’t happen by accident. 

Our goal is to make fuzzing smart contracts a seamless process, so web3’s builders can focus on building and not wasting valuable time setting up security tooling. By dramatically reducing the effort required to use Diligence Fuzzing with Foundry projects, we’re making first steps to bringing seamless UX to all developers—regardless of the development framework.

When we say “seamless”, we really mean it—you can start fuzzing a Foundry project with Diligence Fuzzing in just three steps. Below is a quick rundown of the process (you can also check out the documentation on fuzzing Foundry projects): 

1. Run `pip3 install diligence-fuzzing` in your terminal to install the Diligence Fuzzing CLI (make sure you’re running Python 3.6 or a higher version) 

2. Sign up for a Diligence Fuzzing account and create an API key

3. Run fuzz forge test -k <your_api_key> to create a new fuzzing campaign for your Foundry project

Note that the workflow described above assumes you have installed Foundry and have configured/written some fuzz tests (which makes it super easy to get started). For context, you don’t need to write Scribble annotations to define properties or create extra harnesses to bootstrap a new fuzzing campaign. Since your Foundry property tests (hopefully) contain a lot of existing assertions and invariants, Harvey can immediately start generating fuzzed inputs capable of breaking those properties and invariants.

Compatibility 

Supporting analysis of existing Forge tests required extending the Harvey fuzzer to execute Foundry’s cheat codes. Cheat codes are one of Foundry’s most advanced features and provide developers with fine-grained control over the local EVM testing environment. For example, the `warp` cheat code allows for setting an arbitrary block number in the EVM, and `prank` allows for setting the address of `sender` or `tx.origin` for test transactions. 

In most cases, tests written in Foundry—and the corresponding assertions—will require one or more cheat codes. If an external fuzzer doesn’t support Foundry cheat codes, it may find it difficult to check certain assertions and confirm if the underlying security assumptions hold in different scenarios (which is the point of fuzzing in the first place)

Currently , Diligence Fuzzing offers (experimental) support for important Foundry cheat codes like `warp`, `prank`, `deal`, and `roll` (see full list of supported cheat codes). Support for additional cheat codes is a work in progress and we expect to support even more in the future (you can let us know which cheat codes you’d like to see prioritized!). 

Making industry-grade Fuzzing accessible to all web3 developers 

Supporting Foundry projects in Diligence Fuzzing is the latest step in our journey to make industry-grade fuzzing accessible to web3 builders looking to level up their security game. With improved UX, flexible pricing, and frictionless onboarding, existing Foundry users can now integrate Diligence Fuzzing-as-a-Service into the development stack with minimal investment in time and effort. 

Ready to start fuzzing Foundry projects in Diligence Fuzzing? Sign up for Diligence Fuzzing and read the Fuzzing documentation to get started. If you have any questions, don’t hesitate to reach out to the Diligence team. We also have a form in case you need to share feedback regarding the experience of using Fuzzing-as-a-Service!

All posts