Fuzzer or AI: which finds bugs better? Fuzzilli and BigSleep on JS engines

In this article
Project Zero compares two automated systems Google uses to find vulnerabilities in V8, Chrome's JavaScript engine: Fuzzilli, a fuzzing tool that generates JavaScript code, and BigSleep, an LLM (large language model)-based bug-finding system. Over the past six months, BigSleep found slightly more vulnerabilities (45 vs. 37), but only about 25% of the bugs overlapped. Separate experiments suggest a theoretical overlap of about 60%. Whichever figure is right, a significant share of vulnerabilities can be detected by only one of the two approaches, so the two complement each other.
Background and scope of the comparison
The comparison revolves around three questions: how many bugs each approach finds in the real world, how much the two overlap, and what shape the bugs found by only one side take. For a direct comparison, only two systems are considered, and both run in a production environment, automatically scanning newly added code. Fuzzilli runs continuously on many instances, usually picking up new changes within a few hours. BigSleep currently runs once a day, and is asked to review new commits and write proof-of-concept testcases for the bugs it finds. Both report bugs through Clusterfuzz, a basic crash triage tool that then creates reports in Chrome's issue tracker. The only target is V8, a high-performance JavaScript and WebAssembly engine with a mature testing and fuzzing setup; bugs here are often complex, which makes it suitable for measuring the ability to find deep bugs in well-tested software.
Figures from the production environment
The data comes from the Chromium bug tracker, where each system has its own hotlist, covering April to September 2026, because the automated BigSleep pipeline has only been running since April. The original queries are:
Filtering by type:vulnerability reverses the result: Fuzzilli has 178 reports, BigSleep has 95. However, this data still contains duplicate reports and reports that require no action, such as bugs closed as WAI. In some cases a bug created by Clusterfuzz was closed automatically on the same day because the testcase no longer reproduced, with no interaction from developers. It is likely a duplicate of another bug, but the crash signature, inferred from the stack trace, differed slightly and so was missed.
A more reasonable measure is therefore actionable bugs: those not marked as duplicate or WAI and referenced by at least one fix commit. Counting only Type-Vuln bugs, the result is nearly a tie, with BigSleep slightly ahead: 45 vs. 37. As for cost per bug, a comparison is not yet possible because neither system has been optimized for that ratio.
Overlap between the two systems
Overlap was measured in two ways. The first relies on the bug tracker, looking for bugs that both systems found. The second is a separate experiment, having BigSleep try to rediscover bugs that Fuzzilli had previously found.
With the bug tracker approach, duplicates have to be handled in several layers:
Many duplicates never have an entry in the bug tracker because they were merged at the Clusterfuzz layer, so the Clusterfuzz database must be queried as well. Result: among actionable vulnerability reports, the overlap is about 25% (20 out of 81), with a slightly larger share coming from Fuzzilli finding a bug first and BigSleep finding it later.
Experiment on rediscovering old bugs
The experiment used the BigSleep configuration closest to production, asking the agent to review the commit most likely to have introduced each past Fuzzilli bug. The reverse direction, having Fuzzilli rediscover old BigSleep bugs, is harder to set up because a fuzzer cannot easily accept a similar starting point. In addition, most past BigSleep bugs were found after the fuzzers had already run on the buggy build for months, even years.
The data preparation process went through the following steps:
Take the list of old Fuzzilli bugs from early 2023 to mid-2026, about 1150 raw bugs.
Keep type:vulnerability, leaving about 600 vulnerabilities.
Keep bugs that reproduce on an x64 debug+asan build of the d8 shell from Chrome's public build infrastructure, and that reproduce without special flags. Only the standard flags --fuzzing, --jit-fuzzing and --expose-gc are allowed, because older V8 versions lack a flags-only safe mode and saying in advance which flags to use would amount to a hint. That leaves 132 bugs.
Remove implicit duplicates, leaving 112 bugs.
Convert each bug into a code review problem based on the commit most likely to have caused it, identified for example by bisection. This disadvantages BigSleep because it sometimes picks the wrong, unrelated commit, and there is only one commit per bug whereas in practice the agent could find the bug from a different commit touching similar code.
Run the agent 25 times per commit, asking it to find bugs and prove them with a testcase that crashes the buggy build. Each run stops when it finds a crash, using a configuration equivalent to production at that time, including the underlying model Gemini 3.6 flash.
Post-process by checking whether the testcase crashes the pre-fix build but not the post-fix build, to determine whether the agent found the right bug or a different one.
Result: BigSleep rediscovered about 62% of the original bugs, found a different bug (now fixed) at about 12% of commits, and found no bug at about 26% of commits.
Memorization, difficulty, components and prompts
On difficulty, measured by how long Fuzzilli took to find a bug, the result is statistically significant though the effect size is small, and it runs against expectations: bugs that are hard for fuzzers are also likely to be hard for AI agents. The fuzzers should therefore keep running.
By component, overlap is markedly lower for compiler bugs, the area where Fuzzilli can be most specialized, and higher for the rest of the engine.
On prompts, two additional experiments, run only 5 times per bug on bugs that had not been rediscovered: starting from the buggy file or function instead of the commit, and stating explicitly that assertion failures and similar crashes are also acceptable rather than only memory corruption bugs. In total, 10 more bugs were rediscovered: 8 from a different starting point and 7 from the reworded prompt, of which 5 appeared in both. Combined, the rate of finding the right bug rises to about 70%, and finding some kind of bug at about 84% of starting points. Caution is warranted, since similar tuning for fuzzers could also raise their detection rate.


