Open sourcing erlfuzz
What is it?
erlfuzz is a new tool for generating random Erlang programs. It is designed to test various pieces of the Erlang ecosystem such as the erlc compiler, the BEAM VM, eqWAlizer, dialyzer, infer for Erlang, erlfmt, etc…
It has been under development since last summer, and has just been open sourced: GitHub - WhatsApp/erlfuzz: erlfuzz is a small standalone generator of random erlang programs used to fuzz the erlang compiler and VM, as well as other tools such as erlfmt, dialyzer, eqWAlizer, etc... It does not currently use coverage information or do anything especially clever, except for carefully handling Erlang's scoping rules.
Strengths and limitations
Its main features are the following:
-
Support for most of the language, including some new features such as maybe expressions and map comprehensions.
-
A very fine-grained understanding of Erlang’s scoping rules, allowing it to generate complex code that reuses variables without generating invalid Erlang code.
-
An automated test case minimizer, that usually reduces the test cases from several thousand lines to a couple of lines.
It currently has some limitations to be kept in mind:
- It does not use any coverage information
- It generates quite a few ill-typed programs
- The test case minimizer only works on test cases found by the fuzzer
Results
Erlfuzz has found so far:
- 62 bugs in erlc
- 7 bugs in the VM
- 6 bugs in dialyzer
- 5 bugs in eqwalizer
- 2 bugs in infer
- 1 bug in erlfmt
While most of these are crashes of the tools on rare code patterns, several bugs are more interesting. For example:
- Running the VM in debug mode discovered that it was emitting completely wrong (and memory unsafe!) code in release mode for some guards: [jit] "beam/jit/beam_jit_args.hpp:243:ArgRegister() Assertion failed: isRegister()" · Issue #6634 · erlang/otp · GitHub.
- Comparing the result of running the VM in both interpreter and JIT modes discovered another issue with the JIT where it would in some conditions compute the wrong result for negation of floats ([vm] Floating point discrepancy (-0.0 vs +0.0) between `-emu_flavor emu` and `-emu_flavor jit` · Issue #6717 · erlang/otp · GitHub)
- Comparing the result of running erlang code that had been compiled with and without optimizations discovered that a wrong optimization could cause some
badkeyexceptions to be silently swallowed: erlc folds away an unused map update that should throw an exception · Issue #6960 · erlang/otp · GitHub
erlfuzz has also been successfully used to detect bugs in PRs before they were merged. The most extreme example is Lift restrictions for matching of binaries and maps by bjorng · Pull Request #6415 · erlang/otp · GitHub (which itself was motivated by corner cases of the language discovered by erlfuzz). It went through multiple rounds of revisions as the fuzzer found 8 different bugs in it over time.
A more surprising benefit of this work was to improve the official Erlang documentation. I tried to make Erlfuzz able to generate any string that corresponds to a valid Erlang program according to the documentation, and this revealed several places where the documentation was either ambiguous, or outright missing some rules. E.g. Address a few omissions in documentation of comprehensions by bjorng · Pull Request #6929 · erlang/otp · GitHub.
Finally, Björn Gustavsson made a creative use of the fuzzer, adding crashes on lines of the compiler that were not covered by the test suite, and getting minimized test cases to add to the suite for many of these lines, improving the test coverage: Cover more code in the compiler application by bjorng · Pull Request #6872 · erlang/otp · GitHub.
Future work
The main priority at this point is to integrate erlfuzz into CI for all of the tools mentioned above.
I hope to cover the last few missing parts of the language later on, as well as try to generate more well-typed programs to hit code deeper in tools like eqWAlizer.
Thanks
I owe many thanks to Björn Gustavsson and John Hogberg from Ericsson for fixing with stunning speed all of the erlc and BEAM bugs that I reported.
I’m also grateful to my colleagues at WhatsApp that helped me with various aspects of this work, from getting eqWAlizer and infer working to reviewing all of my patches.