Low-overhead native coverage API for coverage-guided fuzzing

Chat / Discussions
,
1

Hi,

I would like to discuss whether the native coverage support introduced in OTP 27 could expose a lower-level API suitable for coverage-guided fuzzing and other high-frequency tooling.

OTP 27 already provides lightweight native coverage implemented in the JIT. Code can be instrumented for function or line coverage, and coverage can be queried using code:get_coverage/2. For line coverage, modules are compiled with line_coverage, and the VM performs the actual coverage collection.
My use case is an Erlang fuzzer called efz .

At the moment efz implements its own instrumentation. A parse_transform walks the AST, assigns ProbeIds to interesting control-flow locations and injects calls similar to:

efz_cov_rt:hit({Module, BuildId, ProbeId})

The runtime records reached probes in ETS for each execution. After the testcase finishes, the collected probes are converted into a set and compared with global coverage to answer the important fuzzing question:

Did this input discover any new coverage?

Conceptually:

instrumentation
    -> hit()
    -> per-execution ETS
    -> snapshot
    -> set comparison
    -> new coverage?

This works, but calling Erlang code and touching ETS on every probe is relatively expensive.For coverage-guided fuzzing we would ideally want something closer to the model used by AFL++ / SanitizerCoverage:

execute testcase
        |
        v
cheap VM/JIT coverage instrumentation
        |
        v
compact coverage state
        |
        v
current & ~global
        |
        v
new coverage?

The interesting part is that BEAM already performs the expensive/important part — low-overhead instrumentation inside the VM.

The remaining problem seems to be the interface used to read the result.

code:get_coverage/2 returns Erlang terms describing functions/lines and their coverage state. That is perfectly reasonable for normal coverage reporting, but a fuzzer can execute thousands or millions of testcases and needs to query coverage after every execution.

In that scenario we potentially do:

internal compact coverage representation
        ->
construct Erlang list/tuples
        ->
fuzzer converts it back into a bitmap/set
        ->
compare with global state

It would be useful to avoid that conversion completely.

I see three possible APIs.

1. Export compact coverage

Something along the lines of:

code:get_coverage_bitmap(Module) -> binary().

The caller would maintain its own global bitmap and perform novelty detection.

This is flexible, but exposes some representation of the internal coverage map as public API.

2. Take + reset

Something like:

code:take_coverage(Module) -> binary().

which reads the current coverage state and resets it in one operation.

This maps quite naturally to fuzzing:

reset
execute
take coverage
compare
repeat

3. Opaque coverage snapshot

Another possibility is to avoid exposing the bitmap representation at all.

For example:

Snapshot = code:coverage_snapshot(Modules),

run_testcase(),

case code:coverage_has_new(Snapshot) of
    true ->
        interesting;
    false ->
        not_interesting
end.

Or perhaps:

{HasNewCoverage, Snapshot1} =
    code:coverage_check_and_advance(Snapshot0).

In this model the snapshot would be an opaque handle. ERTS would perform the comparison internally.

The caller would only ask:

Has any new coverage appeared compared with this snapshot?

This has an interesting property: the internal coverage representation remains completely private to ERTS and could change between OTP releases.

It would also allow the VM to implement the equivalent of:

current & ~baseline

without constructing potentially large Erlang lists.

My main question is therefore:

Which abstraction would fit Erlang/OTP better?

  • exposing compact coverage as a binary;
  • a take/reset operation;
  • or an opaque snapshot/checkpoint API that performs coverage comparison inside ERTS?

The motivation is coverage-guided fuzzing, but I suspect a low-overhead native coverage API could also be useful for profilers, testing tools and other instrumentation where repeatedly materializing the complete coverage result as Erlang terms is undesirable.

I would be interested in opinions from people familiar with the JIT/native coverage implementation, especially whether an opaque snapshot API would fit the existing coverage architecture or whether exposing a compact representation would be preferable.
Link:
OTP 27 — Native Coverage Suppor
OTP commit introducing native coverage
repo efz-fuzz