Erlperf: benchmarking Erlang code

Recent updates made erlperf a lot more useful. Starting this thread to discuss erlperf - a tool designed to answer a simple question “is X faster than Y in Erlang?”

This tool existed since 2019, but was only targeting relatively slow functions taking micro and milliseconds to execute. With the newest updates, it works well for micro-benchmarking too.

Ask me anything on how to use it - I am bad at documenting things, but good at looking at questions and then putting answers into documentation.

15 Likes

Is there a way to hook into the post-benchmark reporting so that customized reporting on collected statistics could take place? For example: benchee supports plugins that allows for HTML output like this.

As a simplified example of this…

Given a fake comparison between foo(). and bar(). (all the numbers are made up, don’t look too closely at them). Can I customize the output to look like benchee’s and if so, how?

Default erlperf console output:

Code                             ||        QPS       Time     Rel
foo().                            1     100 Mi       5 ns    100%
bar().                            1      50 Mi      10 ns     50%

Default benchee console output:

Name                  ips        average  deviation         median         99th %
foo().              100 M           5 ns       ±10%           5 ns          20 ns
bar().               50 M          10 ns      ±200%          15 ns         200 ns

Comparison:
foo().              100 M
bar().               50 M - 2.00x slower +5 ns

Memory usage statistics:

Name           Memory usage
foo().               100 KB
bar().               200 KB - 2.00x memory usage +100.00 KB
5 Likes

That embarrassing moment when I realise I reinvented the wheel. I wish I was aware about benchee, it might’ve saved me from doing erlperf at all.

erlperf takes a different approach. It is not a framework, so there is no plugin support. There are several ways to produce HTML (or any other format) output:

  1. Use erlperf as a library application. Create your own application, add erlperf as a dependency, and then use erlperf:run/1,2,3 to get the benchmark results programmatically. Essentially run/3 will return a sample or a list of samples that you can format accordingly (or output HTML)
  2. There is also (undocumented) API used erlperf_monitor, allowing to report continuous benchmark samples. Essentially, once a second erlperf_monitor sends monitoring samples to processes registered in a specific pg group. It is possible to implement a handler that would send the data to any telemetry tool around, and they will generate nice reports. Probably I should document that…

I should definitely look at benchee and see what else erlperf is missing or doing wrong. Thanks for the pointer!

9 Likes

Thank you for developing erlperf. I see so many projects written in another BEAM language that could have been developed in plain Erlang.

5 Likes

I love benchee! @PragTob has done truly epic work there. I think there’s room for two in this ecosystem, especially if one (erlperf) has a shot of inclusion in OTP itself. I’ve had a good time using erlperl.

I think one of the main things missing besides nicer output etc. is something that Erlang/OTP doesn’t currently have, the equivalent of .exs files.

A common way of doing benchmarks in an elixir project is to make an exs file you run. The equivalent right now is an escript but something like .exs provides less friction. I don’t have to define a function or pass in flags. I just open a file, define some code (or even modules) and off I go.

2 Likes

I just wanna highlight that Benchee very much is also just a library and no framework. It does pretty much what you describe with 1. - it returns a Suite which includes all needed information, samples, statistics, system information - you name it. You can just call the formatter manually afterwards if you wish. Being able to configure it is done for convenience (you can also pass in anonymous functions), as well as a slight performance improvement (formatters define 2 functions - one pure and one impure for IO, the first function can be run in parallel across all formatters).

Definitely! There’s value to having a benchmarking library in erlang more accessible to the eco system without pulling in the whole elixir eco system. I’ve contemplated porting benchee’s core/parts of benchee’s core for that exact reason. But let’s say… I have some plans to make it even more widely reusable but well… time.

And also, an OSS project is just an OSS project. If one has fun working on it, that’s fine - it’s a hobby.

On the “eco system” side, I think it’s good to know and highlight what differentiates you/what you add to the eco system vs. collaborating on an existing project. I like how beamchmark highlights it. I had a similar highlight directed at benchfella, back when it was the most popular elixir benchmarking solution and I tried to get a clear on mind as to why I’m writing a new one :slight_smile:

4 Likes