Recent updates made erlperf a lot more useful. Starting this thread to discuss erlperf - a tool designed to answer a simple question “is X faster than Y in Erlang?”
This tool existed since 2019, but was only targeting relatively slow functions taking micro and milliseconds to execute. With the newest updates, it works well for micro-benchmarking too.
Ask me anything on how to use it - I am bad at documenting things, but good at looking at questions and then putting answers into documentation.
Is there a way to hook into the post-benchmark reporting so that customized reporting on collected statistics could take place? For example: benchee supports plugins that allows for HTML output like this.
As a simplified example of this…
Given a fake comparison between foo(). and bar(). (all the numbers are made up, don’t look too closely at them). Can I customize the output to look like benchee’s and if so, how?
Default erlperf console output:
Code || QPS Time Rel
foo(). 1 100 Mi 5 ns 100%
bar(). 1 50 Mi 10 ns 50%
Default benchee console output:
Name ips average deviation median 99th %
foo(). 100 M 5 ns ±10% 5 ns 20 ns
bar(). 50 M 10 ns ±200% 15 ns 200 ns
Comparison:
foo(). 100 M
bar(). 50 M - 2.00x slower +5 ns
Memory usage statistics:
Name Memory usage
foo(). 100 KB
bar(). 200 KB - 2.00x memory usage +100.00 KB
That embarrassing moment when I realise I reinvented the wheel. I wish I was aware about benchee, it might’ve saved me from doing erlperf at all.
erlperf takes a different approach. It is not a framework, so there is no plugin support. There are several ways to produce HTML (or any other format) output:
erlperf as a library application. Create your own application, add erlperf as a dependency, and then use erlperf:run/1,2,3 to get the benchmark results programmatically. Essentially run/3 will return a sample or a list of samples that you can format accordingly (or output HTML)erlperf_monitor, allowing to report continuous benchmark samples. Essentially, once a second erlperf_monitor sends monitoring samples to processes registered in a specific pg group. It is possible to implement a handler that would send the data to any telemetry tool around, and they will generate nice reports. Probably I should document that…I should definitely look at benchee and see what else erlperf is missing or doing wrong. Thanks for the pointer!
Thank you for developing erlperf. I see so many projects written in another BEAM language that could have been developed in plain Erlang.
I love benchee! @PragTob has done truly epic work there. I think there’s room for two in this ecosystem, especially if one (erlperf) has a shot of inclusion in OTP itself. I’ve had a good time using erlperl.
I think one of the main things missing besides nicer output etc. is something that Erlang/OTP doesn’t currently have, the equivalent of .exs files.
A common way of doing benchmarks in an elixir project is to make an exs file you run. The equivalent right now is an escript but something like .exs provides less friction. I don’t have to define a function or pass in flags. I just open a file, define some code (or even modules) and off I go.
I just wanna highlight that Benchee very much is also just a library and no framework. It does pretty much what you describe with 1. - it returns a Suite which includes all needed information, samples, statistics, system information - you name it. You can just call the formatter manually afterwards if you wish. Being able to configure it is done for convenience (you can also pass in anonymous functions), as well as a slight performance improvement (formatters define 2 functions - one pure and one impure for IO, the first function can be run in parallel across all formatters).
Definitely! There’s value to having a benchmarking library in erlang more accessible to the eco system without pulling in the whole elixir eco system. I’ve contemplated porting benchee’s core/parts of benchee’s core for that exact reason. But let’s say… I have some plans to make it even more widely reusable but well… time.
And also, an OSS project is just an OSS project. If one has fun working on it, that’s fine - it’s a hobby.
On the “eco system” side, I think it’s good to know and highlight what differentiates you/what you add to the eco system vs. collaborating on an existing project. I like how beamchmark highlights it. I had a similar highlight directed at benchfella, back when it was the most popular elixir benchmarking solution and I tried to get a clear on mind as to why I’m writing a new one 
I implemented this in v. 2.2.0 (with other updates and bugfixes). New command line switch -r (--report) allows switching between basic and full reports. Basic report stays the default when less than 10 samples are collected.
$ ./erlperf 'rand:uniform().' 'crypto:strong_rand_bytes(2).' --samples 10 --warmup 1
Code || Samples Avg StdDev Median P99 Iteration Rel
rand:uniform(). 1 10 16611 Ki 0.20% 16614 Ki 16664 Ki 60 ns 100%
crypto:strong_rand_bytes(2). 1 10 1804 Ki 0.79% 1797 Ki 1829 Ki 554 ns 11%
erlperf 'r(_, S) -> {_, N} = rand:uniform_s(S), N.' --init_runner 'rand:seed(exsss).'
why two args?
erlperf 'r(S) -> {_, N} = rand:uniform_s(S), N.' --init_runner 'rand:seed(exsss).'
one arg has the same result
a bit confusion 
The result is different: two-argument variant generates the actual random number sequence:
./erlperf 'r(_, S) -> {Rand, N} = rand:uniform_s(S), io:format("~p~n", [Rand]), N.' --init_runner 'rand:seed(exsss).' -s 3 -d 10
0.38573201769169996
0.850672148653192
<...>
While single argument variant repeatedly generates the very same number:
./erlperf 'r(S) -> {Rand, N} = rand:uniform_s(S), io:format("~p~n", [Rand]), N.' --init_runner 'rand:seed(exsss).' -s 3 -d 10
0.771681857456621
0.771681857456621
0.771681857456621
0.771681857456621
The difference is explained here.
oh, thanks, I get it
I havn’t find the doc link in github 
BTW, would you like to share your doc maker, it look cool!
It’s Elixir’s ExDoc, running via @starbelly rebar_ex_doc plugin. I haven’t found a good overview on how to use it, so I used @MononcQc recon library as an example.
2 posts were split to a new topic: Rebar3_ex_doc documention feedback
It’s been a while since I updated it… new version, 2.2.2, does not change logic, but finally allows to print generated modules (using --verbose, or -v, option).
When is it useful? Mostly to fight shell escaping, - to know, what is the generated code, after escaping.