Hi,
I was wondering what the difference is in testing tool in elixir vs Erlang and is it something that Erlang could improve?
Have heard that Elixir is better at that and is something that companies discuss when choosing either elixir or Erlang.
8 Likes
eunit allows very powerful and very flexible test definitions and structuring. It’s one of the best unit test frameworks I’ve used, test fixtures are amazing and it’s childsplay to define setups, teardowns and a wide variety of behaviours.
ct, or common test, is also a very neat framework for when you need something bigger than unit tests, and I’ve very successfully used it for all levels of testing, from functional level to node level. Testing interfaces like TCP or HTTP is quite trivial, and with ports it is very possible to make it communicate with systems written in any language directly. Defining groups with different dependencies is simple.
assert.hrl is an MVP in testing. Great usability, detailed error messages.
I’m assuming Elixir’s testing frameworks probably use these under the hood so I doubt it does something that Erlang can’t do, so I don’t really know what would make ExUnit objectively better, but it does have the magic of Elixir macros giving tests a very specific look. For example, a keyword used specifically for defining test functions.
test "Test description" do
assert something
end
which admittedly is a very nice way to avoid the boilerplate of naming a function and providing a description.
something_test_() ->
[{"Test description", fun() -> ?assert(Something) end}].
5 Likes
In ExUnit I really miss the ability to programmatically generate tests from previous results through lazy generators at test runtime.
Even though I rarely use this feature, I think it is missing in elixir.
5 Likes
Maybe you already see this video but still. Take a look to the video Testing Erlang and Elixir through PropEr Modeling with Fred Hebert | Erlang Solutions Webinar - YouTube where @MononcQc provide also info about testing Elixir VS Erlang and compare them.
5 Likes
I would love to hear the opposite too: what is missing in Elixir coming from Erlang (and other langs).
ExUnit is not built on top of ct/eunit, it is its own framework, although I agree the assert macros in them are quite similar.
Something that I like a lot in ExUnit is the error reporting. Take the following test:
defmodule ExampleTest do
use ExUnit.Case, async: true
test "example" do
assert "hello" == "hello!"
end
end
It fails with this report (screenshot):
In the report:
-
The line right after the test name shows the location but you can also copy and paste it into the command line to run that test exclusively, like
mix test test/example_test.exs:5 -
It shows the code that failed and it recognizes the left- and right-hand sides, showing a diff between them
-
We can also compute the diff between patterns. For example, imagine you are asserting on
#{foo := bar} = Map. If it fails, we show which keys match, which keys do not, and bring those values to the top to make it easy to find diffs
The goal is to make it easy for you to spot changes, act on the error reports, and re-run failed tests. I do not have in depth experience with eunit/ct, so I am not sure if those features also exist there, but the error report is my favorite ExUnit feature. 
10 Likes
Do you have an example of any language/framework with similar feature?
4 Likes
I greatly prefer CT. I know there were gaps with ExUnit when I had to use it – one was just related to how CT provides a lot of options for setting up per test case work directories, plus the remote node testing capabilities and test grouping features – that may have been filled in at this point.
I’m also one of the few who like CT’s html output since I can send copy it as an artifact in CI, like in CircleCI and go investigate there, only issue in the case of tools like CircleCI is you don’t have the past runs available in those html files.
There are certainly quirks to CT which we’ve tried to improve in rebar3.
But I also don’t “get” eunit, so exunit likely being more similar to eunit than CT means I came to it already biased :). I know eunit is powerful with its ability to generate tests and thats why there are all the macros and ?_ underscore macros that return functions… but I’ve never taken the time to get that stuff to click.
Are there features of ExUnit people miss or find lacking in CT or EUnit?
Oh, I did learn that ExUnit comes with some assertions that I always have to define myself because they don’t come in assert.hrl, like asserting a message is received. I’ve been meaning to propose/PR some of those to Erlang stdlib to get the ball rolling.
7 Likes
Is it for distributing tests across nodes or for testing code that requires multiple nodes? (or both?). IIRC we got test case work directories in v1.11.
3 Likes
Erlang’s eunit. They call it “lazy generators”.
https://www.erlang.org/doc/apps/eunit/chapter.html#Lazy_generators
5 Likes
I think most people in thread have already said what I would say in regards to things I miss when in CT and things I miss when in exunit.
However, one thing that has been commented on already but I will take a little further is around assertions via assert.hrl.
We don’t get pretty diffs, on failed assertions such as ?assertMatch/2 and ?assertThrow/2 but what’s more in order to get a diff at all we have to put the term right in the macro.
As an example :
?assertThrow({ror,{rebar3_hex_user,bad_local_password}}, rebar3_hex_user:decrypt_write_key(<<"mr_pockets">>, BadKey))
We get :
Failure/Error: ?assertException(throw, , rebar3_hex_user : decrypt_write_key ( << "mr_pockets" >> , BadKey ))
expected: exception { throw , { rror , { rebar3_hex_user , bad_local_password } } , [...] }
got: exception {throw,
{error,{rebar3_hex_user,bad_local_password}},
[{rebar3_hex_user,decrypt_write_key,3,
[{file,
...
Now if I define the expected error in variable right above the assertion like so:
Eh = {ror,{rebar3_hex_user,bad_local_password}},
?assertThrow(Eh, rebar3_hex_user:decrypt_write_key(<<"mr_pockets">>, BadKey))
We get :
Failure/Error: ?assertException(throw, , rebar3_hex_user : decrypt_write_key ( << "mr_pockets" >> , BadKey ))
expected: exception { throw , Eh , [...] }
got: exception {throw,
{error,{rebar3_hex_user,bad_local_password}},
[{rebar3_hex_user,decrypt_write_key,3,
[{file,
Same thing with ?assertMatch/2, etc.
I also miss in CT from exunit one general assert macro. I can say assert :foo = :bar or assert :bar == foo or assert bool and it does the right thing depending on the op.
I remember some of us having a discussion around this and it’s been some time, but I believe the consensus was that a parse transform would have to be written to handle this.
4 Likes
Oh, that’s very neat then!
But yeah, I think the point still stands, eunit and ExUnit do approximately the same stuff, it’s hard to say that one is objectively better than the other. But the ExUnit modules can look very pretty, almost by definition. I think eunit modules can look very nice as well with the use of test_() functions, fixtures and nicely defined lists of test objects, but very often they just look… not so great.
One family of macros that I’ve had to write myself several times is polling asserts, like ?assertEqual(Expected, Actual, Frequency, Timeout). When asserting in CT where the system under test may need some time to reach a wanted state, I don’t want to assert too quickly and fail, but I don’t want to add an error-prone timer:sleep/1 that will almost definitely either fail or waste more time than necessary.
If I write my own polling function it’ll either be ugly or hide all the nice information that the assert macros provide. So my solution has been to steal the ?assert definition and modify it to check recursively before failing.
4 Likes
Yup, I also do an assert like that that I call ?UNTIL in many projects that would be nice to standardize and add to assert.hrl.
3 Likes
Once upon a time I wanted to add Elixir wrapper on top of Common Test, as I find some of it’s features really appealing.
And I would really 
to see the HTML and JUnit reporters built in into ExUnit. At the same time I think that coverage reporting for ExUnit is at the same time too flexible (I doubt that there is any other coverage monitoring system other than :cover) and too rigid (to use multiple reporters, for example covertool together with default CLI reporter, is needlessly complex).
4 Likes