Doctest - a library to test Erlang doc attributes

OTP 27 has implemented the -doc attribute, but what if its examples could be tested?
That’s the purpose of the doctest lib.

The motivation for it is this PR, which fixes some examples of the orddict module.

Example

Take this module:

-module(math).
-moduledoc """
A module for basic arithmetic.
""".

-export([add/2]).

-ifdef(TEST).
-include_lib("doctest/include/doctest.hrl").
% -doctest <see the options section>.
-endif.

-doc """
Adds two numbers together.

_Example_:
```erlang
1> math:add(0, 1).
1
2> math:add(
.. 1,
.. 1
.. ).
2
```
""".
add(A, B) ->
    A+B.

The doctest/include/doctest.hrl header contains a parse_transform definition:

-compile({parse_transform, doctest_parse_transform}).

Including it and compiling via rebar3 as test compile produces a file at `<project_cwd>/test/math_DOCTEST.erl with this content:

%%%---------------------------------------------------------------------
%%% Tests generated via doctest.
%%% - module: math
%%% - file: /home/williamthome/Projects/erlang/doctest_example/src/math.erl
%%%---------------------------------------------------------------------
-module(math_DOCTEST).

-ifdef(TEST).
-include_lib("eunit/include/eunit.hrl").

add_2_test() ->
    [
        ?assertEqual(math:add(0, 1), 1),
        ?assertEqual(math:add( 1, 1 ), 2)
    ].

-endif.

Then by running rebar3 eunit -v:

===> Verifying dependencies...
===> Analyzing applications...
===> Compiling doctest_example
===> Performing EUnit tests...
======================== EUnit ========================
file "doctest_example.app"
  application 'doctest_example'
    module 'math'
math_DOCTEST: add_2_test (module 'math_DOCTEST')...ok
=======================================================
  Test passed.

For more examples, please see the test module and the auto-generated test file.

Options

Some options can be provided via -doctest attribute, e.g.:

  • boolean(): enable or disable tests.
    -doctest true.
    
  • atom(): define the test module name.
    -doctest math_DOCTEST.
    
  • proplists:proplist() | all: define the functions to be tested.
    -doctest [add/2].
    
  • string() | {abs, string()} | {cwd, string()}: define the test file location.
    -doctest {abs, "/tmp/doctests"}.
    
  • map(): define all or partial options.
    -doctest #{
        enabled => true,
        module => math_DOCTEST,
        funs => [add/2],
        location => {abs, "/tmp/doctests"}
    }.
    

Requirements

doctest requires OTP 27 or higher.

Installation

% rebar.config
{profiles, [
    {test, [
        {deps, [{doctest, "0.1.0"}]}
    ]}
]}.

TODO

  • More tests;
  • Ability to test modules via function (maybe also escript) and not only via parse_transform, e.g.:
    doctest:module(math, _Opts = #{}).
    
  • Maybe add a mechanism to test non-exported functions, but probably this makes no sense;
  • Test on umbrella applications;
  • Improve documentation;
  • Implement a -moduledoc test like for -doc by adding a test function called moduledoc_test/0;
  • Add unbound variables support, e.g.:
    1> Foo = foo.
    foo;
    2> foo =:= Foo.
    true
    

I have no doubt that there are bugs. Just a few tests have been implemented.
My idea is to collect suggestions and know what you think about it.
So, suggestions are welcome :slight_smile:

10 Likes

New release

Version 0.2.0 is out!

Changes

Tests

Output to a file to run the tests is no longer required. Tests run directly from the parse_transform.

Options

Only enabled and funs are now valid options, for example:

-doctest #{
    enabled => true,
    funs => [add/2]
}.

Example

Using the same module from the first example…

Running rebar3 eunit:

Finished in 0.018 seconds
2 tests, 0 failures

Now, changing the first test to:

1> math:add(1, 1).
1

And running rebar3 eunit again:

Failures:

  1) doctest_parse_transform:-parse/4-fun-0-/0:26
     Failure/Error: ?assertEqual(1, math:add(1, 1))
       expected: 1
            got: 2
     %% eunit_proc.erl:583:in `eunit_proc:run_group/2`
     Output:
     Output:

Finished in 0.010 seconds
2 tests, 1 failures

Neat! I also started working on something similar since your PR, but take a different approach. I get the docs using code:get_docs/1 and then parse the markdown using shell_docs_markdown to get the code segments out of the markdown.

I’ve not come as far as you, just playing with some ideas of how it could work.

5 Likes

Nice! Didn’t know about those functions.

I can get chunks using code:get_doc, but shell_docs_markdown does not exist to me:

Erlang/OTP 27 [RELEASE CANDIDATE 3] [erts-15.0] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns]

Eshell V15.0 (press Ctrl+G to abort, type help(). for help)
1> shell_d
modules
shell_default:    shell_docs:

How can I use it in doctest?

1 Like

Thanks for the tips, @garazdawi! Now I can run tests via function/shell:

1> doctest:module(foo).
F
Failures:

  1) doctest:-parse/3-fun-0-/0:5
     Failure/Error: ?assertEqual(foo, foo:foo())
       expected: foo
            got: bar
     %% eunit_proc.erl:583:in `eunit_proc:run_group/2`
     Output: 
     Output: 

Finished in 0.008 seconds
1 tests, 1 failures

But I’m not using shell_docs_markdown:parse_md/1. I don’t know why it is undefined to me.

I’ll release a v0.3.0 soon.

1 Like

It was merged after 27.0-rc3 was released, so you need to have a relatively recent master version to use it.

2 Likes

New release

Version 0.3.0 is out!

Changes

doctest:module/1 was added to test modules via function/shell.

Example

Take this module:

-module(foo).

-export([foo/0]).

-doc """
```erlang
1> foo:foo().
foo
```
""".
foo() ->
    bar.

Running it via shell:

1> doctest:module(foo).
F
Failures:

  1) doctest:-parse/3-fun-0-/0:5
     Failure/Error: ?assertEqual(foo, foo:foo())
       expected: foo
            got: bar
     %% eunit_proc.erl:583:in `eunit_proc:run_group/2`
     Output:
     Output:

Finished in 0.008 seconds
1 tests, 1 failures
3 Likes

New release

Version 0.4.0 is out!

Changelog

Now -moduledoc attributes are also testable.

1 Like

New release

v0.5.0

Changelog

  • Added the doctest:module/2 function. The first argument is the module to be tested, and the second is a map with the options:
    • moduledoc :: boolean(): enable or disable -moduledoc test
    • funs :: boolean() | [{atom(), arity()}]: enable or disable -doc tests or define the functions to be tested
  • moduledoc was also added to doctest attribute:
    • {moduledoc, boolean()}: enable or disable -moduledoc test, e.g:
      -doctest {moduledoc, true}.
      
1 Like

New release

v0.6.0

Changelog

This update allows defining EUnit options as described here.

1 Like

New release

v0.7.0

Changelog

Options can now be globally defined.

1 Like

I’m preparing a v0.8 with improvements, bug fixes, and an EUnit reporter/listener to improve test readability.

The reporter is called doctest_eunit_report. It’s in an earlier stage, but it would be nice to hear suggestions about it.

A simple example by running rebar3 eunit:

Comparing it with eunit_progress, the default rebar3 reporter:

doctest_eunit_report provides a bit more information about falling tests and exposes the filename and line for a go-to definition of them. Please see the PR for more info:

If you like this tool, please consider sponsoring me. I’m thankful for your never-ending support :heart:

I also accept coffees :coffee:

"Buy Me A Coffee"

5 Likes

New release

v0.8.0

Changes

The major change of this release is the new EUnit reporter. It also includes bug fixes.

Changelog

3 Likes

New release

v0.9.0

Changes

Support for EDoc @doc tags, comments, and non-exported functions was introduced.
The minimum OTP version is now 24 instead of 27.

Example

  1 β”‚ %% @doc
  2 β”‚ %% Module docs are testable.
  3 β”‚ %% ```
  4 β”‚ %% 1> hello_world:hello().
  5 β”‚ %% "Hello, Joe!"
  6 β”‚ %% '''
  7 β”‚ -module(hello_world).
  8 β”‚
  9 β”‚ -export([hello/0]).
 10 β”‚
 11 β”‚ -ifdef(TEST).
 12 β”‚ -include("doctest.hrl").
 13 β”‚ -endif.
 14 β”‚
 15 β”‚ hello() ->
 16 β”‚     world().
 17 β”‚
 18 β”‚ %% @doc Non-exported functions are testable.
 19 β”‚ %%
 20 β”‚ %% This is true for tests that run via EUnit.
 21 β”‚ %% Testing via 'doctest:module/1,2' only module
 22 β”‚ %% docs and exported functions are testable.
 23 β”‚ %% ```
 24 β”‚ %% 1> % Now comments are supported.
 25 β”‚ %% .. % Bound variables to a value stills valid.
 26 β”‚ %% .. Hello = world().
 27 β”‚ %% "Hello, Mike!"
 28 β”‚ %% 2> Hello.
 29 β”‚ %% "Hello, Robert!"
 30 β”‚ %% '''
 31 β”‚ world() ->
 32 β”‚     "Hello, World!".

Running rebar3 eunit:

 FAIL  ./src/hello_world.erl:4 @moduledoc

   ❌ assertEqual

   Expected: "Hello, Joe!"
   Received: "Hello, World!"

   β”‚
 4 β”‚ %% 1> hello_world:hello().
 5 β”‚ %% "Hello, Joe!"
   β”‚
   └── at ./src/hello_world.erl:4

 FAIL  ./src/hello_world.erl:24 @doc

    ❌ assertEqual

    Expected: "Hello, Mike!"
    Received: "Hello, World!"

    β”‚
 24 β”‚ %% 1> % Now comments are supported.
 25 β”‚ %% .. % Bound variables to a value stills valid.
 26 β”‚ %% .. Hello = world().
 27 β”‚ %% "Hello, Mike!"
    β”‚
    └── at ./src/hello_world.erl:24

 FAIL  ./src/hello_world.erl:28 @doc

    ❌ assertEqual

    Expected: "Hello, Robert!"
    Received: "Hello, World!"

    β”‚
 28 β”‚ %% 2> Hello.
 29 β”‚ %% "Hello, Robert!"
    β”‚
    └── at ./src/hello_world.erl:28



Tests: 3 failed, 3 total
 Time: 0.009 seconds

The extractors option was introduced to extract tests via doctest_extract behavior. By default, [doctest_extract_tag] is defined for OTP < 27, and [doctest_extract_attr, doctest_extract_tag] is defined for OTP >= 27. Custom modules are allowed.

2 Likes

New release

v0.9.3

Changes