Implementation of EEP-59 - Documentation Attributes in Erlang

[kiko]

Implementation of EEP-59 - Documentation Attributes

EEP-59 - Documentation Attributes

• Documentation attributes are added to the binary beam file, following format of EEP-48, via +beam_docs compiler flag

• Warnings related to documentation attributes are dealt with in the beam_docs.erl instead of adding them to erl_lint.erl

Example 1

-module(warn_missing_doc).

-export([test/0, test/1, test/2]).
-export_type([test/0, test/1]).

-type test() :: ok.
-type test(N) :: N.

-callback test() -> ok.

-include("warn_missing_doc.hrl").

test() -> ok.
test(N) -> N.

Using the compiler flag warn_missing_doc will raise a warning when doc. attributes are missing in exported functions, types, and callbacks.

Example 2

-module(doc_with_file).

-export([main/1]).

-moduledoc {file, "README"}.

-doc {file, "FUN"}.
-spec main(Var) -> foo(Var).
main(X) ->
    X.

moduledocs and docs may refer to external files to be embedded.

Example 3 - Warnings and Types

-export([uses_public/0]).
-export_type([public/0]).

-doc false.
-type hidden_type() :: integer().

-type intermediate() :: hidden_type().
-type public() :: intermediate().

-spec uses_public() -> public().
uses_public() ->
    ok.

Compiler warns about exported functions whose specs refer to hidden types. In the example above, the hidden_type() is set as hidden either via -doc false or -doc hidden and public() :> intermediate() :> hidden_type(). When documentation attributes mark a type as hidden, they won’t be part of the documentation. Thus, the warning that the hidden_type() is not part of the documentation, yet used in an exported function.

[jchrist]

Very nice to see this being added to Erlang!

@kiko

-include("warn_missing_doc.hrl").

test() -> ok.
test(N) -> N.

Using the compiler flag warn_missing_doc will raise a warning when doc. attributes are missing in exported functions, types, and callbacks.

Should this example say -compile(warn_missing_doc). to show how the flag can be applied on a per-module basis? Or is it a global compiler flag that applies to all modules?

[rvirding]

I don’t know if I like it. So I have nothing against having lots of documentation describing a system, the more the merrier in fact, but the question I have is where to put it. So I think you can split documentation into 2 groups: the documentation for the user and the documentation of the code.

The code documentation together with the code is of course a must but putting the user documentation in the module can lead to code files that are very difficult to read. There is page after page of documentation and somewhere hidden inside it is the the code. Check out some of the modules in the Elixir libraries.

So I think it is best to keep the code documentation with the code, of course, and the user code in separate files.

[nzok]

• There are two kinds of documentation: documentation for the user and for the maintainer.
  100% agreement.
• User documentation in the code can lead to code that is hard to read.
  100% agreement.

There’s another effect. Since documentation tools tend to expect the documentation in the code to be user documentation, there is pressure to put maintainer documentation nowhere.

I’ve noticed all of these a lot in Java code. Since I’ve started looking at Elixir, I’ve noticed the same thing in Elixir.

There is a huge problem with the Java-style approach of tying documentation to major items in the source code like functions, and that is that maintainers often need to know about stuff that ISN’T there including WHY it isn’t there.

I haven’t read EEP-59 yet, but there’s an obvious way to do things.
-doc_base(“relative-URL”).

-doc(“#foobar-2”).
foobar(…, …) → …

where -doc is interpreted relative to -doc-base and refers to a fragment.

One thing I’ve noticed over the years is that sometimes the external documentation and the code need to change together, but more often they don’t. I wish I could fix a typo in documentation without the build system thinking the code needed recompiling. Again, there’s a perverse incentive here to NOT fix documentation as often or as thoroughly as it should be.

[dischoen]

TL;DR:

• Why switch away from XHTML?
• Can’t the compiler include the edoc run and produce doc chunks?

Here is the original of EEP 59:

erlang/eep/blob/master/eeps/eep-0059.md

I wondered what this is about, given that there is already edoc
with @doc, @docfile, @headerfile and whatnot.

The core of EEP-59 seems to be that the compiler shall be able to
parse the documentation information during compilation and to
create documentation chunks.

I am by no means a compiler expert, but couldn’t that be added
as an additional run for the compiler?

EEP-59 seems to want to change the documentation format from XHTML
to something else. It might not be possible to automatically transform
all existing documentation to that other format(s), which could bring
some effort.
By allowing different backends, vendor lock-in might happen, and different projects
will choose different backends, leading to varying doc build dependencies.

Comparing the ‘@’ tags with what EEP-59 proposes seems to be a step back
in general, with the exception of the doctests.
However, I never used doctests. In erlang I usually need some processes to
verify a function, so a oneliner can’t do much.

About the concerns that the source code will be buried under documentation:
both the existing edoc and eep59 support including of external files, so
the big prose can be somewhere else.

Finally, a remark about the format:

-doc {file, "bla"}

looks a bit awkward. Enclosing the argument(s) in parentheses would look
more like erlang, I think.

-doc(whatever).

[garazdawi]

Thanks for all comments!

@rvirding

The code documentation together with the code is of course a must but putting the user documentation in the module can lead to code files that are very difficult to read. There is page after page of documentation and somewhere hidden inside it is the the code. Check out some of the modules in the Elixir libraries.

We cannot, nor do we want to, solve the age old question about if function docs should live in the same file as the code or not. This is why EEP-59 allows it to be placed in external files if you want to, or inline if you want to.

I’m however convinced that having some standardized annotation in the source pointing to where the function docs are is a good thing thing, as it allows IDEs and other tools to easily know how to find the documentation and display it to the user.

@dischoen

EEP-59 seems to want to change the documentation format from XHTML
to something else. It might not be possible to automatically transform
all existing documentation to that other format(s), which could bring
some effort.

If by XHTML, you mean edoc syntax + XHTML, then in a way I suppose we are trying to move away from it. Or rather, we are trying to make it easier for everyone to write the documentation in the format that they want, rather than the format that edoc prescribes. We plan to update edoc to support reading EEP-48 chunks in edoc syntax, while keeping the current edoc with comments approach for backwards compatibility. So anyone that wants to continue using the old way of writing edoc can do so.

One big motivator for us (The Erlang/OTP team) is to try to unify the tooling around documentation so that we don’t have to support two variants (edoc and erl_docgen). We decided to switch formats to Markdown when migrating away from erl_docgen, not because it is necessarily the best choice, but a compromise between many different factors. I’ve also heard that many other Erlang projects would like to do the same, while others will remain with their current setup.

We will provide a tool that can convert edoc syntax to Markdown, but, as you say, it cannot possibly cope with all the constructs allowed in edoc, so some manual work has to be done.

@dischoen

Finally, a remark about the format:

-doc {file, "bla"}

looks a bit awkward. Enclosing the argument(s) in parentheses would look
more like erlang, I think.

-doc(whatever).

The ( are optional, but allowed, so you can write either -doc("Docs"). or -doc "Docs"., it is up to you.

[rvirding]

@garazdawi

We cannot, nor do we want to, solve the age old question about if function docs should live in the same file as the code or not. This is why EEP-59 allows it to be placed in external files if you want to, or inline if you want to.

I’m however convinced that having some standardized annotation in the source pointing to where the function docs are is a good thing thing, as it allows IDEs and other tools to easily know how to find the documentation and display it to the user

I very much agree with this way of doing it. You can get the best of both worlds by keeping them separate and finding/accessing them when you need to.

[josevalim]

@nzok

Since I’ve started looking at Elixir, I’ve noticed the same thing in Elixir.

Your experience from looking around Elixir then is completely different than mine. Since there is a syntactical distinction between documentation (for users) and code comments (for maintainers), I often find and write both extensively. Of course, I am biased, but I’d go on a limb to say that I have also read much more Elixir code from a wider variety of sources.

It seems Erlang was able to strike a nice balance though, by making both collocated and non-collocated documentation possible, which Elixir currently does not.