How do I organize functions and types into groups using rebar3_ex_doc

Hello, all.

Could you please explain to me how to organize functions and types into groups using rebar3_ex_doc?

Before I start, I would like to thank @starbelly for his help.
But I can’t find a solution yet.
From the description for grouping, I can’t figure out what the filtering function should look like.

Using the example of public_key module from OTP27.

...
    23 -module(public_key).                                                            
    24 -moduledoc """                                                                  
    25 API module for public-key infrastructure.
...                                      
    44 -moduledoc(#{titles =>                                                          
    45                  [{type, <<"Common">>},                                         
    46                   {type,<<"Keys">>},                                            
    47                   {type,<<"PEM files">>},                                       
    48                   {type,<<"Certificates">>},                                    
    49                   {type,<<"Certificate Revocation">>},                          
    50                   {type,<<"Test Data">>},                                       
    51                   {function,<<"PEM API">>},                                     
    52                   {function,<<"Key API">>},
...
   198 -doc(#{title => <<"Keys">>}).                                                   
   199 -doc """                                                                        
   200 Can be provided together with a custom private key, that specifies a key fun, to
   201 provide additional options understood by the fun.                               
   202 """.                                                                            
   203 -type custom_key_opts()      :: [term()].

Next, we see in the documentation here public_key:
879d7f858f10fda15e298bed840dffa99a5a2d17_2_690x388
I’m adding something similar to my code

...
 -moduledoc(#{titles =>                                                          
                  [{type, <<"My Type Group">>},                                                                      
                   {function,<<"My API Group">>},
...      
-doc(#{title => <<"My Type Group">>}).                                                   
-doc """                                                                        
Description.                        
""".                                                                            
-type my_custom_type()      :: term().
...
-doc(#{title => <<"My API Group">>}).                                                   
-doc """                                                                        
Description.                        
""".
-spec my_fun() -> ok.
my_fun() ->
    ok.    

But I see (There is no grouping):
2024-07-02-212158_1920x1080_scrot

Based on this Grouping functions and callbacks from ex_doc:
Functions and callbacks inside a module can also be organized in groups. This is done via the :groups_for_docs configuration which is a keyword list of group titles and filtering functions that receive the documentation metadata of functions as argument. The metadata received will also contain :module, :name, :arity and :kind to help identify which entity is currently being processed.
and on this rebar3 config script
and on this EEP-48: Documentation storage and format
I wrote a script and it doesn’t work.
The contents of the script:

{ok, Modules} = application:get_key(my_app, modules).                                                        
Titles = lists:foldl(                                                           
            fun(Module, Acc) ->                                                 
                case code:get_doc(Module) of                                    
                    %% see eep-48                                               
                    {ok, {docs_v1, _, erlang, _, _, #{titles := Ts}, _}} ->     
                        Acc ++ Ts;                                              
                    _ -> % otherwise                                            
                        Acc                                                     
                end                                                             
            end,                                                                
            [],                                                                 
            Modules).                                                           
Groups = {groups_for_docs,                                                      
          lists:foldl(                                                          
            fun({type, Title}, Acc) ->                                          
                    Acc ++ [{"Types: " ++ binary_to_list(Title),                
                             fun({{Kind, _, _},_ ,_ ,_ , Metadata}) ->          
                                Kind == type andalso                            
                                maps:get(title, Metadata, undefined) =:= Title  
                             end}];                                             
               ({callback, Title}, Acc) ->                                      
                    Acc ++ [{"Callbacks: " ++ binary_to_list(Title),            
                             fun({{Kind, _, _}, _, _, _, Metadata}) ->          
                                Kind == callback andalso                        
                                maps:get(title, Metadata, undefined) =:= Title  
                             end}];                                             
               ({function, Title}, Acc) ->                                      
                    Acc ++ [{binary_to_list(Title),                             
                             fun({{Kind, _, _}, _, _, _, Metadata}) ->          
                                Kind == function andalso                        
                                maps:get(title, Metadata, undefined) =:= Title  
                             end}]                                              
            end,                                                                
            [],                                                                 
            Titles)}.                                                           
{ex_doc, ExDocConf} = lists:keyfind(ex_doc, 1, CONFIG).                         
NewExDocConf = lists:keystore(groups_for_docs, 1, ExDocConf, Groups).           
lists:keystore(ex_doc, 1, CONFIG, {ex_doc, NewExDocConf}).

The contents of the error:

❯ rebar3 ex_doc
===> Error evaluating configuration script at "rebar.config.script":
{error,{2,file,
        {error,{badmatch,undefined},
               [{erl_eval,expr,6,[{file,"erl_eval.erl"},{line,652}]},
                {file,eval_stream2,6,[{file,"file.erl"},{line,2908}]},
                {file,script,2,[{file,"file.erl"},{line,2465}]},
                {rebar_config,consult_and_eval,2,
                              [{file,"/home/username/.cache/yay/rebar3-git/src/rebar3/apps/rebar/src/rebar_config.erl"},
                               {line,320}]},
                {rebar_config,consult_file_,1,
                              [{file,"/home/username/.cache/yay/rebar3-git/src/rebar3/apps/rebar/src/rebar_config.erl"},
                               {line,263}]},
                {rebar_config,consult_file,1,
                              [{file,"/home/username/.cache/yay/rebar3-git/src/rebar3/apps/rebar/src/rebar_config.erl"},
                               {line,245}]},
                {rebar3,init_config,0,
                        [{file,"/home/username/.cache/yay/rebar3-git/src/rebar3/apps/rebar/src/rebar3.erl"},
                         {line,230}]},
                {rebar3,run,1,
                        [{file,"/home/username/.cache/yay/rebar3-git/src/rebar3/apps/rebar/src/rebar3.erl"},
                         {line,100}]}]}}}

I also tried to execute commands from the script separately in the REPL.
Separately, it works, but it does not pass erl_eval.

A keyword list of group titles and filtering functions looks like this

23> {ok, A} = file:consult("rebar.config").
24> {ex_doc, ExDocConf} = lists:keyfind(ex_doc, 1, A).
{ex_doc,[{proglang,erlang},
         {extras,[{"CHANGELOG.md",#{title => "Changelog"}},
                  {"README.md",#{title => "Overview"}},
                  {"LICENSE.md",#{title => "License"}}]},
         ...]}
25> NewExDocConf = lists:keystore(groups_for_docs, 1, ExDocConf, Groups).
[{proglang,erlang},
 {extras,[{"CHANGELOG.md",#{title => "Changelog"}},
          {"README.md",#{title => "Overview"}},
          {"LICENSE.md",#{title => "License"}}]},
...
 {groups_for_docs,[{"API Group1",
                    #Fun<erl_eval.42.39164016>},
                   {"API Group2",#Fun<erl_eval.42.39164016>},
                   {"API Group3",#Fun<erl_eval.42.39164016>},
...
                   {"Types: Type Group1",#Fun<erl_eval.42.39164016>},
                   {"Types: Type Group2",#Fun<erl_eval.42.39164016>},
                   {"Types: Type Group3",#Fun<erl_eval.42.39164016>}]}]
27> lists:keystore(ex_doc, 1, A, {ex_doc, NewExDocConf}).

I suppose that’s exactly what it {"API Group2",#Fun<erl_eval.42.39164016>} should look like.
But something is wrong with the functions.

I would like to consider two variants:

  1. (rebar.config + rebar.config.script)
  2. (rebar.config + rebar.config.script + docs.config)

For the first variant:
The contents of the rebar.config file for rebar3_ex_doc:

...
  8 {ex_doc, [{proglang, erlang},                                                   
  9           {extras, [{"CHANGELOG.md", #{title => "Changelog"}},                  
 10                     {"README.md", #{title => "Overview"}},                      
 11                     {"LICENSE.md", #{title => "License"}}                       
 12                    ]},                                                          
 13           {logo, "/path/to/logo.png"},                               
 14           {homepage_url, "url"},                      
 15           {source_url, "url"}                         
 16          ]}.                                                                    
...

The contents of the rebar.config.script file for rebar3_ex_doc are described above.

For the second variant:
The contents of the rebar.config file for rebar3_ex_doc:

...
  8 {ex_doc, "docs.config"}.
...

The contents of the docs.config file for rebar3_ex_doc:

  1 {proglang, erlang}.                                                                                  
  2 {extras, [{<<"CHANGELOG.md">>, #{title => <<"Changelog">>}},                    
  3           {<<"README.md">>, #{title => <<"Overview">>}},                        
  4           {<<"LICENSE.md">>, #{title => <<"License">>}}                         
  5          ]}.                                                                                         
  6 {logo, <<"/path/to/logo.png">>}.                                                          
  7 {homepage_url, <<"url">>}.                            
  8 {source_url, <<"url">>}.                              

The contents of the rebar.config.script file for rebar3_ex_doc are described above.

Since working with CONFIG binding for the first variant will give the ex_doc configuration from rebar.config like this:

...
{ex_doc, ExDocConf} = lists:keyfind(ex_doc, 1, CONFIG).                         
NewExDocConf = lists:keystore(groups_for_docs, 1, ExDocConf, Groups).           
lists:keystore(ex_doc, 1, CONFIG, {ex_doc, NewExDocConf}).

Then working with CONFIG binding for the second variant will give the ex_doc configuration from rebar.config like this:

[...
{ex_doc, "docs.config"}.
...]

That is, the docs.config file need to be overwritten from the rebar.config.script script.

Hello everyone,

I tried to write another external config file docs.config to configure rebar_ex_doc:

{groups_for_docs,                                                                                                                                                                                                   
 lists:foldl(                                                                   
   fun({type, Title}, Acc) ->                                                   
           Acc ++ [{list_to_binary("Types: " ++ binary_to_list(Title)),         
                    fun(Md) ->                                                  
                       #{kind := Kind} = Md,                                    
                       Kind == type andalso                                     
                       string:equal(binary_to_list(Title), maps:get(title, Md, ""), true, nfd)
                    end}];                                                      
      ({callback, Title}, Acc) ->                                               
           Acc ++ [{list_to_binary("Callbacks: " ++ binary_to_list(Title)),     
                    fun(Md) ->                                                  
                       #{kind := Kind} = Md,                                    
                       Kind == callback andalso                                 
                       string:equal(binary_to_list(Title), maps:get(title, Md, ""), true, nfd)
                    end}];                                                      
      ({function, Title}, Acc) ->                                               
           Acc ++ [{Title,                                                      
                    fun(Md) ->                                                  
                       #{kind := Kind} = Md,                                    
                       Kind == function andalso                                 
                       string:equal(binary_to_list(Title), maps:get(title, Md, ""), true, nfd)
                    end}]                                                       
   end,                                                                         
   [],                                                                          
   lists:foldl(                                                                 
      fun(Module, Acc) ->                                                       
          case code:get_doc(Module) of                                          
              %% see eep-48                                                     
              {ok, {docs_v1, _, erlang, _, _, #{titles := Ts}, _}} ->           
                  Acc ++ Ts;                                                    
              _ -> % otherwise                                                  
                  Acc                                                           
          end                                                                   
      end,                                                                      
      [],                                                                       
      case application:get_key(app, modules) of                                  
        {ok, Modules} -> Modules;                                               
        undefined     -> []                                                     
      end)                                                                      
   )}.                                                                          
{proglang, erlang}.                                                             
{extras, [{<<"CHANGELOG.md">>, #{title => <<"Changelog">>}},                    
          {<<"README.md">>, #{title => <<"Overview">>}},                        
          {<<"LICENSE.md">>, #{title => <<"License">>}}                         
         ]}.                                                                    

But I see another error:

===> ** (RuntimeError) error parsing docs.config: {1, :erl_parse, ~c"bad term"}
    (ex_doc 0.32.2) lib/ex_doc/cli.ex:139: ExDoc.CLI.read_config_erl/1
    (ex_doc 0.32.2) lib/ex_doc/cli.ex:102: ExDoc.CLI.merge_config/1
    (ex_doc 0.32.2) lib/ex_doc/cli.ex:63: ExDoc.CLI.generate/3
    (elixir 1.16.0) lib/kernel/cli.ex:136: anonymous fn/3 in Kernel.CLI.exec_fun/2

Here I will describe why I wrote it that way:

  1. I use functions, so I can create a script rebar.config.script in addition to the rebar.config file for that. I chose the second case i.e. rebar.config + docs.config files, because it says here rebar3_ex_doc setup: “You may also use an external config file by specifying a path instead of a proplist.”

    %% In Elixir format
    {ex_doc, "docs.exs"}.
    
    %% Or in Erlang term, like rebar.config
    {ex_doc, "docs.config"}.
    

    Since I see the use of functions in this file otp/erts/doc/docs.exs, I make the assumption that I can use functions in the docs.config file (i.e., I can not use rebar.config.script file).

  2. Each expression is terminated with a full stop in the docs.config file. If you look at the line from the error (ex_doc 0.32.2) lib/ex_doc/cli.ex:139: ExDoc.CLI.read_config_erl/1 and go to the file lib/ex_doc/cli.ex from ex_doc:

    133   defp read_config_erl(path) do                                                 
    134     case :file.consult(path) do                                                 
    135       {:ok, config} ->                                                          
    136         config                                                                  
    137                                                                                 
    138       {:error, reason} ->                                                       
    139         raise "error parsing #{path}: #{inspect(reason)}"                                            
    140     end                                                                         
    141   end                                                                           
    

    We see the function file:consult/1. From the function description: “Reads Erlang terms, separated by ., from Filename.”

  3. Grouping functions and callbacks. It says here grouping functions and callbacks from the ex_doc: “Functions and callbacks inside a module can also be organized in groups. This is done via the :groups_for_docs configuration which is a keyword list of group titles and filtering functions that receive the documentation metadata of functions as argument. The metadata received will also contain :module, :name, :arity and :kind to help identify which entity is currently being processed.”

    I’m worried about the word also. According to EEP-48: Documentation storage and format, we have:

        {docs_v1,
         Anno :: erl_anno:anno(),
         BeamLanguage :: atom(),
         Format :: binary(),
         ModuleDoc :: #{DocLanguage := DocValue} | none | hidden,
         Metadata :: map(),
         Docs ::
           [{{Kind, Name, Arity},
             Anno :: erl_anno:anno(),
             Signature :: [binary()],
             Doc :: #{DocLanguage := DocValue} | none | hidden,
             Metadata :: map()
            }]} when DocLanguage :: binary(),
                     DocValue :: binary() | term()
    

    and according to code:get_doc/1 we have:

    -spec get_doc(Mod) -> {ok, Res} | {error, Reason}                                                        
                     when                                                              
                         Mod :: module(),                                              
                         Res ::                                                        
                             #docs_v1{anno :: term(),                                  
                                      beam_language :: term(),                         
                                      format :: term(),                                
                                      module_doc :: term(),                            
                                      metadata :: term(),                              
                                      docs :: term()},                                 
                         Reason :: non_existing | missing | file:posix().
    

    We see 2 metadata (under #docs_v1.metadata and under #docs_v1.docs). According to grouping functions and callbacks from the ex_doc #docs_v1.metadata is “a keyword list of group titles”. The metadata 2 #docs_v1.docs remains. But what is it? From the description docs :: term() and Metadata :: map(). OK, let’s execute the function code:get_doc/1 in the REPL.

    1> code:get_doc(crypto).
    {ok,{docs_v1,{24,2},
                 erlang,<<"text/markdown">>,
                 #{<<"en">> =>
                       <<"Crypto Functions\n\nThis module provides a set of cryptographic functions.\n\n- **Hash f"...>>},
                 #{otp_doc_vsn => {1,0,0},
                   titles =>
                       [{function,<<"Cipher API">>},
                        {function,<<"Hash API">>},
                        {function,<<"MAC API">>},
                        {function,<<"Key API">>},
                        {function,<<"Sign/Verify API">>},
                        {function,<<"Random API">>},
                        {function,<<"Utility Functions">>},
                        {function,<<"Engine API">>},
                        {function,<<"Deprecated API">>},
                        {type,<<"Ciphers">>},
                        {type,<<"Digests and hash">>},
                        {type,<<"Elliptic Curves">>},
                        {type,<<"Keys">>},
                        {type,<<"Public/Private K"...>>},
                        {type,<<"Public Key C"...>>},
                        {type,<<"Public K"...>>},
                        {type,<<"Diff"...>>},
                        {type,<<...>>},
                        {type,...}]},
                 [{{type,engine_ref,0},
                   {2991,2},
                   [<<"engine_ref()">>],
                   #{<<"en">> => <<"The result of a call to `engine_load/3`.">>},
                   #{title => <<"Types for Engines">>,exported => true}},
                  {{type,engine_method_type,0},
                   {2985,2},
                   [<<"engine_method_type()">>],
                   none,
                   #{title => <<"Types for Engines">>,exported => false}},
                   ...
    

    We see that metadata 2 is #{title => <<"Types for Engines">>,exported => true}} and so on. But it says here grouping functions and callbacks from the ex_doc: "filtering functions that receive the documentation metadata of functions as argument. The metadata received will also contain :module, :name, :arity and :kind to help identify which entity is currently being processed."

    The following (kind, module, name, arity) are used here otp/erts/doc/docs.exs. Where does this come from?. For my case, there is a similar code here otp/make/ex_doc.exs.

    groups_for_docs =
      Enum.map(
        Access.get(titles, :type, []),
        fn {:type, title} ->
          {"Types: #{title}",
           fn a ->
             a[:kind] == :type && String.equivalent?(Access.get(a, :title, ""), title)
           end}
        end
      ) ++
        [Types: &(&1[:kind] == :type)] ++
        Enum.map(
          Access.get(titles, :callback, []),
          fn {:callback, title} ->
            {"Callbacks: #{title}",
             fn a ->
               a[:kind] == :callback && String.equivalent?(Access.get(a, :title, ""), title)
             end}
          end
        ) ++
          [Callbacks: &(&1[:kind] == :callback)] ++
          Enum.map(
            Access.get(titles, :function, []),
            fn {:function, title} ->
              {"#{title}",
               fn a ->
                 a[:kind] == :function && String.equivalent?(Access.get(a, :title, ""), title)
               end}
            end
          )
    

    Where did the title field (Access.get(a, :title, "")) come from? If I understood correctly, then a is a map().

Could you please explain to me what the metadata data structure (that the filtering function accepts) looks like in Erlang terms?

Hello!

As you have noticed the Erlang/OTP documentation support has some extra features that is not part of ExDoc. The grouping of functions/types/callbacks according to metadata is one such feature. We implemented it there to make it possible to include the grouping in the source code instead of having it in config.

For a smaller/simpler system I would just create the groupings in ex doc config file and not try to use the metadata approach.

If you still want to know how to do it, I’ll find some time once I get back from summer vacation to explain how it all works.

2 Likes

Hello @garazdawi,

Of course I want to know.
I’ll be waiting.
Thank you.

Hi :slight_smile:

I will refrain from commenting on what OTP is doing, though the gist is there.

On the topic of docs.config… this is file that is read in via file:consult/1, thus you can not have functions in it, sans callbacks in the form of a function reference. In other words, fun foo:bar/1. will parse fine, but fun() -> baz end. will not.

On the topic of building up of groups for docs… I agree with Lukas, that if it’s a typical project in size and complexity, then it’s probably not worth the trouble of dynamically building up groups. That said you still should be able to build up this up dynamically using rebar3 config script, but how you are going about it might not work (see the above), but there’s another snag with rebar3_ex_doc at the moment, which I believe you’ll run into.

Additionally, with a rebar3 config script, I would not try to write out a doc.config, instead build up the ex_doc config for rebar3_ex_doc as you did in your first example.

It may be what you’re after is deserving of callbacks either in rebar3_ex_doc or ex_doc, but it’s not clear how niche it is yet (i.e., will a lot of people be after this functionality). That’s something to think about for a bit :slight_smile:

Disclaimer : There may be some caveats around the new doc attributes available in OTP 27 that I’m not aware of (yet), but I don’t see that at the moment.

Hello @starbelly ,

Thanks for the comments.

If I understood correctly, then this is the summary for now:

  1. Recommended combination for dynamic configuration:

    • rebar.config + rebar.config.script only.
    • rebar.config + docs.config only.
  2. Working with functions:

    • rebar.config: Does not support callbacks
      in the form of a function reference and fun expressions
      that begin with the keyword fun and end with the keyword end.
      Example: fun foo:bar/1, fun() -> baz end.

    • docs.config: Supports callbacks in the
      form of a function reference only. Example: fun foo:bar/1.

    • docs.exs: Supports fun expressions
      that begin with the keyword fun and end with the keyword end.
      Example: fun() -> baz end.

Also, syntax highlighting does not work for docs.config,
as for rebar.config inside rebar3.

It’s not that rebar config does not support anonymous functions, it’s that file:consult/1 does not allow it (which is what both rebar and ex_doc utilize to read in simple config files). To be clear, what I shared was information on config files read in via file:consult/1 vs what can and can’t be done with rebar3_ex_doc (and thus ex_doc). You’re only hope for building groups up dynamically is via rebar config script, since it utilizes file:script/1 or as you alluded to, using elixir and .exs files.

Hello @starbelly.

I would like to clarify one point.

From the description file:consult/1:
" Reads Erlang terms, separated by ., from Filename."
From the description file:script/1
" Reads and evaluates Erlang expressions, separated by . (or ,, a sequence of expressions is also an expression), from the file."

Since a function can be a sequence of expressions, the limitations become clear regarding “building groups up dynamically is via rebar config script” since it uses file:script/1.

docs.config works via file:consult/1.
file:consult/1 reads erlang terms, separated by ..
docs.config also allows us to use functions such as fun foo:bar/1.
But since we have a constraint imposed by file:consult/1, then the function bar should look like this?:

bar(Smth) ->
    fun(Smth) ->
        ...,
        ...,
        ...,
    end.

Perhaps the function can only be called in this way from another module, and the functions should not be defined in the docs.config file.

Right, you can not define functions at all in an erlang config file (docs.config, rebar.config, etc.)

To be clear, ex_doc and rebar3_ex_doc by extension currently have no callback for what you’re wanting to do AFAIK, though that may be useful.

Instead, you have to take the OTP approach or with some kind of script to build up those groupings and insert said grouping into the config.

Maybe a better example of a rebar3 config script would help. Here’s one I know of, off the top of my head : lfe/rebar.config.script at develop · lfe/lfe · GitHub

Of course, you could also do what OTP does and just leverage elixir and exs files.

Hello @starbelly ,

Thanks a lot for the example and comments
I’ll try.

1 Like