Can't find include file (Rebar3)

Hello, all.

Could you explain how file inclusion actually works?

I have an umbrella project (Rebar3).
Structure of folders:

myproj = erl_app
├── apps
│   ├── c_app
│   └── erl_app
│       ├── c_src
│       ├── include             %% | CASE 2
│           │   ├── erl_app.hrl %% | CASE 2
│           │   └── x.hrl       %% | CASE 2
│       └── src
│           ├── some_folder
│           │   ├── x.erl
│           │   ├── x.hrl       %% | CASE 3 WITHOUT INCLUDE FOLDER
│           │   └── x_server.erl
│           ├── myproj_app.erl
│           ├── myproj.app.src
│           ├── erl_app.hrl     %% | CASE 3 WITHOUT INCLUDE FOLDER
│           └── myproj_sup.erl
├── config
├── doc
├── erlang_ls.config
├── include         %% | CASE 1
│   ├── erl_app.hrl %% | CASE 1
│   └── x.hrl       %% | CASE 1
├── LICENSE.md
├── priv
├── README.md
├── rebar.config
├── rebar.lock
└── test

According to this 7.4 Directory Structure

src - Required. Contains the Erlang source code, the source of the .app file and internal include files used by the application itself.
include - Optional. Used for public include files that must be reachable from other applications.

The tree shows 3 cases for include files, one of which is actually used.
I have internal include files only.
I am interested in the third case.

According to this 10.1 File Inclusion

If the filename File is absolute (possibly after variable substitution), the include file with that name is included. Otherwise, the specified file is searched for in the following directories, and in this order:
1. The current working directory
2. The directory where the module is being compiled
3. The directories given by the
include option

According to this Erlang Compiler

{i,Dir}
Adds Dir to the list of directories to be searched when including a file. When encountering an -include or -include_lib *directive, the compiler searches for header files in the following directories:
1. ".", the current working directory of the file server
2. The base name of the compiled file
3. The directories specified using option
i ; the directory specified last is searched first

It is noted here Erlang: what is the difference between “include_lib” and “include”?

One difference which is not obvious at first is that -include and -include_lib use a different set of paths when looking for header files. -include_lib in fact uses the code path, not the header file path.

Configuration of the rebar.config file

1 CASE

  • Working case without i compiler option
    Open x.erl from root_dir i.e. "." of myproj
    -include("include/x.hrl"). % The filename File is absolute.
    
  • Non-working case with i compiler option
  1 {erl_opts, [debug_info,                                                         
  2                                                                           
  3             %% For internal include files used by the application itself.       
  4             %% Not the code path is used, but the header file path.             
  5             {i, ["include"]}                                                                  
  6            ]}.
-include("x.hrl"). % E: can't find include file "x.hrl" 

2 CASE

  • Working case without i compiler option
    Open x.erl from root_dir i.e. "." of myproj
    -include("apps/erl_app/include/x.hrl"). % The filename File is absolute.
    
  • Non-working case with i compiler option
  1 {erl_opts, [debug_info,                                                         
  2                                                                           
  3             %% For internal include files used by the application itself.       
  4             %% Not the code path is used, but the header file path.             
  5             {i, ["apps/*/include"]}                                                                  
  6            ]}.
-include("x.hrl"). % E: can't find include file "x.hrl" 

3 CASE

  • Working case without i compiler option
    Open x.erl from root_dir i.e. "." of myproj
    -include("apps/erl_app/src/some_folder/x.hrl"). % The filename File is absolute.
    
  • Non-working case with i compiler option
  1 {erl_opts, [debug_info,                                                         
  2                                                                           
  3             %% For internal include files used by the application itself.       
  4             %% Not the code path is used, but the header file path.             
  5             {i, ["apps/*/src", "apps/*/src/*"]}                                                                  
  6            ]}.
-include("x.hrl"). % E: can't find include file "x.hrl" 

Since all 3 cases use -include, then header file path is used.

According to this Code Path

Environment variable ERL_LIBS (defined in the operating system) can be used to define more library directories to be handled in the same way as the standard OTP library directory described above, except that directories without an ebin directory are ignored.

If I understood correctly, we can use -include_lib as it uses search path aka code path.
For example, for the 2 CASE

  • Working case without i compiler option
    Open x.erl from root_dir i.e. "." of myproj
    -include_lib("erl_app/include/x.hrl").
    

The explanation is here 10.1 File Inclusion

The code server uses code:lib_dir(kernel) to find the directory of the current (latest) version of Kernel, and then the subdirectory include is searched for the file file.hrl.

Only instead of kernel, you need to substitute erl_app.
It should be noted that erl_app should not hide app from $OTPROOT/lib, where $OTPROOT is the installation directory of Erlang/OTP.

Result
Why is {i,Dir} not working (It works, but I don’t understand how.)?

1 Like

Hello, all.

For 3 CASE

...
│       └── src
│           ├── some_folder
│           │   ├── x.erl
│           │   ├── x.hrl       %% | CASE 3 WITHOUT INCLUDE FOLDER
│           │   └── x_server.erl
│           ├── myproj_app.erl
│           ├── myproj.app.src
│           ├── erl_app.hrl     %% | CASE 3 WITHOUT INCLUDE FOLDER
│           └── myproj_sup.erl
...

Open x.erl from root_dir i.e. "." of myproj

...
E  41 -include("x.hrl"). % E: can't find include file "x.hrl"    
...

But the project is being compiled (without {i,Dir}). This means that the corresponding file x.hrl is visible (without defining the absolute file name in the -include()). It can be concluded that directory src is viewed recursively, since the file is located in directory some_folder. However, the error msg is still being reported.

1 Like

First of all:

-include("include/x.hrl"). % The filename File is absolute.

This is not an absolute path, that’s a relative path. An absolute path would look like this:

-include("/home/username/somedir/include/x.hrl"). % The filename File is absolute.

This messes with your first assumption right away for the whole post. All these searches you make for the rest of the post are done with relative paths, and so will start with ./ (which is the project root for Rebar3), then the base name of the file (whatever source directory it has), and then the i options.

Note that for Rebar3, you generally do not have to specify the i option for include/ because Rebar3 will automatically expand them for you, and moreover, it will tend to do it with absolute directory paths for these areas, to avoid mistakenly getting mixed options across applications: rebar3/apps/rebar/src/rebar_compiler_erl.erl at 9880ef3c7604724f8ea81daee1da6e92ccbf1167 · erlang/rebar3 · GitHub (there are also a few tests checking that behavior).

This is a thing we do to avoid messing with compilation contexts across large projects where dependencies might otherwise accidentally pick up the dependencies of their parent applications. This is a nefarious issue when a library author will write a thing that includes, say include/user.hrl and the parent project defines a include/user.hrl at the root of their project (the current working directory with .). What would happen here is that the library author writes a functional app that tests properly, but starts breaking when the user imports it because of unexpected include paths.

To make the library ecosystem more robust, we expand each lib’s relative includes to contain local paths, which prevents accidentally using the parent’s include files in a dependency’s build context within a complex umbrella application.

This sort of protective mechanism clashes with what Erlang documents and does outside the box, because we felt that providing repeatable builds where the user doesn’t have to debug or rewrite their app because they chose a header file name shared with a dep they didn’t write was more important.

However, this will also impact the sort of set-up you’ve done here if you’re trying to reach across these boundaries. So you have to consider new contextual elements:

  • The current working directory (".") is the root of the project as rebar3 does not cd into sub-applications during its build phases.
  • the path names you have described as absolute are in fact relative
  • rebar3 will expand the paths from the {i, ...} options to be less relative and therefore more specific to prevent accidentally breaking dependencies by defining header files with clashing names in their [unrelated] app.
    • note that while the i path is absolute, the -include() path may still be relative.

These are subtle but important changes that will no doubt mess with your scenarios here.


Now looking at your three cases:

Case 1

I’m not sure what you mean by " Open x.erl from root_dir i.e. "." of myproj". there is no x.erl at the root of the project; the "include/x.hrl" file should however be visible from the root.

"x.hrl" however is not a path that exists at the root of the project. That relative path would be visible for apps/erl_app/src/x.erl if it declared -include("x.hrl"). for example (which should grab apps/erl_app/src/x.hrl), but not the include/x.hrl at the project root.

Case 2
I don’t believe "apps/*/include" makes sense as an i option.

You used -include("apps/erl_app/include/x.hrl"). and it works, but it’s brittle. If you suddenly move erl_app to say, libs/, it will break.

Here, using -include_lib("erl_app/include/x.hrl"). will tell the build system to find erl_app’s include/x.hrl no matter where in the system that library is located (so long as it’s visible to the build tool).

It is therefore always a better idea to do it with -include_lib(), unless you’re somehow having multiple versions of the same app in your path and want to build one but include files from another.

Case 3

Same comment as for Case 2 where I don’t think the directory syntax makes sense.

Note that in this case, -include_lib(...) would be inadequate because you’re trying to load a private header file (within src/ rather than in include/) and that’s outside of conventions, so the use of -include(...) is adequate.

(The file path is still relative, it’s just properly defined as relative to the project root, which is where rebar3 executes from).


Why is {i,Dir} not working (It works, but I don’t understand how.)?

Hopefully, the details I provided about how Rebar3 expands the paths to prevent cross-application clashes explains some of the confusing behavior you have noticed here. Also of note is that I don’t believe wildcards work in the path specifications, and all the options are literal.

1 Like

Hello, Fred.

I am very grateful for the detailed explanation.

I’m not sure what you mean by " Open x.erl from root_dir i.e. "." of myproj".*

I open vim in the current working directory (".") i.e. the root of the project, then open the x.erl file using nerdtree.

If I understood correctly, then in order for the error to go away, I must determine
in rebar.config {i, "/home/username/path/to/myproj/apps/erl_app/src/some_folder"} % an absolute path
in x.erl -include("x.hrl") % a relative path

The error msg does not go away.
E 41 -include("x.hrl"). % E: can't find include file "x.hrl"

But the project is being compiled.

1 Like

no, -include("x.hrl") is a relative path regardless of the i options. If you want it to be absolute, it would need to be specified as -include("/home/username/path/to/myproj/apps/erl_app/src/some_folder/x.hrl").

1 Like

The error msg does not go away.
I tried to move x.hrl from the some_folder to the src folder (one level higher) and made changes to the rebar.config, it still doesn’t go away.
I
rebar.config: {i, "/home/username/path/to/myproj/apps/erl_app/src/some_folder/x.hrl"}.
x.erl: E 41 -include("x.hrl"). % E: can't find include file "x.hrl"
II
rebar.config: {i, "/home/username/path/to/myproj/apps/erl_app/src/x.hrl"}.
x.erl: E 41 -include("x.hrl"). % E: can't find include file "x.hrl"

I open it via vim, via nerdtree.
The project is still being compiled.

I still have erlang_ls installed. Maybe it has some effect?

I don’t know that your editor opening the file has an impact, unless your editor is doing the building–I generally build all my projects by calling rebar3 compile in the terminal, which is fully disjoint from my editor or IDE.

If your editor does the compiling, then that’s an extra environment to debug for sure. If you have erlang_ls enabled, it may have an impact, because it might be the tool actually compiling the code and not rebar3.

I have erlang_ls installed, but I compile the code with the command rebar3 compile.
There is an explanation in the vim below E1516: can't find include file "x.hrl"
I checked it out

❯ git clone https://github.com/erlang-ls/erlang_ls
❯ cd erlang_ls
❯ grep -HRs E1516 .
./apps/els_lsp/test/els_diagnostics_SUITE.erl:            code => <<"E1516">>,
./apps/els_lsp/src/els_compiler_diagnostics.erl:    <<"E1516">>;

I have a file erlang_ls.yaml in the root of the project.
I have commented out all its contents, but there is no effect.
Vim is used for editing code only.
Maybe to reset the erlang_ls settings, it must be rebooted if it has any effect.
If this does not help, then it is unclear what the root cause is.

erlang_ls settings:

    1 #apps_dirs:                                                                                        
    2 #  - "apps/*"                                                                   
>>  3 #deps_dirs:                                                                     
    4 #  - "_build/default/lib/*"                                                     
    5 #include_dirs:                                                                  
    6 #  - "apps"                                                                     
    7 #  - "apps/*/include"                                                           
    8 #  - "_build/default/lib/"                                                      
    9 #  - "_build/default/lib/*/include"

I rebooted PC. The same.
This error message E1516: can't find include file "x.hrl" does not always appear.
Now it is reported like this can't find include file "x.hrl".

I’m pretty sure that isn’t a rebar3 error, since we don’t number error messages.

I changed the erlang version from 27.0-rc2 to 26.2.3 and completely disabled erlang_ls. The same thing. Next, I changed the erlang version from 26.2.3 to 27.0-rc2 again and enabled erlang_ls. The same thing. Now there is an error, but without a number.
In the line of the directive include() in vim:

E  41 -include("x.hrl"). % E: can't find include file "x.hrl"

At the very bottom in vim:

 NORMAL  <c/some_folder/x.erl   erlang     utf-8    55% :41/74☰ ℅:1  23:00  E:4(L41) 
can't find include file "x.hrl"

I thought that maybe the absolute path was written incorrectly in the rebar.config file. I checked it several times. When typing, auto-completion is triggered. Maybe it’s not the rebar3 error. I don’t know why that is.
I get so
Whatever the directory structure, there is a rebar.config file and it contains the absolute path to the .hrl file and there is a source code file including a header file using the directive -include() that defines the relative path. The relative path should expand to the absolute path, but this does not happen.

rebar.config:

  1 {erl_opts, [debug_info,                                                                              
  2             warn_export_vars, warn_shadow_vars, warn_obsolete_guard,            
  3             warn_missing_spec, warn_untyped_record,                             
  4                                                                                 
  5             %% For internal include files used by the application itself.       
  6             %% Not the code path is used, but the header file path.             
  7             {i, "/home/username/path/to/myproj/apps/erl_app/src/some_folder/x.hrl"}, % myproj = erl_app
  8             {i, "/home/username/path/to/myproj/apps/erl_app/src/erl_app.hrl"}        
  9            ]}.                                                                  
 10 {deps, []}.

erlang_ls.yaml:

    1 apps_dirs:                                                                                         
    2   - "apps/*"                                                                    
>>  3 deps_dirs:                                                                      
    4   - "_build/default/lib/*"                                                      
    5 include_dirs:                                                                   
    6   - "apps"                                                                      
    7   - "apps/*/include"                                                            
    8   - "_build/default/lib/"                                                       
    9   - "_build/default/lib/*/include" 

Also, the name of my project matches the name of one of the erlang libraries.

❯ asdf where erlang
/home/username/.asdf/installs/erlang/27.0-rc2
❯ ll /home/username/.asdf/installs/erlang/27.0-rc2/lib
total 136K
drwxr-xr-x 7 username username 4.0K Mar 21 20:17 asn1-5.3
drwxr-xr-x 7 username username 4.0K Mar 21 20:17 common_test-1.27
drwxr-xr-x 4 username username 4.0K Mar 21 20:17 compiler-8.5
...

Let’s say it’s like if my project was called compiler, but at least the versions don’t match. So I don’t think it should have any impact.

vim ~/.vim/coc-settings.json:

   18         "erlang": {                                                             
   19             "command": "erlang_ls",                                             
   20             "filetypes": ["erlang"]                                             
   21         },  
❯ rebar3 -v
rebar 3.22.0 on Erlang/OTP 27 Erts 14.3
❯ erlang_ls -v
Version: 0.49.0
❯ asdf current
erlang          27.0-rc2 

Sometimes (when switching to the previous version and back) on version 27.0-rc2, this appears during compilation:

===> Analyzing applications...                                                                                                                                                                                      
===> Compiling pc                                                                  
===> Compiling ../../../../.cache/rebar3/plugins/pc/src/pc_port_env.erl failed  
../../../../.cache/rebar3/plugins/pc/src/pc_port_env.erl:190:10: code:lib_dir/2 is deprecated; this functionality will be removed in a future release
                                                                                   
===> Errors loading plugin pc. Run rebar3 with DEBUG=1 set to see errors.

It helps
rebar3 as global plugins upgrade pc

Global rebar3 config vim ~/.config/rebar3/rebar.config:

1 {plugins, [rebar3_auto, rebar3_ex_doc]}.

Additionally, 3 plugins have been removed from .vimrc:

   52 "Plug 'vim-erlang/vim-erlang-omnicomplete' " autocompletion                     
   53 "Plug 'vim-erlang/vim-erlang-compiler' " syntax checking and compiler                              
   54 "Plug 'vim-erlang/vim-erlang-runtime' " indentation, syntax and ftplugin scripts

And the rebar3 cache has been cleared:

❯ rm -r ~/.cache/rebar3

The error msg does not go away.

Here’s a demo app showing all the kinds of include paths you may want, with zero extra config: GitHub - ferd/incl_chk: a demo repo for a forum discussion

.
├── a.hrl
├── apps
│   ├── alt
│   │   ├── include
│   │   │   └── g.hrl
│   │   └── src
│   │       ├── alt.app.src
│   │       ├── alt.erl
│   │       └── h.hrl
│   └── incl_chk
│       ├── include
│       │   └── c.hrl
│       └── src
│           ├── d.hrl
│           ├── incl_chk_app.erl
│           ├── incl_chk.app.src
│           ├── incl_chk_sup.erl
│           └── sub
│               ├── chk.erl
│               ├── e.hrl
│               └── f.hrl
├── config
│   ├── sys.config
│   └── vm.args
└── include
    └── b.hrl

See incl_chk/apps/incl_chk/src/sub/chk.erl:

-module(chk).
-export([f/0]).

-include("a.hrl").
-include("include/b.hrl").
-include("include/c.hrl").
-include("d.hrl").
-include("sub/e.hrl").
-include("f.hrl").
-include_lib("alt/include/g.hrl").
-include("apps/alt/src/h.hrl").

f() ->
    [?A, ?B, ?C, ?D, ?E, ?F, ?G, ?H].

Run with:

→ rebar3 shell --apps=
===> Verifying dependencies...
===> Analyzing applications...
===> Compiling incl_chk
===> Compiling alt
Erlang/OTP 26 [erts-14.0] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [jit:ns]

Eshell V14.0 (press Ctrl+G to abort, type help(). for help)
1> chk:f().
[a,b,c,d,e,f,g,h]

This works out of the box.

2 Likes

Thank you very much, Fred. I will try.

File should be named erlang_ls.config, not erlang_ls.yaml - maybe your erlang_ls config is not even used. How you can trivially check that is to inject syntax error in that yaml format - if it gets reported during erlang_ls startup then config is used, if it doesn’t than your configuration is not used by erlang_ls.

If you get error messages from your LSP client in vim, it doesn’t have to be that the project doesn’t compile. You can get error messages from LSP client and be able to compile a project. You can also have no errors from LSP, but project could not compile.

Hello, all.

For @mmin:

"File should be named erlang_ls.config, not erlang_ls.yaml"

According to erlang_ls configuration:

It is possible to customize the behaviour of the erlang_ls server via a configuration file, named erlang_ls.config or erlang_ls.yaml.

Thanks a lot, @MononcQc for the GitHub - ferd/incl_chk: a demo repo for a forum discussion.

I experimented with the demo. The results are as follows:
I use vim and some set of plugins.
If open vim in the current working directory (".") i.e. the root of the project, then open incl_chk/apps/incl_chk/src/sub/chk.erl file using nerdtree then


According to what @MononcQc said above:

I’m pretty sure that isn’t a rebar3 error, since we don’t number error messages.

In the same window, let’s go down a little lower


We see that the error is already without a number, but we do not yet know what is causing it. Ok.

Disable reporting in-line warnings and errors from the Erlang compiler. Description here erlang_ls-configuration-diagnostics:
Add in the erlang_ls.yaml file:

1 diagnostics:                                                                                         
2   disabled:                                                                     
3     - compiler

Let’s open vim in the current working directory (".") i.e. the root of the project, then open incl_chk/apps/incl_chk/src/sub/chk.erl file using nerdtree again


We can see that on line 6 there is an error message with no number already. Ok.

Now turn off the plugin ALE (Asynchronous Lint Engine)
Add to .vimrc:

...
67 let g:ale_enabled = 0
...

Let’s open vim in the current working directory (".") i.e. the root of the project, then open incl_chk/apps/incl_chk/src/sub/chk.erl file using nerdtree again

1 Like

Thanks to everyone who took the time to look at and answer the question.

BR, Denis.

Hello, all.

Minimal Erlang Integration for this case

ALE

Add to the Vim configuration file ~/.vimrc

" === ale ===                                                                      
"ALE Erlang Integration:                                                           
"Enable only a particular set of linters for erlang files.                         
let g:ale_linters = {                                                              
\   'erlang': ['dialyzer', 'erlang_ls', 'erlc']                                    
\}                                                                                 
let g:ale_erlang_erlc_options = "$(find apps -type d -exec echo '-I {}' ';')" 

Run :ALEInfo to see which linters are available after telling ALE to run Erlang linters.
For more information, see Vim help files for ALE and in particular Erlang-specific Vim help file for ALE.

coc.nvim

Add to the coc.nvim configuration file ~/.vim/coc-settings.json:

{
      "languageserver": {                                                            
          "erlang": {                                                                
              "command": "erlang_ls",                                                
              "filetypes": ["erlang"]                                                
         }                                                 
      },                                                                                                 
      "diagnostic.displayByAle": true                                                
  }  

For more information, see Coc plugin configuration and The easiest way to get both plugins (coc.nvim + ale) to work together.

erlang_ls

Add to the erlang_ls.yaml or erlang_ls.config configuration file:

apps_dirs:                                                                      
  - "apps/*"                                                                    
deps_dirs:                                           
  - "_build/default/lib/*"                                                                             
# These paths are prepended to all other paths  
# given by  an -include or -include_lib directive.                                 
include_dirs:                                                                      
  - "apps"                                                                         
  - "apps/*"                                                                       
  - "apps/*/src"                                                                   
  - "_build/default/lib/"                                                          
  - "_build/default/lib/*/include"  

For more information, see erlang_ls configuration.

Tested on GitHub - ferd/incl_chk: a demo repo for a forum discussion.