Trouble with -include_lib: "can't find include lib"

[BrettHemes]

The command:

-include_lib("my_local_lib/include/the_header.hrl")

is causing the error "can't find include lib "...". This error then triggers many others within the file due to missing definitions.

I am using rebar3. The dependency from which I am trying to include the hrl file from is under development locally and thus is linked (via relative soft link) in the _checkouts folder. Could this be causing the issue? The project compiles and runs fine with rebar despite the error.

Thanks

[hcs42]

Hi,

include_lib should work in the _checkouts directory too.

1. Please call rebar3 compile and see if that fixes the error (i.e. if you save the file again, the error will disappear).

   It fixes it for me:

   $ rebar3 new release
   ===> Writing myapp/apps/myapp/src/myapp_app.erl
   ===> Writing myapp/apps/myapp/src/myapp_sup.erl
   ===> Writing myapp/apps/myapp/src/myapp.app.src
   ===> Writing myapp/rebar.config
   ===> Writing myapp/config/sys.config
   ===> Writing myapp/config/vm.args
   ===> Writing myapp/.gitignore
   ===> Writing myapp/LICENSE
   ===> Writing myapp/README.md
   
   $ cd myapp/
   
   $ vim rebar.config
   
   # Add lager as a dependency:
   #
   #   {deps, [
   #           {lager, {git, "git://github.com/erlang-lager/lager.git"}}
   #          ]}.
   
   $ mkdir _checkouts
   
   $ cd _checkouts
   
   $ git clone git://github.com/erlang-lager/lager.git lager
   Cloning into 'lager'...
   remote: Enumerating objects: 30, done.
   remote: Counting objects: 100% (30/30), done.
   remote: Compressing objects: 100% (21/21), done.
   remote: Total 4611 (delta 15), reused 20 (delta 9), pack-reused 4581
   Receiving objects: 100% (4611/4611), 2.01 MiB | 3.43 MiB/s, done.
   Resolving deltas: 100% (3157/3157), done.
   
   $ ls
   lager
   
   $ cd ..
   
   $ vim myapp/apps/myapp/src/myapp_app.erl
   
   # Add the following line:
   #
   # -include_lib("lager/include/lager.hrl").
   
   # Try to save -> the error comes
   
   $ rebar3 compile
   ===> Verifying dependencies...
   ===> Fetching goldrush v0.1.9
   ===> Version cached at /home/hcs/.cache/rebar3/hex/hexpm/packages/goldrush-0.1.9.tar is up to date, reusing it
   ===> Compiling goldrush
   ===> Compiling lager
   ===> Compiling myapp
   
   $ vim myapp/apps/myapp/src/myapp_app.erl
   
   # Try to save -> no error
   

2. If it doesn't fix it for you, then please try how the above "lager" example works in your environment.

   Then please go to the problematic source file and call the erlang_check script manually:

   $ cd /home/hcs/_/2020_07_22/myapp/apps/myapp/src
   
   $ /path/to/vim-erlang-compiler/compiler/erlang_check.erl --outdir . myapp_app.erl -v
   Verbose mode on.
   Found project root: "/home/hcs/_/2020_07_22/myapp/apps/myapp"
   Try build system: rebar3
   Found build file: ["/home/hcs/_/2020_07_22/myapp"] "/home/hcs/_/2020_07_22/myapp/rebar.lock"
   Found build file: ["/home/hcs/_/2020_07_22/myapp"] "/home/hcs/_/2020_07_22/myapp/rebar.config"
   rebar.config read: /home/hcs/_/2020_07_22/myapp/rebar.config
   rebar3 executable found: /home/hcs/bin/rebar3
   Absolute path to rebar3 executable: /home/hcs/bin/rebar3
   Call: QUIET=1 "/home/hcs/bin/rebar3" as "default" path
   Result: /home/hcs/_/2020_07_22/myapp/_build/default/lib/goldrush/ebin /home/hcs/_/2020_07_22/myapp/_build/default/lib/myapp/ebin
   Code paths: ["/home/hcs/_/2020_07_22/myapp/_checkouts/lager/ebin",
                "/home/hcs/_/2020_07_22/myapp/_build/default/lib/myapp/ebin",
                "/home/hcs/_/2020_07_22/myapp/_build/default/lib/goldrush/ebin",
                ".","/usr/lib/erlang/lib/kernel-6.5.1/ebin",
                "/usr/lib/erlang/lib/stdlib-3.11.2/ebin",
                "/usr/lib/erlang/lib/xmerl-1.3.23/ebin",
                ...
                "/usr/lib/erlang/lib/asn1-5.0.9/ebin"]
   Compiling: compile:file("/home/hcs/_/2020_07_22/myapp/apps/myapp/src/myapp_app.erl",
       [warn_export_all,warn_export_vars,warn_shadow_vars,warn_obsolete_guard,
        warn_unused_import,report,debug_info,debug_info,
        {outdir,"/home/hcs/_/2020_07_22/myapp/."},
        {i,"/home/hcs/_/2020_07_22/myapp/apps/myapp/src/../include"},
        {i,"/home/hcs/_/2020_07_22/myapp/apps/myapp/include"}])
   

   Do you see _checkouts/my_local_lib/bin in the "Code paths"?

[BrettHemes]

The "lager" MWE fails in the same way even after a $ rebar3 compile.

Here is the output from running erlang_check on the "lager" example:

me@thispc:~/erlang/myapp/apps/myapp/src$ ~/.vim/bundle/vim-erlang-compiler/compiler/erlang_check.erl --outdir . myapp_app.erl -v
Verbose mode on.
Found project root: "/home/me/erlang/myapp/apps/myapp"
Try build system: rebar3
Found build file: ["/home/me/erlang/myapp"] "/home/me/erlang/myapp/rebar.lock"
Found build file: ["/home/me/erlang/myapp"] "/home/me/erlang/myapp/rebar.config"
rebar.config read: /home/me/erlang/myapp/rebar.config
rebar3 executable found: /home/me/.cache/rebar3/bin/rebar3
Absolute path to rebar3 executable: /home/me/.cache/rebar3/bin/rebar3
Call: QUIET=1 "/home/me/.cache/rebar3/bin/rebar3" as "default" path
Result: /home/me/erlang/myapp/_build/default/lib/goldrush/ebin /home/me/erlang/myapp/_build/default/lib/myapp/ebin
Code paths: ["/home/me/erlang/myapp/_build/default/lib/myapp/ebin",
"/home/me/erlang/myapp/_build/default/lib/goldrush/ebin",
".","/usr/lib/erlang/lib/kernel-6.5.1/ebin",
"/usr/lib/erlang/lib/stdlib-3.11.2/ebin",
"/usr/lib/erlang/lib/xmerl-1.3.23/ebin",
"/usr/lib/erlang/lib/wx-1.9/ebin",
"/usr/lib/erlang/lib/tools-3.3/ebin",
"/usr/lib/erlang/lib/tftp-1.0.2/ebin",
"/usr/lib/erlang/lib/syntax_tools-2.2.1/ebin",
"/usr/lib/erlang/lib/ssl-9.5.3/ebin",
"/usr/lib/erlang/lib/ssh-4.8.2/ebin",
"/usr/lib/erlang/lib/snmp-5.4.5/ebin",
"/usr/lib/erlang/lib/sasl-3.4.1/ebin",
"/usr/lib/erlang/lib/runtime_tools-1.14/ebin",
"/usr/lib/erlang/lib/reltool-0.8/ebin",
"/usr/lib/erlang/lib/public_key-1.7.1/ebin",
"/usr/lib/erlang/lib/parsetools-2.1.8/ebin",
"/usr/lib/erlang/lib/os_mon-2.5.1/ebin",
"/usr/lib/erlang/lib/odbc-2.12.4/ebin",
"/usr/lib/erlang/lib/observer-2.9.3/ebin",
"/usr/lib/erlang/lib/mnesia-4.16.2/ebin",
"/usr/lib/erlang/lib/megaco-3.18.7/ebin",
"/usr/lib/erlang/lib/jinterface-1.10.1/ebin",
"/usr/lib/erlang/lib/inets-7.1.2/ebin",
"/usr/lib/erlang/lib/hipe-3.19.2/ebin",
"/usr/lib/erlang/lib/ftp-1.0.4/ebin",
"/usr/lib/erlang/lib/eunit-2.4/ebin",
"/usr/lib/erlang/lib/et-1.6.4/ebin",
"/usr/lib/erlang/lib/erts-10.6.4/ebin",
"/usr/lib/erlang/lib/erl_interface-3.13.1/ebin",
"/usr/lib/erlang/lib/erl_docgen-0.11/ebin",
"/usr/lib/erlang/lib/eldap-1.2.8/ebin",
"/usr/lib/erlang/lib/edoc-0.11/ebin",
"/usr/lib/erlang/lib/diameter-2.2.1/ebin",
"/usr/lib/erlang/lib/dialyzer-4.1.1/ebin",
"/usr/lib/erlang/lib/debugger-4.2.8/ebin",
"/usr/lib/erlang/lib/crypto-4.6.4/ebin",
"/usr/lib/erlang/lib/compiler-7.5.2/ebin",
"/usr/lib/erlang/lib/common_test-1.18.1/ebin",
"/usr/lib/erlang/lib/asn1-5.0.9/ebin"]
Compiling: compile:file("/home/me/erlang/myapp/apps/myapp/src/myapp_app.erl",
[warn_export_all,warn_export_vars,warn_shadow_vars,warn_obsolete_guard,
warn_unused_import,report,debug_info,debug_info,
{outdir,"/home/me/erlang/myapp/."},
{i,"/home/me/erlang/myapp/apps/myapp/src/../include"},
{i,"/home/me/erlang/myapp/apps/myapp/include"}])
/home/me/erlang/myapp/apps/myapp/src/myapp_app.erl:8: can't find include lib "lager/include/lager.hrl"

[hcs42]

In erlang_check.erl, the following line should find the _checkouts/lager/ebin directory and add it to the code path (which does not happen in your environment):

code:add_pathsa(filelib:wildcard(absname(ConfigPath, "_checkouts") ++ "/*/ebin")),

1. What is your vim-erlang-compiler version? I.e. what does git log -n1 say? I assume git status says "nothing to commit, working tree clean".

2. Do you have a _checkouts/lager/ebin directory?

3. If you go into the main myapp directory start an Erlang shell, what is the result of the following call?

   > filelib:wildcard("_checkouts/*/ebin").
   ["_checkouts/lager/ebin"]
   

[BrettHemes]

Sorry for the delayed response. Thanks for the help; I would very much like to figure out the issue.

What is your vim-erlang-compiler version? I.e. what does git log -n1 say? I assume git status says "nothing to commit, working tree clean".

commit 7ca6f2a9ed97a41891bc48cfb798efc51d240cc0 (HEAD -> master, origin/master, origin/HEAD)
Merge: 5d61763 763be9a
Author: Csaba Hoch <[email protected]>
Date:   Tue Jun 26 11:18:44 2018 +0200

    Merge pull request #43 from bullno1/fix-extra-profile-loading
    
    Fix badmatch in erlang_check:process_rebar3_config/2

This is in my .vim folder and was grabbed through vundle

Do you have a _checkouts/lager/ebin directory?

I have _checkouts/lager directory but no ebin subdirectory

If you go into the main myapp directory start an Erlang shell, what is the result of the following call?

1> filelib:wildcard("_checkouts/*/ebin").
[]

[hcs42]

Ok, so the difference is that in your environment, the dependency (lager in the example) is not compiled into its ebin directory.

Could you execute this command to find where the BEAM files are created?

$ find -name lager.beam
./_checkouts/lager/ebin/lager.beam

Which rebar version do you use? I tried 3.13.2 and 3.7.5, both with the same result.

$ rebar3 --version
rebar 3.13.2 on Erlang/OTP 22 Erts 10.6.4

[BrettHemes]

My build files seem to be going into the following folder:

./_build/default/checkouts/lager/ebin/

and

$ find -name lager.beam
./_build/default/checkouts/lager/ebin/lager.beam

And my rebar3 version was recently installed on Ubuntu 20.04 from source via the following website instructions:

$ git clone https://github.com/erlang/rebar3.git
$ cd rebar3
$ ./bootstrap

$ rebar3 --version
rebar 3.14.0-rc2+build.4688.refd92809aa on Erlang/OTP 22 Erts 10.6.4

[hcs42]

I found the root cause. Rebar3 changed the output directory for the _checkouts dependencies since the last rebar3 release (3.13.2). See this commit: output checkouts compilation to _build/<profile>/checkouts/

Could you edit the vim-erlang-compiler/compiler/erlang_check.erl file and add the following line after line 638:

 % _checkouts -> code_path (see
 % https://www.rebar3.org/docs/dependencies#section-checkout-dependencies)
 code:add_pathsa(filelib:wildcard(absname(ConfigPath, "_checkouts") ++ "/*/ebin")),
+code:add_pathsa(filelib:wildcard(absname(ConfigPath, "_build") ++ "/default/checkouts/*/ebin")),
 
 lists:foreach(
   fun({ProfileName, Deps}) ->

Then please check whether it solves the problem (it solved it for me in my lager example).

It's not an ideal solution because the profile name ("default") is hard-coded, but you could use vim-erlang-compiler while I find the proper solution (and it would confirm that we are on the right track).

[BrettHemes]

This indeed addresses the problem on my setup using the lager example as a test.

[BrettHemes]

Although I am still having trouble with my original example. I will dig a bit more.

[BrettHemes]

OK, made a bit of progress... my other project doesn't have a rebar.lock file and thus process_rebar3_config/2 wasn't getting called. This output from vim-erlang-compiler is what tipped me off:

Try build system: rebar3
Try build system: rebar

Adding an empty rebar.lock fixed the issue and executes the new code:add_pathsa line.

I am too new to rebar(3) to know why my project didn't have a rebar.lock file or it that is even an issue. Can you comment?

[hcs42]

Thank you for the feedback.

A simple rebar3 compile should add the rebar.lock file:

$ rebar3 new release
===> Writing myapp/apps/myapp/src/myapp_app.erl
===> Writing myapp/apps/myapp/src/myapp_sup.erl
===> Writing myapp/apps/myapp/src/myapp.app.src
===> Writing myapp/rebar.config
===> Writing myapp/config/sys.config
===> Writing myapp/config/vm.args
===> Writing myapp/.gitignore
===> Writing myapp/LICENSE
===> Writing myapp/README.md

$ cd myapp/

$ ls
apps  config  LICENSE  README.md  rebar.config

$ rebar3 compile
===> Verifying dependencies...
===> Compiling myapp

$ ls
apps  _build  config  LICENSE  README.md  rebar.config  rebar.lock

Maybe this also works differently in the master branch of rebar3, I haven't checked that.

[BrettHemes]

Indeed, I get the lock file in the lager post-compile but not in my from-scratch project. Here is the myapp lager example lock file contents:

$ cat rebar.lock 
{"1.2.0",
[{<<"goldrush">>,{pkg,<<"goldrush">>,<<"0.1.9">>},1}]}.
[
{pkg_hash,[
 {<<"goldrush">>, <<"F06E5D5F1277DA5C413E84D5A2924174182FB108DABB39D5EC548B27424CD106">>}]},
{pkg_hash_ext,[
 {<<"goldrush">>, <<"99CB4128CFFCB3227581E5D4D803D5413FA643F4EB96523F77D9E6937D994CEB">>}]}
].

Perhaps if there are no locks then the file is not created?

Thanks again for the help; vim-erlang is an awesome project!

[hcs42]

Perhaps if there are no locks then the file is not created?

It seems that this is another behaviour that changed in the rebar3 master branch.

• In the latest rebar3 release, the rebar.lock file was created even if there were no dependencies (see the the output I produced in my previous comment).
• But with the rebar3 master branch, the same steps don't produce a rebar.lock file. Only if I add a dependency too.

Creating an empty rebar.lock file looks like a good workaround. For the proper solution, I will need to figure out how to distinguish between rebar2 and rebar3 projects, and how to find the project root. (So far the existence and location of the rebar.lock file was a convenient way to do these.)

vim-erlang is an awesome project!

Thanks :)

[hcs42]

1. Regarding the issue of the new _checkouts ebin location: I pushed the fix I described in my previous comment. It works as long as the dependency is present in the default profile, which I imagine covers most of the use cases.

2. Regarding the issue of the rebar.lock file (described in my other previous comment): I asked the rebar3 developers and the next rebar release will again always produce a rebar.lock file. So no change in vim-erlang-compiler is needed.