Slow compilation of projects with yrl files on rebar 3.7.0-rc2

[lpil]

Hi! Since moving to 3.7.0-rc2 my project has taken longer to compile.

I think that the yrl file is failing to be cached so that rebar3 is generating the erl file for the yrl file every time, even when the yrl file does not change, though I do not know how to confirm this.

Here are the details:

louispilfold@penguin:~/tmp/rebar3_7_0$ tree src/
src/
├── gleam_parser.erl
├── gleam_parser.yrl
├── myapp.app.src
└── myapp.erl

Here is the yrl file I'm compiling.

Nonterminals
source module statements statement import
function test enum enum_defs enum_def external_fn external_type
exprs expr elems args call_args
type type_args
case_clauses case_clause field fields.

Terminals
'(' ')' '[' ']' '::' '{' '}'
',' '=' '|' '|>' '->'
'<=' '<' '>' '>=' '==' '!='
'.' ':'
'/' '*' '+' '-' '/.' '*.' '+.' '-.'
int float atom string
hole name upname
kw_fn kw_case kw_test kw_raise kw_throw kw_pub kw_enum kw_external kw_type
kw_import.

Rootsymbol source.

Left 150 '!='.
Left 150 '=='.
Left 160 '<'.
Left 160 '<='.
Left 160 '>'.
Left 160 '>='.
Left 180 '|>'.
Left 210 '+'.
Left 210 '+.'.
Left 210 '-'.
Left 210 '-.'.
Left 220 '*'.
Left 220 '*.'.
Left 220 '/'.
Left 220 '/.'.
Left 230 '.'.
Left 240 ':'.
Left 300 '('.
Right 60 '::'.
Right 70 '|'.

source -> module : '$1'.
source -> exprs  : '$1'.

module -> statements : ok.

statements -> statement             : ['$1'].
statements -> statement statements  : ['$1' | '$2'].

statement -> test          : '$1'.
statement -> enum          : '$1'.
statement -> import        : '$1'.
statement -> function      : '$1'.
statement -> external_fn   : '$1'.
statement -> external_type : '$1'.

enum -> kw_pub kw_enum upname              '=' enum_defs : ok.
enum -> kw_pub kw_enum upname '(' args ')' '=' enum_defs : ok.
enum ->        kw_enum upname              '=' enum_defs : ok.
enum ->        kw_enum upname '(' args ')' '=' enum_defs : ok.

enum_defs -> enum_def           : ['$1'].
enum_defs -> enum_def enum_defs : ['$1' | '$2'].

enum_def -> '|' upname                   : ok.
enum_def -> '|' upname '(' type_args ')' : ok.

import -> kw_import name : ok.

type_args -> type               : ['$1'].
type_args -> type ',' type_args : ['$1' | '$3'].

type -> upname '(' type_args ')' : ok.
type -> upname                   : ok.
type -> name                     : ok.

external_fn -> kw_external kw_fn name '(' ')' '->' 'type' '=' atom atom : ok.
external_fn -> kw_pub kw_external kw_fn name '(' ')' '->' 'type' '=' atom atom : ok.
external_fn -> kw_external kw_fn name '(' type_args ')' '->' 'type' '=' atom atom : ok.
external_fn -> kw_pub kw_external kw_fn name '(' type_args ')' '->' 'type' '=' atom atom : ok.

external_type ->        kw_external kw_type upname : ok.
external_type -> kw_pub kw_external kw_type upname : ok.

function -> kw_pub kw_fn name '('      ')' '{' exprs '}' : ok.
function -> kw_pub kw_fn name '(' args ')' '{' exprs '}' : ok.
function ->        kw_fn name '('      ')' '{' exprs '}' : ok.
function ->        kw_fn name '(' args ')' '{' exprs '}' : ok.

test -> kw_test name '{' exprs '}' : ok.

exprs -> expr '=' expr exprs : ok.
exprs -> expr                : '$1'.
exprs -> expr exprs          : ok.

expr -> atom                       : ok.
expr -> int                        : ok.
expr -> float                      : ok.
expr -> string                     : ok.
expr -> hole                       : ok.
expr -> upname                     : ok.
expr -> upname '(' elems ')'       : ok.
expr -> '{' elems '}'              : ok.
expr -> '[' ']'                    : ok.
expr -> '[' elems ']'              : ok.
expr -> '{' '}'                    : ok.
expr -> '{' fields '}'             : ok.
expr -> '{' expr '|' fields '}'    : ok.
expr -> name                       : ok.
expr -> expr '.' name              : ok.
expr -> expr ':' name              : ok.
expr -> expr '(' ')'               : ok.
expr -> expr '(' call_args ')'     : ok.
expr -> expr '.' '(' ')'           : ok.
expr -> expr '.' '(' call_args ')' : ok.
expr -> kw_raise expr ')'          : ok.
expr -> kw_throw expr ')'          : ok.
expr -> expr '::' expr             : ok.
expr -> expr '|>' expr             : ok.
expr -> expr '+' expr              : ok.
expr -> expr '-' expr              : ok.
expr -> expr '*' expr              : ok.
expr -> expr '/' expr              : ok.
expr -> expr '+.' expr             : ok.
expr -> expr '-.' expr             : ok.
expr -> expr '*.' expr             : ok.
expr -> expr '/.' expr             : ok.
expr -> expr '<=' expr             : ok.
expr -> expr '<'  expr             : ok.
expr -> expr '>'  expr             : ok.
expr -> expr '>=' expr             : ok.
expr -> expr '==' expr             : ok.
expr -> expr '!=' expr             : ok.
expr -> kw_case expr '{' case_clauses '}' : ok.
expr -> kw_fn '('      ')' '{' exprs '}'  : ok.
expr -> kw_fn '(' args ')' '{' exprs '}'  : ok.

case_clauses -> case_clause              : ['$1'].
case_clauses -> case_clause case_clauses : ['$1'|'$2'].

case_clause -> '|' expr '->' expr : ok.

call_args -> expr               : ['$1'].
call_args -> expr ','           : ['$1'].
call_args -> expr ',' call_args : ['$1' | '$3'].

args -> name          : [ok].
args -> name ','      : [ok].
args -> name ',' args : [ok | '$3'].

elems -> expr           : ['$1'].
elems -> expr ','       : ['$1'].
elems -> expr ',' elems : ['$1' | '$3'].

fields -> field             : ['$1'].
fields -> field ','         : ['$1'].
fields -> field ',' fields  : ['$1' | '$3'].

field -> name '=' expr      : ok.

$ rebar3 version
rebar 3.7.0-rc2+build.4175.ref83d01b52 on Erlang/OTP 21 Erts 10.1

$ time rebar3 compile
===> Verifying dependencies...
===> Compiling myapp

real	0m2.346s
user	0m1.858s
sys	0m0.504s

$ time rebar3 compile
===> Verifying dependencies...
===> Compiling myapp

real	0m2.394s
user	0m1.908s
sys	0m0.487s

$ rebar3 version
rebar 3.6.2 on Erlang/OTP 21 Erts 10.1

$ time rebar3 compile
===> Verifying dependencies...
===> Compiling myapp

real	0m1.154s
user	0m0.736s
sys	0m0.452s

$ time rebar3 compile
===> Verifying dependencies...
===> Compiling myapp

real	0m1.030s
user	0m0.545s
sys	0m0.409s

[ferd]

Interesting. I'm guessing this might be related to the new compiler structure @tsloughter added. One interesting thing is that I guess they are possibly missing some checks for a changed target, which would prompt for the re-compiling?

[tsloughter]

@ferd yea, pretty sure it is likely a screw up with the digraph in the compile changes. Looking today.

[tsloughter]

err, nevermind, looks like yrl files were never in the digraph. Must just be the check against the time the erl file was created. Will find it.

[tsloughter]

I've got a fix and should have a PR soon. It turns out we actually have a test for this.. so I'm now trying to figure out why the test didn't catch it.

I'm not happy with the fix, I want to make it more generic, but it may be good enough to merge for now.

[tsloughter]

This makes no sense. The test checks if the beam file's last modifies time stamp changes. Which it should if the erl file's last modified time stamp changes. I added into the test a check on the erl file changing and it does indeed then fail. But have to figure out why that isn't causing the beam file to change...

[tsloughter]

D'oh. Got it. Wasn't looking up the beam file on the right path! Which then last_modified just returns 0 both times and 0 == 0!

Ok, have a shitty fix PR coming in a minute.