[CRITICAL] v2.4 breaks includet which breaks Genie

[essenciary]

We have noticed that the loading of Genie apps suddenly broke, with the loading just hanging indefinitely, with high Julia CPU usage. Sometimes the whole REPL hangs, forcing the killing of the Julia process.

I traced the issue to Revise.includet starting with v2.4 of LoweredCodeUtils.

For replicating see attached zip of a Genie app. Unzip and pkg> instantiate then run using Revise; Revise.includet("app.jl"). The REPL will hang. CTRL+C will result in something in the lines of:

ERROR: InterruptException:
Stacktrace:
 [1] includet(mod::Module, file::String)
   @ Revise ~/.julia/packages/Revise/7HQ7u/src/packagedef.jl:1027
 [2] includet(file::String)
   @ Revise ~/.julia/packages/Revise/7HQ7u/src/packagedef.jl:1032
 [3] top-level scope
   @ REPL[1]:1

caused by: InterruptException:
Stacktrace:
  [1] copy
    @ ./array.jl:365 [inlined]
  [2] copy
    @ ~/.julia/packages/LoweredCodeUtils/ZMqe2/src/codeedges.jl:716 [inlined]
  [3] enumerate_paths!(paths::Vector{LoweredCodeUtils.Path}, cfg::Core.Compiler.CFG, path::LoweredCodeUtils.Path) (repeats 72 times)
    @ LoweredCodeUtils ~/.julia/packages/LoweredCodeUtils/ZMqe2/src/codeedges.jl:756
  [4] enumerate_paths(cfg::Core.Compiler.CFG)
    @ LoweredCodeUtils ~/.julia/packages/LoweredCodeUtils/ZMqe2/src/codeedges.jl:744
  [5] lines_required!(isrequired::Vector{Bool}, objs::Set{Union{GlobalRef, Symbol}}, src::Core.CodeInfo, edges::LoweredCodeUtils.CodeEdges; norequire::Tuple{})
    @ LoweredCodeUtils ~/.julia/packages/LoweredCodeUtils/ZMqe2/src/codeedges.jl:612
  [6] lines_required!
    @ ~/.julia/packages/LoweredCodeUtils/ZMqe2/src/codeedges.jl:606 [inlined]
  [7] #lines_required!#33
    @ ~/.julia/packages/LoweredCodeUtils/ZMqe2/src/codeedges.jl:586 [inlined]
  [8] lines_required!
    @ ~/.julia/packages/LoweredCodeUtils/ZMqe2/src/codeedges.jl:584 [inlined]
  [9] minimal_evaluation!(predicate::Any, methodinfo::Revise.CodeTrackingMethodInfo, src::Core.CodeInfo, mode::Symbol)
    @ Revise ~/.julia/packages/Revise/7HQ7u/src/lowered.jl:105
 [10] minimal_evaluation!
    @ ~/.julia/packages/Revise/7HQ7u/src/lowered.jl:111 [inlined]
 [11] minimal_evaluation!
    @ ~/.julia/packages/Revise/7HQ7u/src/lowered.jl:115 [inlined]
 [12] methods_by_execution!(recurse::Any, methodinfo::Revise.CodeTrackingMethodInfo, docexprs::Dict{Module, Vector{Expr}}, mod::Module, ex::Expr; mode::Symbol, disablebp::Bool, always_rethrow::Bool, kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:skip_include,), Tuple{Bool}}})
    @ Revise ~/.julia/packages/Revise/7HQ7u/src/lowered.jl:188
 [13] methods_by_execution!
    @ ~/.julia/packages/Revise/7HQ7u/src/lowered.jl:173 [inlined]
 [14] #eval_with_signatures#91
    @ ~/.julia/packages/Revise/7HQ7u/src/packagedef.jl:471 [inlined]
 [15] eval_with_signatures
    @ ~/.julia/packages/Revise/7HQ7u/src/packagedef.jl:468 [inlined]
 [16] instantiate_sigs!(modexsigs::OrderedDict{Module, OrderedDict{Revise.RelocatableExpr, Union{Nothing, Vector{Any}}}}; mode::Symbol, kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:skip_include,), Tuple{Bool}}})
    @ Revise ~/.julia/packages/Revise/7HQ7u/src/packagedef.jl:479
 [17] instantiate_sigs!
    @ ~/.julia/packages/Revise/7HQ7u/src/packagedef.jl:475 [inlined]
 [18] track(mod::Module, file::String; mode::Symbol, kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:skip_include,), Tuple{Bool}}})
    @ Revise ~/.julia/packages/Revise/7HQ7u/src/packagedef.jl:915
 [19] track
    @ ~/.julia/packages/Revise/7HQ7u/src/packagedef.jl:890 [inlined]
 [20] includet(mod::Module, file::String)
    @ Revise ~/.julia/packages/Revise/7HQ7u/src/packagedef.jl:1010
 [21] includet(file::String)
    @ Revise ~/.julia/packages/Revise/7HQ7u/src/packagedef.jl:1032

IrisClustering-main 2.zip

[timholy]

Thanks for the reproducer. This is likely a consequence of my response to #87 (fixed by #88).

Here's the cfg and diagnosis:

julia> cfg
CFG with 221 blocks:
  bb 1 (stmt 1) → bb 22, 2
  bb 2 (stmts 2:3) → bb 4, 3
  bb 3 (stmts 4:5) → bb 21
  bb 4 (stmts 6:19) → bb 20, 5
  bb 5 (stmts 20:25) → bb 20, 6
  bb 6 (stmts 26:28) → bb 20, 7
  bb 7 (stmts 29:31) → bb 9, 8
  bb 8 (stmts 32:33) → bb 9
  bb 9 (stmt 34) → bb 10
  bb 10 (stmt 35) → bb 13, 11
  bb 11 (stmts 36:39) → bb 12
  bb 12 (stmt 40) → bb 15
  bb 13 (stmt 41) → bb 14
  bb 14 (stmts 42:45) → bb 15
  bb 15 (stmt 46) → bb 20, 16
  bb 16 (stmts 47:50) → bb 18, 17
  bb 17 (stmts 51:52) → bb 19
  bb 18 (stmt 53) → bb 19
  bb 19 (stmt 54) → bb 20
  bb 20 (stmt 55) → bb 21
  bb 21 (stmt 56) → bb 23
  bb 22 (stmts 57:59) → bb 23
  bb 23 (stmt 60) → bb 44, 24
  bb 24 (stmts 61:62) → bb 26, 25
  bb 25 (stmt 63) → bb 43
  bb 26 (stmts 64:77) → bb 42, 27
  bb 27 (stmts 78:83) → bb 42, 28
  bb 28 (stmts 84:86) → bb 42, 29
  bb 29 (stmts 87:89) → bb 31, 30
  bb 30 (stmts 90:91) → bb 31
  bb 31 (stmt 92) → bb 32
  bb 32 (stmt 93) → bb 35, 33
  bb 33 (stmts 94:97) → bb 34
  bb 34 (stmt 98) → bb 37
  bb 35 (stmt 99) → bb 36
  bb 36 (stmts 100:103) → bb 37
  bb 37 (stmt 104) → bb 42, 38
  bb 38 (stmts 105:108) → bb 40, 39
  bb 39 (stmts 109:110) → bb 41
  bb 40 (stmt 111) → bb 41
  bb 41 (stmt 112) → bb 42
  bb 42 (stmt 113) → bb 43
  bb 43 (stmt 114) → bb 45
  bb 44 (stmts 115:117) → bb 45
  bb 45 (stmt 118) → bb 66, 46
  bb 46 (stmts 119:120) → bb 48, 47
  bb 47 (stmt 121) → bb 65
  bb 48 (stmts 122:135) → bb 64, 49
  bb 49 (stmts 136:141) → bb 64, 50
  bb 50 (stmts 142:144) → bb 64, 51
  bb 51 (stmts 145:147) → bb 53, 52
  bb 52 (stmts 148:149) → bb 53
  bb 53 (stmt 150) → bb 54
  bb 54 (stmt 151) → bb 57, 55
  bb 55 (stmts 152:155) → bb 56
  bb 56 (stmt 156) → bb 59
  bb 57 (stmt 157) → bb 58
  bb 58 (stmts 158:161) → bb 59
  bb 59 (stmt 162) → bb 64, 60
  bb 60 (stmts 163:166) → bb 62, 61
  bb 61 (stmts 167:168) → bb 63
  bb 62 (stmt 169) → bb 63
  bb 63 (stmt 170) → bb 64
  bb 64 (stmt 171) → bb 65
  bb 65 (stmt 172) → bb 67
  bb 66 (stmts 173:175) → bb 67
  bb 67 (stmt 176) → bb 88, 68
  bb 68 (stmts 177:178) → bb 70, 69
  bb 69 (stmt 179) → bb 87
  bb 70 (stmts 180:193) → bb 86, 71
  bb 71 (stmts 194:199) → bb 86, 72
  bb 72 (stmts 200:202) → bb 86, 73
  bb 73 (stmts 203:205) → bb 75, 74
  bb 74 (stmts 206:207) → bb 75
  bb 75 (stmt 208) → bb 76
  bb 76 (stmt 209) → bb 79, 77
  bb 77 (stmts 210:213) → bb 78
  bb 78 (stmt 214) → bb 81
  bb 79 (stmt 215) → bb 80
  bb 80 (stmts 216:219) → bb 81
  bb 81 (stmt 220) → bb 86, 82
  bb 82 (stmts 221:224) → bb 84, 83
  bb 83 (stmts 225:226) → bb 85
  bb 84 (stmt 227) → bb 85
  bb 85 (stmt 228) → bb 86
  bb 86 (stmt 229) → bb 87
  bb 87 (stmt 230) → bb 89
  bb 88 (stmts 231:233) → bb 89
  bb 89 (stmt 234) → bb 110, 90
  bb 90 (stmts 235:236) → bb 92, 91
  bb 91 (stmt 237) → bb 109
  bb 92 (stmts 238:251) → bb 108, 93
  bb 93 (stmts 252:257) → bb 108, 94
  bb 94 (stmts 258:260) → bb 108, 95
  bb 95 (stmts 261:263) → bb 97, 96
  bb 96 (stmts 264:265) → bb 97
  bb 97 (stmt 266) → bb 98
  bb 98 (stmt 267) → bb 101, 99
  bb 99 (stmts 268:271) → bb 100
  bb 100 (stmt 272) → bb 103
  bb 101 (stmt 273) → bb 102
  bb 102 (stmts 274:277) → bb 103
  bb 103 (stmt 278) → bb 108, 104
  bb 104 (stmts 279:282) → bb 106, 105
  bb 105 (stmts 283:284) → bb 107
  bb 106 (stmt 285) → bb 107
  bb 107 (stmt 286) → bb 108
  bb 108 (stmt 287) → bb 109
  bb 109 (stmt 288) → bb 111
  bb 110 (stmts 289:291) → bb 111
  bb 111 (stmt 292) → bb 132, 112
  bb 112 (stmts 293:294) → bb 114, 113
  bb 113 (stmts 295:296) → bb 131
  bb 114 (stmts 297:310) → bb 130, 115
  bb 115 (stmts 311:316) → bb 130, 116
  bb 116 (stmts 317:319) → bb 130, 117
  bb 117 (stmts 320:322) → bb 119, 118
  bb 118 (stmts 323:324) → bb 119
  bb 119 (stmt 325) → bb 120
  bb 120 (stmt 326) → bb 123, 121
  bb 121 (stmts 327:330) → bb 122
  bb 122 (stmt 331) → bb 125
  bb 123 (stmt 332) → bb 124
  bb 124 (stmts 333:336) → bb 125
  bb 125 (stmt 337) → bb 130, 126
  bb 126 (stmts 338:341) → bb 128, 127
  bb 127 (stmts 342:343) → bb 129
  bb 128 (stmt 344) → bb 129
  bb 129 (stmt 345) → bb 130
  bb 130 (stmt 346) → bb 131
  bb 131 (stmt 347) → bb 133
  bb 132 (stmts 348:351) → bb 133
  bb 133 (stmt 352) → bb 154, 134
  bb 134 (stmts 353:354) → bb 136, 135
  bb 135 (stmts 355:360) → bb 153
  bb 136 (stmts 361:374) → bb 152, 137
  bb 137 (stmts 375:380) → bb 152, 138
  bb 138 (stmts 381:383) → bb 152, 139
  bb 139 (stmts 384:386) → bb 141, 140
  bb 140 (stmts 387:388) → bb 141
  bb 141 (stmt 389) → bb 142
  bb 142 (stmt 390) → bb 145, 143
  bb 143 (stmts 391:394) → bb 144
  bb 144 (stmt 395) → bb 147
  bb 145 (stmt 396) → bb 146
  bb 146 (stmts 397:400) → bb 147
  bb 147 (stmt 401) → bb 152, 148
  bb 148 (stmts 402:405) → bb 150, 149
  bb 149 (stmts 406:407) → bb 151
  bb 150 (stmt 408) → bb 151
  bb 151 (stmt 409) → bb 152
  bb 152 (stmt 410) → bb 153
  bb 153 (stmt 411) → bb 155
  bb 154 (stmts 412:419) → bb 155
  bb 155 (stmt 420) → bb 176, 156
  bb 156 (stmts 421:422) → bb 158, 157
  bb 157 (stmts 423:425) → bb 175
  bb 158 (stmts 426:439) → bb 174, 159
  bb 159 (stmts 440:445) → bb 174, 160
  bb 160 (stmts 446:448) → bb 174, 161
  bb 161 (stmts 449:451) → bb 163, 162
  bb 162 (stmts 452:453) → bb 163
  bb 163 (stmt 454) → bb 164
  bb 164 (stmt 455) → bb 167, 165
  bb 165 (stmts 456:459) → bb 166
  bb 166 (stmt 460) → bb 169
  bb 167 (stmt 461) → bb 168
  bb 168 (stmts 462:465) → bb 169
  bb 169 (stmt 466) → bb 174, 170
  bb 170 (stmts 467:470) → bb 172, 171
  bb 171 (stmts 471:472) → bb 173
  bb 172 (stmt 473) → bb 173
  bb 173 (stmt 474) → bb 174
  bb 174 (stmt 475) → bb 175
  bb 175 (stmt 476) → bb 177
  bb 176 (stmts 477:481) → bb 177
  bb 177 (stmt 482) → bb 198, 178
  bb 178 (stmts 483:484) → bb 180, 179
  bb 179 (stmts 485:487) → bb 197
  bb 180 (stmts 488:501) → bb 196, 181
  bb 181 (stmts 502:507) → bb 196, 182
  bb 182 (stmts 508:510) → bb 196, 183
  bb 183 (stmts 511:513) → bb 185, 184
  bb 184 (stmts 514:515) → bb 185
  bb 185 (stmt 516) → bb 186
  bb 186 (stmt 517) → bb 189, 187
  bb 187 (stmts 518:521) → bb 188
  bb 188 (stmt 522) → bb 191
  bb 189 (stmt 523) → bb 190
  bb 190 (stmts 524:527) → bb 191
  bb 191 (stmt 528) → bb 196, 192
  bb 192 (stmts 529:532) → bb 194, 193
  bb 193 (stmts 533:534) → bb 195
  bb 194 (stmt 535) → bb 195
  bb 195 (stmt 536) → bb 196
  bb 196 (stmt 537) → bb 197
  bb 197 (stmt 538) → bb 199
  bb 198 (stmts 539:543) → bb 199
  bb 199 (stmt 544) → bb 220, 200
  bb 200 (stmts 545:546) → bb 202, 201
  bb 201 (stmts 547:549) → bb 219
  bb 202 (stmts 550:563) → bb 218, 203
  bb 203 (stmts 564:569) → bb 218, 204
  bb 204 (stmts 570:572) → bb 218, 205
  bb 205 (stmts 573:575) → bb 207, 206
  bb 206 (stmts 576:577) → bb 207
  bb 207 (stmt 578) → bb 208
  bb 208 (stmt 579) → bb 211, 209
  bb 209 (stmts 580:583) → bb 210
  bb 210 (stmt 584) → bb 213
  bb 211 (stmt 585) → bb 212
  bb 212 (stmts 586:589) → bb 213
  bb 213 (stmt 590) → bb 218, 214
  bb 214 (stmts 591:594) → bb 216, 215
  bb 215 (stmts 595:596) → bb 217
  bb 216 (stmt 597) → bb 217
  bb 217 (stmt 598) → bb 218
  bb 218 (stmt 599) → bb 219
  bb 219 (stmt 600) → bb 221
  bb 220 (stmts 601:605) → bb 221
  bb 221 (stmts 606:626)

julia> paths = LoweredCodeUtils.Path[]
LoweredCodeUtils.Path[]

julia> LoweredCodeUtils.enumerate_paths!(paths, cfg, LoweredCodeUtils.Path(1))
^CERROR: InterruptException:
Stacktrace:
 [1] copy
   @ Base ./array.jl:411 [inlined]
 [2] copy
   @ LoweredCodeUtils ~/.julia/dev/LoweredCodeUtils/src/codeedges.jl:716 [inlined]
 [3] enumerate_paths!(paths::Vector{LoweredCodeUtils.Path}, cfg::Core.Compiler.CFG, path::LoweredCodeUtils.Path) (repeats 85 times)
   @ LoweredCodeUtils ~/.julia/dev/LoweredCodeUtils/src/codeedges.jl:758
 [4] top-level scope
   @ REPL[10]:1

julia> length(paths)
1079289

I hit Ctrl-C after about 5s. That's a lot of paths through the control-flow graph! Clearly I'm going to need to design a more efficient algorithm. Until then, best to pin LoweredCodeUtils at 2.3. Sorry for the inconvenience.

[essenciary]

Yes, I'm sticking to 2.3.2 for now, thanks!

[essenciary]

In case it's relevant, we noticed the issue on Julia 1.10 initially, but ultimately we reproduced the issue on Julia 1.9 as well.

[timholy]

The control-flow graph for this example is pretty enough to be worth sharing:

(See #87 for details on its generation; this time I flipped the arrows to be from predecessor to successor.)

It can clearly be partitioned (cuts at nodes 45, 67, 89, ...) but some of the lobes are sufficiently complex that it will probably be better to try to come up with a polynomial algorithm.

[essenciary]

Thank you!