Type intersection stack overflow testing StructuredOptimization.jl

[maleadt]

The following MWE reproduces a stack overflow as seen on PkgEval, https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_date/2022-12/11/StructuredOptimization.primary.log:

pkg> activate --temp
pkg> add StructuredOptimization, FFTW

julia> using StructuredOptimization, FFTW
julia> using StructuredOptimization: ProximalOperators, AbstractOperators

julia> x = Tuple{Type{ProximalOperators.Precompose{_A, _B, _C, _D, _E, _F}} where _F where _E where _D where _C where _B where _A, ProximalOperators.NormL2{Int64}, AbstractOperators.DCT{1, Float64, FFTW.DCTPlan{Float64, 5, false}, FFTW.DCTPlan{Float64, 4, false}}, Float64, Union{AbstractArray{C, N} where N, C} where C<:Union{Base.Complex{R}, R} where R<:Real}
julia> y = Tuple{Type{ProximalOperators.Precompose{T, R, C, U, V, M}}, T, M, U, V} where M where V<:Union{AbstractArray{C, N} where N, C} where U<:Union{AbstractArray{R, N} where N, R} where C<:Union{Base.Complex{R}, R} where R<:Real where T<:ProximalOperators.ProximableFunction
julia> typeintersect(x, y)

ERROR: StackOverflowError:
Stacktrace:
 [1] typeintersect(a::Any, b::Any)
   @ Base ./reflection.jl:712
 [2] top-level scope
   @ REPL[23]:1

Looking at the rr trace (included in the PkgEval log), type intersection seems stuck:

(rr) f 56
#56 0x00007fc07b382c01 in subtype (x=0x1d86ae0, y=0x1d868e0, e=0x7ffe7f161fa0, param=<optimized out>) at /cache/build/default-amdci4-3/julialang/julia-master/src/subtype.c:1231
1231            return var_lt((jl_tvar_t*)x, y, e, param);
(rr) f 62
#62 0x00007fc07b382c01 in subtype (x=0x1d86ae0, y=0x1d868e0, e=0x7ffe7f161fa0, param=<optimized out>) at /cache/build/default-amdci4-3/julialang/julia-master/src/subtype.c:1231
1231            return var_lt((jl_tvar_t*)x, y, e, param);

(rr) C<:Union{Base.Complex{R<:Real}, R<:Real}
(rr) Union{AbstractArray{C<:Union{Base.Complex{R<:Real}, R<:Real}, N} where N, C<:Union{Base.Complex{R<:Real}, R<:Real}}

This also happens on 1.8, so isn't a regression, but it's consistently listed as a crash at the top of PkgEval reports so would be good to fix.

[giordano]

This also happens on 1.8, so isn't a regression

Might be a regression compared to older versions as tests were reportedly passing on nightly and 1.6 in July 2021: https://github.com/JuliaFirstOrder/StructuredOptimization.jl/commits/master

[giordano]

@N5N3 was this fixed by #48029?

[N5N3]

No, that PR just fix the MWE above and my local test shows many remaining stack overflow.

Edit: MWE for remaining intersect issue

S1 = Tuple{Int, Vararg{Val{C} where C<:Union{Complex{R}, R}}} where R
T1 = Tuple{Any, Vararg{Val{C} where C<:Union{Complex{R}, R}}} where R<:Real
julia> typeintersect(S1, T1)
ERROR: StackOverflowError: