promote_typejoin with abstract types

[nalimilan]

Currently, promote_typejoin(Missing, T) returns Any when T isn't concrete. This can be annoying in cases where the type information is necessary to perform some operation (and not just for performance), like computing zero(T):

julia> nt1 = NamedTuple{(:x, :y),Tuple{Union{Missing, Int},Union{Missing, Int}}}((missing, missing));

julia> eltype(nt1)
Union{Missing, Int64}

julia> sum(skipmissing(nt1))
0

julia> nt2 = NamedTuple{(:x, :y),Tuple{Union{Missing, Int},Union{Missing, Float64}}}((missing, missing));

julia> eltype(nt2)
Any

julia> sum(skipmissing(nt2))
ERROR: MethodError: no method matching zero(::Type{Any})
[...]

This would work if promote_typejoin returned Union{Missing, Real} since zero is defined for that type.

This situation happens in practice, for example when a row or group of rows happens to contain only missing values (see JuliaData/DataFrames.jl#2196).

Would it make sense to change promote_typejoin to stop special-casing concrete types?

Cc: @JeffBezanson @vtjnash

[pdeffebach]

This would work if promote_type returned Union{Missing, Real} since zero is defined for that type.

I think you mean promote_typejoin rather than promote_type.

julia> promote_type(Union{Missing, Int}, Float64)
Union{Missing, Float64}

promote_typejoin is what's used when we get eltype(nt::NamedTuple) here. I don't understand the distinction, though.

[nalimilan]

Woops, that was a typo. I was indeed talking about promote_typejoin everywhere.

(promote_type is very different as it's allowed to return a type which isn't a supertype of both inputs. It's used notably by vcat or by fallback methods for mathematical operators e.g. to return Float64 when the inputs mix integers and Float64.)

[pdeffebach]

The solution seems to be altering this block here to add a method like

_promote_typejoin(a::Type{S}, b::Type{Union{T, Missing}}) = Union{typejoin(S, T), MIssing}

[nalimilan]

The isconcretetype(T) check can simply be removed. That's mostly a design decision.

[pdeffebach]

But we aren't joining to Missing, right? Those methods are just for joining Missing with something. Here, we are joinning Union{T, MIssing} with something that is not missing.

[pdeffebach]

Would it be possible to get something that fixes this into 1.6?

[nalimilan]

I've filed #36939 to get the ball rolling.

[nalimilan]

This just got merged into Julia master, so it should be in 1.6.

[pdeffebach]

Wonderful! Now basically all egen behaviors are easy in Julia!