Don't make join output DataValueArray unless there are NAs

[shashi]

No description provided.

[piever]

I used the same strategy in unstack (assume DataValue even if all values are there) following join example: I should probably change it there as well.

[shashi]

This PR turned out to be unrelated to _promote_op removal though... What it does is it never creates DataValueArray but instead maintains indices where a region (left or right) of the output is null, then creates DataValueArray at the end.

Getting rid of _promote_op considerably increases the complexity of the _join!... Especially harder once we have missing (i.e. the type may change from the first element to the second) in which case you have to use something like try; push!(..); catch err @goto retry end to meaningfully handle this -- this is a performance disaster.

[piever]

We're not the only ones encountering this issue: there is a closely related issue #25925 in Julia Base by @nalimilan: the issue is explained very clearly there but I'm not 100% sure what are the ideas to solve it.

From what I understood the general plan seems to improve inference so that it can figure out which field of the output named tuples could be missing: if this succeeds, preallocate correctly. If this fails (you get type ANY or something unusable), start with some guess and widen as needed.

[nalimilan]

Not really knowing what this PR does, I don't think it's related. Inference isn't needed for joins, as the type of the data is perfectly known: it's that of the input (without any function in-between). What we do in DataFrames is that columns have Union{T, Missing} eltype if one of the input columns allowed for missing values, or when some values have to be filled with missing values for joints other than inner (JuliaData/DataFrames.jl#1316).

[piever]

The main difference is that join in IndexedTables can accept an arbitrary joining function:

join(f, x, y)

where f defaults to be the concatenation function (joining the two data rows) but it could be an arbitrary function taking two Tuples as input and returning a Tuple.

[nalimilan]

I see, thanks. Then indeed the problem is very similar to #25925.