similar and traits

[StefanKarpinski]

The situation for the similar function feels a little messy these days (perhaps I'm wrong about that and it's all under control, in which case, consider this a documentation request). It seems like the similar API could be clarified with appropriate traits, e.g.

similar(a, Dense) # same dimensions and element type as `a` but dense
similar(a, Sparse) # same dimensions and element type as `a` but sparse
similar(a, Mutable) # same dimensions and element type as `a` but fully mutable
similar(a, ElType{T}) # same dimensions and storage type as `a` but with element type `T`
similar(a, Dimensions{...}) # same storage and element type as `a` but with different dimensions

This is sort of already what we're doing but the current system is a bit more ad hoc: type arguments are treated as an implicit element type trait and a tuple of integers is treated collectively as a dimensions trait. It might be fruitful to have a more explicit trait-based core API and translate the various shorthands into that.

[andyferris]

This seems good! While we are thinking about this I would also love to see some solutions for arrays of fixed size (the compiler might need to know the sizes as constants, e.g. as Val{}s or preferably just using constant propagation), and also some way of dealing with immutable arrays (e.g. have similar return Ref of an array? I'm not sure how to have an API that works for both immutable and mutable arrays otherwise).

[lstagner]

Also it would be good to start thinking about adding something like similar_type.

Having something like this would be very helpful for some packages. (JuliaDiff/ForwardDiff.jl#143, SimonDanisch/FixedSizeArrays.jl#118)

In practice similar_type's interface should mirror similar and similar_type(x,args...) == typeof(similar(x,args...)).

[andyferris]

+1 for similar_type. For now we can have it just change the eltype, in Base, and extend it in StaticArrays for sizes.

I've also been thinking that more traits would be really useful for dispatch on array types. Many functions assume, e.g., mutability, while you might want separate methods for arrays of fixed size and other traits. Dimensions{...} could be a great trait for this, e.g. the dimensions of Matrix might be Dimensions{(:,:)} while for SMatrix{3,3} it could be Dimensions({3,3}). Or something.

[StefanKarpinski]

These would all be new signatures, so this could even be added post 1.0. In the meantime, this is unlikely to undergo a massive overhaul in the next few months.

[nalimilan]

How about requiring that any AbstractArray type A support similar(::Type{A{T}}, dims), and defining a fallback similar(a::A, dims) = similar(typeof(A), dims)? That would be needed to provide a promotion mechanism for concatenation (see fourth bullet at #20815), and should require very limited changes.

[Sacha0]

Ref. #13731 (regarding similar and wrapped/structured/sparse matrices).

[Sacha0]

Ref. #10889 (regarding traits to characterize storage).

[Sacha0]

Ref. #23270 (regarding determining types for allocation / allocation of unwrapped storage similar to a wrapped type) and downstream #24148.

[Sacha0]

Ref. #11573 (regarding a formal interface/contract for similar).

[Sacha0]

Ref. #10064 (regarding a formal interface/contract AbstractArrays, which touches on storage layout and allocation).

[Sacha0]

Ref. #12441 (regarding a formal interface/contract for convert, which touches on allocating storage for copies, generating copies with different el or container type, and copying immutables).

[Sacha0]

Ref. #11610 (regarding the interplay between similar and immutables).

[Sacha0]

Ref. #18772 (regarding similar and non-Base array types).

[Sacha0]

Ref. #15198 (regarding similar and structured/wrapped matrix types).

[andyferris]

Ref. #24019 (regarding similar having the same semantics for Associative as AbstractArray, e.g. currently for arrays it preserves indices while for associatives it discards indices)

[nalimilan]

Another use case appeared at JuliaData/DataFrames.jl#1257: for a CategoricalArray or a PooledArray, one may want an array similar in shape, element type but also with the same levels/pool (including its custom ordering). Of course in most cases the pool shouldn't be preserved when calling similar since one could want to store different values in the newly allocated array. Not sure how this could fit in the general system.

[Sacha0]

Ref. #22218 (regarding similar-/copy_oftype-like functionality: allocating an array of the same type as the argument array but with different eltype).

[timholy]

We do have some of these (e.g., #18161 (comment)), and they are documented:

julia> similar(Array{Float16}, (2, 2))
2×2 Array{Float16,2}:
 1.0e-7  0.0
 0.0     0.0

[nalimilan]

Right. I suggest we document this as part of the AbstractArray interface under the "Interfaces" section of the manual, and define fallbacks for all other similar variants which use it.