Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error with v2.6.0 #458

Closed
baumgold opened this issue Jun 7, 2023 · 9 comments
Closed

Error with v2.6.0 #458

baumgold opened this issue Jun 7, 2023 · 9 comments

Comments

@baumgold
Copy link
Member

baumgold commented Jun 7, 2023

After upgrading to Arrow v2.6.0 I get the following error in one of my apps:

MethodError: Cannot `convert` an object of type Vector{NamedTuple{(:QuoteSetID, :UnderlyingSecurityID, :OfferSize, :BidSize, :SecurityID, :QuoteEntryID, :OfferPx, :BidPx), Tuple{UInt16, Union{Missing, Int32}, Union{Missing, UInt32}, Union{Missing, UInt32}, Int32, UInt32, Union{Missing, FixedPointDecimals.FixedDecimal{Int64, 9}}, Union{Missing, FixedPointDecimals.FixedDecimal{Int64, 9}}}}} to an object of type SubArray{NamedTuple{(:QuoteSetID, :UnderlyingSecurityID, :OfferSize, :BidSize, :SecurityID, :QuoteEntryID, :OfferPx, :BidPx), Tuple{UInt16, Union{Missing, Int32}, Union{Missing, UInt32}, Union{Missing, UInt32}, Int32, UInt32, Union{Missing, FixedPointDecimals.FixedDecimal{Int64, 9}}, Union{Missing, FixedPointDecimals.FixedDecimal{Int64, 9}}}}, 1, Arrow.Struct{NamedTuple{(:QuoteSetID, :UnderlyingSecurityID, :OfferSize, :BidSize, :SecurityID, :QuoteEntryID, :OfferPx, :BidPx), Tuple{UInt16, Union{Missing, Int32}, Union{Missing, UInt32}, Union{Missing, UInt32}, Int32, UInt32, Union{Missing, FixedPointDecimals.FixedDecimal{Int64, 9}}, Union{Missing, FixedPointDecimals.FixedDecimal{Int64, 9}}}}, Tuple{Arrow.Primitive{UInt16, Vector{UInt16}}, Arrow.Primitive{Union{Missing, Int32}, Vector{Int32}}, Arrow.Primitive{Union{Missing, UInt32}, Vector{UInt32}}, Arrow.Primitive{Union{Missing, UInt32}, Vector{UInt32}}, Arrow.Primitive{Int32, Vector{Int32}}, Arrow.Primitive{UInt32, Vector{UInt32}}, Arrow.Primitive{Union{Missing, FixedPointDecimals.FixedDecimal{Int64, 9}}, Vector{Arrow.Decimal{19, 9, Int128}}}, Arrow.Primitive{Union{Missing, FixedPointDecimals.FixedDecimal{Int64, 9}}, Vector{Arrow.Decimal{19, 9, Int128}}}}}, Tuple{UnitRange{Int64}}, true}

Closest candidates are:
  convert(::Type{T}, ::LinearAlgebra.Factorization) where T<:AbstractArray
   @ LinearAlgebra ~/julia/julia-1.9.0/share/julia/stdlib/v1.9/LinearAlgebra/src/factorization.jl:59
  convert(::Type{T}, ::T) where T<:AbstractArray
   @ Base abstractarray.jl:16
  convert(::Type{T}, ::T) where T
   @ Base Base.jl:64
  ...

Stacktrace:
  [1] push!
    @ ./array.jl:1060
  [2] #ToList#16
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/list.jl:117
  [3] ToList
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/list.jl:100 [inlined]
  [4] #arrowvector#18
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/list.jl:214
  [5] arrowvector
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/list.jl:211 [inlined]
  [6] #arrowvector#10
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/arraytypes.jl:97
  [7] #arrowvector#3
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/arraytypes.jl:74
  [8] #49
    @ ./none:0
  [9] iterate
    @ ./generator.jl:47 [inlined]
 [10] collect_to!
    @ ./array.jl:840
 [11] collect_to!
    @ ./array.jl:848
 [12] collect_to!
    @ ./array.jl:848
 [13] collect_to_with_first!
    @ ./array.jl:818
 [14] collect
    @ ./array.jl:792
 [15] _totuple
    @ ./tuple.jl:401 [inlined]
 [16] Tuple
    @ ./tuple.jl:369 [inlined]
 [17] #arrowvector#48
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/struct.jl:93
 [18] arrowvector
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/struct.jl:89 [inlined]
 [19] #arrowvector#10
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/arraytypes.jl:97
 [20] #arrowvector#3
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/arraytypes.jl:74
 [21] arrowvector
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/arraytypes.jl:58 [inlined]
 [22] #toarrowvector#2
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/arraytypes.jl:37
 [23] toarrowvector
    @ ~/.julia/packages/Arrow/86p6m/src/arraytypes/arraytypes.jl:34 [inlined]
 [24] #145
    @ ~/.julia/packages/Arrow/86p6m/src/write.jl:329
 [25] eachcolumn
    @ ~/.julia/packages/Tables/AcRIE/src/utils.jl:70
 [26] toarrowtable
    @ ~/.julia/packages/Arrow/86p6m/src/write.jl:326
 [27] macro expansion
    @ ~/.julia/packages/Arrow/86p6m/src/write.jl:187 [inlined]
 [28] macro expansion
    @ ./task.jl:476 [inlined]
 [29] write
    @ ~/.julia/packages/Arrow/86p6m/src/write.jl:177
 [30] #122
    @ ~/.julia/packages/Arrow/86p6m/src/write.jl:58
 [31] #open#409
    @ ./io.jl:395
 [32] open
    @ ./io.jl:392 [inlined]
 [33] #write#121
    @ ~/.julia/packages/Arrow/86p6m/src/write.jl:57 [inlined]
 [34] write
    @ ~/.julia/packages/Arrow/86p6m/src/write.jl:56 [inlined]
@Moelf
Copy link
Contributor

Moelf commented Jun 7, 2023

you probably manually asserted (initialized) Vector{} of something at some point, see #452

@Moelf
Copy link
Contributor

Moelf commented Jun 7, 2023

https://github.com/JuliaHEP/UnROOT.jl/actions/runs/5202197179/jobs/9383461173?pr=258#step:6:382

hmm, I hit this in a much more general case -- I convert to Arrow and read back, and it works. But some how it hits this thing during == which is worrisome, will provide a reproducer later.

@Moelf
Copy link
Contributor

Moelf commented Jun 7, 2023 •
edited
Loading

yeah ok this seems super broken now @quinnj

julia> vector_vector_int = [[[1]], [[1], [2]], [[1], [2], [3]], [[1], [2], [3], [4]], [[1], [2], [3], [4], [5]]]
5-element Vector{Vector{Vector{Int64}}}:
...

julia> df = DataFrame(; vector_vector_int)
5×1 DataFrame
 Row │ vector_vector_int
     │ Array…
─────┼───────────────────────────
   1 │ [[1]]
   2 │ [[1], [2]]
   3 │ [[1], [2], [3]]
   4 │ [[1], [2], [3], [4]]
   5 │ [[1], [2], [3], [4], [5]]

julia> Arrow.write(path, df)
"/tmp/jl_i5KrCRaB6b"

julia> DataFrame(Arrow.Table(path))
Error showing value of type DataFrame:
ERROR: MethodError: no method matching SubArray{Vector{Int64}, 1, Arrow.List{SubArray{Int64, 1, Arrow.Primitive{Int64, Vector{Int64}}, Tuple{UnitRange{Int64}}, true}, Int32, Arrow.Primitive{Int64, Vector{Int64}}}, Tuple{UnitRange{Int64}}, true}(::SubArray{SubArray{Int64, 1, Arrow.Primitive{Int64, Vector{Int64}}, Tuple{UnitRange{Int64}}, true}, 1, Arrow.List{SubArray{Int64, 1, Arrow.Primitive{Int64, Vector{Int64}}, Tuple{UnitRange{Int64}}, true}, Int32, Arrow.Primitive{Int64, Vector{Int64}}}, Tuple{UnitRange{Int64}}, true})

Closest candidates are:
  SubArray{T, N, P, I, L}(::Any, ::Any, ::Any, ::Any) where {T, N, P, I, L}
   @ Base subarray.jl:19

@Moelf Moelf mentioned this issue Jun 7, 2023
Closed
@quinnj
Copy link
Member

quinnj commented Jun 7, 2023

It'd be nice to have a repro from @baumgold's original error, because it seems like a different thing than what @Moelf is showing later on.

@quinnj
Copy link
Member

quinnj commented Jun 7, 2023

I have a relatively easy fix for @Moelf's case. I've been trying to figure out a way to reproduce @baumgold's error, but have been unsuccessful so far.

@baumgold
Copy link
Member Author

baumgold commented Jun 7, 2023

I’ve been working to come up with a simple reproducible example. Been tied up with some other stuff today but hope to spend more time tonight and tomorrow. Will let you know as soon as I have something. Thanks, all.

@baumgold baumgold mentioned this issue Jun 8, 2023
Merged
@baumgold
Copy link
Member Author

baumgold commented Jun 8, 2023 •
edited
Loading

@quinnj - I think I managed to do it. See my latest comment in #456. I'm not quite sure how we should solve this. Any suggestions?

quinnj added a commit that referenced this issue Jun 8, 2023
@quinnj
Fix case of recursively nested lists
a988fb9 
Fixes issue reported by @Moelf in #458.

The issue is we need to get the eltype from the recursively built
child array instead of as a top-level translation of the current field type.
@quinnj quinnj mentioned this issue Jun 8, 2023
Merged
@quinnj
Copy link
Member

quinnj commented Jun 8, 2023

Ok, I have a fix for @Moelf's issue here: #459. I think we also have a good fix for @baumgold's issue, so let's merge those 2 and get a new release kicked off. These errors are involving pretty complicated types, so it doesn't seem worth reverting/yanking anything IMO, let's just move forward with a patch fix.

quinnj added a commit that referenced this issue Jun 8, 2023
@quinnj
Fix case of recursively nested lists (#459)
074c7a7 
Fixes issue reported by @Moelf in #458.

The issue is we need to get the eltype from the recursively built child
array instead of as a top-level translation of the current field type.
@quinnj
Copy link
Member

quinnj commented Jun 9, 2023

Fixed in #456

@quinnj quinnj closed this as completed Jun 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@quinnj @baumgold @Moelf