DataLoader transducers reducible API #75
Conversation
|
|
||
| dloader = DataLoader(1:1000; batchsize = 2, shuffle = true) | ||
| @test copy(Map(x -> x[1]), Vector{Int}, dloader) != collect(1:2:1000) | ||
| end |
There was a problem hiding this comment.
So now the @inferred tests in this file are passing?
There was a problem hiding this comment.
They don't, I've not made any changes to the behavior of the iterator API. Good point on inference in general though, I changed __foldl__ to infer correctly and added a test for it.
Codecov Report
@@ Coverage Diff @@
## main #75 +/- ##
==========================================
- Coverage 90.36% 89.16% -1.20%
==========================================
Files 13 13
Lines 498 526 +28
==========================================
+ Hits 450 469 +19
- Misses 48 57 +9
Continue to review full report at Codecov.
|
|
Found remaining allocations during iteration was due to the Any chance for final review and merge soon? |
|
I don't understand the Fold-iverse very well, but could this help simplify some of the code in https://github.com/JuliaML/MLUtils.jl/blob/main/src/parallel.jl as well? |
I suspect it may, but I haven't used the parallel transducers API myself yet |
|
I don't quite have the feeling of how often in practice the transducer protocol would be used and if that it is worth carrying another package dependence |
|
Haven't followed this closely, just wanted to point out that the code in |
|
Any chance to consider merging this, please? |
|
Since we already indirectly depend on Transducers.jl it would be ok to also support its API, so we can do this. But the dataloader implementation has changed a bit in the meanwhile, there are merge conflicts. Also depending on the outcome of the discussion in #90 the DataLoader could be moved out of this repo. |
|
@ancapdev we decided to keep the Transducer dependency, if you want to rebase this PR we can merge it |
With the set of runtime options available to
DataLoader, the iteration protocol is not type stable.This PR adds
Transducersreducible support toDataLoader, to invert control flow, and type stabilize over iteration, as an alternative API.Not directly related, I've added some inbounds propagation where I found it missing.