-
-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add mapAccumulate to Traverse #4209
Conversation
797337f
to
eb1f1e1
Compare
Okay, I think I made all the requested changes; please let me know if I forgot something or if there is something else. |
BTW: also I think you can override this for list, vector, lazy list and stream that doesn't use state and is likely far faster. |
One more thing: I think this is a very nice function. |
While trying to override the current implementation for What do you all think @satorg @armanbilge @johnynek? |
Don't forget But it is going to be quite a big chunk of additional work. I would suggest to do it in a separate PR, if possible. |
Traverse already forces a LazyList doesn't it? Otherwise we would need fs2 as much. Am I mistaken? |
That's correct. But in fact,
So we cannot emit the outer But seems that it would not be the case for |
Just a note: seems that
Not sure if in such a case it should really be enforced... |
I can think about several options on how we could address it.
|
The more I think about this the less I like it, I think the safest is just to not leak the |
But there is no Traverse instance for fs2.Stream right? There can't be for the same reason you can't have one for IO. I'm disappointed that the more theoretic argument about what the signature should be (the one that composes and does lifting of the
|
Regarding composition I mentioned in my last comment. Recall the functions here: #4209 (comment) So if we have: val mapList: [X, Y] => (X => Y) => (List[X] => List[Y]) = ...
val mapOption: [X, Y] => (X => Y) => (Option[X] => Option[Y]) = ...
val mapListOption: [X, Y] => (X => Y) => (List[Option[X]] => List[Option[Y]]) =
[X, Y] => (fn: X => Y) => mapList(mapOption(fn)) so this is what we mean when we say Functor composes. If you look at the shapes of the functions in #4209 (comment) , you can see which ones are kind of "shape preserving" meaning the results are the same shape as the original with an Now, look at the case we have here: val mapAccList: [X, S, Y] => ((X, S) => (Y, S)) => ((List[X], S) => (List[Y], S) ...
val mapAccOption: [X, S, Y] => ((X, S) => (Y, S)) => ((Option[X], S) => (Option[Y], S)) = ...
val mapAccListOption: [X, S, Y] => ((X, S) => (Y, S)) => ((List[Option[X]], S) => (List[Option[Y], S)) =
[X, Y] => (fn: (X, S) => (Y, S)) => mapAccList(mapAccOption(fn)) So, in this way, if we have a We can't keep our optimizations under compositions if we discard the final S. I can see the motivations to discard the S in many cases, but I think if we really think we want that we could add a second function that just drops the S off. But to me, the theoretical reasons should be the highest priority here since they tend to guide us to universal principles, not just the aesthetics of one person or another. |
Okay, thank you all for the feedback, I think the definition of the method is ready to go. Let me know what you all think and if there is any other detail to discuss :) |
can we also override mapAccumulate here: cats/core/src/main/scala/cats/Composed.scala Line 118 in 708cbab
and:
|
9eccf9d
to
7707e2b
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking good. Two minor issues.
Not saying we should do it here, but I wonder if there should be something like
Then let strict collections only implement this typeclass. This would allow us to represent the idea of "strictness" it a typefull way (not just by hiding it somewhere inside Cats). |
@satorg but traverse itself is strict isn't it? How can we lazily implement |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is pretty great.
Very nice generalization of mapWithIndex.
Actually, this is not what I meant – I didn't think it through enough. Yes, you're right – it is strict itself. What I really meant: seems we could define an abstract internal mix-in trait |
def mapAccumulate[S, B](init: S)(f: (S, A) => (S, B)): (S, NonEmptySeq[B]) = { | ||
var state = init | ||
|
||
val fb = iterator.map { a => | ||
val (newState, b) = f(state, a) | ||
|
||
state = newState | ||
|
||
b | ||
}.toIndexedSeq | ||
|
||
(state, new NonEmptySeq(fb)) | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is that .toIndexedSeq
still 100% safe here? I mean, IndexedSeq
is an interface. Although all IndexedSeq
-s from the Scala library might be strict collections, but we cannot completely exclude a case when someone can try to create a NonEmptySeq
out of their own IndexedSeq
implementation that have iterator.toIndexedSeq
returning a non-strict collection. E.g. imagine some storage-efficient sparse-array implementation. If it happens then state
won't be properly calculated.
I'd suggest to iterate the iterator directly via hasNext
/next
methods – that will definitely guarantee the enforced evaluation. Also it will probably make the implementation even more efficient since there will be no closure involved anymore.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, there is one in the stdlib IndexedSeqView
which means yeah this is unsafe.
val fb = fa.iterator.map { a => | ||
val (newState, b) = f(state, a) | ||
|
||
state = newState | ||
|
||
b | ||
}.toIndexedSeq |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The same comment as for the NonEmptySeq
above.
Actually, I think that the only implementation should be put here and then re-used from those for Traverse[NonEmptySeq]
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also I still think that it would be better to preserve the original Seq's concrete type. The only thing, it should be a bit different for Scala 2.12 and 2.13+:
val builder = fa.companion.newBuilder[B] // 2.12
OR
val builder = fa.iterableFactory.newBuilder[B] // 2.13+
@satorg I am sorry, but I really don't have the mental energy to deal with |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One question, otherwise LGTM! Thanks for all your work :)
As promised in discord :)