This repository has been archived by the owner on Jan 20, 2022. It is now read-only.
Scalding batch tests #296
Merged
Scalding batch tests #296
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Conflicts: summingbird-scalding/src/test/scala/com/twitter/summingbird/scalding/ScaldingLaws.scala
|
Looks like that failure was a storm failure. Restarted. We need to get the false positives down on our failure detector. Maybe the stuff @ianoc did on the security manager is ready to merge. |
|
@jnievelt this was work I started before we do more optimizations of the scalding side. I want to have better confidence that we don't regress as we improve the planning. |
| @@ -79,8 +88,9 @@ class TestStore[K, V](store: String, inBatcher: Batcher, initBatch: BatchID, ini | |||
| }.toMap | |||
|
|
|||
| // Call this after you compute to check the results of the | |||
| def lastToIterable(b: BatchID): Iterable[(K, V)] = | |||
| sourceToBuffer(batches(b)).toIterable.map { tup => tconv(new TupleEntry(tup)) } | |||
| def lastToIterable: Iterable[(K, V)] = { | |||
There was a problem hiding this comment.
:) yep. I saw that on github after I pushed. I had added them to get a val, I later removed.
snoble
pushed a commit
to snoble/summingbird
that referenced
this pull request
Sep 8, 2017
…rsFromSB Feature/migrate async summers from sb
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The point of this code is to start testing scalding with a random batcher. Previously, we put everything in one batch so it was easy to write the test code (which needs more refactoring). Now, we generate a random batcher that tries up to 5 batches.
This exposed a lot of little assumptions in the testing, but no bugs in summingbird yet.
The next step is to use this better testing to verify optimizations to the scalding side don't break the logic.
First up: a better algorithm that puts all the heavy lifting into one Map/Reduce job (so run 100 batches in 1 MR step). Until we get template-tap support is scalding, after that step there will be a 100 concurrent Map-only jobs to write to the correct partitions, but that should be really fast, and will be solved when we close:
twitter/scalding#654
or
twitter/scalding#484