feat(subscriptions): Executor uses the ExecuteQuery strategy #2280
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Integrates the ExecuteQuery strategy in the subscriptions executor. Note that currently the `executor_sample_rate` option is not set so all scheduled subscriptions will be filtered at the start of the pipeline, and even their offsets will not be committed. This is to avoid running duplicate queries on ClickHouse until we are ready to begin testing.
Codecov Report
@@ Coverage Diff @@
## master #2280 +/- ##
==========================================
- Coverage 93.20% 93.06% -0.14%
==========================================
Files 551 553 +2
Lines 24989 25245 +256
==========================================
+ Hits 23290 23494 +204
- Misses 1699 1751 +52
Continue to review full report at Codecov.
|
fpacifici
reviewed
Dec 15, 2021
| def create( | ||
| self, commit: Callable[[Mapping[Partition, Position]], None] | ||
| ) -> ProcessingStrategy[KafkaPayload]: | ||
| return Noop(commit) | ||
| executor = ThreadPoolExecutor(self.__max_concurrent_queries) |
There was a problem hiding this comment.
You do not need to create a threadpool each time you create the step, which can happen often. Please keep it static.
There was a problem hiding this comment.
Moved this line out of the processing strategy factory and passing it in now so it doesn't get recreated on partition assignment, etc
fpacifici
approved these changes
Dec 16, 2021
lynnagara
added a commit
that referenced
this pull request
Dec 16, 2021
Integrates the ExecuteQuery strategy in the subscriptions executor. Note that currently the `executor_sample_rate` option is not set so all scheduled subscriptions will be filtered at the start of the pipeline, and even their offsets will not be committed. This is to avoid running duplicate queries on ClickHouse until we are ready to begin testing.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Integrates the ExecuteQuery strategy in the subscriptions executor.
Note that currently the
executor_sample_rateoption is not setso all scheduled subscriptions will be filtered at the start of the
pipeline, and even their offsets will not be committed.
This is to avoid running duplicate queries on ClickHouse until we
are ready to begin testing.