Karafka & Waterdrop: support different configurations per topic #5107
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Drowze
force-pushed
the
karafka-configuration-per-topic
branch
2 times, most recently
from
0aee689 to
0faf894
Compare
By the time that `Karafka::App#initialized!` is called,
`Karafka.producer` was already initialized.
On top of that, there's really nowhere we can hook in the Karafka
initialization where we can be sure that `Karafka.producer` wasn't yet
initiaized - that's because the Karafka initialization is NOT
necessarily tied to the WaterDrop initialization. For instance, the
following is a possible scenario:
```
$producer = WaterDrop::Producer.new { ... }
Datadog.configure do |c|
c.tracing.instrument :karafka
end
# note that the producer was initialized before the Karafka app (and in
# this case, even before datadog was configured)
Karafka::App.setup do |c|
c.producer = $producer
end
```
So instead of trying to hook somewhere before `Karafka.producer` is
initialized, let's simply listen to a Karafka after-initialization event
and append our middleware to the producer when that event is triggered.
Drowze
force-pushed
the
karafka-configuration-per-topic
branch
from
0faf894 to
4eae5f9
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
NOTE: this pull request is blocked until #4876 and #5120 are merged
What does this PR do?
Supports different karafka/waterdrop configurations per topic.
Motivation:
Most of our Karafka consumers simply iterate through a kafka batch of messages and process them individually. Some of our consumers however have a some logic to pre-process a batch (e.g.: split in smaller batches based on some logic) to process more efficiently later.
In such cases we want to selectively disable Datadog's Waterdrop/Karafka tracing and handle tracing manually ourselves.
Change log entry
Additional Notes:
How to test the change?
Automated tests provided.