KAFKA-18026: KIP-1112, ProcessorWrapper API with PAPI and partial DSL implementation #17892
+622
−16
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ableegoldman
requested review from
bbejeck,
cadonna,
lucasbru and
guozhangwang
ableegoldman
changed the title
ableegoldman
commented
Nov 21, 2024
| @@ -714,8 +714,10 @@ public org.apache.kafka.streams.processor.api.Processor<Object, Object, Object, | |||
| public synchronized <KIn, VIn, KOut, VOut> Topology addProcessor(final String name, | |||
| final ProcessorSupplier<KIn, VIn, KOut, VOut> supplier, | |||
| final String... parentNames) { | |||
| internalTopologyBuilder.addProcessor(name, supplier, parentNames); | |||
| final Set<StoreBuilder<?>> stores = supplier.stores(); | |||
| final ProcessorSupplier<KIn, VIn, KOut, VOut> wrapped = internalTopologyBuilder.wrapProcessorSupplier(name, supplier); | |||
There was a problem hiding this comment.
this is the entire implementation for PAPI apps
ableegoldman
commented
Nov 21, 2024
| dslStoreSuppliers = Utils.newInstance( | ||
| config.getClass(StreamsConfig.DSL_STORE_SUPPLIERS_CLASS_CONFIG), | ||
| DslStoreSuppliers.class); | ||
| dslStoreSuppliers = config.getConfiguredInstance( |
There was a problem hiding this comment.
side fix: we weren't configuring this with the app configs
ableegoldman
commented
Nov 21, 2024
| for (final StoreBuilder<?> storeBuilder : processorSupplier.stores()) { | ||
| ApiUtils.checkSupplier(processorSupplier); | ||
|
|
||
| final ProcessorSupplier<KIn, VIn, KOut, VOut> wrapped = |
There was a problem hiding this comment.
this is how the DSL implementation will work -- processors are wrapped when building the DSL into a Topology, specifically in this method where they are added to the InternalTopologyBuilder.
Note this PR is only a partial implementation since not all DSL operators go through this class, some just write their processors to the InternalTopologyBuilder directly. Followup PRs will tackle converting these operators and enforcing that any new ones added in the future will have to go through this method and be wrapped
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR includes the API for KIP-1112 and a partial implementation, which wraps any processors added through the PAPI and the DSL processors that are written to the topology through the ProcessorParameters#addProcessorTo method.
Further PRs will complete the implementation by converting the remaining DSL operators to using the #addProcessorTo method, and future-proof the processor writing mechanism to prevent new DSL operators from being implemented incorrectly/without the wrapper
Unit tests are WIP -- will be added to this PR shortly