Skip to content

KAFKA-9046: Use top-level worker configs for connector admin clients #7525

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
C0urante wants to merge 4 commits into from
Closed

KAFKA-9046: Use top-level worker configs for connector admin clients #7525

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5f31ce2
KAFKA-9046: Use top-level worker configs for connector admin clients
C0urante Oct 15, 2019
575f9e0
KAFKA-9046: Address review comments
C0urante Oct 23, 2019
6ef7a4b
KAFKA-9046: Address review comments
C0urante Oct 23, 2019
ae232d2
KAFKA-9046: Address review comments
C0urante Nov 13, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -606,7 +606,9 @@ static Map<String, Object> adminConfigs(ConnectorTaskId id,
ConnectorConfig connConfig,
Class<? extends Connector> connectorClass,
ConnectorClientConfigOverridePolicy connectorClientConfigOverridePolicy) {
Map<String, Object> adminProps = new HashMap<>();
// Use the top-level worker configs to retain backwards compatibility with older releases which
// did not require a prefix for connector admin client configs in the worker configuration file
Map<String, Object> adminProps = new HashMap<>(config.originals());
adminProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, Utils.join(config.getList(WorkerConfig.BOOTSTRAP_SERVERS_CONFIG), ","));
// User-specified overrides
adminProps.putAll(config.originalsWithPrefix("admin."));
Expand Down
1 change: 1 addition & 0 deletions connect/runtime/src/test/java/org/apache/kafka/connect/runtime/WorkerTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1144,6 +1144,7 @@ public void testAdminConfigsClientOverridesWithAllPolicy() {
connConfig.put("metadata.max.age.ms", "10000");

Map<String, String> expectedConfigs = new HashMap<>();
expectedConfigs.putAll(props);

@ncliang ncliang Oct 22, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The admin configs after overrides will contain both "admin.metadata.max.age.ms", and "metadata.max.age.ms", with different values. I feel like that might be confusing to debug. Could we include just the configs that don't start with "admin." prefix so they would not be overridden?

@C0urante C0urante Oct 23, 2019

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Part of this test is to ensure that overrides happen as expected; leaving out to-be-overridden configs would leave that case uncovered. If anything, it might make sense to expand the tests here to make sure that three distinct sources of configs for the admin client are recognized and handled with the expected precedence. In ascending order, that would be: from the worker config with no prefix, from the worker config with a (stripped) prefix of admin., and from the connector config with a (stripped) prefix of admin.override..

@C0urante C0urante Oct 23, 2019 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or are you suggesting changes in the actual Worker class as opposed to this test class?

@ncliang ncliang Oct 23, 2019 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, yeah, the comment was not about the test, just happen to realize this when looking at the testcase. Could we have something like originalsWithoutPrefix on AbstractConfig and use it to pre-populate adminProps with just original configs not containing override ones? Or even just calling originalsWithPrefix without stripping the prefix and removing those keys from adminProps

@C0urante C0urante Oct 23, 2019

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that should be fine. Probably not worth it to alter the AbstractConfig since that's public API but engaging in some effort to make sure that both admin.foo and foo aren't present in the props given to the admin client seems like a good idea.

expectedConfigs.put("bootstrap.servers", "localhost:9092");
expectedConfigs.put("client.id", "testid");
expectedConfigs.put("metadata.max.age.ms", "10000");
Expand Down