Kafka-mesh filter by adamkotwasinski · Pull Request #11936 · envoyproxy/envoy

adamkotwasinski · 2020-07-07T21:22:33Z

Commit message: Kafka-mesh filter
Additional Description:
Kafka mesh filter.
Can be put between native Kafka client and intercept produce requests to forward them to upstream Kafka clusters as per its fowarding rules. Right now there's only one type of rule - prefix by topic name (so e.g. we can send messages to topics a* and have them in cluster 1, while others go to cluster 2). Only produce requests for now. Support for consumers would be more complex or require more limitations (as Kafka consumers keep state, so it's not like Redis - or actually it's like Redis pub/sub).
Doesn't work on Windows because of librdkafka compilation issues (and I don't have a Windows machine).
Risk Level: Low
Testing: Unit tests + integration test (2 kafka clusters are started + proxy in between + producer) + manual testing.
Docs Changes: New .rst & .proto files, as we have a new filter.
Release Notes: N/A

Without getting into too much details:

Kafka producers (Java, Python, etc.) send records to Envoy (the filter contains enough code to establish communication, first it needs to advertise itself it's a Kafka server through api-versions + metadata requests).
Kafka-mesh filter instance receives the records, parses them, forwards them to rules which decide where the records should go.
The records are forwarded to producers dedicated for each of clusters.
librdkafka producers send the records where they should have gone (librdkafka has its own threads and we use dispatcher.post to have the delivery confirmation back to normal worker thread).

Other PRs for kafka-mesh work (merged independently):
#17202
#17374
#17475
#17597
#17783
#17818
#17839
#17887

repokitteh-read-only · 2020-07-07T21:22:41Z

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to (api/envoy[\w/]*/(v1alpha\d?|v1|v2alpha\d?|v2))|(api/envoy/type/(matcher/)?\w+.proto).
CC @envoyproxy/api-shepherds: Your approval is needed for changes made to api/envoy/.
CC @envoyproxy/api-watchers: FYI only for changes made to api/envoy/.

🐱

Caused by: #11936 was opened by adamkotwasinski.

see: more, trace.

stale · 2020-08-01T08:58:12Z

This pull request has been automatically marked as stale because it has not had activity in the last 7 days. It will be closed in 7 days if no further activity occurs. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions!

repokitteh-read-only · 2020-08-07T21:53:59Z

Retrying Azure Pipelines, to retry CircleCI checks, use /retest-circle.

🐱

Caused by: a #11936 (comment) was created by @adamkotwasinski.

see: more, trace.

adamkotwasinski · 2021-08-19T19:17:28Z

/wait

Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

…esh-public Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

…o kafka-mesh-public

Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

…o kafka-mesh-public Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

adamkotwasinski · 2021-09-09T00:44:54Z

Ready for review - also ran some Java tests.

mattklein123

Cool, thanks. A few small comments.

/wait

mattklein123 · 2021-09-09T15:43:38Z

docs/root/configuration/listeners/network_filters/kafka_mesh_filter.rst

+clusters, depending on configured forwarding rules. Corresponding message versions from
+Kafka 2.4.0 are supported.
+
+.. attention::


Can you also add an attention/warning here that this is only available on contrib builds with a link? See what we do for squash and some of the other contrib filters and please unify as needed, thanks.

done - comments and missing link added; same for kafka-broker

mattklein123 · 2021-09-09T15:44:33Z

source/extensions/filters/network/well_known_names.h

+  // Kafka Mesh filter
+  const std::string KafkaMesh = "envoy.filters.network.kafka_mesh";


we are trying to get rid of this file. Can you just inline this?

Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

repokitteh-read-only · 2021-09-09T21:03:07Z

Retrying Azure Pipelines:
Retried failed jobs in: envoy-presubmit

🐱

Caused by: a #11936 (comment) was created by @adamkotwasinski.

see: more, trace.

mattklein123

Thanks!

repokitteh-read-only bot added v2-freeze api labels Jul 7, 2020

adamkotwasinski force-pushed the kafka-mesh-public branch 12 times, most recently from 7e7ea5f to 2ecbd19 Compare July 10, 2020 20:54

stale bot added the stale stalebot believes this issue/PR has not been touched recently label Aug 1, 2020

adamkotwasinski force-pushed the kafka-mesh-public branch from 8ea61ac to a35ea7c Compare August 7, 2020 20:20

stale bot removed the stale stalebot believes this issue/PR has not been touched recently label Aug 7, 2020

adamkotwasinski force-pushed the kafka-mesh-public branch 10 times, most recently from d6e2bd6 to e49dbde Compare August 17, 2020 20:46

adamkotwasinski added 3 commits August 19, 2021 16:40

Add missing test references

53e50b7

Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

Merge remote-tracking branch 'envoy/main' into kafka-mesh-public

a70ba4e

Real merge after upstream kafka facade got into mainline

943351b

Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

adamkotwasinski mentioned this pull request Aug 21, 2021

Does Envoy Proxy's Kafka Network Filter Support Routing to Multiple Kafka Brokers? #17791

Closed

adamkotwasinski added 17 commits August 24, 2021 10:49

kafka: produce request for mesh-filter

b315a0b

Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

kafka: record extractor for produce requests in mesh-filter

91575de

Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

Merge remote-tracking branch 'github/kafka-mesh-produce' into kafka-m…

bf706df

…esh-public Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

More merge fixes

4d2962e

Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

Merge remote-tracking branch 'github/kafka-mesh-record-extractor' int…

df17fd0

…o kafka-mesh-public

Merge remote-tracking branch 'envoy/main' into kafka-mesh-public

e49c1ee

kafka: record extractor for produce requests in mesh-filter

f5120eb

Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

Code cleanup

7078ecb

Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

Merge remote-tracking branch 'github/kafka-mesh-record-extractor' int…

2839755

…o kafka-mesh-public Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

Merge remote-tracking branch 'envoy/main' into kafka-mesh-public

ea3b318

Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

Merge remote-tracking branch 'envoy/main' into kafka-mesh-public

1ef67e9

Formatting

ed36384

Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

Merge remote-tracking branch 'envoy/main' into kafka-mesh-public

34c11fe

Fixes

c01ded7

Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

Cleanup

4b267e8

Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

Cleanup

73549a0

Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

Cleanup

d3b549d

Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

mattklein123 requested changes Sep 9, 2021

View reviewed changes

adamkotwasinski added 3 commits September 9, 2021 11:09

Merge remote-tracking branch 'envoy/main' into kafka-mesh-public

7b4ec98

Docs fixes; inlined filter name

57ff9c8

Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

Remove well_known_names reference and other includes

3b6b3cf

Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

mattklein123 approved these changes Sep 10, 2021

View reviewed changes

adamkotwasinski mentioned this pull request Mar 9, 2023

Kafka protocol filter #2852

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kafka-mesh filter#11936

Kafka-mesh filter#11936
mattklein123 merged 114 commits intoenvoyproxy:mainfrom
adamkotwasinski:kafka-mesh-public

adamkotwasinski commented Jul 7, 2020 •

edited

Loading

Uh oh!

repokitteh-read-only bot commented Jul 7, 2020

Uh oh!

stale bot commented Aug 1, 2020

Uh oh!

repokitteh-read-only bot commented Aug 7, 2020

Uh oh!

adamkotwasinski commented Aug 19, 2021

Uh oh!

adamkotwasinski commented Sep 9, 2021

Uh oh!

mattklein123 left a comment

Uh oh!

mattklein123 Sep 9, 2021

Uh oh!

adamkotwasinski Sep 9, 2021

Uh oh!

mattklein123 Sep 9, 2021

Uh oh!

adamkotwasinski Sep 9, 2021

Uh oh!

repokitteh-read-only bot commented Sep 9, 2021

Uh oh!

mattklein123 left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		// Kafka Mesh filter
		const std::string KafkaMesh = "envoy.filters.network.kafka_mesh";

Conversation

adamkotwasinski commented Jul 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

repokitteh-read-only bot commented Jul 7, 2020

Uh oh!

stale bot commented Aug 1, 2020

Uh oh!

repokitteh-read-only bot commented Aug 7, 2020

Uh oh!

adamkotwasinski commented Aug 19, 2021

Uh oh!

adamkotwasinski commented Sep 9, 2021

Uh oh!

mattklein123 left a comment

Choose a reason for hiding this comment

Uh oh!

mattklein123 Sep 9, 2021

Choose a reason for hiding this comment

Uh oh!

adamkotwasinski Sep 9, 2021

Choose a reason for hiding this comment

Uh oh!

mattklein123 Sep 9, 2021

Choose a reason for hiding this comment

Uh oh!

adamkotwasinski Sep 9, 2021

Choose a reason for hiding this comment

Uh oh!

repokitteh-read-only bot commented Sep 9, 2021

Uh oh!

mattklein123 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

adamkotwasinski commented Jul 7, 2020 •

edited

Loading