Set kafka message size and compression as configurable #173
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
terryyylim
reviewed
Mar 7, 2022
terryyylim
approved these changes
Mar 7, 2022
krithika369
reviewed
Mar 10, 2022
| @@ -52,6 +52,10 @@ type KafkaConfig struct { | |||
| Topic string `json:"topic"` | |||
| // Serialization Format used for the messages | |||
| SerializationFormat SerializationFormat `json:"serialization_format"` | |||
| // Producer Config - Max message byte to send to broker | |||
| MaxMessageBytes int `json:"max_message_bytes"` | |||
There was a problem hiding this comment.
IMO, these values can just be applied at deploy time, like many other default values we have in RouterDefaults (since they are currently applicable to all routers and not user-configured). Any reason we are saving this to the DB?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
This MR is a follow up from the previous, it allows
max.request.sizeandcompression.typeof the Kafka producer configuration to be set in helm values when deploying Turing Server.Modification
api/turing/api/request/request.go- AppendedMaxMessageBytesandCompressionTypeduring building of new router version when logger type is kafka.api/turing/cluster/servicebuilder/router.go- Creating router with env varAPP_KAFKA_MAX_MESSAGE_BYTESandAPP_KAFKA_COMPRESSION_TYPEengines/router/missionctl/log/resultlog/kafka.go- Router's logger to init with config, which are read from env var during initialisation.Testing
Setting router default in Turing Server helms value
Router container env var set with the global values
