Cu 86by7bekc create new kafka topics for validate and match rules kenya #254
Conversation
WalkthroughThe recent changes introduce Kafka-based asynchronous messaging to the JeMPI application. Specifically, new Kafka topics for validation and matching interactions were added, along with corresponding producers in the Changes
Sequence Diagram(s) (Beta)sequenceDiagram
participant LibMPI
participant KafkaProducer
participant KafkaTopicValidate
participant LinkerDWH
participant KafkaTopicMatch
LibMPI->>KafkaProducer: Create Validation Message
KafkaProducer->>KafkaTopicValidate: Send Validation Message
link KafkaTopicValidate: Kafka Topic for Validation
LinkerDWH->>KafkaProducer: Create MatchNotification Message
KafkaProducer->>KafkaTopicMatch: Send MatchNotification Message
link KafkaTopicMatch: Kafka Topic for Matching
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configration File (
|
…date-and-Match-RulesKenya
…date-and-Match-RulesKenya
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
Outside diff range and nitpick comments (1)
devops/linux/docker/helper/topics/topics-config.sh (1)
Line range hint
52-72: Consider using$(...)notation instead of legacy backticks for command substitution for better readability and nesting capabilities.- RETENTION_MS[$TOPIC_INTERACTION_ETL]=`echo "1*24*60*60*1000" | bc` + RETENTION_MS[$TOPIC_INTERACTION_ETL]=$(echo "1*24*60*60*1000" | bc)
Review Details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (6)
- JeMPI_Apps/JeMPI_LibMPI/src/main/java/org/jembi/jempi/libmpi/LibMPI.java (3 hunks)
- JeMPI_Apps/JeMPI_LibShared/src/main/java/org/jembi/jempi/shared/models/GlobalConstants.java (1 hunks)
- JeMPI_Apps/JeMPI_LibShared/src/main/java/org/jembi/jempi/shared/models/MatchNotification.java (1 hunks)
- JeMPI_Apps/JeMPI_LibShared/src/main/java/org/jembi/jempi/shared/models/Validation.java (1 hunks)
- JeMPI_Apps/JeMPI_Linker/src/main/java/org/jembi/jempi/linker/backend/LinkerDWH.java (3 hunks)
- devops/linux/docker/helper/topics/topics-config.sh (6 hunks)
Files skipped from review due to trivial changes (2)
- JeMPI_Apps/JeMPI_LibShared/src/main/java/org/jembi/jempi/shared/models/MatchNotification.java
- JeMPI_Apps/JeMPI_LibShared/src/main/java/org/jembi/jempi/shared/models/Validation.java
Additional Context Used
ShellCheck (35)
devops/linux/docker/helper/topics/topics-config.sh (35)
[warning] 14-14: TOPICS appears unused. Verify use (or export if used externally).
[warning] 15-15: Quote to prevent word splitting/globbing, or split robustly with mapfile or read -a.
[warning] 16-16: Quote to prevent word splitting/globbing, or split robustly with mapfile or read -a.
[warning] 17-17: Quote to prevent word splitting/globbing, or split robustly with mapfile or read -a.
[warning] 18-18: Quote to prevent word splitting/globbing, or split robustly with mapfile or read -a.
[warning] 19-19: Quote to prevent word splitting/globbing, or split robustly with mapfile or read -a.
[warning] 20-20: Quote to prevent word splitting/globbing, or split robustly with mapfile or read -a.
[warning] 21-21: Quote to prevent word splitting/globbing, or split robustly with mapfile or read -a.
[warning] 22-22: Quote to prevent word splitting/globbing, or split robustly with mapfile or read -a.
[warning] 23-23: Quote to prevent word splitting/globbing, or split robustly with mapfile or read -a.
[warning] 24-24: Quote to prevent word splitting/globbing, or split robustly with mapfile or read -a.
[warning] 37-37: PARTITIONS appears unused. Verify use (or export if used externally).
[warning] 49-49: REPLICATION appears unused. Verify use (or export if used externally).
[style] 52-52: Use $(...) notation instead of legacy backticks
....
[style] 53-53: Use $(...) notation instead of legacy backticks
....
[style] 54-54: Use $(...) notation instead of legacy backticks
....
[style] 55-55: Use $(...) notation instead of legacy backticks
....
[style] 56-56: Use $(...) notation instead of legacy backticks
....
[style] 57-57: Use $(...) notation instead of legacy backticks
....
[style] 58-58: Use $(...) notation instead of legacy backticks
....
[style] 59-59: Use $(...) notation instead of legacy backticks
....
[style] 60-60: Use $(...) notation instead of legacy backticks
....
[warning] 61-61: RETENTION_MS appears unused. Verify use (or export if used externally).
[style] 61-61: Use $(...) notation instead of legacy backticks
....
[style] 64-64: Use $(...) notation instead of legacy backticks
....
[style] 65-65: Use $(...) notation instead of legacy backticks
....
[style] 66-66: Use $(...) notation instead of legacy backticks
....
[style] 67-67: Use $(...) notation instead of legacy backticks
....
[style] 68-68: Use $(...) notation instead of legacy backticks
....
[style] 69-69: Use $(...) notation instead of legacy backticks
....
[style] 70-70: Use $(...) notation instead of legacy backticks
....
[style] 71-71: Use $(...) notation instead of legacy backticks
....
[style] 72-72: Use $(...) notation instead of legacy backticks
....
[warning] 73-73: SEGMENT_BYTES appears unused. Verify use (or export if used externally).
[style] 73-73: Use $(...) notation instead of legacy backticks
....
Additional comments not posted (6)
devops/linux/docker/helper/topics/topics-config.sh (3)
7-8: Added new Kafka topics for validation and matching as per PR objectives.
Line range hint
14-14: Verify the usage or export of arraysTOPICS,PARTITIONS,REPLICATION,RETENTION_MS, andSEGMENT_BYTESto ensure they are not unused.Also applies to: 37-37, 49-49, 61-61, 73-73
Verification successful
The arrays
TOPICS,PARTITIONS,REPLICATION,RETENTION_MS, andSEGMENT_BYTESare indeed used in various scripts within the repository. Here are the relevant locations:
devops/linux/docker/helper/topics/topics-config.shdevops/linux/docker/helper/topics/topics-create.shdevops/linux/docker/helper/topics/topics-describe.shdevops/linux/docker/helper/topics/topics-delete.shdevops/windows/base-docker-linux/helper/topics/topics-create.shdevops/windows/base-docker-linux/helper/topics/topics-describe.shdevops/windows/base-docker-linux/helper/topics/topics-delete.shdevops/windows/base-docker-wsl/helper/topics/topics-create.shdevops/windows/base-docker-wsl/helper/topics/topics-describe.shdevops/windows/base-docker-wsl/helper/topics/topics-delete.shThese arrays are actively used and not unused.
Scripts Executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify the usage or export of the arrays. # Test: Search for the usage or export of the arrays. Expect: Usage in scripts or export statements. rg --type sh $'TOPICS|PARTITIONS|REPLICATION|RETENTION_MS|SEGMENT_BYTES'Length of output: 14418
19-20: Ensure that the new topics are correctly integrated into the system and are being used as intended.Also applies to: 32-33, 44-45, 56-57, 68-69
Verification successful
The new Kafka topics
JeMPI-interaction-validateandJeMPI-interaction-matchare defined inGlobalConstants.javaandtopics-config.sh. Additionally, they are referenced inLinkerDWH.javaandLibMPI.java, indicating their integration into the system.
JeMPI-interaction-validateis used inLibMPI.java.JeMPI-interaction-matchis used inLinkerDWH.java.These references suggest that the new topics are being utilized as intended in the producer and consumer configurations.
Scripts Executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify the usage of new Kafka topics in the system. # Test: Search for the usage of new topics. Expect: Occurrences in producer and consumer configurations. rg --type sh --type java $'JeMPI-interaction-validate|JeMPI-interaction-match'Length of output: 643
Script:
#!/bin/bash # Description: Verify the usage of new Kafka topics in producer and consumer configurations. # Test: Search for the usage of new topics in Java files, focusing on producer and consumer configurations. rg --type java 'TOPIC_INTERACTION_VALIDATE|TOPIC_INTERACTION_MATCH'Length of output: 779
JeMPI_Apps/JeMPI_LibShared/src/main/java/org/jembi/jempi/shared/models/GlobalConstants.java (1)
10-11: Added new constants for Kafka topics as per PR objectives.JeMPI_Apps/JeMPI_Linker/src/main/java/org/jembi/jempi/linker/backend/LinkerDWH.java (2)
38-38: Added a new Kafka producer for match notifications. Ensure proper error handling and integration with the rest of the system.Also applies to: 206-222
Verification successful
The initialization and usage of
matchNoficationProducerinLinkerDWH.javaappear to be properly integrated. The producer is initialized with necessary configurations and used to produce messages. Error handling is also present in the usage.
Initialization:
matchNoficationProducer = new MyKafkaProducer<>(AppConfig.KAFKA_BOOTSTRAP_SERVERS, GlobalConstants.TOPIC_INTERACTION_MATCH, stringSerializer(), matchNotificationSerializer(), "LinkerDWH-INTERACTION-MATCH-NOTIFICATIONS");Usage:
matchNoficationProducer.produceSync(UUID.randomUUID().toString(), new MatchNotification(interaction, new GoldenRecordWithScore(workCandidate.goldenRecord, workCandidate.score))); } catch (JsonProcessingException e) { LOGGER.error(e.getLocalizedMessage(), e); }The integration seems correct, and error handling is in place.
Scripts Executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify the integration and usage of the new Kafka producer for match notifications. # Test: Search for the usage of `matchNoficationProducer`. Expect: Occurrences in method calls and configurations. rg --type java $'matchNoficationProducer'Length of output: 705
Script:
#!/bin/bash # Extract the relevant code segments for detailed review of initialization and usage of matchNoficationProducer. # Extract the initialization code rg --type java -A 5 'matchNoficationProducer = new MyKafkaProducer' # Extract the usage code rg --type java -A 5 'matchNoficationProducer.produceSync'Length of output: 2179
431-433: Added a serializer forMatchNotification. Ensure that it is correctly implemented and used.Verification successful
The
matchNotificationSerializermethod is correctly implemented and used. TheJsonPojoSerializerclass properly serializes objects to JSON, and theMyKafkaProducerclass correctly utilizes the serializer for producing Kafka messages.
JsonPojoSerializerimplementation is found inJeMPI_Apps/JeMPI_LibShared/src/main/java/org/jembi/jempi/shared/serdes/JsonPojoSerializer.java.MyKafkaProducercorrectly initializes a Kafka producer with the provided serializers.Scripts Executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify the implementation and usage of the `matchNotificationSerializer`. # Test: Search for the usage of `matchNotificationSerializer`. Expect: Occurrences in method calls and configurations. rg --type java $'matchNotificationSerializer'Length of output: 389
Script:
#!/bin/bash # Extract lines around the usage of matchNotificationSerializer in LinkerDWH.java rg --type java -A 5 -B 5 $'matchNotificationSerializer' JeMPI_Apps/JeMPI_Linker/src/main/java/org/jembi/jempi/linker/backend/LinkerDWH.java # Locate the implementation of JsonPojoSerializer rg --type java 'class JsonPojoSerializer'Length of output: 1376
Script:
#!/bin/bash # Extract the implementation of JsonPojoSerializer rg --type java -A 20 -B 0 'class JsonPojoSerializer' JeMPI_LibShared/src/main/java/org/jembi/jempi/shared/serdes/JsonPojoSerializer.java # Locate the implementation of MyKafkaProducer rg --type java 'class MyKafkaProducer'Length of output: 451
Script:
#!/bin/bash # Locate the correct path for JsonPojoSerializer fd --type f --name 'JsonPojoSerializer.java' # Extract the implementation of MyKafkaProducer rg --type java -A 20 -B 0 'class MyKafkaProducer' JeMPI_Apps/JeMPI_LibShared/src/main/java/org/jembi/jempi/shared/kafka/MyKafkaProducer.javaLength of output: 1347
Script:
#!/bin/bash # Correct the fd command to locate the correct path for JsonPojoSerializer fd 'JsonPojoSerializer.java' # Extract the full implementation of MyKafkaProducer rg --type java -A 50 -B 0 'class MyKafkaProducer' JeMPI_Apps/JeMPI_LibShared/src/main/java/org/jembi/jempi/shared/kafka/MyKafkaProducer.javaLength of output: 1960
Script:
#!/bin/bash # Extract the implementation of JsonPojoSerializer rg --type java -A 50 -B 0 'class JsonPojoSerializer' JeMPI_Apps/JeMPI_LibShared/src/main/java/org/jembi/jempi/shared/serdes/JsonPojoSerializer.javaLength of output: 776
| @@ -15,12 +15,14 @@ | |||
| import java.time.LocalDateTime; | |||
| import java.util.List; | |||
| import java.util.Locale; | |||
| import java.util.UUID; | |||
There was a problem hiding this comment.
Tip
Codebase Verification
The verification script did not find any try-catch blocks in the LibMPI.java file, indicating a potential lack of explicit error handling around the Kafka producer usage.
- Ensure that error handling is robust around the Kafka producer usage to handle any potential exceptions.
Analysis chain
Added a new Kafka producer for validation messages. Ensure that error handling is robust and that the producer is properly integrated and used throughout the system.
Also applies to: 25-46, 399-408
Scripts Executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Verify the integration and usage of the new Kafka producer for validation messages.
# Test: Search for the usage of `topicValidation`. Expect: Occurrences in method calls and configurations.
rg --type java $'topicValidation'
Length of output: 458
Script:
#!/bin/bash
# Description: Verify error handling around the Kafka producer usage.
# Test: Search for try-catch blocks around the usage of `topicValidation`.
rg --type java -A 5 -B 5 'topicValidation.produceAsync'
Length of output: 1574
Script:
#!/bin/bash
# Description: Verify the presence of try-catch blocks in the `LibMPI.java` file.
# Test: Search for try-catch blocks in the `LibMPI.java` file.
rg --type java 'try\s*\{|\}\s*catch\s*\(' JeMPI_Apps/JeMPI_LibMPI/src/main/java/org/jembi/jempi/libmpi/LibMPI.java
Length of output: 116
MatthewErispe
deleted the
CU-86by7bekc_Create-New-Kafka-Topics-for-Validate-and-Match-RulesKenya
branch
Summary by CodeRabbit
New Features
Enhancements
Infrastructure