Implemented a new id strategy in the Sink Connector #87
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ct id as string in a field and then it will be converted to ObjectId inside the connector
|
hi, thanks for the PR @crossleyjuan , we have exactly the same need / constraints in our actual product. Any guys in the mongodb core team can evaluate this pr / need ? |
|
@crossleyjuan That's really great !! Waiting to be merged :) |
| @@ -0,0 +1,20 @@ | |||
| # MongoDB Kafka BsonObjectId | |||
|
|
|||
| This implementaion allows the MongoDB Kafka sink connector to convert an existing _id that is string as ObjectId | |||
There was a problem hiding this comment.
Suggested change
| This implementaion allows the MongoDB Kafka sink connector to convert an existing _id that is string as ObjectId | |
| This implementation allows the MongoDB Kafka sink connector to convert an existing _id that is string as ObjectId |
| VALUE | ||
| } | ||
|
|
||
| private ProvidedIn where; |
There was a problem hiding this comment.
Suggested change
| private ProvidedIn where; | |
| private final ProvidedIn where; |
|
Hi there, |
|
|
||
| abstract class BsonOidProvidedStrategy implements IdStrategy { | ||
|
|
||
| protected enum ProvidedIn { |
| () -> | ||
| new DataException( | ||
| "Provided id strategy is used but the document structure either contained" | ||
| + " no _id field or it was null")); |
|
Hi team, |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Recently I'd a request from my customer where they explained that when the _id is provided in the kafka stream there's no way to convert this to ObjectId to keep it consistent with the other records in the collection, checking the different Id Providers it looks like there's no implementation of the ObjectId when it's provided, as such I've implemented a custom Id Provider that receives the string either in the Key or in the Value and converts this to an ObjectId, this way the customer will keep consistency with the previous records created using standard driver.