Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: redis sink using depot #193

Merged
merged 12 commits into from
Nov 9, 2022
Merged

feat: redis sink using depot #193

merged 12 commits into from
Nov 9, 2022

Conversation

sumitaich1998
Copy link
Collaborator

@sumitaich1998 sumitaich1998 commented Sep 21, 2022

We are extracting out Redis sink code from firehose and using the Redis sink connector from ODPF Depot library.

Here are the changed configs -

  • we are replacing INPUT_SCHEMA_PROTO_TO_COLUMN_MAPPING config with the config SINK_REDIS_HASHSET_FIELD_TO_COLUMN_MAPPING . The Proto field index in the mapping must be replaced by the proto field name.
  • we are replacing SINK_REDIS_LIST_DATA_PROTO_INDEX config with the config SINK_REDIS_LIST_DATA_FIELD_NAME . The Proto field index must be replaced by the proto field name
  • we are replacing SINK_REDIS_LIST_DATA_PROTO_INDEX config with the config SINK_REDIS_LIST_DATA_FIELD_NAME . The Proto field index must be replaced by the proto field name.
  • SINK_REDIS_KEY_TEMPLATE will now accept proto field names as the argument instead of the proto index.

@kevinbheda kevinbheda self-assigned this Oct 19, 2022
Copy link
Contributor

@kevinbheda kevinbheda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sumitaich1998 sumitaich1998 self-assigned this Nov 4, 2022
@kevinbheda
Copy link
Contributor

dont need the prom sink commit in this branch, please remove it.

@kevinbheda kevinbheda merged commit 01de086 into main Nov 9, 2022
@kevinbheda kevinbheda deleted the redis-depo-sink branch November 9, 2022 07:34
sumitaich1998 added a commit that referenced this pull request Nov 23, 2022
lavkesh pushed a commit that referenced this pull request Nov 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants