Replication job to trigger setup and carbon flow for replica tables [WIP]

[rohitkum2506]

Summary

New job workflow to run Replication setup process on Airflow. Applies to primary tables with defined ReplicationConfig.

Design decisions:

1. The Replication job does not use JobsClient to trigger a job.
   Instead it will use Airflow client to trigger and manage lifecycle of a Airflow job.
2. Replication will have the task definition in Li-Openhouse side since it needs to leverage AirflowClient.
3. As there can be multiple ReplicationConfigs for a table. The ReplicationTask goes over each config sequentially to trigger a setup job corresponding to the config.

Future work:

1. Develop AirflowClient which will allow triggering, managing state of Airflow jobs and integrate with Replication job run.
2. Develop CarbonClient which can trigger carbon jobs to setup scheduled replication flows.
3. Integrate CarbonClient with Replica table setup job

Changes

• Client-facing API Changes
• Internal API Changes
• Bug Fixes
• New Features
• Performance Improvements
• Code Style
• Refactoring
• Documentation
• Tests
  For all the boxes checked, please include additional details of the changes made in this pull request.

Testing Done

• Manually Tested on local docker setup. Please include commands ran, and their output.
• Added new tests for the changes made.
• Updated existing tests to reflect the changes made.
• No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
• Some other form of testing like staging or soak time in production. Please explain.

Tested on Local docker setup:

Ran the new Replication Job with Local Docker setup. Added a table with replicationConfig.
Observed:

1. The new job gets picked up by the JobScheduler and follows the task flow
2. Only Primary tables with defined replicationConfig are considered and others are filtered out

For all the boxes checked, include a detailed description of the testing done for the changes made in this pull request.

Additional Information

• Breaking Changes
• Deprecations
• Large PR broken into smaller PRs, and PR plan linked in the description.

For all the boxes checked, include additional details of the changes made in this pull request.

[abhisheknath2011]

Thanks @rohitkum2506. Did initial pass and added some comments.