control-service: data jobs synchronizer initial implementation #2633

mivanov1988 · 2023-09-01T12:54:55Z

Why
As part of the VEP-2272, we have to introduce a process to synchronize the data jobs from the database to the Kubernetes.

What
The initial implementation of the Data Jobs Synchronizer provides methods for synchronizing jobs between the database and Kubernetes. It introduces new classes suffixed with 'V2' to improve the separation of the old and new approaches.

Please take note that this is just the first phase of implementation. The following enhancements are planned for future PRs:

We will annotate the method DataJobsSynchronizer.synchronizeDataJobs() with @scheduled.
Improved exception handling will be integrated.
An additional deployment table will be introduced to effectively manage the desired and actual deployment state.
More tests will be included in subsequent updates.
ThreadPool configuration will be tuned and exposed through the application.properties.

Testing Done
Integration tests

Signed-off-by: Miroslav Ivanov [email protected]

...nes_control_service/src/main/java/com/vmware/taurus/service/deploy/DataJobsSynchronizer.java

...l_service/src/main/java/com/vmware/taurus/service/deploy/DeploymentNotificationHelperV2.java

...nes_control_service/src/main/java/com/vmware/taurus/service/deploy/DeploymentProgressV2.java

...nes_control_service/src/main/java/com/vmware/taurus/service/deploy/DataJobsSynchronizer.java

...ines_control_service/src/main/java/com/vmware/taurus/service/deploy/DeploymentServiceV2.java

...es_control_service/src/main/java/com/vmware/taurus/service/monitoring/DeploymentMonitor.java

...ce/src/integration-test/java/com/vmware/taurus/service/deploy/DataJobDeploymentCrudITV2.java

antoniivanov

I don't think my comments at this stage are a blocker. So I will let you decide how to handle them - in this PR or subsequent. There's another planned work.

...es_control_service/src/main/java/com/vmware/taurus/service/monitoring/DeploymentMonitor.java

...lines_control_service/src/main/java/com/vmware/taurus/service/deploy/JobImageDeployerV2.java

...ines_control_service/src/main/java/com/vmware/taurus/service/deploy/DeploymentServiceV2.java

...nes_control_service/src/main/java/com/vmware/taurus/service/deploy/DataJobsSynchronizer.java

mivanov1988 · 2023-09-26T12:32:50Z

@antoniivanov, @dakodakov thank you for the review!

Why As part of the VEP-2272, we need to introduce a process which will syncronize the data jobs from the database to the Kubernetes. What TODO Testing Done TODO Signed-off-by: Miroslav Ivanov [email protected]

vmwclabot added the cla-not-required label Sep 1, 2023

mivanov1988 force-pushed the person/miroslavi/implement-kubernetes-cron-jobs-synchronizer branch from 7a7dbfd to cc76404 Compare September 25, 2023 14:10

mivanov1988 changed the title ~~[DRAFT] control-service: data jobs synchronizer~~ control-service: data jobs synchronizer initial implementation Sep 25, 2023

antoniivanov reviewed Sep 25, 2023

View reviewed changes

mivanov1988 force-pushed the person/miroslavi/implement-kubernetes-cron-jobs-synchronizer branch 4 times, most recently from 0aee91c to f45a468 Compare September 26, 2023 09:16

antoniivanov reviewed Sep 26, 2023

View reviewed changes

...ce/src/integration-test/java/com/vmware/taurus/service/deploy/DataJobDeploymentCrudITV2.java Show resolved Hide resolved

antoniivanov approved these changes Sep 26, 2023

View reviewed changes