-
Notifications
You must be signed in to change notification settings - Fork 193
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
kie-issues#2789: [sonataflow-operator] Implement DB Migrator Changes into SonataFlow Operator #2790
base: main
Are you sure you want to change the base?
Conversation
packages/sonataflow-operator/api/v1alpha08/sonataflowplatform_services_types.go
Outdated
Show resolved
Hide resolved
packages/sonataflow-operator/internal/controller/platform/dbMigratorJob.go
Outdated
Show resolved
Hide resolved
const ( | ||
dbMigrationJobName = "sonataflow-db-migrator-job" | ||
dbMigrationContainerName = "db-migration-container" | ||
dbMigratorToolImage = "quay.io/rhkp/incubator-kie-kogito-service-db-migration-postgresql:latest" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This image must come from the image package. So can you bring that here? Or we can merge the image first and then you review this const. I'm about to change this versioning/image naming in the operator package to read this info from ENV/Configuration.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated to "docker.io/apache/incubator-kie-kogito-db-migrator-tool:latest"
as named in DB Migrator Image package here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For now, unresolving this and reverting to old image for sake of seeing how CI behaves.
packages/sonataflow-operator/internal/controller/platform/dbMigratorJob.go
Outdated
Show resolved
Hide resolved
packages/sonataflow-operator/internal/controller/platform/dbMigratorJob.go
Outdated
Show resolved
Hide resolved
packages/sonataflow-operator/internal/controller/platform/dbMigratorJob.go
Outdated
Show resolved
Hide resolved
packages/sonataflow-operator/internal/controller/platform/dbMigratorJob.go
Outdated
Show resolved
Hide resolved
packages/sonataflow-operator/internal/controller/platform/services/secrets.go
Outdated
Show resolved
Hide resolved
packages/sonataflow-operator/internal/controller/platform/services/services.go
Show resolved
Hide resolved
...est/e2e/testdata/platform/persistence/service_based_db_migration/02-sonataflow_platform.yaml
Outdated
Show resolved
Hide resolved
packages/sonataflow-operator/internal/controller/platform/db_migrator_Job.go
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @rhkp I have added some comments, but couldn't finish the review.
Will continue on this after Christmas.
Many thanks!
packages/sonataflow-operator/api/v1alpha08/sonataflow_persistence_types.go
Show resolved
Hide resolved
packages/sonataflow-operator/internal/controller/platform/db_migrator_job.go
Outdated
Show resolved
Hide resolved
packages/sonataflow-operator/internal/controller/platform/db_migrator_job.go
Outdated
Show resolved
Hide resolved
packages/sonataflow-operator/internal/controller/platform/db_migrator_job.go
Outdated
Show resolved
Hide resolved
packages/sonataflow-operator/internal/controller/platform/db_migrator_job.go
Show resolved
Hide resolved
packages/sonataflow-operator/internal/controller/platform/db_migrator_job.go
Outdated
Show resolved
Hide resolved
packages/sonataflow-operator/internal/controller/platform/db_migrator_job.go
Outdated
Show resolved
Hide resolved
packages/sonataflow-operator/internal/controller/platform/db_migrator_job.go
Outdated
Show resolved
Hide resolved
503f490
to
ab197e9
Compare
Can someone please help me understand the relationship between this PR and #2697? This PR's issue has no description, which makes it hard to follow what's the overall plan for the DB Migrator image. Thanks! |
90cdbd4
to
00e2a1d
Compare
packages/sonataflow-operator/config/manager/controllers_cfg.yaml
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the dbMigrationStrategy must be added to the DI and JS in this file too.
dbMigrationStrategy:
default: service
description: |-
DB Migration approach for service?
job: use job based approach
service: service itself shall migrate the db
none: no db migration needed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@wmedvede Where are DI and JS in this file? Please advise.
packages/sonataflow-operator/internal/controller/cfg/controllers_cfg.go
Outdated
Show resolved
Hide resolved
packages/sonataflow-operator/internal/controller/platform/services/secrets.go
Outdated
Show resolved
Hide resolved
} | ||
|
||
if jsJobsBasedDBMigration { | ||
quarkusDataSourceJobService = getQuarkusDataSourceFromPersistence(ctx, platform, platform.Spec.Services.JobService.Persistence, "jobs-service") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same as for the DI, see comment above.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
and same comment for default schema name calculation, see: j.GetServiceName()
return defaultSchemaName | ||
} | ||
|
||
func getJdbcUrl(postgresql *operatorapi.PersistencePostgreSQL, defaultDBSchemaName string) string { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe this calculations are kind of duplicated here:
Line 44 in c690d46
if postgresql.ServiceRef != nil { |
I think we should explore moving this calculation postgresql.go and reuse.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This file needs tests, the same as we have for other ones.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @wmedvede!
They are included here e.g.
Please let me know if you are referring to something else.
MigrationCmd string | ||
} | ||
|
||
func getDBSchemaName(persistencePostgreSQL *operatorapi.PersistencePostgreSQL, defaultSchemaName string) string { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
tests please
return dataSourceURL | ||
} | ||
|
||
func getQuarkusDataSourceFromPersistence(ctx context.Context, platform *operatorapi.SonataFlowPlatform, persistence *operatorapi.PersistenceOptionsSpec, defaultSchemaName string) *QuarkusDataSource { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Important:
Persistence configuration for a service can be taken not only from eg. spec.services.jobsService but also from the SPF global persistence configuration.
see:
Along this PR, we must consider that potential case too.
I.e, the service don't have "persistence", but the SPF do.
Examples:
apiVersion: sonataflow.org/v1alpha08
kind: SonataFlowPlatform
metadata:
name: sonataflow-platform
spec:
services:
# no persistence is defined for the DI, it will take it from the SFP if any.
# and apply the by default DBMigrationStrategy
dataIndex:
enabled: true
jobService:
# The only persistence setting we do is the DBMigrationStrategy
# So, the service grabs the persistence configuation from the SPF and the given DBMigrationStrategy
# from here.
enabled: true
persistence:
dbMigrationStrategy: "job"
# global SPF persitence configuration.
persistence:
postgresql:
secretRef:
name: postgres-secrets
userKey: POSTGRESQL_USER
passwordKey: POSTGRESQL_PASSWORD
serviceRef:
name: postgres
databaseName: sonataflow
@rhkp take look please and propagate in all places where needed.
@ricardozanini see my considerations for this particular case, where a service takes the persistence from the SPF, but we still can mark at the service level the dbMigrationStrategy
packages/sonataflow-operator/internal/controller/platform/db_migrator_job.go
Outdated
Show resolved
Hide resolved
packages/sonataflow-operator/internal/controller/platform/db_migrator_job.go
Outdated
Show resolved
Hide resolved
packages/sonataflow-operator/internal/controller/platform/db_migrator_job.go
Outdated
Show resolved
Hide resolved
diJobsBasedDBMigration := false | ||
jsJobsBasedDBMigration := false | ||
|
||
if pshDI.IsPersistenceEnabledtInSpec() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consider my comments regarding a potential persistence configuration at the SFP level
Closes #2789
Implement DB Migrator Changes into SonataFlow Operator.