Skip to content

Conversation

@mansehajsingh
Copy link
Contributor

@mansehajsingh mansehajsingh commented Apr 21, 2025

Currently the polaris-synchronizer tool performs a full replication onto the target Polaris instance. This includes overwriting entities with the same name on the source and target, as well as removing entities from the target that do not exist on the source.

This change introduces a new option, --strategy to specify a level of modification the tool should perform on the target. There are three supported strategies:

  • CREATE_ONLY: only create entities on the target instance, no overwrites, no removes
  • CREATE_AND_OVERWRITE: create new entities on the target instance, overwrite entities with the same name across the source and target, no removes
  • REPLICATE: full replication, including reflecting removals from the source instance onto the target

The lowest level of modification, CREATE_ONLY is the default.

This PR also adds a flag --delta-only to make modification aware synchronization opt-in. If this flag is not specified,

  • entities across the source and target will not be compared, just overwritten
  • namespace metadata will not be compared across the source and target
  • tables will not be loaded with etag header

Copy link

@travis-bowen travis-bowen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These changes make it a lot easier to configure with basic scenarios!


@CommandLine.Option(
names = {"--strategy"},
defaultValue = "CREATE_ONLY",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to use the enum values here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, we can't use the enum value. I could declare a separate set of constants for them that are assigned strings with the same value, if you think that's worth adding for maintainability? Something like

enum Strategy {
    CREATE_ONLY,
    ...

    class Names {
         public static final String CREATE_ONLY = "CREATE_ONLY";
         ...
    }

}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why doesn't e.g. CREATE_ONLY.name() work?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can only use a String constant in an annotation.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, makes sense. Bummer

Copy link
Contributor

@eric-maynard eric-maynard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, nice!

@eric-maynard eric-maynard merged commit 46fe5e0 into apache:main Apr 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants