Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure faithful duplication of columns for backfilling #227

Open
andrew-farries opened this issue Jan 12, 2024 · 0 comments
Open

Ensure faithful duplication of columns for backfilling #227

andrew-farries opened this issue Jan 12, 2024 · 0 comments
Assignees
Labels
bug Something isn't working
Milestone

Comments

@andrew-farries
Copy link
Collaborator

andrew-farries commented Jan 12, 2024

Various pgroll operations need to duplicate (and backfill) a column on migration start. The general pattern for these operations is:

On migration start:

  • Create a duplicate of the affected column
  • Backfill the duplicated column with values from the original.

On migration complete:

  • Drop the original column
  • Rename the duplicated column to the original column name

Duplication of columns currently loses a lot of information about the source column. These means that when such a migration completes, information (such as constraints and defaults) on the original column is lost. Currently only the column's type is preserved.

The operations that duplicate a column are:

  • set NOT NULL.
  • change type.
  • drop a constraint.
  • add a CHECK constraint.
  • add a FOREIGN KEY constraint
  • add a UNIQUE constraint

The information that needs to be preserved when duplicating a column for backfilling is:

  • columnDEFAULT.
  • foreign keys defined on the column.
  • indexes defined on the column.
  • constraints defined on the column.
  • nullability

Support matrix

For each property of a column, p, and each operation type, o, we need to preserve property p when running operation o:

property/operation set NOT NULL drop NOT NULL change type drop constraint add CHECK constraint add FK constraint add UNIQUE constraint
column DEFAULT 🟢 🟢 🟢 🟢 🟢 🟢 🟢
FKs on column 🟢 🟢 🟢 🟢 🟢 🟢 🟢
indexes on column 🔴 🔴 🔴 🔴 🔴 🔴 🔴
check constraints on column 🟢 🟢 🟢 🟢 🟢 🟢 🟢
unique constraints on column 🟢 🟢 🟢 🟢 🟢 🟢 [1]
column nullability n/a n/a 🟢 🟢 🟢 🟢 🟢
column comments 🟢 🟢 🟢 🟢 🟢 🟢 🟢

[1] - It isn't currently possible to create unique constraints covering more than one column so it's not possible to test that existing UNIQUE constraints on a column are preserved when adding another.

@andrew-farries andrew-farries added the bug Something isn't working label Jan 12, 2024
@andrew-farries andrew-farries self-assigned this Jan 12, 2024
andrew-farries added a commit that referenced this issue Jan 15, 2024
#230)

When duplicating a column for backfilling to add `NOT NULL` constraint,
ensure that any foreign keys on the duplicated column are preserved.

This fixes the issue where adding `NOT NULL` to an FK column would drop
the FK from the column.

This is part of a larger class of issues where duplicated columns are
not faithfully preserving all properties of the original, tracked in
#227.
andrew-farries added a commit that referenced this issue Jan 15, 2024
Preserve any foreign key constraints defined on columns when changing a
column's type.

Uses the `Duplicator` type added in #230 in the 'change type' operation
to ensure that FK constraints are preserved when the column is
duplicated for backfilling.

#230 did the same thing for
preserving FK constraints when a column has a `NOT NULL` constraint
added.

Part of #227
andrew-farries added a commit that referenced this issue Jan 16, 2024
When duplicating a column for backfilling to change a column's type,
ensure that any default on the column is preserved on the duplicated
column.

This fixes the issue where changing type would lose any `DEFAULT`
defined on the column

This is part of a larger class of issues where duplicated columns are
not faithfully preserving all properties of the original, tracked in
#227.
andrew-farries added a commit that referenced this issue Jan 16, 2024
Add a check to ensure that any `DEFAULT` on a column that is set to `NOT
NULL` is preserved by the set `NOT NULL` operation.

This already works as a consequence of earlier PRs for
#227 but it's worth explicitly
testing this behaviour.
andrew-farries added a commit that referenced this issue Jan 16, 2024
Preserve properties of columns when duplicating them for backfilling to
add a`CHECK` constraint. Currently, the column properties that are
preserved are:

* `DEFAULT`s
* foreign key constraints

but this list will grow as more work is done on #227.
andrew-farries added a commit that referenced this issue Jan 16, 2024
Preserve properties of columns when duplicating them for backfilling to
add a `UNIQUE` constraint. Currently, the column properties that are
preserved are:

* `DEFAULT`s
* foreign key constraints

but this list will grow as more work is done on
#227.
andrew-farries added a commit that referenced this issue Jan 16, 2024
…238)

Preserve properties of columns when duplicating them for backfilling to
add a `FOREIGN KEY` constraint. Currently, the column properties that
are preserved are:

* `DEFAULT`s
* foreign key constraints

but this list will grow as more work is done on
#227.
andrew-farries added a commit that referenced this issue Jan 19, 2024
When duplicating a column for backfilling ensure that any `CHECK`
constraints on the original column are re-created on the duplicated
column. The `CHECK` constraint is initially created as `NOT VALID` then
validated after migration completion.

This is part of #227

As of this PR, column properties that are preserved when duplicating a
column for backfilling are:

* `DEFAULT` values
* `FOREIGN KEY` constraints
* `CHECK` constraints
andrew-farries added a commit that referenced this issue Jan 22, 2024
When duplicating a column for backfilling ensure that a `NOT NULL`
attribute on the original column is re-created on the duplicated column.
A `CHECK IS NOT NULL` constraint is initially created as `NOT VALID`
then validated and converted to a `NOT NULL` column attribute after
migration completion.

This is part of #227

As of this PR, column properties that are preserved when duplicating a
column for backfilling are:

* `DEFAULT` values
* `FOREIGN KEY` constraints
* `CHECK` constraints
* `NOT NULL` attributes
andrew-farries added a commit that referenced this issue Jan 22, 2024
…247)

Add uniqueness information to each index recorded in `pgroll`'s internal
schema representation.

With the extra information, the entry for indexes in the schema
representation now looks like:

```json
"indexes": {                                                           
    "products_name_unique": {                                          
        "name": "products_name_unique",                                
        "unique": true,                                                                                                              
    },                                                                  
    "_pgroll_new_products_pkey": {                                     
        "name": "_pgroll_new_products_pkey",                           
        "unique": true,                                                                                                          
    }                                                                  
}
```

Having index uniqueness recorded in the schema representation will help
with preserving uniqueness constraints on columns duplicated for
backfilling (see #227).
andrew-farries added a commit that referenced this issue Jan 25, 2024
When duplicating a column for backfilling ensure that any `UNIQUE`
constraints on the original column are re-created on the duplicated
column. The `UNIQUE` constraint is initially created as an index
concurrently and then converted to a constraint on migration completion.

This is part of #227

As of this PR, column properties that are preserved when duplicating a
column for backfilling are:

* `DEFAULT` values
* `FOREIGN KEY` constraints
* `CHECK` constraints
* `UNIQUE` constraints
andrew-farries added a commit that referenced this issue Jan 26, 2024
Ensure that properties of a column are preserved when duplicating a
column for backfilling as part of the drop constraint operation.

The properties that are preserved are:

* `NOT NULL`
* `DEFAULT`
* `FOREIGN KEY` constraints
* `CHECK` constraints
* `UNIQUE` constraints

Part of #227
@andrew-farries andrew-farries pinned this issue Jan 26, 2024
andrew-farries added a commit that referenced this issue Mar 20, 2024
Ensure that column comments are preserved when a column is duplicated.

Part of #227
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant