feat: generate ts schema from pg serial field #2952

p5quared · 2024-10-14T20:11:45Z

Description of changes

Update TS schema generator to properly annotate serial fields from Postgres datasources.

e.g.:
myField SERIAL -> myField: a.integer().default()

Explanation:
We already populate field.default when importing a DB schema, but don't really do anything with it prior to this change.

Now, we check if the field default is DB_GENERATED, and the value for the generated field indicates it is a SERIAL column. If this is the case, we annotate the TS schema field with .default().

CDK / CloudFormation Parameters Changed

N/A

Issue #, if available

N/A

Description of how you validated changes

See test cases in

packages/amplify-graphql-schema-generator/src/__tests__/generate-ts-data-schema.test.ts
packages/amplify-graphql-schema-generator/src/__tests__/pg-string-datasource-adapter.test.ts

Checklist

PR description included
yarn test passes
E2E test run linked
Tests are changed or added
Relevant documentation is changed or added (and PR referenced)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

p5quared · 2024-10-14T22:51:23Z

packages/amplify-graphql-schema-generator/src/__tests__/__snapshots__/generate.test.ts.snap

 "
 `;

+exports[`generate graphqlSchemaFromSQLSchema creates graphql schema from "postgres" "serial" schema 1`] = `


This snapshot is not intentional as we are not intending to support GQL generation here. It is generated automatically to test against anyway from the schemas in __utils__.

p5quared · 2024-10-14T22:54:09Z

packages/amplify-graphql-schema-generator/src/__tests__/generate-ts-data-schema.test.ts

    expect(graphqlSchema).toMatchSnapshot();
  });

+  it.each([


These cases are not exhaustive over every Field that could exist but cover the most unique cases.

… a SERIAL field

p5quared · 2024-10-14T23:07:01Z

packages/amplify-graphql-schema-generator/src/ts-schema-generator/helpers.ts

-const createDataType = (type: FieldType): ts.Node => {
-  if (type.kind === 'Scalar') {
+const createDataType = (field: Field): ts.Node => {
+  const sequenceRegex = /^nextval\(.+::regclass\)$/;


Regex Explain

I went back and forth on a few approaches to this check but I eventually felt that a quick regex is the simplest surefire way to identify SERIAL fields. In the future I think this logic could be extracted into something like if (canLetDBGenerate(field)) which evaluates rules isSerial(field) || isUserDefinedFunction(field) etc... But I felt that might be premature for this implementation.

For more context, here's what a value for column_default looks like on a serial field when we query the DB schema:
nextval('serial_table_number_seq'::regclass).

Can we find documentation that this is the canonical way to determine whether a field is serial? In the past we've relied on experimentally-verified "rules" that turned out to be incomplete, so I'd love to get ahead of that this time.

Added main reply with my research/argument later in thread. I don't believe there is a truly canonical way to identify serial but we can identify if a column is backed by a sequence.

p5quared · 2024-10-14T23:23:48Z

...fy-graphql-schema-generator/src/__tests__/__snapshots__/generate-ts-data-schema.test.ts.snap

+    \\"CoffeeQueue\\": a.model({
+        id: a.integer().default(),
+        name: a.string(),
+        orderNumber: a.integer().default()


This is the desired behavior of this PR.

p5quared · 2024-10-14T23:44:12Z

...aphql-schema-generator/src/__tests__/__snapshots__/pg-string-datasource-adapter.test.ts.snap

 ]
 `;
+
+exports[`testPostgresStringDataSourceAdapter sets the model field correctly from a schema with a SERIAL field 1`] = `


This behavior is not new but I am codifying it since generation depends on it now.

sundersc · 2024-10-17T17:04:59Z

packages/amplify-graphql-schema-generator/src/__tests__/__snapshots__/generate.test.ts.snap

+}
+
+
+type SerialTable @refersTo(name: \\"      serial_table\\") @model {


Do we know why we have blank spaces before the table name for refersTo directive?

Whitespace in my template string with the field info csv was causing this--resolved.

sundersc · 2024-10-17T17:09:20Z

packages/amplify-graphql-schema-generator/src/__tests__/pg-string-datasource-adapter.test.ts


+  it('sets the model field correctly from a schema with a SERIAL field', () => {
+    const adapter = new PostgresStringDataSourceAdapter(schemas.postgres.serial);
+    expect(adapter.getModels()[0].getFields()).toMatchSnapshot();


Before this validation, can we add a check to make sure getModels contains an element? If there are failures, the current implementation will throw a generic error like trying to access a property of undefined which could be hard to troubleshoot.

I modified this to match the other tests, e.g. just adapter.getModels(). I think it's clearer and it also retains info that about the PK:

"primaryKey": Index { "fields": Array [ "id", ], "name": "PRIMARY_KEY", },

sundersc · 2024-10-17T17:11:50Z

packages/amplify-graphql-schema-generator/src/ts-schema-generator/helpers.ts

-  if (type.kind === 'Scalar') {
+const createDataType = (field: Field): ts.Node => {
+  const sequenceRegex = /^nextval\(.+::regclass\)$/;
+  if (field.default?.kind === 'DB_GENERATED' && sequenceRegex.test(field.default.value.toString())) {


We can possible move the check sequenceRegex.test(field.default.value.toString() to a separate method so that it can reused in the future like isSequenceField(field).

the text alignment was causing weird names in snaphot

palpatim · 2024-10-21T20:20:17Z

packages/amplify-graphql-schema-generator/src/ts-schema-generator/helpers.ts

+  const sequenceRegex = /^nextval\(.+::regclass\)$/;
+  return field.default?.kind === 'DB_GENERATED' && sequenceRegex.test(field.default.value.toString());


(Copying this from main conversation comment)

Can we find documentation that this is the canonical way to determine whether a field is serial? In the past we've relied on experimentally-verified "rules" that turned out to be incomplete, so I'd love to get ahead of that this time.

In summary we cannot determine if a column was declared serial, we can identify sequences though, and from this, infer serial fields.

Ultimately serial doesn't really exist as a datatype we can directly introspect. In PG it is basically a macro for "Create a column as an integer and create a sequence to back that column as a default". We end up with an ordinary non-null integer with a special default value.

We can tell if a column is backed by a sequence since its default will be nextval(_), one of the special sequence functions.

Reference.

Thanks for the thorough explanation!

* chore: add missing types * test: ts schema serial field gen happy case * test: ts schema serial field gen sad cases * chore: add DEFAULT_METHOD to TS schema constants * feat: intepret pg serial fields in ts schema gen * test: do not handle db provided int constants * refactor: simpler serial field default condition * test: pg string adapter sets model field correctly from a schema with a SERIAL field * test: align text in template string the text alignment was causing weird names in snaphot * test: serial gen snapshots against all of getModels() * refactor: extract sequence field identification logic * chore: no-op change to force rebuild

p5quared added 6 commits October 11, 2024 11:00

chore: add missing types

2147702

test: ts schema serial field gen happy case

995b74d

test: ts schema serial field gen sad cases

3c8f132

chore: add DEFAULT_METHOD to TS schema constants

c8f39ed

feat: intepret pg serial fields in ts schema gen

28a83ed

test: do not handle db provided int constants

e02d2d7

p5quared force-pushed the feat/serial-schema-generation branch 2 times, most recently from 8d4f420 to f411e70 Compare October 14, 2024 22:48

p5quared commented Oct 14, 2024

View reviewed changes

refactor: simpler serial field default condition

102c544

p5quared force-pushed the feat/serial-schema-generation branch from f411e70 to 09e08e8 Compare October 14, 2024 23:03

test: pg string adapter sets model field correctly from a schema with…

b793733

… a SERIAL field

p5quared force-pushed the feat/serial-schema-generation branch from 09e08e8 to b793733 Compare October 14, 2024 23:03

p5quared commented Oct 14, 2024

View reviewed changes

p5quared marked this pull request as ready for review October 14, 2024 23:20

p5quared requested review from a team as code owners October 14, 2024 23:20

p5quared changed the title ~~Feat/serial schema generation~~ feat: generate ts schema from pg serial field Oct 14, 2024

p5quared commented Oct 14, 2024

View reviewed changes

p5quared mentioned this pull request Oct 14, 2024

chore: cleanup imports and update deprecated jest functions #2953

Merged

4 tasks

sundersc reviewed Oct 17, 2024

View reviewed changes

p5quared added 3 commits October 17, 2024 13:39

test: align text in template string

5e5e3c1

the text alignment was causing weird names in snaphot

test: serial gen snapshots against all of getModels()

2c89d64

refactor: extract sequence field identification logic

759789e

sundersc approved these changes Oct 18, 2024

View reviewed changes

palpatim approved these changes Oct 21, 2024

View reviewed changes

p5quared added 2 commits October 21, 2024 18:02

chore: no-op change to force rebuild

07e6c3a

Merge branch 'main' into feat/serial-schema-generation

a1fa7df

palpatim merged commit 6b87331 into main Oct 22, 2024
7 checks passed

palpatim deleted the feat/serial-schema-generation branch October 22, 2024 14:55

p5quared mentioned this pull request Oct 29, 2024

update schema generation aws-amplify/amplify-backend#2164

Merged

4 tasks

		}


		type SerialTable @refersTo(name: \\" serial_table\\") @model {

		const sequenceRegex = /^nextval\(.+::regclass\)$/;
		return field.default?.kind === 'DB_GENERATED' && sequenceRegex.test(field.default.value.toString());

feat: generate ts schema from pg serial field #2952

feat: generate ts schema from pg serial field #2952

Uh oh!

Conversation

p5quared commented Oct 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of changes

CDK / CloudFormation Parameters Changed

Issue #, if available

Description of how you validated changes

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

p5quared Oct 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

p5quared Oct 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

p5quared commented Oct 14, 2024 •

edited

Loading

p5quared Oct 14, 2024 •

edited

Loading

p5quared Oct 21, 2024 •

edited

Loading