Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Uniformize datasources #5196

Merged
merged 1 commit into from
Apr 27, 2024
Merged

Uniformize datasources #5196

merged 1 commit into from
Apr 27, 2024

Conversation

charlesBochet
Copy link
Member

@charlesBochet charlesBochet commented Apr 27, 2024

Context

We recently enabled the option to bypass SSL certificate authority validation when establishing a connection to PostgreSQL. Previously, if this validation failed, the server would revert to unencrypted traffic. Now, it maintains encryption even if the SSL certificate check fails. In the process, we overlooked a few DataSource setups, prompting a review of DataSource creation within our code.

Current State

Our DataSource initialization is distributed as follows:

  • Database folder: Contains 'core', 'metadata', and 'raw' DataSources. The 'core' and 'metadata' DataSources manage migrations and static resolver calls to the database. The 'raw' DataSource is utilized in scripts and commands that require handling both aspects.
  • typeorm.service.ts script: These DataSources facilitate multi-schema connections.

Vision for Discussion

  • SystemSchema (formerly core) DataSource: Manages system schema migrations and system resolvers/repos. The 'core' schema will be renamed to 'system' as the Core API will include parts of the system and workspace schemas.
  • MetadataSchema DataSource: Handles metadata schema migrations and metadata API resolvers/repos.
  • (Dynamic) WorkspaceSchema DataSource: Will be used in the Twenty ORM to access a specific workspace schema.

We currently do not support cross-schema joins, so maintaining these DataSources separately should be feasible. Core API resolvers will select the appropriate DataSource based on the field context.

  • To be discussed: The potential need for an AdminDataSource (akin to 'Raw'), which would be used in commands, setup scripts, and the admin panel to connect to any database schema without loading any model. This DataSource should be reserved for cases where utilizing metadata, system, or workspace entities is impractical.

In This PR

  • Ensuring all existing DataSources are compliant with the SSL update.
  • Introducing RawDataSource to eliminate the need for declaring new DataSource() instances in commands.

@charlesBochet
Copy link
Member Author

@magrinj @Weiko FYI, would be great to know your thought on that

@@ -89,6 +89,11 @@ export class TypeORMService implements OnModuleInit, OnModuleDestroy {
? ['query', 'error']
: ['error'],
schema,
ssl: this.environmentService.get('PG_SSL_ALLOW_SELF_SIGNED')
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this datasource is not used as we have disable the 1 datasource per workspace pattern which we would need if we want to introduce 1 user per workspace (pg_graphql needs it for performance + great for security/privacy). Likely to be deprecated mid term with twenty orm effort

@charlesBochet charlesBochet merged commit e976a1b into main Apr 27, 2024
5 checks passed
@charlesBochet charlesBochet deleted the uniformize-datasources branch April 27, 2024 09:43
arnavsaxena17 pushed a commit to arnavsaxena17/twenty that referenced this pull request Oct 6, 2024
## Context

We recently enabled the option to bypass SSL certificate authority
validation when establishing a connection to PostgreSQL. Previously, if
this validation failed, the server would revert to unencrypted traffic.
Now, it maintains encryption even if the SSL certificate check fails. In
the process, we overlooked a few DataSource setups, prompting a review
of DataSource creation within our code.

## Current State

Our DataSource initialization is distributed as follows:
- **Database folder**: Contains 'core', 'metadata', and 'raw'
DataSources. The 'core' and 'metadata' DataSources manage migrations and
static resolver calls to the database. The 'raw' DataSource is utilized
in scripts and commands that require handling both aspects.
- **typeorm.service.ts script**: These DataSources facilitate
multi-schema connections.

## Vision for Discussion
- **SystemSchema (formerly core) DataSource**: Manages system schema
migrations and system resolvers/repos. The 'core' schema will be renamed
to 'system' as the Core API will include parts of the system and
workspace schemas.
- **MetadataSchema DataSource**: Handles metadata schema migrations and
metadata API resolvers/repos.
- **(Dynamic) WorkspaceSchema DataSource**: Will be used in the Twenty
ORM to access a specific workspace schema.

We currently do not support cross-schema joins, so maintaining these
DataSources separately should be feasible. Core API resolvers will
select the appropriate DataSource based on the field context.
- **To be discussed**: The potential need for an AdminDataSource (akin
to 'Raw'), which would be used in commands, setup scripts, and the admin
panel to connect to any database schema without loading any model. This
DataSource should be reserved for cases where utilizing metadata,
system, or workspace entities is impractical.

## In This PR
- Ensuring all existing DataSources are compliant with the SSL update.
- Introducing RawDataSource to eliminate the need for declaring new
DataSource() instances in commands.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant