feat: twenty orm for standard and custom objects #6178

magrinj · 2024-07-09T14:26:25Z

Overview

This PR builds upon #5153, adding the ability to get a repository for custom objects. The entitySchema is now generated for both standard and custom objects based on metadata stored in the database instead of the decorated WorkspaceEntity in the code. This change ensures that standard objects with custom fields and relations can also support custom objects.

Implementation Details

Key Changes:

Dynamic Schema Generation: The entitySchema for standard and custom objects is now dynamically generated from the metadata stored in the database. This shift allows for greater flexibility and adaptability, particularly for standard objects with custom fields and relations.

Custom Object Repository Retrieval: A repository for a custom object can be retrieved using TwentyORMManager based on the object's name. Here's an example of how this can be achieved:

const repository = await this.twentyORMManager.getRepository('custom');
/*
 * `repository` variable will be typed as follows, ensuring that standard fields and relations are properly typed:
 * const repository: WorkspaceRepository<CustomWorkspaceEntity & {
 *    [key: string]: any;
 * }>
 */
const res = await repository.find({});

Fix #6179

…m-object

greptile-apps

PR Summary

Dynamic Schema Generation: Refactored EntitySchemaColumnFactory, EntitySchemaRelationFactory, and EntitySchemaFactory to support dynamic schema generation from database metadata.
Custom Object Repository: Updated TwentyORMManager to retrieve repositories for custom objects using metadata names.
Storage Management: Introduced ObjectEntitiesStorage for managing entity schemas and metadata, removed ObjectLiteralStorage.
Dependency Updates: Added lodash.lowerfirst to package.json for handling object names.
Scoped Workspace Handling: Renamed and simplified ScopedWorkspaceDatasourceFactory to ScopedWorkspaceIdFactory.

_{12 file(s) reviewed, 1 comment(s)}
_{Edit PR Review Bot Settings}

packages/twenty-server/src/engine/twenty-orm/factories/entity-schema.factory.ts

greptile-apps

PR Summary

(updates since last review)

Updated workspace.repository.ts to use ObjectEntitiesStorage.getObjectMetadataEntity for dynamic schema generation.
Renamed methods in object-entities.storage.ts for clarity.
Added methods in object-entities.storage.ts to retrieve collections and entities by metadata name.

No major changes found since last review.

_{2 file(s) reviewed, no comment(s)}
_{Edit PR Review Bot Settings}

greptile-apps

PR Summary

(updates since last review)

Updated packages/twenty-server/src/engine/twenty-orm/factories/entity-schema.factory.ts to support dynamic schema generation from metadata.
Added getObjectMetadataByName method in entity-schema.factory.ts for fetching metadata.
Modified packages/twenty-server/src/engine/twenty-orm/repository/workspace.repository.ts to use ObjectEntitiesStorage.getObjectMetadataEntity.
Renamed methods in packages/twenty-server/src/engine/twenty-orm/storage/object-entities.storage.ts for clarity.
Enhanced object-entities.storage.ts with new methods for retrieving metadata and entity schemas.

_{3 file(s) reviewed, 1 comment(s)}
_{Edit PR Review Bot Settings}

greptile-apps · 2024-07-09T15:33:19Z

packages/twenty-server/src/engine/twenty-orm/factories/entity-schema.factory.ts

+    if (!objectMetadata) {
+      throw new Error('Object metadata not found');


🧠 logic: Throws an error if objectMetadata is not found.

…m-object

greptile-apps

PR Summary

(updates since last review)

Modified packages/twenty-server/src/database/typeorm/typeorm.module.ts to support dynamic schema generation.
Introduced cacheVersion parameter in packages/twenty-server/src/engine/twenty-orm/factories/workspace-datasource.factory.ts.
Renamed ScopedWorkspaceIdFactory to ScopedWorkspaceContextFactory in packages/twenty-server/src/engine/twenty-orm/factories/index.ts.
Added logic to destroy outdated data sources in packages/twenty-server/src/engine/twenty-orm/storage/data-source.storage.ts.
Updated packages/twenty-server/src/engine/twenty-orm/twenty-orm.manager.ts for dynamic schema generation and cache version management.

_{10 file(s) reviewed, 1 comment(s)}
_{Edit PR Review Bot Settings}

greptile-apps · 2024-07-09T15:51:35Z

packages/twenty-server/src/engine/twenty-orm/storage/data-source.storage.ts

@@ -7,6 +7,16 @@ export class DataSourceStorage {
    new Map();

  public static getDataSource(key: string): WorkspaceDataSource | undefined {
+    // Destroy all data sources that belong to the workspace if the cache version change
+    for (const [dataSourceKey, dataSource] of this.dataSources.entries()) {
+      const workspaceId = key.split('-')?.[0];


🪶 style: Consider adding a null check for key.split('-') to avoid potential runtime errors.

greptile-apps

PR Summary

(updates since last review)

Added UpgradeTo0_22CommandModule in packages/twenty-server/src/database/commands/database-command.module.ts
Removed UpdateBooleanFieldsNullDefaultValuesAndNullValuesCommand in packages/twenty-server/src/database/commands/database-command.module.ts
Introduced FixObjectMetadataIdStandardIdCommand in packages/twenty-server/src/database/commands/upgrade-version/0-22/0-22-fix-object-metadata-id-standard-id.command.ts
Renamed and updated command in packages/twenty-server/src/database/commands/upgrade-version/0-22/0-22-update-boolean-fields-null-default-values-and-null-values.command.ts
Added UpgradeTo0_22Command and module in packages/twenty-server/src/database/commands/upgrade-version/0-22/0-22-upgrade-version.command.ts and 0-22-upgrade-version.module.ts

_{5 file(s) reviewed, no comment(s)}
_{Edit PR Review Bot Settings}

greptile-apps

PR Summary

(updates since last review)

Added Kubernetes and Terraform configurations for deploying TwentyCRM (packages/twenty-docker/k8s/).
Enhanced link handling logic in LinksFieldInput.tsx and LinksFieldMenuItem.tsx (packages/twenty-front/src/modules/object-record/record-field/meta-types/input/components/).
Renamed 'emailAliases' to 'handleAliases' across multiple files for consistency (packages/twenty-server/src/modules/connected-account/, packages/twenty-server/src/modules/messaging/).
Modified validation schema to allow empty strings as valid URLs (packages/twenty-front/src/utils/validation-schemas/absoluteUrlSchema.ts).
Updated dropdown rendering logic to use portals for better positioning (packages/twenty-front/src/modules/ui/layout/dropdown/components/Dropdown.tsx).

_{45 file(s) reviewed, no comment(s)}
_{Edit PR Review Bot Settings}

greptile-apps

PR Summary

(updates since last review)

Switched from InjectWorkspaceRepository to TwentyORMManager for repository management (packages/twenty-server/src/engine/core-modules/calendar/timeline-calendar-event.service.ts)
Updated methods to retrieve repositories dynamically (packages/twenty-server/src/engine/core-modules/calendar/timeline-calendar-event.service.ts)
Introduced dynamic schema generation from database metadata (packages/twenty-server/src/engine/twenty-orm/twenty-orm.manager.ts)
Added custom object repository retrieval (packages/twenty-server/src/engine/twenty-orm/twenty-orm.manager.ts)
New method getRepositoryForWorkspace for specific workspace repositories (packages/twenty-server/src/engine/twenty-orm/twenty-orm.manager.ts)

_{2 file(s) reviewed, no comment(s)}
_{Edit PR Review Bot Settings}

greptile-apps

PR Summary

(updates since last review)

Replaced direct repository injections with dynamic retrieval using TwentyORMManager (packages/twenty-server/src/engine/core-modules/auth/services/google-apis.service.ts)
Updated repository retrieval to use TwentyORMManager for dynamic schema generation (packages/twenty-server/src/modules/calendar/blocklist-manager/jobs/blocklist-item-delete-calendar-events.job.ts)
Refactored services to dynamically obtain repositories via TwentyORMManager (packages/twenty-server/src/modules/calendar/calendar-event-cleaner/services/calendar-event-cleaner.service.ts)
Enhanced flexibility by dynamically fetching repositories in various services (packages/twenty-server/src/modules/calendar/calendar-event-import-manager/services/calendar-events-import.service.ts)
Aligned repository management with new ORM capabilities for custom objects (packages/twenty-server/src/modules/messaging/message-cleaner/services/messaging-message-cleaner.service.ts)

_{18 file(s) reviewed, no comment(s)}
_{Edit PR Review Bot Settings}

This reverts commit 4a496ab.

greptile-apps

PR Summary

(updates since last review)

Replaced WorkspaceSchemaStorageModule with WorkspaceCacheStorageModule for dynamic schema generation (packages/twenty-server/src/engine/api/graphql/core-graphql-api.module.ts)
Updated WorkspaceSchemaStorageService to WorkspaceCacheStorageService for dynamic schema generation (packages/twenty-server/src/engine/api/graphql/workspace-schema.factory.ts)
Introduced CacheManager class for caching dynamic schemas (packages/twenty-server/src/engine/twenty-orm/storage/cache-manager.storage.ts)
Refactored repository injections to use InjectWorkspaceRepository instead of TwentyORMManager (packages/twenty-server/src/modules/calendar/blocklist-manager/jobs/blocklist-item-delete-calendar-events.job.ts)
Added TestModule and TestResolver to demonstrate dynamic schema generation (packages/twenty-server/src/engine/core-modules/test-module/test.module.ts, packages/twenty-server/src/engine/core-modules/test-module/test.resolver.ts)

_{34 file(s) reviewed, 3 comment(s)}
_{Edit PR Review Bot Settings}

packages/twenty-server/src/engine/core-modules/test-module/test.resolver.ts

packages/twenty-server/src/engine/twenty-orm/storage/cache-manager.storage.ts

greptile-apps

PR Summary

(updates since last review)

Added internalContext initialization in createEntityManager method (packages/twenty-server/src/engine/twenty-orm/datasource/workspace.datasource.ts)
Introduced internalContext in WorkspaceEntityManager constructor and updated getRepository method (packages/twenty-server/src/engine/twenty-orm/entity-manager/entity.manager.ts)
Added workspaceId parameter to create method for dynamic schema generation (packages/twenty-server/src/engine/twenty-orm/factories/entity-schema-column.factory.ts)
Removed getRelationType method and centralized relation details logic (packages/twenty-server/src/engine/twenty-orm/factories/entity-schema-relation.factory.ts)
Deleted data-source.storage.ts file, indicating a refactor in data source management (packages/twenty-server/src/engine/twenty-orm/storage/data-source.storage.ts)

_{18 file(s) reviewed, 5 comment(s)}
_{Edit PR Review Bot Settings}

packages/twenty-server/src/engine/twenty-orm/datasource/workspace.datasource.ts

packages/twenty-server/src/engine/twenty-orm/entity-manager/entity.manager.ts

packages/twenty-server/src/engine/twenty-orm/factories/entity-schema.factory.ts

packages/twenty-server/src/engine/twenty-orm/storage/cache-manager.storage.ts

packages/twenty-server/src/engine/twenty-orm/utils/determine-relation-details.util.ts

…m-object

greptile-apps

PR Summary

(updates since last review)

Removed TestModule and related files (packages/twenty-server/src/engine/core-modules/core-engine.module.ts, packages/twenty-server/src/engine/core-modules/test-module/test.module.ts, packages/twenty-server/src/engine/core-modules/test-module/test.resolver.ts)
Improved WorkspaceEntityManager initialization and dynamic schema generation (packages/twenty-server/src/engine/twenty-orm/datasource/workspace.datasource.ts)
Cleaned up logging in WorkspaceEntityManager (packages/twenty-server/src/engine/twenty-orm/entity-manager/entity.manager.ts)
Enhanced dynamic schema generation using WorkspaceEntitiesStorage (packages/twenty-server/src/engine/twenty-orm/factories/entity-schema.factory.ts)
Introduced dynamic schema generation and repository retrieval for custom objects (packages/twenty-server/src/engine/twenty-orm/repository/workspace.repository.ts, packages/twenty-server/src/engine/twenty-orm/twenty-orm.manager.ts)

_{10 file(s) reviewed, 2 comment(s)}
_{Edit PR Review Bot Settings}

greptile-apps · 2024-07-16T14:55:10Z

packages/twenty-server/src/engine/twenty-orm/factories/entity-schema.factory.ts

+    if (!objectMetadata) {
+      throw new Error('Object metadata not found');


Logic: Throws an error if objectMetadata is not found.

greptile-apps · 2024-07-16T14:58:01Z

packages/twenty-server/src/engine/twenty-orm/twenty-orm.manager.ts

+  ): Promise<WorkspaceRepository<T>>;
+
+  async getRepository<T extends ObjectLiteral>(
+    entityClassOrobjectMetadataName: Type<T> | string,


Style: Consider renaming entityClassOrobjectMetadataName to entityClassOrObjectMetadataName for better readability.

…m-object

magrinj added 4 commits July 4, 2024 11:51

feat: wip twenty orm custom objects

443fc2b

Merge remote-tracking branch 'origin/main' into feat/twenty-orm-custo…

1c20a1d

…m-object

feat: twenty-orm based on metadata

f494d5e

Merge remote-tracking branch 'origin/main' into feat/twenty-orm-custo…

4bab95e

…m-object

magrinj requested review from charlesBochet and Weiko July 9, 2024 14:34

magrinj added the -PR: awaiting review label Jul 9, 2024

magrinj marked this pull request as ready for review July 9, 2024 14:35

greptile-apps bot reviewed Jul 9, 2024

View reviewed changes

packages/twenty-server/src/engine/twenty-orm/factories/entity-schema.factory.ts Show resolved Hide resolved

fix: wrong function naming

d9d8480

magrinj force-pushed the feat/twenty-orm-custom-object branch from 8785138 to d9d8480 Compare July 9, 2024 15:32

greptile-apps bot reviewed Jul 9, 2024

View reviewed changes

fix: add TODO

a2648d0

greptile-apps bot reviewed Jul 9, 2024

View reviewed changes

magrinj added 2 commits July 9, 2024 17:48

fix: cache datasource by workspaceId and cacheVersion

e92d713

Merge remote-tracking branch 'origin/main' into feat/twenty-orm-custo…

359f2d8

…m-object

greptile-apps bot reviewed Jul 9, 2024

View reviewed changes

charlesBochet self-assigned this Jul 10, 2024

Merge branch 'main' into feat/twenty-orm-custom-object

9833355

charlesBochet approved these changes Jul 10, 2024

View reviewed changes

greptile-apps bot reviewed Jul 10, 2024

View reviewed changes

Fixes

c5b32af

greptile-apps bot reviewed Jul 10, 2024

View reviewed changes

fix more workspace repositories

4a496ab

greptile-apps bot reviewed Jul 10, 2024

View reviewed changes

magrinj added 2 commits July 11, 2024 10:49

Revert "fix more workspace repositories"

651852b

This reverts commit 4a496ab.

fix: wip

40c8847

greptile-apps bot reviewed Jul 12, 2024

View reviewed changes

charlesBochet added the -PR: wip label Jul 14, 2024

charlesBochet added -PR: awaiting author and removed -PR: awaiting review labels Jul 14, 2024

fix: wip

413707c

greptile-apps bot reviewed Jul 15, 2024

View reviewed changes

magrinj added 2 commits July 16, 2024 16:52

fix: workspace entities storage

d5cf3b7

Merge remote-tracking branch 'origin/main' into feat/twenty-orm-custo…

1cbd505

…m-object

magrinj added -PR: awaiting review and removed -PR: wip -PR: awaiting author labels Jul 16, 2024

magrinj requested a review from charlesBochet July 16, 2024 14:53

magrinj added 2 commits July 16, 2024 16:55

fix: typo

98e5e1b

fix: tests

770e92a

greptile-apps bot reviewed Jul 16, 2024

View reviewed changes

magrinj and others added 11 commits July 16, 2024 17:26

fix: drop lowerfirst

40c51db

fix: login not working

39f788f

fix: cacheVersion is null

094a450

Merge remote-tracking branch 'origin/main' into feat/twenty-orm-custo…

861a80a

…m-object

Merge branch 'main' into feat/twenty-orm-custom-object

7ab6f89

fix: missing await in save method

aa0887a

fix: missing await

b32b375

fix: target instance change over time so comparaison stop working

13fe6ad

Merge remote-tracking branch 'origin/main' into feat/twenty-orm-custo…

dfb1ca4

…m-object

Merge branch 'main' into feat/twenty-orm-custom-object

0772fdd

Fixes

8d96c9e

charlesBochet merged commit 088d061 into main Jul 19, 2024
3 of 4 checks passed

charlesBochet deleted the feat/twenty-orm-custom-object branch July 19, 2024 16:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: twenty orm for standard and custom objects #6178

feat: twenty orm for standard and custom objects #6178

magrinj commented Jul 9, 2024 •

edited

Loading

greptile-apps bot left a comment

greptile-apps bot left a comment

greptile-apps bot left a comment

greptile-apps bot Jul 9, 2024

greptile-apps bot left a comment

greptile-apps bot Jul 9, 2024

greptile-apps bot left a comment

greptile-apps bot left a comment

greptile-apps bot left a comment

greptile-apps bot left a comment

greptile-apps bot left a comment

greptile-apps bot left a comment

greptile-apps bot left a comment

greptile-apps bot Jul 16, 2024

greptile-apps bot Jul 16, 2024

		if (!objectMetadata) {
		throw new Error('Object metadata not found');

feat: twenty orm for standard and custom objects #6178

feat: twenty orm for standard and custom objects #6178

Conversation

magrinj commented Jul 9, 2024 • edited Loading

Overview

Implementation Details

Key Changes:

greptile-apps bot left a comment

Choose a reason for hiding this comment

PR Summary

greptile-apps bot left a comment

Choose a reason for hiding this comment

PR Summary

greptile-apps bot left a comment

Choose a reason for hiding this comment

PR Summary

greptile-apps bot Jul 9, 2024

Choose a reason for hiding this comment

greptile-apps bot left a comment

Choose a reason for hiding this comment

PR Summary

greptile-apps bot Jul 9, 2024

Choose a reason for hiding this comment

greptile-apps bot left a comment

Choose a reason for hiding this comment

PR Summary

greptile-apps bot left a comment

Choose a reason for hiding this comment

PR Summary

greptile-apps bot left a comment

Choose a reason for hiding this comment

PR Summary

greptile-apps bot left a comment

Choose a reason for hiding this comment

PR Summary

greptile-apps bot left a comment

Choose a reason for hiding this comment

PR Summary

greptile-apps bot left a comment

Choose a reason for hiding this comment

PR Summary

greptile-apps bot left a comment

Choose a reason for hiding this comment

PR Summary

greptile-apps bot Jul 16, 2024

Choose a reason for hiding this comment

greptile-apps bot Jul 16, 2024

Choose a reason for hiding this comment

magrinj commented Jul 9, 2024 •

edited

Loading