6181 workflows create a custom code executor #6235

martmull · 2024-07-12T09:53:14Z

Testing

download Altair graphql dev tool https://altairgraphql.dev/#download
create a file locally test.ts containing:

export const handler = async (event: object, context: object) => {
  return { test: 'toto', data: event['data'] };
}

play those requests in Altair:
mutation UpsertFunction($file: Upload!) {
upsertFunction(name: "toto", file: $file)
}

mutation ExecFunction {
executeFunction(name:"toto", payload: {data: "titi"})
}

it will run the local driver, add those env variable to test with lambda driver

CUSTOM_CODE_ENGINE_DRIVER_TYPE=lambda
LAMBDA_REGION=eu-west-2
LAMBDA_ROLE=<ASK_ME>

github-actions · 2024-07-12T09:54:12Z

	Warnings
⚠️	Changes were made to the environment variables, but not to the documentation - Please review your changes and check if a change needs to be documented!

Generated by 🚫 dangerJS against 2ec302a

greptile-apps

PR Summary

Added dependencies for AWS Lambda and file handling in package.json
Introduced CUSTOM_CODE_ENGINE_DRIVER_TYPE in .env.example
Integrated FunctionModule into core-engine.module.ts
Made FileUploadModule globally available in file-upload.module.ts
Added Function entry to FileFolder enum in file-folder.interface.ts

_{29 file(s) reviewed, 16 comment(s)}
_{Edit PR Review Bot Settings}

packages/twenty-server/src/engine/core-modules/function/function.module.ts

packages/twenty-server/src/engine/core-modules/function/function.service.ts

packages/twenty-server/src/engine/integrations/custom-code-engine/custom-code-engine.service.ts

...ne/integrations/custom-code-engine/drivers/interfaces/custom-code-engine-driver.interface.ts

greptile-apps · 2024-07-12T10:03:59Z

packages/twenty-server/src/engine/integrations/custom-code-engine/drivers/local.driver.ts

+        if (code && code !== 0) {
+          reject(new Error(`Child process exited with code ${code}`));
+          fs.unlink(tmpFilePath);
+        }


🧠 logic: Ensure fs.unlink handles errors to avoid unhandled promise rejections.

packages/twenty-server/src/engine/integrations/custom-code-engine/utils/compile-typescript.ts

...y-server/src/engine/integrations/custom-code-engine/utils/temporary-lambda-folder-manager.ts

packages/twenty-server/src/engine/integrations/file-storage/utils/read-file-content.ts

greptile-apps

PR Summary

(updates since last review)

Fixed typos in import paths across multiple files for FunctionWorkspaceEntity
Implemented AWS Lambda driver for custom code execution (lambda.driver.ts)
Implemented local driver for custom code execution (local.driver.ts)
Added new fields and relations in function.workspace-entity.ts to support custom code executor
Ensured correct initialization of injected driver in custom-code-engine.service.ts

_{9 file(s) reviewed, 6 comment(s)}
_{Edit PR Review Bot Settings}

greptile-apps · 2024-07-12T13:09:07Z

packages/twenty-server/src/engine/integrations/custom-code-engine/drivers/lambda.driver.ts

+    this.fileUploadService = options.fileUploadService;
+  }
+
+  async upsert(file: FileUpload) {


🪶 style: Consider adding error handling for file reading and TypeScript compilation.

greptile-apps · 2024-07-12T13:09:24Z

packages/twenty-server/src/engine/integrations/custom-code-engine/drivers/local.driver.ts

+  }
+
+  async upsert({ createReadStream, filename, mimetype }: FileUpload) {
+    const typescriptCode = await readFileContent(createReadStream());


🧠 logic: Ensure readFileContent handles potential errors from the stream.

greptile-apps · 2024-07-12T13:09:24Z

packages/twenty-server/src/engine/integrations/custom-code-engine/drivers/local.driver.ts

+
+    const { path: builtSourcePath } = await this.fileUploadService.uploadFile({
+      file: javascriptCode,
+      filename: '.js',


🪶 style: Filename should include the original filename with a .js extension for clarity.

greptile-apps · 2024-07-12T13:09:25Z

packages/twenty-server/src/engine/integrations/custom-code-engine/drivers/local.driver.ts

+      child.on('message', (message: object) => {
+        resolve(message);
+        child.kill();
+        fs.unlink(tmpFilePath);


🧠 logic: Ensure fs.unlink handles errors to avoid unhandled promise rejections.

greptile-apps · 2024-07-12T13:09:26Z

packages/twenty-server/src/engine/integrations/custom-code-engine/drivers/local.driver.ts

+      child.on('error', (error) => {
+        reject(error);
+        child.kill();
+        fs.unlink(tmpFilePath);


🧠 logic: Ensure fs.unlink handles errors to avoid unhandled promise rejections.

greptile-apps · 2024-07-12T13:09:27Z

packages/twenty-server/src/engine/integrations/custom-code-engine/drivers/local.driver.ts

+        if (code && code !== 0) {
+          reject(new Error(`Child process exited with code ${code}`));
+          fs.unlink(tmpFilePath);
+        }


🧠 logic: Ensure fs.unlink handles errors to avoid unhandled promise rejections.

greptile-apps

PR Summary

(updates since last review)

Removed JavaScript code upload in packages/twenty-server/src/engine/integrations/custom-code-engine/drivers/lambda.driver.ts
Set builtSourcePath to undefined in LambdaDriver class
Updated filename handling logic in packages/twenty-server/src/engine/integrations/custom-code-engine/drivers/local.driver.ts
Improved temporary file creation and cleanup in local.driver.ts

_{2 file(s) reviewed, 6 comment(s)}
_{Edit PR Review Bot Settings}

packages/twenty-server/src/engine/integrations/custom-code-engine/drivers/lambda.driver.ts

packages/twenty-server/src/engine/integrations/custom-code-engine/drivers/local.driver.ts

greptile-apps

PR Summary

(updates since last review)

Introduced optional builtSourcePath in custom-code-engine-driver.interface.ts
Removed JavaScript code upload in lambda.driver.ts
Set builtSourcePath to undefined in LambdaDriver class
Updated filename handling logic in local.driver.ts to ensure correct JavaScript filename generation
Handled upload of compiled JavaScript file with correct filename in local.driver.ts

_{3 file(s) reviewed, 1 comment(s)}
_{Edit PR Review Bot Settings}

packages/twenty-server/src/engine/integrations/custom-code-engine/drivers/lambda.driver.ts

greptile-apps

PR Summary

(updates since last review)

Renamed upsert to generateExecutable in packages/twenty-server/src/engine/core-modules/function/function.service.ts
Updated generateExecutable method in packages/twenty-server/src/engine/integrations/custom-code-engine/custom-code-engine.service.ts
Modified generateExecutable method in packages/twenty-server/src/engine/integrations/custom-code-engine/drivers/interfaces/custom-code-engine-driver.interface.ts
Enhanced Lambda function deployment logic in packages/twenty-server/src/engine/integrations/custom-code-engine/drivers/lambda.driver.ts
Improved TypeScript compilation and execution in packages/twenty-server/src/engine/integrations/custom-code-engine/drivers/local.driver.ts

_{5 file(s) reviewed, 3 comment(s)}
_{Edit PR Review Bot Settings}

packages/twenty-server/src/engine/integrations/custom-code-engine/drivers/local.driver.ts

greptile-apps

PR Summary

(updates since last review)

Modified upsertFunction to update existing functions by name in packages/twenty-server/src/engine/core-modules/function/function.service.ts
Minor refactoring for readability and maintainability in packages/twenty-server/src/engine/core-modules/function/function.service.ts

_{1 file(s) reviewed, no comment(s)}
_{Edit PR Review Bot Settings}

packages/twenty-website/src/content/developers/self-hosting/self-hosting-var.mdx

packages/twenty-server/src/modules/function/standard-objects/function.workspace-entity.ts

Devessier

That's impressive what you have made in a few days. Congrats!

I left a few comments about implementation details.

packages/twenty-server/src/engine/core-modules/code-engine/drivers/lambda.driver.ts

packages/twenty-server/src/engine/core-modules/code-engine/utils/build-directory-manager.ts

packages/twenty-server/src/engine/core-modules/code-engine/utils/create-zip-file.ts

packages/twenty-server/src/engine/integrations/file-storage/utils/read-file-content.ts

...server/src/database/typeorm/metadata/migrations/1721040829256-createFunctionMetadataTable.ts

charlesBochet

That's great. Overall, I have 4 concerns:

functionMetadata is not the right naming actually. Ideally it should be function but this is a keyword reserved in the language. So I think we should actually find something else: customFunction, functionBlock, codeFunction?
code-engine is confusing with the twenty-engine. Maybe code-runner ? not a fan as we clearly have a build/compile and a run/execute phase
-,I think we should not pass the file-storage to the drivers
and drivers should be in the integrations folder

packages/twenty-server/src/engine/core-modules/code-engine/code-engine-module.factory.ts

...server/src/database/typeorm/metadata/migrations/1721040829256-createFunctionMetadataTable.ts

packages/twenty-server/src/engine/core-modules/code-engine/drivers/lambda.driver.ts

packages/twenty-server/src/engine/core-modules/code-engine/utils/build-directory-manager.ts

...-server/src/engine/workspace-manager/workspace-sync-metadata/constants/standard-field-ids.ts

...server/src/engine/workspace-manager/workspace-sync-metadata/constants/standard-object-ids.ts

...rkspace-manager/workspace-sync-metadata/services/workspace-sync-function-metadata.service.ts

FelixMalfait · 2024-07-16T20:15:32Z

I don't like customFunction (everything is custom), ok for codeFunction on my side, I don't love it but don't have anything better to offer

martmull · 2024-07-17T08:21:29Z

@charlesBochet @FelixMalfait what do you think about Script or CodeSnippet to rename Function? CodeFunction seems pleonastic to me

martmull · 2024-07-17T08:30:51Z

looks like we go for serverlessFunction

charlesBochet

Another masterpiece!

FelixMalfait · 2024-07-17T16:57:49Z

Bravo @martmull !!!!

martmull added 12 commits July 10, 2024 11:33

Add function workspace entity

54b94a9

Add function core modul

3deafc3

Add workspace member relation to function entity

80362c0

Create function record at function upsert

a414092

Remove useless parameter

8b6335f

Improve js compilation

f07ab8f

Add a code-executor

9402091

Clean cod

75c70f5

Move code executor

e1c25b7

Rename code executor

4c3025c

Add an abstraction layer for custom code engine

3f090b9

Add lambda driver

0612291

martmull linked an issue Jul 12, 2024 that may be closed by this pull request

[Workflows] Create a custom code executor #6181

Closed

martmull marked this pull request as ready for review July 12, 2024 09:59

greptile-apps bot reviewed Jul 12, 2024

View reviewed changes

Fix typo

ebdc85e

greptile-apps bot reviewed Jul 12, 2024

View reviewed changes

Code review return

48599a5

martmull force-pushed the 6181-workflows-create-a-custom-code-executor branch from a99bc5f to 48599a5 Compare July 12, 2024 13:24

greptile-apps bot reviewed Jul 12, 2024

View reviewed changes

packages/twenty-server/src/engine/integrations/custom-code-engine/drivers/lambda.driver.ts Outdated Show resolved Hide resolved

Rename method properly

85ed537

greptile-apps bot reviewed Jul 12, 2024

View reviewed changes

Use upsert

7cdccd0

greptile-apps bot reviewed Jul 12, 2024

View reviewed changes

FelixMalfait reviewed Jul 12, 2024

View reviewed changes

packages/twenty-website/src/content/developers/self-hosting/self-hosting-var.mdx Show resolved Hide resolved

FelixMalfait reviewed Jul 12, 2024

View reviewed changes

packages/twenty-server/src/modules/function/standard-objects/function.workspace-entity.ts Outdated Show resolved Hide resolved

FelixMalfait reviewed Jul 12, 2024

View reviewed changes

packages/twenty-server/src/modules/function/standard-objects/function.workspace-entity.ts Outdated Show resolved Hide resolved

FelixMalfait reviewed Jul 12, 2024

View reviewed changes

packages/twenty-server/src/modules/function/standard-objects/function.workspace-entity.ts Outdated Show resolved Hide resolved

martmull added 3 commits July 16, 2024 11:34

Add metadata generated types

825bad6

Fix deprecated field type

550da0f

Inject code engine module

787808b

Devessier reviewed Jul 16, 2024

View reviewed changes

martmull mentioned this pull request Jul 16, 2024

[Timebox] Custom Code Execution #6283

Closed

martmull added 3 commits July 16, 2024 15:46

Code review return: use async fs function

b39ea08

Code review return: use await streal chunks

811901c

Code review return: use pipeline to zip file

5f45533

charlesBochet reviewed Jul 16, 2024

View reviewed changes

martmull force-pushed the 6181-workflows-create-a-custom-code-executor branch 2 times, most recently from 7aa6df7 to b461901 Compare July 17, 2024 12:25

Rename functionMetadata to serverlessFunction

fca54cb

martmull force-pushed the 6181-workflows-create-a-custom-code-executor branch from b461901 to fca54cb Compare July 17, 2024 13:54

martmull added 7 commits July 17, 2024 15:57

Remove useless parameter

6db50ee

Tyde up code

d97f890

Turn BuildDirectoryManager utils into service

c383306

Remove wrongly named parameter

eddf462

Remove fileUploadService from serverless integration

62abc03

Create sourceCodeHash column in serverlessFunction

a946892

Remove useless decorator

00be140

martmull force-pushed the 6181-workflows-create-a-custom-code-executor branch from 272adc5 to 00be140 Compare July 17, 2024 15:40

martmull added 2 commits July 17, 2024 17:46

Rename column

903e9df

Remove useless stuff

2ec302a

charlesBochet approved these changes Jul 17, 2024

View reviewed changes

charlesBochet merged commit 47ddc7b into main Jul 17, 2024
12 of 13 checks passed

charlesBochet deleted the 6181-workflows-create-a-custom-code-executor branch July 17, 2024 15:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

6181 workflows create a custom code executor #6235

6181 workflows create a custom code executor #6235

martmull commented Jul 12, 2024 •

edited

Loading

github-actions bot commented Jul 12, 2024 •

edited

Loading

greptile-apps bot left a comment

greptile-apps bot Jul 12, 2024

greptile-apps bot left a comment

greptile-apps bot Jul 12, 2024

greptile-apps bot Jul 12, 2024

greptile-apps bot Jul 12, 2024

greptile-apps bot Jul 12, 2024

greptile-apps bot Jul 12, 2024

greptile-apps bot Jul 12, 2024

greptile-apps bot left a comment

greptile-apps bot left a comment

greptile-apps bot left a comment

greptile-apps bot left a comment

Devessier left a comment

charlesBochet left a comment

FelixMalfait commented Jul 16, 2024

martmull commented Jul 17, 2024

martmull commented Jul 17, 2024

charlesBochet left a comment

FelixMalfait commented Jul 17, 2024

6181 workflows create a custom code executor #6235

6181 workflows create a custom code executor #6235

Conversation

martmull commented Jul 12, 2024 • edited Loading

Testing

github-actions bot commented Jul 12, 2024 • edited Loading

greptile-apps bot left a comment

Choose a reason for hiding this comment

PR Summary

greptile-apps bot Jul 12, 2024

Choose a reason for hiding this comment

greptile-apps bot left a comment

Choose a reason for hiding this comment

PR Summary

greptile-apps bot Jul 12, 2024

Choose a reason for hiding this comment

greptile-apps bot Jul 12, 2024

Choose a reason for hiding this comment

greptile-apps bot Jul 12, 2024

Choose a reason for hiding this comment

greptile-apps bot Jul 12, 2024

Choose a reason for hiding this comment

greptile-apps bot Jul 12, 2024

Choose a reason for hiding this comment

greptile-apps bot Jul 12, 2024

Choose a reason for hiding this comment

greptile-apps bot left a comment

Choose a reason for hiding this comment

PR Summary

greptile-apps bot left a comment

Choose a reason for hiding this comment

PR Summary

greptile-apps bot left a comment

Choose a reason for hiding this comment

PR Summary

greptile-apps bot left a comment

Choose a reason for hiding this comment

PR Summary

Devessier left a comment

Choose a reason for hiding this comment

charlesBochet left a comment

Choose a reason for hiding this comment

FelixMalfait commented Jul 16, 2024

martmull commented Jul 17, 2024

martmull commented Jul 17, 2024

charlesBochet left a comment

Choose a reason for hiding this comment

FelixMalfait commented Jul 17, 2024

martmull commented Jul 12, 2024 •

edited

Loading

github-actions bot commented Jul 12, 2024 •

edited

Loading