Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make GUI and deployment scripts use compiling service #2848

Open
wants to merge 22 commits into
base: master
Choose a base branch
from

Conversation

bobbai00
Copy link
Collaborator

@bobbai00 bobbai00 commented Sep 19, 2024

This PR change the GUI, backend, and shell scripts under core/scripts to incorporate with the TexeraWorkflowCompilingService introduced in #2796

How to test this PR

To test with the user-sys enabled, set the user-sys.jwt.256-bit-secret to be a 256-bit string in application.conf.

For backend, launch 3 services

  • TexeraWebApplication
  • TexeraRunWorker
  • TexeraWorkflowCompilingService (runs at 8082)

For frontend, launch the GUI normally.

Core changes

1. GUI

WorkflowCompilingService is added

Workflow Compiling Service provides mainly 3 functionalities:

  1. autocomplete attribute property of operators (previously done by the SchemaPropagationService)
  2. receive static errors (previously done by sending EditingTimeCompilationRequest and saving in the ExecutionStateInfo)
  3. manage PhysicalPlan

/core/gui/src/app/common/type/proto protoc-generated ts files are added

These files are generated and added based on proto files in /core/amber/src/main/protobuf. The main purpose is to correctly de-serialize the PhysicalPlan given by the WorkflowCompilingService

This service will talk to the TexeraWorkflowCompilingService in a certain debounce time when:

  • operator add, delete, property changed, disabled
  • link add, delete

SchemaPropagationService is removed.

Schema propagation feature now relies on the WorkflowCompilingService's endpoint to accomplish

EditingTimeCompilationRequest and OperatorReuseCacheStatusService are removed

The main purpose of EditingTimeCompilationRequest and OperatorReuseCacheStatusService is to report the static errors Now this relies on the WorkflowCompilingService to accomplish.

2. Backend

Some bugs in WorkflowCompilationResource are fixed

These bugs are found when frontend is actually using it. This PR fixes those issues.

EditingTimeCompilationRequest and its handler are removed

Now the compilation is done by TexeraWorkflowCompilingService.

SchemaPropagationResource is removed

Now the schema propagation is done by the TexeraWorkflowCompilingService.

3. scripts

workflow-compiling-service.sh is added

This script is mainly used to launch the TexeraWorkflowCompilingService.

Related build and deploy shell scripts are updated

These scripts are updated to include the launching and terminating of TexeraWorkflowCompilingService

@bobbai00 bobbai00 changed the title Make GUI use compiling service Make GUI and deployment scripts use compiling service Sep 20, 2024
@bobbai00 bobbai00 self-assigned this Sep 20, 2024
@bobbai00 bobbai00 added the refactoring Refactor the code label Sep 20, 2024
@bobbai00 bobbai00 marked this pull request as ready for review September 20, 2024 00:53
@bobbai00 bobbai00 force-pushed the jiadong-use-compiling-service branch from 557a9aa to e7f6b46 Compare October 1, 2024 13:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
refactoring Refactor the code
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant