feat: ui build in one single http request #3020

nicoloboschi · 2024-07-29T11:51:40Z

Problem

Currently the backend that serves the frontend can't be scaled up to use multiple workers. The main limitation is that the playground builds the component with multiple requests, expecting to always hit the same backend. The current mitigation is to put a distributed cache between the backend (redis the only one supported). While this is doable, it adds a lot of complexity both from the coding and operational perspective.

Solution

This pull request introduces a new endpoint for building the flow in the playground in a single http request, removing the need for the state to be distributed across workers.
The main requirements for this endpoint are:

Provide "updates" to the client after each node has been built.
Being able to be stopped if the client decides so (e.g. infinite flow loops)

To fullfill this request, the endpoint serves the response in streaming-like fashion. (NDJSON format)
After each vertex is built a JSON containing the build details is sent to the client, that reflects the state in the UI.
If the client interrupts the connection, all the build tasks are stopped.

All build layers is now calculated server-side only, this improves testability and performance. For each layer, each vertex is built in parallel using asyncio.

The final result is that the build is faster (mainly due to removed network overhead); from user perspective sometimes is too fast that components seem to be skipped (they get the "in-progress" state for a couple of milliseconds). To improve the user experience the UI calculates the delta between the actual progress time, making each component to enter the "in-progress" state for at least 300 milliseconds. This gives to the user the progressive build feeling in all the cases.

Since the frontend now calls a different endpoint, it'd be a breaking change (e.g. you upgrade frontend first and then backend returns 404 since it doesn't know that endpoint yet). To overcome that, the old logic is still present and used as fallback. However it's recommended to run the same version for the frontend and backend.

One implementation detail is that axios doesn't support streaming responses and I had to use fetch , which doesn't pass through the common interceptors and could lead to unexpected error handling. (however they should be handled correctly)

Discarded alternatives

using SSE instead of NDJSON streaming: there's a huge limitation in modern browser for EventSource using http 1.1. To make langflow more predictable, I've discarded this solution.
Using a file-based cache in the backend. While this could have helped with multiple workers, it would't have had any effect in deployment with machine-isolated backends (e.g. Kubernetes pods)

github-actions · 2024-07-29T11:52:23Z

Pull Request Validation Report

This comment is automatically generated by Conventional PR

Whitelist Report

Whitelist	Active	Result
Pull request is a draft and should be ignored	✅	✅
Pull request is made by a whitelisted user and should be ignored	❌	❌
Pull request is submitted by a bot and should be ignored	✅	❌
Pull request is submitted by administrators and should be ignored	❌	❌

Result

Pull request matches with one (or more) enabled whitelist criteria. Pull request validation is skipped.

_{Last Modified at 29 Jul 24 11:52 UTC}

aws-amplify-sa-east-1 · 2024-07-29T11:55:00Z

This pull request is automatically being deployed by Amplify Hosting (learn more).

Access this pull request here: https://pr-3020.dmtpw4p5recq1.amplifyapp.com

lucaseduoli · 2024-07-31T14:22:56Z

src/frontend/src/stores/flowStore.ts

@@ -116,6 +120,7 @@ const useFlowStore = create<FlowStoreType>((set, get) => ({
      newFlowPool[nodeId].push(data);
    }
    get().setFlowPool(newFlowPool);
+    get().updateFlowPool(nodeId, data);


Maybe the lines 122 and 123 are doing the same thing

correct. removed now

lucaseduoli

The changes look good from the Frontend's side, but I would wait for Gabriel's review of the backend.

ogabrielluiz · 2024-08-01T12:14:46Z

src/backend/base/langflow/api/v1/chat.py

+            # and return the same structure but only with the ids
+            components_count = len(graph.vertices)
+            vertices_to_run = list(graph.vertices_to_run.union(get_top_level_vertices(graph, graph.vertices_to_run)))
+            await chat_service.set_cache(str(flow_id), graph)


I tried testing the Sequential Tasks starter project and something seems to be happening to the OpenAI component. I think it is the due to caching. I am not sure we need caching anymore since even streaming can be done in this endpoint.

I've removed the cache usage completely

Cristhianzl · 2024-08-01T16:13:23Z

hey @nicoloboschi,
I tried to run FE tests locally, It seems like is missing the session after building the flow:

Before:

After:

nicoloboschi · 2024-08-01T17:07:18Z

@Cristhianzl nice catch! This has been fixed now

Cristhianzl · 2024-08-01T20:06:40Z

@nicoloboschi

We have a feature called "Freeze Path", when I'm trying to run a flow with this activate, It throws me an error:

Could you please check for us?

nicoloboschi · 2024-08-02T12:21:36Z

@Cristhianzl I've fixed it. I've also found and fixed a related bug in another PR - #3158

Cristhianzl · 2024-08-02T13:37:18Z

@nicoloboschi the frontend e2e tests passed (good job!)

Just one point, I tried to run backend tests and I've got a few errors. could you please run "make tests" and check for us?
thank you!

github-actions bot added the enhancement New feature or request label Jul 29, 2024

github-actions bot added enhancement New feature or request and removed enhancement New feature or request labels Jul 29, 2024

nicoloboschi force-pushed the new-flow-build branch 2 times, most recently from 4ef0d1c to 205fdd7 Compare July 31, 2024 12:25

nicoloboschi marked this pull request as ready for review July 31, 2024 12:25

dosubot bot added the size:XXL This PR changes 1000+ lines, ignoring generated files. label Jul 31, 2024

github-actions bot added enhancement New feature or request and removed enhancement New feature or request labels Jul 31, 2024

nicoloboschi force-pushed the new-flow-build branch from 44e8548 to bc79582 Compare July 31, 2024 12:49

dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:XXL This PR changes 1000+ lines, ignoring generated files. labels Jul 31, 2024

NadirJ requested review from ogabrielluiz, Cristhianzl and lucaseduoli July 31, 2024 14:19

lucaseduoli reviewed Jul 31, 2024

View reviewed changes

ogabrielluiz reviewed Aug 1, 2024

View reviewed changes

nicoloboschi force-pushed the new-flow-build branch from 6cdfb8f to b78719f Compare August 1, 2024 14:58

nicoloboschi force-pushed the new-flow-build branch 2 times, most recently from 6a817d1 to 75328f1 Compare August 1, 2024 16:35

nicoloboschi force-pushed the new-flow-build branch from d777f07 to 3157947 Compare August 1, 2024 17:06

nicoloboschi added 3 commits August 2, 2024 10:40

feat: ui build in one single http request

388bf7a

fix use session_id

ddc6ab4

fix frozen

7b8f5ec

nicoloboschi force-pushed the new-flow-build branch from ec1c871 to 7b8f5ec Compare August 2, 2024 08:42

autofix-ci bot and others added 2 commits August 2, 2024 08:43

[autofix.ci] apply automated fixes

acc9ada

prettier

3bd2b9c

nicoloboschi added the lgtm This PR has been approved by a maintainer label Aug 2, 2024

nicoloboschi and others added 4 commits August 2, 2024 14:17

add tests

7bf5a0a

add tests

96ed47a

fix mypy

628fdbc

[autofix.ci] apply automated fixes

1b7d3bf

ogabrielluiz approved these changes Aug 2, 2024

View reviewed changes

ogabrielluiz merged commit f311a6d into langflow-ai:main Aug 2, 2024
51 checks passed

anovazzi1 mentioned this pull request Aug 3, 2024

fix: unnecessary split method in vertex_id that breaks playground #3182

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: ui build in one single http request #3020

feat: ui build in one single http request #3020

nicoloboschi commented Jul 29, 2024 •

edited

Loading

github-actions bot commented Jul 29, 2024

aws-amplify-sa-east-1 bot commented Jul 29, 2024

lucaseduoli Jul 31, 2024

nicoloboschi Aug 1, 2024

lucaseduoli left a comment

ogabrielluiz Aug 1, 2024

nicoloboschi Aug 1, 2024

Cristhianzl commented Aug 1, 2024

nicoloboschi commented Aug 1, 2024

Cristhianzl commented Aug 1, 2024 •

edited

Loading

nicoloboschi commented Aug 2, 2024

Cristhianzl commented Aug 2, 2024

feat: ui build in one single http request #3020

feat: ui build in one single http request #3020

Conversation

nicoloboschi commented Jul 29, 2024 • edited Loading

Problem

Solution

Discarded alternatives

github-actions bot commented Jul 29, 2024

Pull Request Validation Report

Whitelist Report

aws-amplify-sa-east-1 bot commented Jul 29, 2024

lucaseduoli Jul 31, 2024

Choose a reason for hiding this comment

nicoloboschi Aug 1, 2024

Choose a reason for hiding this comment

lucaseduoli left a comment

Choose a reason for hiding this comment

ogabrielluiz Aug 1, 2024

Choose a reason for hiding this comment

nicoloboschi Aug 1, 2024

Choose a reason for hiding this comment

Cristhianzl commented Aug 1, 2024

nicoloboschi commented Aug 1, 2024

Cristhianzl commented Aug 1, 2024 • edited Loading

nicoloboschi commented Aug 2, 2024

Cristhianzl commented Aug 2, 2024

nicoloboschi commented Jul 29, 2024 •

edited

Loading

Cristhianzl commented Aug 1, 2024 •

edited

Loading