Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Implement graph combination method #3224

Merged
merged 134 commits into from
Aug 7, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
134 commits
Select commit Hold shift + click to select a range
3927bf5
feat(utils.py): add escape_json_dump function to escape JSON strings …
ogabrielluiz Aug 5, 2024
6c40f7b
refactor(Output): streamline add_types method to prevent duplicate en…
ogabrielluiz Aug 5, 2024
a744489
feat(data.py): add classmethod decorator to validate_data for enhance…
ogabrielluiz Aug 5, 2024
8254275
feat(setup.py): implement retry logic for loading starter projects to…
ogabrielluiz Aug 5, 2024
fd770f2
fix(input_mixin.py): improve model_config formatting and update field…
ogabrielluiz Aug 5, 2024
22fd048
feat(types.py): refactor vertex constructors to use NodeData and add …
ogabrielluiz Aug 5, 2024
e75d06e
feat(schema.py): add NodeData and Position TypedDicts for improved ty…
ogabrielluiz Aug 5, 2024
7e65921
feat(base.py): update Vertex to use NodeData type and add to_data met…
ogabrielluiz Aug 5, 2024
862ae9c
refactor(schema.py): update TargetHandle and SourceHandle models to i…
ogabrielluiz Aug 5, 2024
ac5e541
Add TypedDict classes for graph schema serialization in `schema.py`
ogabrielluiz Aug 5, 2024
3006402
Refactor `Edge` class to improve handle validation and data handling
ogabrielluiz Aug 5, 2024
cf69f83
Refactor `Edge` class to improve handle validation and data handling
ogabrielluiz Aug 5, 2024
6ea2715
Refactor: Standardize attribute naming and add `to_data` method in Ed…
ogabrielluiz Aug 5, 2024
c6f31f8
Refactor: Update Edge class to consistently use snake_case for attrib…
ogabrielluiz Aug 5, 2024
8983260
Refactor: Change node argument type in add_node and _create_vertex me…
ogabrielluiz Aug 5, 2024
5f24db8
Refactor: Implement JSON serialization for graph data with `dumps` an…
ogabrielluiz Aug 5, 2024
4239ab6
Refactor: Add pytest fixtures for ingestion and RAG graphs, enhance t…
ogabrielluiz Aug 5, 2024
46fec80
Refactor: Add pytest fixtures for memory_chatbot_graph tests and impr…
ogabrielluiz Aug 5, 2024
78ff654
Refactor: Remove unused methods in ComponentVertex class to streamlin…
ogabrielluiz Aug 5, 2024
cc3fc12
Refactor: Remove unnecessary line in ComponentVertex class to enhance…
ogabrielluiz Aug 5, 2024
8d0140f
refactor: Add utility functions for getting handle IDs in CustomNodes
ogabrielluiz Aug 5, 2024
5f763d5
refactor: Add type for escaped handle IDs in edges to improve type sa…
ogabrielluiz Aug 5, 2024
26961a0
feat: Add function to escape handle IDs in edges, enhancing edge mana…
ogabrielluiz Aug 5, 2024
d5d0ea1
feat: Add function to check edges without escaped handle IDs, improvi…
ogabrielluiz Aug 5, 2024
2b264da
feat: Enhance edge processing in reactflowUtils to handle edges witho…
ogabrielluiz Aug 5, 2024
0590d26
feat: Add layoutUtils module for handling node layout using elkjs
ogabrielluiz Aug 5, 2024
711f258
feat: update processDataFromFlow to add layout to nodes if needed
ogabrielluiz Aug 5, 2024
d21843e
feat: Update flowsManagerStore to parse flow data from file before pr…
ogabrielluiz Aug 5, 2024
c9a6c34
Refactor import paths to use 'initialize' module in 'base.py'
ogabrielluiz Aug 5, 2024
db0a021
feat: Add method to set class source code and integrate it with front…
ogabrielluiz Aug 5, 2024
d088421
refactor: Update sourceHandle dataType to use custom component class …
ogabrielluiz Aug 5, 2024
685cc8c
fix: Raise error for unknown vertex types instead of returning defaul…
ogabrielluiz Aug 5, 2024
e216287
refactor: Remove redundant call to _import_vertex_types() in VertexTy…
ogabrielluiz Aug 5, 2024
afc9a3f
refactor: Simplify add_code_field by removing unnecessary field_confi…
ogabrielluiz Aug 5, 2024
bb88027
feat: Add elkjs dependency to package.json and package-lock.json for …
ogabrielluiz Aug 5, 2024
80a2b05
refactor: Update fields type in Template class to use InputTypes for …
ogabrielluiz Aug 5, 2024
a7820f3
refactor: Update import path for DefaultPromptField to improve code o…
ogabrielluiz Aug 5, 2024
ac27030
refactor: Reorganize imports in __init__.py for better structure and …
ogabrielluiz Aug 5, 2024
dff6f5e
refactor: Clean up imports in types.py for better organization and co…
ogabrielluiz Aug 5, 2024
cf5d719
refactor: Change vertex type annotations to strings for better compat…
ogabrielluiz Aug 5, 2024
5128701
refactor: Update component instantiation to include _code parameter a…
ogabrielluiz Aug 5, 2024
fe33a50
refactor: Remove unused CustomComponent import from __init__.py for c…
ogabrielluiz Aug 5, 2024
9aadcc3
refactor: Modify custom_component instantiation to include _code argu…
ogabrielluiz Aug 5, 2024
220a1c1
refactor: Update CustomComponent import in __init__.py for improved m…
ogabrielluiz Aug 5, 2024
beaf5e2
refactor: Update launch.json to include correct path for backend sour…
ogabrielluiz Aug 5, 2024
13e1cfb
refactor: Update dependencies in poetry.lock to latest versions and r…
ogabrielluiz Aug 5, 2024
e6f06b6
Merge branch 'main' into feat/elkjslayout
ogabrielluiz Aug 6, 2024
c689330
[autofix.ci] apply automated fixes
autofix-ci[bot] Aug 6, 2024
121c9c0
refactor: Remove unnecessary line in test_memory_chatbot.py
ogabrielluiz Aug 6, 2024
6e09dc3
refactor: Update dataType assignment in Component class to use compon…
ogabrielluiz Aug 6, 2024
a05a7bc
refactor: Correct flow_id reference in MemoryComponent to improve cla…
ogabrielluiz Aug 6, 2024
8e7c9ad
refactor: Update import path for DefaultPromptField to improve code o…
ogabrielluiz Aug 6, 2024
3c76f7b
refactor: Add loading module to __init__.py for improved organization…
ogabrielluiz Aug 6, 2024
73613e1
refactor: Clean up imports in base.py and enforce edge validation in …
ogabrielluiz Aug 6, 2024
ab8cdec
refactor: Remove edge component additions in test_base.py to streamli…
ogabrielluiz Aug 6, 2024
f21ff76
refactor: Mark @clack/prompts is-unicode-supported as extraneous in p…
ogabrielluiz Aug 6, 2024
13a5383
refactor: Update dataType assignment in Component class to use compon…
ogabrielluiz Aug 6, 2024
de11978
refactor: Fix edge existence check in Graph class to use correct vari…
ogabrielluiz Aug 6, 2024
7a0b10e
refactor: Add test for graph with edge and improve graph preparation …
ogabrielluiz Aug 6, 2024
c9e46ac
refactor: Set default node type to "genericNode" in getLayoutedNodes …
ogabrielluiz Aug 6, 2024
52d10cd
create consts for node widht and height
anovazzi1 Aug 6, 2024
3d6765e
refactor: Catch and log errors when processing flow data in flowsMana…
anovazzi1 Aug 6, 2024
a19fb9f
[autofix.ci] apply automated fixes
autofix-ci[bot] Aug 6, 2024
216cded
Merge branch 'main' into feat/elkjslayout
anovazzi1 Aug 6, 2024
e628904
fix: Validate custom components for source and target vertices in Gra…
ogabrielluiz Aug 6, 2024
b2c1b15
test: Add fixture for client and raise TypeError for invalid class pa…
ogabrielluiz Aug 6, 2024
4fce970
test: Add unit test for listing flows as Flow objects in custom compo…
ogabrielluiz Aug 6, 2024
8653833
test: Update assertions for memory chatbot component types in unit tests
ogabrielluiz Aug 6, 2024
b381e01
test: Refactor assertions to use updated component names in vector st…
ogabrielluiz Aug 6, 2024
1bc4403
fix: Change error handling to return default Vertex for unknown node …
ogabrielluiz Aug 6, 2024
c95ad5a
[autofix.ci] apply automated fixes
autofix-ci[bot] Aug 6, 2024
165c0d5
test: Add pytest fixture for CustomComponent in unit tests to enhance…
ogabrielluiz Aug 6, 2024
084acf4
chore: Update component names in vector store RAG unit tests
ogabrielluiz Aug 6, 2024
9429478
test: Refactor imports and make flow name generation unique in databa…
ogabrielluiz Aug 6, 2024
84836ae
chore: Add new attributes to Edge class for improved state management…
ogabrielluiz Aug 7, 2024
5a20bf6
chore: Implement addition methods for Graph class to combine vertices…
ogabrielluiz Aug 7, 2024
f9244c6
chore: Extend serialization in Graph class to include additional inte…
ogabrielluiz Aug 7, 2024
26d189a
chore: Call initialize method in prepare for proper setup before vali…
ogabrielluiz Aug 7, 2024
9ea2fb8
chore: Add test to validate graph combination in vector store RAG, en…
ogabrielluiz Aug 7, 2024
05ca09e
refactor: Add utility functions for getting handle IDs in CustomNodes
ogabrielluiz Aug 5, 2024
373e280
refactor: Add type for escaped handle IDs in edges to improve type sa…
ogabrielluiz Aug 5, 2024
4271692
feat: Add function to escape handle IDs in edges, enhancing edge mana…
ogabrielluiz Aug 5, 2024
a179422
feat: Add function to check edges without escaped handle IDs, improvi…
ogabrielluiz Aug 5, 2024
f2f5c4e
feat: Enhance edge processing in reactflowUtils to handle edges witho…
ogabrielluiz Aug 5, 2024
b1504db
feat: Add layoutUtils module for handling node layout using elkjs
ogabrielluiz Aug 5, 2024
cd985c9
feat: update processDataFromFlow to add layout to nodes if needed
ogabrielluiz Aug 5, 2024
1c507d8
Refactor import paths to use 'initialize' module in 'base.py'
ogabrielluiz Aug 5, 2024
613a84f
feat: Add method to set class source code and integrate it with front…
ogabrielluiz Aug 5, 2024
5d24d7e
refactor: Update sourceHandle dataType to use custom component class …
ogabrielluiz Aug 5, 2024
59919cc
fix: Raise error for unknown vertex types instead of returning defaul…
ogabrielluiz Aug 5, 2024
06c8626
refactor: Remove redundant call to _import_vertex_types() in VertexTy…
ogabrielluiz Aug 5, 2024
280527a
refactor: Simplify add_code_field by removing unnecessary field_confi…
ogabrielluiz Aug 5, 2024
e46ebbc
feat: Add elkjs dependency to package.json and package-lock.json for …
ogabrielluiz Aug 5, 2024
269f46b
refactor: Update fields type in Template class to use InputTypes for …
ogabrielluiz Aug 5, 2024
3691fbd
refactor: Reorganize imports in __init__.py for better structure and …
ogabrielluiz Aug 5, 2024
1e3be3f
refactor: Clean up imports in types.py for better organization and co…
ogabrielluiz Aug 5, 2024
bb6b331
refactor: Change vertex type annotations to strings for better compat…
ogabrielluiz Aug 5, 2024
1a287c1
refactor: Update component instantiation to include _code parameter a…
ogabrielluiz Aug 5, 2024
b3bcfa3
refactor: Remove unused CustomComponent import from __init__.py for c…
ogabrielluiz Aug 5, 2024
f465f27
refactor: Modify custom_component instantiation to include _code argu…
ogabrielluiz Aug 5, 2024
6d185c5
refactor: Update CustomComponent import in __init__.py for improved m…
ogabrielluiz Aug 5, 2024
6897ec8
refactor: Update launch.json to include correct path for backend sour…
ogabrielluiz Aug 5, 2024
5d1ef23
refactor: Update dependencies in poetry.lock to latest versions and r…
ogabrielluiz Aug 5, 2024
312945c
refactor: Update dataType assignment in Component class to use compon…
ogabrielluiz Aug 6, 2024
47a03e4
refactor: Correct flow_id reference in MemoryComponent to improve cla…
ogabrielluiz Aug 6, 2024
5e41abe
refactor: Update import path for DefaultPromptField to improve code o…
ogabrielluiz Aug 6, 2024
6178058
refactor: Add loading module to __init__.py for improved organization…
ogabrielluiz Aug 6, 2024
36cabbf
refactor: Clean up imports in base.py and enforce edge validation in …
ogabrielluiz Aug 6, 2024
0586682
refactor: Remove edge component additions in test_base.py to streamli…
ogabrielluiz Aug 6, 2024
25b080e
refactor: Mark @clack/prompts is-unicode-supported as extraneous in p…
ogabrielluiz Aug 6, 2024
07de10c
refactor: Update dataType assignment in Component class to use compon…
ogabrielluiz Aug 6, 2024
25046ab
refactor: Fix edge existence check in Graph class to use correct vari…
ogabrielluiz Aug 6, 2024
bdfb3d8
refactor: Add test for graph with edge and improve graph preparation …
ogabrielluiz Aug 6, 2024
4457775
refactor: Set default node type to "genericNode" in getLayoutedNodes …
ogabrielluiz Aug 6, 2024
02707e9
create consts for node widht and height
anovazzi1 Aug 6, 2024
906e687
refactor: Catch and log errors when processing flow data in flowsMana…
anovazzi1 Aug 6, 2024
5b5dc24
[autofix.ci] apply automated fixes
autofix-ci[bot] Aug 6, 2024
5df5af9
fix: Validate custom components for source and target vertices in Gra…
ogabrielluiz Aug 6, 2024
8099f92
test: Add fixture for client and raise TypeError for invalid class pa…
ogabrielluiz Aug 6, 2024
684af41
test: Add unit test for listing flows as Flow objects in custom compo…
ogabrielluiz Aug 6, 2024
b17b29f
test: Update assertions for memory chatbot component types in unit tests
ogabrielluiz Aug 6, 2024
a0b6916
test: Refactor assertions to use updated component names in vector st…
ogabrielluiz Aug 6, 2024
8f0789a
fix: Change error handling to return default Vertex for unknown node …
ogabrielluiz Aug 6, 2024
617f15b
[autofix.ci] apply automated fixes
autofix-ci[bot] Aug 6, 2024
a2bf7e4
test: Add pytest fixture for CustomComponent in unit tests to enhance…
ogabrielluiz Aug 6, 2024
969a857
chore: Update component names in vector store RAG unit tests
ogabrielluiz Aug 6, 2024
9ed053e
test: Refactor imports and make flow name generation unique in databa…
ogabrielluiz Aug 6, 2024
473de91
chore: Remove unused upload and flow management functions from flowsM…
ogabrielluiz Aug 7, 2024
fa8c44e
chore: Await processDataFromFlow in useAddFlow hook
ogabrielluiz Aug 7, 2024
496e7a3
chore: Correct NODE_HEIGHT calculation to use NODE_WIDTH constant for…
ogabrielluiz Aug 7, 2024
7d12ea0
chore: Remove extraneous flag for is-unicode-supported in package-loc…
ogabrielluiz Aug 7, 2024
5315eac
Merge branch 'feat/elkjslayout' into feat/dunderadd
ogabrielluiz Aug 7, 2024
9356c79
Merge branch 'main' into feat/dunderadd
ogabrielluiz Aug 7, 2024
4f43e15
Merge branch 'main' into feat/dunderadd
ogabrielluiz Aug 7, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions src/backend/base/langflow/graph/edge/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,13 @@ def __setstate__(self, state):
self.target_param = state["target_param"]
self.source_handle = state.get("source_handle")
self.target_handle = state.get("target_handle")
self._source_handle = state.get("_source_handle")
self._target_handle = state.get("_target_handle")
self._data = state.get("_data")
self.valid_handles = state.get("valid_handles")
self.source_types = state.get("source_types")
self.target_reqs = state.get("target_reqs")
self.matched_type = state.get("matched_type")

def validate_edge(self, source, target) -> None:
# If the self.source_handle has base_classes, then we are using the legacy
Expand Down
33 changes: 33 additions & 0 deletions src/backend/base/langflow/graph/graph/base.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import asyncio
import copy
import json
import uuid
from collections import defaultdict, deque
Expand Down Expand Up @@ -100,6 +101,29 @@ def __init__(
if (start is not None and end is None) or (start is None and end is not None):
raise ValueError("You must provide both input and output components")

def __add__(self, other):
if not isinstance(other, Graph):
raise TypeError("Can only add Graph objects")
# Add the vertices and edges from the other graph to this graph
new_instance = copy.deepcopy(self)
for vertex in other.vertices:
# This updates the edges as well
new_instance.add_vertex(vertex)
new_instance.build_graph_maps(new_instance.edges)
new_instance.define_vertices_lists()
return new_instance

def __iadd__(self, other):
if not isinstance(other, Graph):
raise TypeError("Can only add Graph objects")
# Add the vertices and edges from the other graph to this graph
for vertex in other.vertices:
# This updates the edges as well
self.add_vertex(vertex)
self.build_graph_maps(self.edges)
self.define_vertices_lists()
return self

def dumps(
self,
name: Optional[str] = None,
Expand Down Expand Up @@ -779,6 +803,14 @@ def __getstate__(self):
"vertices_to_run": self.vertices_to_run,
"stop_vertex": self.stop_vertex,
"vertex_map": self.vertex_map,
"_run_queue": self._run_queue,
"_first_layer": self._first_layer,
"_vertices": self._vertices,
"_edges": self._edges,
"_is_input_vertices": self._is_input_vertices,
"_is_output_vertices": self._is_output_vertices,
"_has_session_id_vertices": self._has_session_id_vertices,
"_sorted_vertices_layers": self._sorted_vertices_layers,
}

def __setstate__(self, state):
Expand Down Expand Up @@ -1450,6 +1482,7 @@ def _create_vertex(self, frontend_data: NodeData):
return vertex_instance

def prepare(self, stop_component_id: Optional[str] = None, start_component_id: Optional[str] = None):
self.initialize()
if stop_component_id and start_component_id:
raise ValueError("You can only provide one of stop_component_id or start_component_id")
self.validate_stream()
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import copy
from textwrap import dedent

import pytest
Expand Down Expand Up @@ -211,6 +212,73 @@ def test_vector_store_rag_dump_components_and_edges(ingestion_graph, rag_graph):
assert (source, target) in expected_rag_edges, f"Edge {source} -> {target} not found"


def test_vector_store_rag_add(ingestion_graph, rag_graph):
ingestion_graph_copy = copy.deepcopy(ingestion_graph)
rag_graph_copy = copy.deepcopy(rag_graph)
ingestion_graph_copy += rag_graph_copy

assert (
len(ingestion_graph_copy.vertices) == len(ingestion_graph.vertices) + len(rag_graph.vertices)
), f"Vertices mismatch: {len(ingestion_graph_copy.vertices)} != {len(ingestion_graph.vertices)} + {len(rag_graph.vertices)}"
assert len(ingestion_graph_copy.edges) == len(ingestion_graph.edges) + len(
rag_graph.edges
), f"Edges mismatch: {len(ingestion_graph_copy.edges)} != {len(ingestion_graph.edges)} + {len(rag_graph.edges)}"

combined_graph_dump = ingestion_graph_copy.dump(
name="Combined Graph", description="Graph for data ingestion and RAG", endpoint_name="combined"
)

combined_data = combined_graph_dump["data"]
combined_nodes = combined_data["nodes"]
combined_edges = combined_data["edges"]

# Sort nodes by id to check components
combined_nodes = sorted(combined_nodes, key=lambda x: x["id"])

# Expected components in the combined graph (both ingestion and RAG nodes)
expected_nodes = sorted(
[
{"id": "file-123", "type": "File"},
{"id": "openai-embeddings-123", "type": "OpenAIEmbeddings"},
{"id": "text-splitter-123", "type": "SplitText"},
{"id": "vector-store-123", "type": "AstraDB"},
{"id": "chatinput-123", "type": "ChatInput"},
{"id": "chatoutput-123", "type": "ChatOutput"},
{"id": "openai-123", "type": "OpenAIModel"},
{"id": "openai-embeddings-124", "type": "OpenAIEmbeddings"},
{"id": "parse-data-123", "type": "ParseData"},
{"id": "prompt-123", "type": "Prompt"},
{"id": "rag-vector-store-123", "type": "AstraDB"},
],
key=lambda x: x["id"],
)

for expected_node, combined_node in zip(expected_nodes, combined_nodes):
assert combined_node["data"]["type"] == expected_node["type"]
assert combined_node["id"] == expected_node["id"]

# Expected edges in the combined graph (both ingestion and RAG edges)
expected_combined_edges = [
("file-123", "text-splitter-123"),
("text-splitter-123", "vector-store-123"),
("openai-embeddings-123", "vector-store-123"),
("chatinput-123", "rag-vector-store-123"),
("openai-embeddings-124", "rag-vector-store-123"),
("chatinput-123", "prompt-123"),
("rag-vector-store-123", "parse-data-123"),
("parse-data-123", "prompt-123"),
("prompt-123", "openai-123"),
("openai-123", "chatoutput-123"),
]

assert len(combined_edges) == len(expected_combined_edges), combined_edges

for edge in combined_edges:
source = edge["source"]
target = edge["target"]
assert (source, target) in expected_combined_edges, f"Edge {source} -> {target} not found"


def test_vector_store_rag_dump(ingestion_graph, rag_graph):
# Test ingestion graph dump
ingestion_graph_dump = ingestion_graph.dump(
Expand Down
1 change: 1 addition & 0 deletions src/frontend/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading