feat: truncate parsed uploads to prevent database and frontend blocking caused by excessively large files #3914
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…MB to 100MB to allow larger file uploads
…t message to display the correct file size limit of 100 bytes instead of 10 bytes
…n VertexBuildResponse class 📝 (schemas.py): Add a new truncate_text helper function to safely truncate text in nested dictionaries 📝 (model.py): Add a new field_serializer method to serialize outputs in TransactionBase class 📝 (model.py): Add a new truncate_text helper function to safely truncate text in nested dictionaries 📝 (model.py): Add a new field_serializer method to serialize data and artifacts in VertexBuildBase class 📝 (model.py): Add a new truncate_text helper function to safely truncate text in nested dictionaries
… instead of 99999 🐛 (model.py): fix truncation length of text fields to 10 characters instead of 99999 🐛 (model.py): fix truncation length of text fields to 10 characters instead of 99999 🐛 (index.tsx): truncate resultMessage to 99999 characters and add message if text is too long
…transformations for performance optimization
🐛 (model.py): Fix typo in the path for 'base_retriever' data field 🐛 (model.py): Fix typo in the path for 'base_retriever' data field 🐛 (model.py): Fix typo in the path for 'base_retriever' data field 🐛 (index.tsx): Fix logic to correctly handle resultMessageMemoized when it is an object
…s for better clarity and consistency 📝 (model.py): update serialize_outputs and serialize_artifacts functions to use truncate_long_strings for string truncation 📝 (model.py): introduce MAX_TEXT_LENGTH constant for defining the maximum length of text to truncate in the models
… class to use a new helper function truncate_long_strings for better code readability and maintainability
dosubot
bot
added
size:L
github-actions
bot
added
enhancement
…te module to improve code organization and reusability 🔧 (model.py): Import the `truncate_long_strings` function from the correct module to fix the reference error 🔧 (model.py): Import the `truncate_long_strings` function from the correct module to fix the reference error
dosubot
bot
added
size:M
…te long strings in dictionaries and lists to prevent exceeding the maximum text length.
dosubot
bot
added
size:L
…efining maximum text length allowed in the application
…o match new location in util module
ogabrielluiz
requested changes
Sep 25, 2024
...end/src/CustomNodes/GenericNode/components/outputModal/components/switchOutputView/index.tsx
Outdated
Show resolved
Hide resolved
...end/src/CustomNodes/GenericNode/components/outputModal/components/switchOutputView/index.tsx
Outdated
Show resolved
Hide resolved
…function truncate_long_strings to ensure correct behavior when truncating long strings in various data structures 🐛 (switchOutputView/index.tsx): Fix truncation logic to correctly truncate long strings by adding ellipsis at the end instead of displaying additional text about truncation.
…truncate_long_strings function 📝 (test_truncate_long_strings_on_objects.py): Add additional tests for handling negative, zero, and small max_length values in truncate_long_strings function
♻️ (AssemblyAIFormatTranscript.py): Change typing annotations to lowercase for consistency ♻️ (AssemblyAIListTranscripts.py): Change typing annotations to lowercase for consistency ♻️ (LangChainHubPrompt.py): Remove duplicate import statement to improve code readability ♻️ (model.py): Remove unnecessary blank lines to enhance code readability ♻️ (model.py): Remove unnecessary blank lines to enhance code readability ♻️ (constants.py): Update MAX_TEXT_LENGTH constant to have consistent spacing ♻️ (util.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string
… size for uploads in MB
…ettings function to allow configuring maximum file size for uploads
… size upload from utility store to improve code modularity and reusability 🐛 (inputFileComponent/index.tsx): fix error handling logic to display error message when uploading a file fails
…ger used and update MAX_TEXT_LENGTH constant to a higher value
…value from config response
…ileSizeUpload function to handle maximum file size upload in bytes
…Upload method to handle maximum file size upload functionality in the UtilityStoreType
…ove code readability and consistency by removing unnecessary whitespace and aligning assignment operators.
…code readability and maintain consistency
github-actions
bot
added
enhancement
…e is larger than the maximum allowed size in MB
…pload state 🐛 (chatView/index.tsx): add validation to check if file size exceeds maxFileSizeUpload limit before uploading 🐛 (chatView/index.tsx): handle error response when uploading file and display error message
…yStore to check file size before uploading to prevent exceeding the maximum allowed file size 📝 (chatInput/index.tsx): add error handling for file size exceeding the maximum allowed size and display an error alert with the appropriate message
…ng file size validation errors 🔧 (FileInput/index.tsx): Update file input component to use global alert store and utility store for error handling and max file size configuration
…load a file larger than the specified limit of 0.001MB. This test includes mocking API response, checking for required environment variables, interacting with the page elements, and validating error message display.
…found issue in test case
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces a feature to limit the size of parsed files during uploads. The primary goal is to prevent the database and the front end from being blocked by excessively large files, which can cause performance issues and degrade user experience.
Implemented a recursive function to analyze schemas for fields containing excessive character lengths and truncate them accordingly.
Introduced a frontend and backend safeguard to truncate excessively long values, preventing potential crashes in the user interface.
Added a new .env variable LANGFLOW_MAX_FILE_SIZE_UPLOAD with a default value of 100 MB. This allows users to customize the maximum file upload size according to their needs.