feat: Add GitLoader Component with advanced filtering options #2850

danielgines · 2024-07-20T00:59:17Z

This commit enhances the GitLoaderComponent, providing users with the ability to load files from a Git repository with advanced filtering options, including dynamic input handling and content-based filtering.

GitLoader Component:

Implementation: Utilizes the langchain_community.document_loaders.git.GitLoader module to load files from a Git repository.
Dynamic Inputs: Changed inputs from StrInput to MessageTextInput to support dynamic usage with agents.
Advanced Filtering:
- File Filter: Supports glob patterns for including or excluding specific files based on their extensions or other criteria.
- Content Filter: Added content_filter field to enable content-based filtering using regex patterns.
- Binary File Exclusion: Implemented a filter to remove binary files from queries, ensuring relevance to the agent's purpose.

Examples of file_filter usage:

Include only .py files: *.py
Exclude .py files: !*.py

Examples of content_filter usage:

Search for 'pathlib' in any file: pathlib

Features:

Support for loading documents from Git repositories.
Dynamic input handling for better flexibility with agents.
Advanced file filtering options using glob patterns.
Content-based filtering using regex.
Automatic exclusion of binary files to focus on relevant text files.

This commit introduces the GitLoaderComponent, enabling users to load files from a Git repository with advanced filtering options. GitLoader Component: - Implementation of the GitLoaderComponent to load files from a Git repository using the `langchain_community.document_loaders.git.GitLoader` module. - Advanced filtering option using `file_filter` to include or exclude specific files based on their extensions or other criteria. Examples of `file_filter` usage: - Include only .py files: `lambda file_path: file_path.endswith('.py')` - Exclude .py files: `lambda file_path: not file_path.endswith('.py')` This component ensures a flexible and customizable approach for loading documents from Git repositories, enhancing the user experience with advanced filtering capabilities. Features: - Support for loading documents from Git repositories. - Advanced file filtering options to include or exclude specific files.

github-actions · 2024-07-20T01:00:01Z

Pull Request Validation Report

This comment is automatically generated by Conventional PR

Whitelist Report

Whitelist	Active	Result
Pull request is a draft and should be ignored	✅	❌
Pull request is made by a whitelisted user and should be ignored	❌	❌
Pull request is submitted by a bot and should be ignored	✅	❌
Pull request is submitted by administrators and should be ignored	❌	❌

Result

Pull request does not satisfy any enabled whitelist criteria. Pull request will be validated.

Validation Report

Validation	Active	Result
All commits in this pull request has valid messages	❌	✅
Pull request does not introduce too many changes	❌	✅
Pull request has a valid title	✅	✅
Pull request has mentioned issues	❌	✅
Pull request has valid branch name	❌	✅
Pull request should have a non-empty body	✅	✅

Result

Pull request satisfies all enabled pull request rules.

_{Last Modified at 20 Jul 24 01:00 UTC}

aws-amplify-sa-east-1 · 2024-07-20T01:02:39Z

This pull request is automatically being deployed by Amplify Hosting (learn more).

Access this pull request here: https://pr-2850.dmtpw4p5recq1.amplifyapp.com

This commit introduces the GitLoaderComponent, enabling users to load files from a Git repository with advanced filtering options. GitLoader Component: - Implementation of the GitLoaderComponent to load files from a Git repository using the `langchain_community.document_loaders.git.GitLoader` module. - Advanced filtering option using `file_filter` to include or exclude specific files based on their extensions or other criteria. Examples of `file_filter` usage: - Include only .py files: `lambda file_path: file_path.endswith('.py')` - Exclude .py files: `lambda file_path: not file_path.endswith('.py')` This component ensures a flexible and customizable approach for loading documents from Git repositories, enhancing the user experience with advanced filtering capabilities. Features: - Support for loading documents from Git repositories. - Advanced file filtering options to include or exclude specific files.

…derComponent This commit fixes the issue where the GitLoaderComponent would fail if the file_filter input was not evaluated correctly. Changes include: - Added a check to ensure that file_filter is a valid string before calling eval. - Ensured that the evaluated file_filter is callable, otherwise it defaults to None.

src/backend/base/langflow/components/documentloaders/GitLoader.py

- Changed inputs from `StrInput` to `MessageTextInput` to enable dynamic use with agents. - Added `content_filter` field to allow additional content filtering using regex. - Updated `file_filter` to support glob format, simplifying usage for users. - Implemented binary file removal filter to exclude binary files from queries, aligning with the agent's purpose.

danielgines

@ogabrielluiz

I applied the suggestion you recommended and took the opportunity to make a small improvement to the agent based on my use case. I look forward to your feedback.

ogabrielluiz

That's a lot nicer.

Thanks for all the help, @danielgines

…ow-ai#2850) * feat: Add GitLoader Component with advanced filtering options This commit introduces the GitLoaderComponent, enabling users to load files from a Git repository with advanced filtering options. GitLoader Component: - Implementation of the GitLoaderComponent to load files from a Git repository using the `langchain_community.document_loaders.git.GitLoader` module. - Advanced filtering option using `file_filter` to include or exclude specific files based on their extensions or other criteria. Examples of `file_filter` usage: - Include only .py files: `lambda file_path: file_path.endswith('.py')` - Exclude .py files: `lambda file_path: not file_path.endswith('.py')` This component ensures a flexible and customizable approach for loading documents from Git repositories, enhancing the user experience with advanced filtering capabilities. Features: - Support for loading documents from Git repositories. - Advanced file filtering options to include or exclude specific files. * feat: Add GitLoader Component with advanced filtering options This commit introduces the GitLoaderComponent, enabling users to load files from a Git repository with advanced filtering options. GitLoader Component: - Implementation of the GitLoaderComponent to load files from a Git repository using the `langchain_community.document_loaders.git.GitLoader` module. - Advanced filtering option using `file_filter` to include or exclude specific files based on their extensions or other criteria. Examples of `file_filter` usage: - Include only .py files: `lambda file_path: file_path.endswith('.py')` - Exclude .py files: `lambda file_path: not file_path.endswith('.py')` This component ensures a flexible and customizable approach for loading documents from Git repositories, enhancing the user experience with advanced filtering capabilities. Features: - Support for loading documents from Git repositories. - Advanced file filtering options to include or exclude specific files. * fix: Ensure proper evaluation and validation of file_filter in GitLoaderComponent This commit fixes the issue where the GitLoaderComponent would fail if the file_filter input was not evaluated correctly. Changes include: - Added a check to ensure that file_filter is a valid string before calling eval. - Ensured that the evaluated file_filter is callable, otherwise it defaults to None. * [autofix.ci] apply automated fixes * feat: Enhance GitLoaderComponent with dynamic inputs, content filtering - Changed inputs from `StrInput` to `MessageTextInput` to enable dynamic use with agents. - Added `content_filter` field to allow additional content filtering using regex. - Updated `file_filter` to support glob format, simplifying usage for users. - Implemented binary file removal filter to exclude binary files from queries, aligning with the agent's purpose. * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> (cherry picked from commit d108ca1)

dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. enhancement New feature or request labels Jul 20, 2024

github-actions bot added enhancement New feature or request and removed enhancement New feature or request labels Jul 20, 2024

danielgines added 5 commits July 19, 2024 22:05

Merge branch 'langflow-ai:main' into feat/documentloaders/git

f349d3d

Merge branch 'langflow-ai:main' into feat/documentloaders/git

d5a784f

Merge branch 'main' into feat/documentloaders/git

761b07d

ogabrielluiz requested changes Jul 25, 2024

View reviewed changes

src/backend/base/langflow/components/documentloaders/GitLoader.py Outdated Show resolved Hide resolved

danielgines and others added 8 commits July 25, 2024 11:05

Merge branch 'langflow-ai:main' into feat/documentloaders/git

9d9ac4e

[autofix.ci] apply automated fixes

7a7037b

Merge branch 'langflow-ai:main' into feat/documentloaders/git

3986377

Merge branch 'main' into feat/documentloaders/git

3d506f2

Merge remote-tracking branch 'origin/main' into feat/documentloaders/git

d9d4d5f

[autofix.ci] apply automated fixes

0744f7c

[autofix.ci] apply automated fixes (attempt 2/3)

ce85091

danielgines commented Jul 29, 2024

View reviewed changes

danielgines requested a review from ogabrielluiz July 29, 2024 18:51

github-actions bot added enhancement New feature or request and removed enhancement New feature or request labels Jul 29, 2024

ogabrielluiz approved these changes Jul 30, 2024

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label Jul 30, 2024

ogabrielluiz enabled auto-merge (squash) July 30, 2024 01:50

ogabrielluiz merged commit d108ca1 into langflow-ai:main Jul 30, 2024
59 of 60 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add GitLoader Component with advanced filtering options #2850

feat: Add GitLoader Component with advanced filtering options #2850

danielgines commented Jul 20, 2024 •

edited

Loading

github-actions bot commented Jul 20, 2024

aws-amplify-sa-east-1 bot commented Jul 20, 2024

danielgines left a comment

ogabrielluiz left a comment

feat: Add GitLoader Component with advanced filtering options #2850

feat: Add GitLoader Component with advanced filtering options #2850

Conversation

danielgines commented Jul 20, 2024 • edited Loading

GitLoader Component:

Examples of file_filter usage:

Examples of content_filter usage:

Features:

github-actions bot commented Jul 20, 2024

Pull Request Validation Report

Whitelist Report

Validation Report

aws-amplify-sa-east-1 bot commented Jul 20, 2024

danielgines left a comment

Choose a reason for hiding this comment

ogabrielluiz left a comment

Choose a reason for hiding this comment

danielgines commented Jul 20, 2024 •

edited

Loading