Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add fallback options to search #49

Merged
merged 20 commits into from
Feb 13, 2025
Merged

Add fallback options to search #49

merged 20 commits into from
Feb 13, 2025

Conversation

ppinchuk
Copy link
Collaborator

@ppinchuk ppinchuk commented Feb 7, 2025

Refactor search to allow falling back to other SE options if initial (likely Google) search fails. The fallback list/order of search engines is entirely configurable by the user (as an input to the search function).

Also added an option to blacklist URL parts. This can be useful to further customize the web search piece

@ppinchuk ppinchuk self-assigned this Feb 7, 2025
@ppinchuk
Copy link
Collaborator Author

@spodgorny9 @rolson2 This should be ready for review.

@spodgorny9, I added an option to silence those poppler errors. Updated the notebook to run it quietly.

Copy link
Collaborator

@rolson2 rolson2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested the fallback by running the ords example pipeline and forcing the Google search to fail. The fallback seems to be working. The process initiated DuckDuckGo after Google failed.

@ppinchuk ppinchuk merged commit d2e56d3 into main Feb 13, 2025
15 checks passed
@ppinchuk ppinchuk deleted the pp/search_with_fallbacks branch February 13, 2025 23:02
github-actions bot pushed a commit that referenced this pull request Feb 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants