Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a retry mechanism for scraping failure #42

Closed
brafdlog opened this issue May 5, 2020 · 3 comments
Closed

Add a retry mechanism for scraping failure #42

brafdlog opened this issue May 5, 2020 · 3 comments

Comments

@brafdlog
Copy link
Owner

brafdlog commented May 5, 2020

Since scraping is a very integrative process it can fail from many causes. Instead of making the user retry himself, it would be good if there was a retry mechanism in the code that would retry x times if the scraping failed.

@brafdlog brafdlog added enhancement New feature or request Server - node labels May 5, 2020
brafdlog pushed a commit that referenced this issue Jun 5, 2020
@baruchiro
Copy link
Collaborator

baruchiro commented Apr 4, 2021

I will check more in-depth later, but for now, I see two options:

  1. Use Enable/Disable state for importers like we have for exporters.
  2. Add a retry button on the importer status (in the middle of the screen).

I think option 2 is more user-friendly, since the scraping, by definition, is not stable so you need to retry one of the importers in many cases.

But this option is more complicated since then you have more than one button to trigger the process, so you need to create a locking architecture to avoid a parallel process.
Also, using enable/disable checkbox is keeps the way we are working today- manage the config and save it, and use it as is when clicking on Run. Use a retry button require to create a one-time config object and use it to scrape (it is not a bad thing, I just describing why I want the to implement option 1)

@baruchiro
Copy link
Collaborator

The third option we discussed is to identify the events that require us to retry- the timeout events, for example, and automatically retry the importer.

Copy link
Contributor

Issue has been marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the Stale label Nov 10, 2024
@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Nov 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants