-
Notifications
You must be signed in to change notification settings - Fork 58
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[mula] Rate limiting by interval and grouper as set in boefje manifest. #1317
Comments
underdarknl
changed the title
[mula] Rate limiting by iterval and grouper as set in boefje manifest.
[mula] Rate limiting by interval and grouper as set in boefje manifest.
Jul 3, 2023
Implementation on the scheduler side has been formed. #1413 For this PR is only the information needed from the katalogus regarding the grouping and the rate limit. |
TODO: write down what the current solution is in the PR |
Open
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
We need some way of doing rate-limiting when accessing external services. The easiest way for now is to keep track of which jobs are given out to the runtimes and make sure we dont hit the various services too often.
Describe the solution you'd like
Implement the following (low hanging fruit) rate-limiter:
Describe in the boefjes-manifest (if required) two settings:
** Either a string (eg, shodan.com)
** or a placeholder that we can replace with data from the referenced inputOOI. (eg, [ipaddress])
When a worker requests a job, check the boefje manifest and set a timer (if needed) for the duration interval, and key it to the given grouping target.
If a subsequent worker requests a job, make sure to see if the grouping target already has a timer, and if it is still within the duration interval, if so, select the next job. If not give it out, and reset the timer to the current time.
Some cleaning of obsolete timers can be done periodically.
We could probably do this filtering in the job query to avoid repeated database queries when asking for jobs for a given boefje worker.
Describe alternatives you've considered
None currently spring to mind.
Additional context
The text was updated successfully, but these errors were encountered: