Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[mula] Rate limiting by interval and grouper as set in boefje manifest. #1317

Open
underdarknl opened this issue Jul 3, 2023 · 2 comments · May be fixed by #1611
Open

[mula] Rate limiting by interval and grouper as set in boefje manifest. #1317

underdarknl opened this issue Jul 3, 2023 · 2 comments · May be fixed by #1611
Assignees
Labels
mula Issues related to the scheduler
Milestone

Comments

@underdarknl
Copy link
Contributor

Is your feature request related to a problem? Please describe.
We need some way of doing rate-limiting when accessing external services. The easiest way for now is to keep track of which jobs are given out to the runtimes and make sure we dont hit the various services too often.

Describe the solution you'd like
Implement the following (low hanging fruit) rate-limiter:

Describe in the boefjes-manifest (if required) two settings:

  • rate limit interval
  • rate limit group target:
    ** Either a string (eg, shodan.com)
    ** or a placeholder that we can replace with data from the referenced inputOOI. (eg, [ipaddress])

When a worker requests a job, check the boefje manifest and set a timer (if needed) for the duration interval, and key it to the given grouping target.
If a subsequent worker requests a job, make sure to see if the grouping target already has a timer, and if it is still within the duration interval, if so, select the next job. If not give it out, and reset the timer to the current time.

Some cleaning of obsolete timers can be done periodically.
We could probably do this filtering in the job query to avoid repeated database queries when asking for jobs for a given boefje worker.

Describe alternatives you've considered
None currently spring to mind.

Additional context

@underdarknl underdarknl added the mula Issues related to the scheduler label Jul 3, 2023
@underdarknl underdarknl added this to KAT Jul 3, 2023
@github-project-automation github-project-automation bot moved this to Incoming features / Need assessment in KAT Jul 3, 2023
@underdarknl underdarknl changed the title [mula] Rate limiting by iterval and grouper as set in boefje manifest. [mula] Rate limiting by interval and grouper as set in boefje manifest. Jul 3, 2023
@jpbruinsslot jpbruinsslot moved this from Incoming features / Need assessment to In Progress in KAT Jul 18, 2023
@jpbruinsslot jpbruinsslot linked a pull request Jul 18, 2023 that will close this issue
@jpbruinsslot
Copy link
Contributor

Implementation on the scheduler side has been formed. #1413 For this PR is only the information needed from the katalogus regarding the grouping and the rate limit.

@jpbruinsslot jpbruinsslot moved this from In Progress to Review in KAT Aug 8, 2023
@jpbruinsslot jpbruinsslot moved this from Review to In Progress in KAT Aug 14, 2023
@jpbruinsslot jpbruinsslot linked a pull request Aug 14, 2023 that will close this issue
@jpbruinsslot jpbruinsslot moved this from In Progress to Review in KAT Aug 15, 2023
@underdarknl underdarknl added this to the OpenKAT v1.12 milestone Aug 17, 2023
@jpbruinsslot jpbruinsslot moved this from Review to Blocked / To be discussed in KAT Aug 28, 2023
@dekkers dekkers modified the milestones: OpenKAT v1.12, OpenKAT v1.13 Sep 14, 2023
@jpbruinsslot
Copy link
Contributor

TODO: write down what the current solution is in the PR

@jpbruinsslot jpbruinsslot moved this from To be discussed to Backlog / To do in KAT May 23, 2024
@jpbruinsslot jpbruinsslot removed a link to a pull request Jun 4, 2024
@underdarknl underdarknl removed this from the OpenKAT v1.17 milestone Aug 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
mula Issues related to the scheduler
Projects
Status: Backlog / To do
Development

Successfully merging a pull request may close this issue.

4 participants