Skip to content

Conversation

@mattnowzari
Copy link
Contributor

Closes #2298

This PR creates a new configurable field

service.idle_jobs_threshold

which is the amount of time (in seconds) before a sync job is considered 'idle'.

We make this field configurable by grabbing it from the service configurations and passing the value to SyncJobIndex::idle_jobs() as an new argument - previously this was a hardcoded global variable.

TODO

  • Setting this value to zero should disable 'idle' jobs fully (?)
  • Possibly provide a log message as to why this timeout occurred

Checklists

Pre-Review Checklist

  • this PR does NOT contain credentials of any kind, such as API keys or username/passwords (double check config.yml.example)
  • this PR has a meaningful title
  • this PR links to all relevant github issues that it fixes or partially addresses
  • if there is no GH issue, please create it. Each PR should have a link to an issue
  • this PR has a thorough description
  • Covered the changes with automated tests
  • Tested the changes locally
  • Added a label for each target release version (example: v7.13.2, v7.14.0, v8.0.0)
  • Considered corresponding documentation changes
  • Contributed any configuration settings changes to the configuration reference

@mattnowzari mattnowzari added enhancement New feature or request sdh-driven labels Apr 21, 2025
@artem-shelkovnikov
Copy link
Member

There's one thing to keep in mind here - Kibana has its own threshold where it'll also mark jobs as "Idle" in UI. Don't remember where exactly, but somewhere in our module.

Connectors fail the job once idle threshold is reached, while Kibana just signalises that it haven't seen any updates for a bit and job might be stuck - might be worth to re-visit this part too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve job idle timeout logic

4 participants