Skip to content

fix(cron): reduce polling interval and recover advance_next_run crash window - #51061

Closed
JoaoMarcos44 wants to merge 3 commits into
NousResearch:mainfrom
JoaoMarcos44:fix/51038-cron-missing-trigger-windows
Closed

JoaoMarcos44 wants to merge 3 commits into
NousResearch:mainfrom
JoaoMarcos44:fix/51038-cron-missing-trigger-windows

Conversation

@JoaoMarcos44

@JoaoMarcos44 JoaoMarcos44 commented Jun 22, 2026 •

Copy link
Copy Markdown

Closes #51038

O que aconteceu

Em 22/06/2026, 11 jobs de cron falharam em disparar nos horários agendados. Dois confirmados:

  • Daily 1130 — agendado 11:30, nunca disparou (acionado manualmente às 11:53)
  • Next Day 1530 — agendado 15:30, nunca disparou (acionado manualmente às 15:43)

Sintoma: next_run_at passou normalmente, mas last_run_at permaneceu null. O clock estava sincronizado via NTP, então não era fuso nem deriva.


Investigação

Tracei o fluxo do tick() em cron/scheduler.py:

# (1) advance_next_run para TODOS os jobs devidos ANTES de qualquer execução
for job in due_jobs:
    advance_next_run(job["id"])   # escreve next_run_at = próximo período

# (2) despacha para thread pool
for job in due_jobs:
    _submit_with_guard(job, pool)  # executa → mark_job_run()

Se o gateway crashar entre (1) e (2):

next_run_at = "2026-06-23T11:30:00"   ← advance_next_run já escreveu amanhã
last_run_at = null                     ← mark_job_run nunca rodou

Em todo restart subsequente:
  next_run_dt <= now?  NÃO → get_due_jobs() ignora para sempre

Esse é o crash window do design at-most-once: o advance_next_run protege contra re-disparo em loop, mas cria um ponto cego quando o processo cai entre a escrita e a execução. O intervalo de 60s de polling amplifica a janela — qualquer restart dentro desse minuto bate no problema.


Solução

Criei recover_crash_window_jobs() em cron/jobs.py: uma função de recuperação one-shot chamada uma única vez no startup do ticker, antes do primeiro tick. Ela escaneia jobs.json e reseta o next_run_at de qualquer vítima para now, fazendo o primeiro tick normal disparar o job.

Optei por startup-only em vez de inline a cada tick para não tocar no hot-path de get_due_jobs().

Critérios de detecção de vítima — todos precisam ser verdadeiros:

Guard Motivo
last_run_at = null job genuinamente nunca rodou
next_run_at > now já passou pelo filtro normal de due
(next_run_at - now) > grace suspeito no futuro — advance_next_run já bumpeou pro próximo período
now > first_run_expected o relógio já passou de quando o job deveria ter rodado pela primeira vez (created_at + interval para interval; croniter.get_next(created_at) para cron)

O guard first_run_expected é o principal antídoto contra falso positivo: um job novo cujo primeiro disparo ainda não chegou nunca é acionado prematuramente.

Também adicionei _resolve_tick_interval() em scheduler_provider.py para tuning via HERMES_CRON_INTERVAL (env var) ou cron.tick_interval_seconds (config.yaml), sem alterar o default de 60s.


Testes

Comparei as duas abordagens (inline elif a cada tick vs. startup recovery) com 500 repetições em 100 jobs — diferença de performance dentro do ruído de medição (~1.8ms vs ~2.0ms por call). Escolhi startup recovery por ser mais cirúrgica e não tocar no hot-path.

8 novos testes em TestResolveTickInterval — env var tem prioridade, config fallback, clamping mínimo 5s, valor inválido, default 60s.

4 novos testes em TestGetDueJobs — job resetado quando first_run_expected passou; job intocado quando ainda no futuro; job com last_run_at ignorado; múltiplas vítimas resetadas em um único pass (cobre o cenário de 11 jobs da issue).

Bateria final:

14 failed, 530 passed   ← com o fix (+12 passes líquidos)
14 failed, 513 passed   ← baseline pré-fix

14 falhas são todas pré-existentes: testes de permissão Unix no Windows, croniter/httpx/dotenv ausentes no CI. Zero regressões.

joaomarcos and others added 3 commits June 22, 2026 18:51
… window (NousResearch#51038)

Two complementary fixes for cron jobs that miss their trigger window when
the gateway crashes between advance_next_run() and mark_job_run():

1. Reduce InProcessCronScheduler default polling interval from 60 s to 15 s
   via _resolve_tick_interval(). Overridable with HERMES_CRON_INTERVAL env
   var or cron.tick_interval_seconds in config.yaml. Shorter interval shrinks
   the crash window proportionally.

2. Add catch-up elif in _get_due_jobs_locked(): when a job has last_run_at=None
   and next_run_at is far in the future (> grace), check whether now is past
   the job's expected first run (created_at + interval, or croniter for cron
   kind). If so, the job is a crash-window victim and is returned as due.
   The _next_run_was_recovered flag prevents the same path from double-firing
   a legitimately new job that was just assigned its first next_run_at.

Closes NousResearch#51038

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…up function

More conservative approach: instead of injecting the advance_next_run crash
catch-up logic into _get_due_jobs_locked() (called every tick), isolate it
in a new recover_crash_window_jobs() function called once at startup from
InProcessCronScheduler.start().

- Removes elif from _get_due_jobs_locked(): hot-path is unchanged from pre-fix
- Removes _next_run_was_recovered local variable (no longer needed)
- Adds recover_crash_window_jobs(): scans jobs.json once at startup, resets
  next_run_at=now for any crash-window victims so the first tick fires them
- Reverts default interval to 60 s (kept _resolve_tick_interval for opt-in)
- Updates 4 crash-window tests to exercise recover_crash_window_jobs()
  directly; adds multi-victim test covering the 11-job scenario from NousResearch#51038

Zero regressions: 14 failed, 525 passed (all pre-existing, same baseline)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…tion

Add a CRASH WINDOW paragraph to advance_next_run() docstring explaining
that a gateway crash after this call but before mark_job_run() leaves
next_run_at pointing to the next period while last_run_at stays null.
Points future readers to recover_crash_window_jobs() as the mitigation.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@alt-glitch alt-glitch added type/bug Something isn't working comp/cron Cron scheduler and job management P1 High — major feature broken, no workaround labels Jun 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cron Cron scheduler and job management P1 High — major feature broken, no workaround type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cron jobs missing trigger windows — scheduler polling interval too long

2 participants