ProactiveTriggers service has race condition when iterating over triggers dict

[zzstoatzz]

Bug Description

The ProactiveTriggers service can crash with RuntimeError: dictionary changed size during iteration when the triggers dictionary is modified by the automation change listener while being iterated over.

Error Details

19:40:16.023 | ERROR   | prefect.server.services.proactivetriggers - Unexpected error in: RuntimeError('dictionary changed size during iteration')
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prefect/server/services/base.py", line 249, in start
    await self.run_once()
  File "/usr/local/lib/python3.11/site-packages/prefect/server/events/services/triggers.py", line 81, in run_once
    await triggers.evaluate_proactive_triggers()
  File "/usr/local/lib/python3.11/site-packages/prefect/server/events/triggers.py", line 1212, in evaluate_proactive_triggers
    for trigger in triggers.values():
RuntimeError: dictionary changed size during iteration

Root Cause

At line 1212 in src/prefect/server/events/triggers.py, the code iterates over triggers.values() without making a copy. The triggers dict can be modified concurrently by the automation change listener, causing this error.

Proposed Solutions

1. Quick fix: Iterate over a copy: for trigger in list(triggers.values()):
2. Better fix: Use a lock (asyncio.Lock or threading.Lock) to synchronize access to the triggers dict between the evaluation loop and the automation change listener

Impact

While the service recovers and continues running after the error, it may miss trigger evaluations during the error, potentially causing automations to not fire when expected.

[desertaxle]

@claude submit a PR to fix this

[claude]

Fixing ProactiveTriggers Race Condition

Working on fixing the race condition where the triggers dictionary changes size during iteration.

Todo List:

• Read repository setup instructions (CLAUDE.md) - no CLAUDE.md found
• Examine the problematic code in src/prefect/server/events/triggers.py
• Implement the fix for the race condition
• Check for related tests and update if needed
• Run linting/testing as specified in repo guidelines
• Commit and push changes
• Provide PR creation link

Analysis:

• Found the issue at line 1212: for trigger in triggers.values():
• The triggers dict is modified by load_automation() and forget_automation() functions
• These functions are protected by _automations_lock() but evaluate_proactive_triggers() doesn't use the lock
• Simple fix: iterate over a copy with list(triggers.values())

Implementing the fix now...

---

Job Run | Branch