The Scheduler service eats indecent amounts of CPU to schedule RRule schedules

[VincentAntoine]

Bug summary

Short description of the problem

On a self-hosted Prefect server, the Prefect Scheduler service eats an indecent amount of CPU when there are deployments using RRule schedules.

Details

I'm running a self-hosted Prefect server on machine A which has 2 vCPUs and 8GB RAM. Prefect is deployed using Docker compose with the following containers :

• Postgresql 15
• Redis
• Prefect API (--no-services)
• Prefect Scheduler only
• Prefect services (all services except Scheduler)

A single worker running on a separate machine B polls the Prefect API and runs flows on the same machine B.

I have ~40 deployments, most of which use CRON schedules, and 3 of which use RRule schedules.

On machine A I initially used a single container for all Prefect background services, but that container constantly showed at at 100% CPU in docker stats so I split services between two containers (using environment variables to toggle services on or off) to try to pinpoint which service was eating the CPU. Thus I found out that the culprit was the Scheduler service.

Excerpt of the Scheduler service container logs :

09:58:40.189 | WARNING | prefect.server.services.recentdeploymentsscheduler - RecentDeploymentsScheduler took 49.552764 seconds to run, which is longer than its loop interval of 30.0 seconds.
09:59:58.494 | WARNING | prefect.server.services.recentdeploymentsscheduler - RecentDeploymentsScheduler took 48.302719 seconds to run, which is longer than its loop interval of 30.0 seconds.
10:00:29.829 | WARNING | prefect.server.services.recentdeploymentsscheduler - RecentDeploymentsScheduler took 31.333974 seconds to run, which is longer than its loop interval of 30.0 seconds.
10:01:25.774 | WARNING | prefect.server.services.recentdeploymentsscheduler - RecentDeploymentsScheduler took 55.943919 seconds to run, which is longer than its loop interval of 30.0 seconds.
10:03:39.531 | WARNING | prefect.server.services.recentdeploymentsscheduler - RecentDeploymentsScheduler took 49.124948 seconds to run, which is longer than its loop interval of 30.0 seconds.
10:05:58.731 | WARNING | prefect.server.services.recentdeploymentsscheduler - RecentDeploymentsScheduler took 49.19527 seconds to run, which is longer than its loop interval of 30.0 seconds.

And docker stats (prefect-background-services-a is the one with the scheduler service):

I already increased the scheduler loop to 60s and the recent scheduler loop to 30s and reduce max_runs to 10, and it's still having a really hard time to keep up.

Now if I turn off the 3 RRule based schedules, the CPU goes to 0% most of the time and everything is fine.

Version info

Version:             3.4.14
API version:         0.8.4
Python version:      3.12.11
Git commit:          9e2d422b
Built:               Thu, Aug 21, 2025 04:06 PM
OS/Arch:             linux/x86_64
Profile:             ephemeral
Server type:         server
Pydantic version:    2.11.7
Integrations:
  prefect-redis:     0.2.4

Additional context

docker-compose.yml for Prefect server :

services:
  postgres:
    image: postgres:15
    container_name: prefect-db
    environment:
      POSTGRES_USER: prefect
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?error}
      POSTGRES_DB: prefect
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./postgres.conf:/etc/postgresql/postgresql.conf
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U prefect"]
      interval: 10s
      timeout: 5s
      retries: 5
    ports:
      - "5432:5432"
    command: postgres -c config_file=/etc/postgresql/postgresql.conf
    restart: always
    deploy:
      resources:
        reservations:
          cpus: '0.5'
          memory: 500M

  redis:
    image: redis:7

  prefect-api:
    image: prefecthq/prefect:3-python3.12
    container_name: prefect-server
    depends_on:
      postgres:
        condition: service_healthy
      migrate:
        condition: service_completed_successfully
      redis:
        condition: service_started
    environment:
      # Prefect API and UI configuration
      PREFECT_API_URL: ${PREFECT_API_URL:?error}
      PREFECT_UI_URL: ${PREFECT_UI_URL:?error}
      
      # Database configuration
      PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:${POSTGRES_PASSWORD:?error}@postgres:5432/prefect?ssl=disable
      PREFECT_API_DATABASE_MIGRATE_ON_START: "false"
      # Redis
      PREFECT_MESSAGING_BROKER: prefect_redis.messaging
      PREFECT_MESSAGING_CACHE: prefect_redis.messaging
      PREFECT_SERVER_EVENTS_CAUSAL_ORDERING: prefect_redis.ordering
      PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE: prefect_redis.lease_storage
      PREFECT_REDIS_MESSAGING_HOST: redis
      PREFECT_REDIS_MESSAGING_PORT: "6379"
      PREFECT_LOGGING_LEVEL: DEBUG
    ports:
      - "4200:4200"
    restart: always
    command: ["prefect", "server", "start", "--host", "0.0.0.0", "--no-services"]

  migrate:
    image: prefecthq/prefect:3-python3.12
    depends_on:
      postgres:
        condition: service_healthy
    command: prefect server database upgrade -y
    environment:
      PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:${POSTGRES_PASSWORD:?error}@postgres:5432/prefect

  prefect-background-a:
    image: prefecthq/prefect:3-python3.12
    container_name: prefect-background-services-a
    depends_on:
      postgres:
        condition: service_healthy
      migrate:
        condition: service_completed_successfully
      redis:
        condition: service_started
    command: prefect server services start
    environment:
      # Database configuration
      PREFECT_SERVER_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:${POSTGRES_PASSWORD:?error}@postgres:5432/prefect?ssl=disable
      PREFECT_SERVER_DATABASE_MIGRATE_ON_START: "false"

      # Redis configuration
      PREFECT_MESSAGING_BROKER: prefect_redis.messaging
      PREFECT_MESSAGING_CACHE: prefect_redis.messaging
      PREFECT_SERVER_EVENTS_CAUSAL_ORDERING: prefect_redis.ordering
      PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE: prefect_redis.lease_storage
      PREFECT_REDIS_MESSAGING_HOST: redis
      PREFECT_REDIS_MESSAGING_PORT: "6379"

      # A
      PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED: "true"
      PREFECT_SERVER_SERVICES_SCHEDULER_LOOP_SECONDS: 60
      PREFECT_SERVER_SERVICES_SCHEDULER_MAX_RUNS: 10
      PREFECT_SERVER_SERVICES_SCHEDULER_MIN_SCHEDULED_TIME: PT5M
      PREFECT_SERVER_SERVICES_SCHEDULER_MAX_SCHEDULED_TIME: P14D
      PREFECT_SERVER_SERVICES_SCHEDULER_RECENT_DEPLOYMENTS_LOOP_SECONDS: 30
      # B
      PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED: "false"
      PREFECT_SERVER_SERVICES_LATE_RUNS_ENABLED: "false"
      PREFECT_SERVER_SERVICES_PAUSE_EXPIRATIONS_ENABLED: "false"
      PREFECT_SERVER_SERVICES_CANCELLATION_CLEANUP_ENABLED: "false"
      PREFECT_SERVER_SERVICES_FOREMAN_ENABLED: "false"
      PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED: "false"
      PREFECT_SERVER_SERVICES_EVENT_LOGGER_ENABLED: "false"
      PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED: "false"
      PREFECT_API_EVENTS_STREAM_OUT_ENABLED: "false"  # Distributor

      # Always off
      PREFECT_SERVER_ANALYTICS_ENABLED: "false"       # Telemetry

    restart: unless-stopped

  prefect-background-b:
    image: prefecthq/prefect:3-python3.12
    container_name: prefect-background-services-b
    depends_on:
      postgres:
        condition: service_healthy
      migrate:
        condition: service_completed_successfully
      redis:
        condition: service_started
    command: prefect server services start
    environment:
      # Database configuration
      PREFECT_SERVER_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:${POSTGRES_PASSWORD:?error}@postgres:5432/prefect?ssl=disable
      PREFECT_SERVER_DATABASE_MIGRATE_ON_START: "false"

      # Redis configuration
      PREFECT_MESSAGING_BROKER: prefect_redis.messaging
      PREFECT_MESSAGING_CACHE: prefect_redis.messaging
      PREFECT_SERVER_EVENTS_CAUSAL_ORDERING: prefect_redis.ordering
      PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE: prefect_redis.lease_storage
      PREFECT_REDIS_MESSAGING_HOST: redis
      PREFECT_REDIS_MESSAGING_PORT: "6379"

      # A
      PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED: "false"

      # B
      PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED: "true"
      PREFECT_SERVER_SERVICES_LATE_RUNS_ENABLED: "true"
      PREFECT_SERVER_SERVICES_PAUSE_EXPIRATIONS_ENABLED: "true"
      PREFECT_SERVER_SERVICES_CANCELLATION_CLEANUP_ENABLED: "true"
      PREFECT_SERVER_SERVICES_FOREMAN_ENABLED: "true"
      PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED: "true"
      PREFECT_SERVER_SERVICES_EVENT_LOGGER_ENABLED: "true"
      PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED: "true"
      PREFECT_API_EVENTS_STREAM_OUT_ENABLED: "true"  # Distributor

      # Always off
      PREFECT_SERVER_ANALYTICS_ENABLED: "false"       # Telemetry


    restart: unless-stopped

volumes:
  postgres_data:

Start a docker worker :

prefect worker start --pool my_work_pool --type docker --with-healthcheck

Then register any deployment with the RRule schedule, for instance "FREQ=MINUTELY;BYSECOND=45"