Skip to content

feat: cache warmer in memory switchover #237

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
SkArchon merged 10 commits into from
Feb 9, 2026
Merged

feat: cache warmer in memory switchover #237

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
461219b
feat: cache warmer in memory switchover
SkArchon Jan 21, 2026
57e856c
fix: review comments
SkArchon Jan 30, 2026
2fbd503
fix: review comments
SkArchon Jan 30, 2026
c8172dd
fix: review comments
SkArchon Jan 30, 2026
46b8bc0
fix: review comments
SkArchon Jan 30, 2026
614e2fa
fix: review comments
SkArchon Jan 30, 2026
76edfcd
fix: review comments
SkArchon Jan 30, 2026
42701a2
fix: review comments
SkArchon Jan 31, 2026
11e6be5
fix: review comments
SkArchon Jan 31, 2026
fe91d49
fix: added tradeoffs
SkArchon Feb 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions docs/concepts/cache-warmer.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,3 +69,41 @@ Users can manually recompute slow queries from the Cosmo Studio. Currently, reco
<Frame>
<img src="/images/concepts/image-4.png" />
</Frame>

## In-Memory Switchover Cache Warming

The in-memory switchover feature provides automatic cache persistence across router restarts, ensuring zero cold starts when deploying new configurations or updates.
Comment thread
StarpTech marked this conversation as resolved.
Outdated

### How It Works

The router can restart due to two reasons, either due to a config change or due to a schema change. Due to the structure of the router internals we have two slight variations on how we handle the in-memory switchover cache warming:

1. **Before Shutdown**: In case of config changes, the router extracts all queries from the current plan cache, preserving the "hot" queries that were actively being used.

2. **During Startup**: The router with the updated config receives the queries from the previous plan cache and uses them to warm up its plan cache before serving traffic.
Comment thread
StarpTech marked this conversation as resolved.
Outdated

3. **Result**: The updated router starts with a fully warmed cache, eliminating latency spikes that would normally occur during cold starts.

When using the in memory switchover, the first start will still experience a cold start, as there is no prior populated planner cache. Subsequent restarts will benefit from the in-memory switchover.

### When can you use the In-Memory Switchover

The in-memory switchover is a fallback method. Which means if your cosmo account has access to the Cosmo Cloud CDN cache warmer, that will be used instead of the in-memory switchover. However, if this is not the case, the in-memory switchover will be used as the primary cache warming method.
Comment thread
StarpTech marked this conversation as resolved.
Outdated

<Note>
The in-memory switchover won't be used as a fallback for sources other than the Cosmo Cloud CDN cache warmer, which is the default source.
</Note>

### Configuration

The in-memory switchover is **enabled by default when cache warming is enabled**. It automatically activates when:

* Cache warming is enabled on the router configuration (`cache_warmup.enabled: true`)
* Cosmo CDN cache warmer is not available or disabled for the account
* `in_memory_switchover_fallback` is set to `true` (default)

```yaml
cache_warmup:
enabled: true
in_memory_switchover_fallback: true # Enabled by default
```
26 changes: 15 additions & 11 deletions docs/router/configuration.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2173,13 +2173,14 @@ apollo_router_compatibility_flags:

## Cache warmer

| Environment Variable | YAML | Required | Description | Default Value |
| -------------------- | ---------------- | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- |
| | enabled | <Icon icon="square" /> | Enable the cache warmer. | false |
| | workers | <Icon icon="square" /> | The number of workers for the cache warmup to run in parallel. Higher numbers decrease the time to warm up the cache but increase the load on the system. | 8 |
| | items_per_second | <Icon icon="square" /> | The number of cache warmup items to process per second. Higher numbers decrease the time to warm up the cache but increase the load on the system. | 50 |
| | timeout | <Icon icon="square" /> | The timeout for warming up the cache. This can be used to limit the amount of time cache warming will block deploying a new config. The period is specified as a string with a number and a unit, e.g. 10ms, 1s, 1m, 1h. The supported units are 'ms', 's', 'm', 'h'. | 30s |
| | source | <Icon icon="square" /> | The source of the cache warmup items. Only one can be specified. If empty, the cache warmup source is the Cosmo CDN and it requires a graph to be set. | |
| Environment Variable | YAML | Required | Description | Default Value |
| --------------------------------------------------- | ------------------------------ | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- |
| CACHE_WARMUP_ENABLED | enabled | <Icon icon="square" /> | Enable the cache warmer. | false |
| CACHE_WARMUP_ITEMS_PER_SECOND | items_per_second | <Icon icon="square" /> | The number of cache warmup items to process per second. Higher numbers decrease the time to warm up the cache but increase the load on the system. | 50 |
| CACHE_WARMUP_TIMEOUT | timeout | <Icon icon="square" /> | The timeout for warming up the cache. This can be used to limit the amount of time cache warming will block deploying a new config. The period is specified as a string with a number and a unit, e.g. 10ms, 1s, 1m, 1h. The supported units are 'ms', 's', 'm', 'h'. | 30s |
| CACHE_WARMUP_WORKERS | workers | <Icon icon="square" /> | The number of workers for the cache warmup to run in parallel. Higher numbers decrease the time to warm up the cache but increase the load on the system. | 8 |
| | source | <Icon icon="square" /> | The source of the cache warmup items. Only one can be specified. If empty, the cache warmup source is the Cosmo CDN and it requires a graph to be set. | |
| CACHE_WARMUP_IN_MEMORY_SWITCHOVER_FALLBACK_ENABLED | in_memory_switchover_fallback | <Icon icon="square" /> | Enable in-memory switchover fallback that persists the plan cache across router restarts. When enabled, the router will use queries from the previous instance's plan cache to warm up the new cache on restart. Applicable when Cosmo is used but the cache warmer on cosmo is not enabled. | true |

### Example YAML config:

Expand All @@ -2191,15 +2192,16 @@ cache_warmup:
workers: 8
items_per_second: 50
timeout: 30s
in_memory_switchover_fallback: true
```

### Source

The source of the cache warmup items. Only one can be specified. If empty, the cache warmup source is the Cosmo CDN and it requires a graph to be set.

| Environment Variable | YAML | Required | Description | Default Value |
| -------------------- | ---- | ---------------------- | ------------------------------------------------------------ | ------------- |
| | path | <Icon icon="square" /> | The path to the directory containing the cache warmup items. | |
| Environment Variable | YAML | Required | Description | Default Value |
| -------------------- | -------------- | ---------------------- | ------------------------------------------------------------ | ------------- |
| | filesystem.path | <Icon icon="square" /> | The path to the directory containing the cache warmup items. | |

### Example YAML config:

Expand All @@ -2211,6 +2213,8 @@ cache_warmup:
workers: 8
items_per_second: 50
timeout: 30s
in_memory_switchover_fallback: true
source:
path: "./cache-warmer/operations"
filesystem:
path: "./cache-warmer/operations"
```
6 changes: 4 additions & 2 deletions docs/router/deployment/config-hot-reload.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@ This will behave similarly to the CDN polling with regard to router lifecycle.

WebSocket or SSE (Server-Sent Events) connections are long-lived and will be closed immediately after the active graph instance is shut down. Clients must reconnect to restore the connection.

### Latency Implications
### Latency Implications and Cache Warming

When the configuration is swapped, a new graph is created internally. This implies that the planner cache is invalidated, as a plan is coupled to a specific graph. In high-traffic scenarios, this can increase latency. We are currently working on a solution to mitigate this by warming up the cache with the latest top 100 operations during the initial router start and configuration changes.
When the configuration is swapped, a new graph is created internally. This implies that the planner cache is invalidated, as a plan is coupled to a specific graph. In high-traffic scenarios, this can increase latency.

To mitigate this issue, the router supports **cache warming**. For more details on cache warming, see the [Cache Warmer documentation](/concepts/cache-warmer).