Skip to content

Cache generated images in S3 - #10

Merged
iHiD merged 1 commit into
mainfrom
cache-generated-images-in-s3
Aug 11, 2026
Merged

Cache generated images in S3#10
iHiD merged 1 commit into
mainfrom
cache-generated-images-in-s3

Conversation

@iHiD

@iHiD iHiD commented Aug 11, 2026

Copy link
Copy Markdown
Member

Why

image_generator was ~$455 of July's $459 Lambda bill (836,207 invocations). Most of that was the Cloudflare outage, now fixed — but the underlying shape is the real problem: cost scales with requests, not with how many images exist.

Edge caches can't fix that on their own. Cloudflare and CloudFront are both per-PoP, they evict the long tail (most preview images are fetched a handful of times ever, so each PoP refills from scratch), and a flood of requests for distinct URLs misses them entirely. That last part is not hypothetical — it's what late July's bot traffic did.

The decisive number is what a miss costs:

Origin Cost per miss
Lambda (~3.5s x 2GB) ~$0.00012
S3 GET ~$0.0000004

Roughly 300x. Writing through to S3 means a given URL is only ever rendered once, so cost becomes O(images) instead of O(requests) — and a bot wave stops being able to run the bill up.

What this does

  • On invocation, look for the image in S3 first. On a miss, generate as today and write it through before returning.
  • Timestamped URLs (-{timestamp}.jpg) are immutable, so the stored copy is used indefinitely. Legacy untimestamped URLs point at mutable content, so a stored copy is only reused for 24h — matching the Cache-Control we already hand the CDN.
  • Both reads and writes fail soft. A missing bucket or policy logs and falls through to generating the image, so this degrades to exactly today's behaviour rather than breaking.

Also included, because they're the same cost story:

  • Explicit navigation/selector timeouts (6s each). puppeteer defaults both to 30s — longer than the Lambda's own 20s timeout, which is precisely why the recent outage burned a full 20s at 2GB on every single failure instead of failing fast.
  • browser.close() moved into a finally. Now that a render can fail rather than taking the container down with it, the browser would otherwise leak into the next warm invocation.

Bucket choice

Defaults to exercism-v3-assets — it's in eu-west-2 alongside the Lambda (no cross-region latency or transfer), it's a general-purpose assets bucket, and it already carries a screenshots/ prefix. exercism-assets, exercism-static and exercism-uploads are all eu-west-1; exercism-v3-icons is the right region but semantically wrong for generated user content.

Everything is env-configurable (IMAGE_BUCKET, IMAGE_KEY_PREFIX, NAVIGATION_TIMEOUT_MS, SELECTOR_TIMEOUT_MS), so overriding is a one-line change.

Required before this does anything

The Lambda's execution role needs s3:GetObject and s3:PutObject on arn:aws:s3:::exercism-v3-assets/generated-images/*. That's a separate terraform change.

This is safe to merge first — without the policy it fails soft and behaves exactly as it does today, so there's no ordering constraint.

Testing

No test suite exists in this repo, so this is unverified beyond a syntax check and a smoke test of the error path (unmappable path returns 500 without launching Chrome or touching S3). The S3 hit/miss paths have not been exercised against real AWS. Worth a careful look at fetchFromS3, and worth watching the invocation count drop after deploy to confirm the cache is being populated.

Rendering an image costs a few seconds of headless Chrome at 2GB, and
until now we paid that on every cache miss. Edge caches can't prevent
that on their own: they're per-PoP, they evict the long tail, and a
flood of requests for distinct URLs misses them entirely. July's bill
showed what that costs when someone points a bot at it.

Write each generated image through to S3 so a given URL is only ever
rendered once. Cost becomes a function of how many images exist rather
than how many times they're requested. Reads and writes both fail soft,
so a missing bucket or policy degrades to today's behaviour rather than
breaking image generation.

Timestamped URLs are immutable, so their stored copy is used forever.
Legacy untimestamped URLs point at mutable content, so a stored copy is
only reused for 24h — matching the Cache-Control we already return.

Also set explicit navigation and selector timeouts. puppeteer defaults
both to 30s, which is longer than the Lambda's own 20s timeout, so a
hung render burned the full 20s at 2GB rather than failing fast. Since a
render can now fail instead of taking the container down with it, the
browser is closed in a finally block so it can't leak into the next warm
invocation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018iQj4EdMfNsWP6NtFTqwUU
@iHiD
iHiD requested a review from a team as a code owner August 11, 2026 22:24
@iHiD
iHiD merged commit bb1e488 into main Aug 11, 2026
1 check passed
@iHiD
iHiD deleted the cache-generated-images-in-s3 branch August 11, 2026 22:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant