fix: give supabase-storage the S3 protocol credentials its clients sign with (#1282)

[sakibsadmanshajib]

Fixes #1282.

Cause, read from the running box rather than guessed

The error log added for issue #1255 made this one command away. On the demo box:

ERROR file upload to object storage failed bucket=hive-files size=18 content_type=application/octet-stream
error="s3 PUT /storage/v1/s3/hive-files/<account>/<file-id>/probe.jsonl failed with status 403:
<Error><Code>AccessDenied</Code>
<Message>Missing S3 Protocol Access Key ID or Secret Key Environment variables</Message></Error>"

That message is Storage's own, from createServerSignature in its signature-v4 plugin. In single tenant mode Storage verifies an S3 request by recomputing the AWS SigV4 signature from S3_PROTOCOL_ACCESS_KEY_ID and S3_PROTOCOL_ACCESS_KEY_SECRET, and with neither set it cannot verify anything, so it refuses every signed request. The supabase-storage service in docker-compose.enterprise.yml has never had either variable, confirmed against the running container's environment.

This was configuration, not code. edge-api and control-plane were correct throughout: correct endpoint, correct region, correct bucket names, buckets present as rows in storage.buckets, container healthy. Nothing on the client side could have been changed to make it work.

The prior guidance made it worse rather than random. .env.example and docker-compose.yml both said to set S3_ACCESS_KEY and S3_SECRET_KEY to the ENTERPRISE_SERVICE_ROLE_KEY, and the box followed it: both values were byte identical service_role JWTs. A service_role JWT is not an S3 credential in single tenant mode, and identical halves are their own problem, since SigV4 sends the access key id in the clear inside the Authorization header, so every request would have disclosed the service_role key to anything on the path.

The change

supabase-storage now receives:

• S3_PROTOCOL_ACCESS_KEY_ID and S3_PROTOCOL_ACCESS_KEY_SECRET, from the same S3_ACCESS_KEY and S3_SECRET_KEY variables the consumers sign with. Deliberately the same two: a separate pair lets the signing half and the verifying half drift with no boot error on either side, and the only symptom is a 403 from a service that looks configured.
• S3_PROTOCOL_PREFIX: /storage/v1. Caddyfile.supabase strips that prefix with handle_path before Storage sees the request, and Storage rebuilds the canonical URI as s3ProtocolPrefix + request.url, so leaving it empty fails every request with SignatureDoesNotMatch instead, which reads as a wrong key rather than a wrong path.

.env.example and the docker-compose.yml comment now say to generate a distinct pair, and say why.

Guards

Two new checks in scripts/test_selfhost_supabase_seam.py, which CI already runs through make test-scripts:

• Storage is handed the same credentials the consumers sign with, and both consumers still read those variables.
• The prefix, the S3_ENDPOINT the operator is told to use, and the gateway's handle_path route all agree.

Both were confirmed to go red against the unfixed compose file before being kept.

Evidence against the deployed box

Same API key, same file, same public endpoint, before and after.

Before:

$ curl -X POST https://api-hive.scubed.co/v1/files -H "Authorization: Bearer <key>" -F purpose=batch -F file=@probe.jsonl
HTTP 500
{"error":{"message":"Failed to store file","type":"api_error","param":null,"code":null}}

After:

HTTP 200
{"id":"file-3e6cfaeb-aae0-4eba-8fe6-b1fea3e87f70","object":"file","bytes":18,
 "created_at":1787984071,"filename":"probe1282.jsonl","purpose":"batch","status":"uploaded"}

Round trip, not just the write:

GET /v1/files/file-3e6cfaeb.../content   HTTP 200   {"custom_id":"a"}
GET /v1/files                            HTTP 200   the object listed

The two dependent surfaces

Batches: the storage half is fixed, one unrelated failure remains. A batch submission now gets all the way past every storage dependent stage: the input file uploads, ownership resolves, validateJSONL downloads and parses the object out of hive-files, and the credit reservation is taken. It then fails in control-plane at route selection, identically for every alias offered by the deployment:

POST /internal/batches/create -> 500 {"error":"select batch route: routing: no eligible routes: alias hive-fast has no eligible routes"}

That is alias independent, has nothing to do with object storage, and matches CLAUDE.md Known Issues item 4: no provider in the current mix has a batch API. It is pre-existing and out of scope here. Worth its own issue, and worth noting that the live conformance suite will keep reporting Batches red for this second reason after this merges.

RAG document upload never depended on object storage. apps/edge-api/internal/rag forwards parsed content to control-plane, which chunks, embeds and writes to pgvector. No Upload call anywhere in either package, and the chat front end stores its own uploads on local disk, with no S3 variables in the Open WebUI container at all. Probing POST /v1/rag/documents against production returns 403 ACCESS_DENIED from featuregate, which is the tenant feature flag for the probe account and not a storage failure. So the claim that this 500 also broke the RAG upload beat does not hold, and no tenant flag was changed to check.

What was done to the running box, exactly

1. ~/hive/.env: S3_ACCESS_KEY and S3_SECRET_KEY replaced with a freshly generated distinct pair, replacing the two identical service_role JWTs. Previous file backed up on the box.
2. A temporary untracked deploy/docker/docker-compose.fix1282.yml carrying the same three variables this PR adds, so the fix could be proven ahead of the merge without touching a tracked file, which would have broken the deploy's git pull --ff-only. It is passed only to the manual command below and is deleted after verification; the deploy workflow's compose flags never reference it.
3. docker compose ... up -d --no-deps --no-build supabase-storage edge-api control-plane. Three containers recreated and nothing else. supabase-storage to receive the new variables, edge-api and control-plane because their S3 credentials changed. All three came back healthy within twenty seconds.

Once this merges, the deploy recreates those services from the tracked file with the same values from the box's .env, so the state converges rather than depending on the override.

What this does not close

#1324 stays open, and this is not its fix. That issue is about CI, where the S3_ENDPOINT repository secret still names the deleted Supabase Cloud project. Production never pointed there: the box has pointed at http://caddy-supabase/storage/v1/s3 since the cutover, and its failure was the missing server side credentials, not a dead hostname. Two separate causes producing the same blind 500 in two places.

I cannot close the CI half from here, and it is not a secret rotation either: the box's Storage is not reachable from a GitHub hosted runner, so there is no correct value for the current secret. The real repair is for the throwaway CI stack to run its own supabase-storage container with these same three variables. Related, and part of why nobody noticed: ci.yml points the local stack's S3_ENDPOINT at http://127.0.0.1:9 with a stub key, so the always on sdk-tests-js lane cannot exercise an upload either, and reports green while doing so.

Also noticed and left alone: control-plane logs auditarchive: cold storage configured bucket=hive-audit-cold, and no such bucket exists. supabase-init creates hive-files and hive-images only. Nothing fails today because that cron only moves records older than ninety hot days, but it will fail the first time it runs. Separate issue, not folded in here.

Buglog entry

{"date":"2026-08-29","title":"Supabase Storage refused every S3 request because the server had no S3 protocol credentials","error_message":"s3 PUT /storage/v1/s3/hive-files/... failed with status 403: AccessDenied: Missing S3 Protocol Access Key ID or Secret Key Environment variables","root_cause":"The self-hosted supabase-storage service was never given S3_PROTOCOL_ACCESS_KEY_ID, S3_PROTOCOL_ACCESS_KEY_SECRET or S3_PROTOCOL_PREFIX. In single tenant mode Storage recomputes the SigV4 signature from that pair, so with neither set it refuses every signed request while staying healthy with the buckets present. The documented recipe compounded it by telling operators to point both client variables at the service_role key, which is not an S3 credential and discloses itself through the access key id that SigV4 sends in the clear.","fix":"Hand supabase-storage S3_PROTOCOL_ACCESS_KEY_ID and S3_PROTOCOL_ACCESS_KEY_SECRET from the same S3_ACCESS_KEY and S3_SECRET_KEY the consumers sign with, plus S3_PROTOCOL_PREFIX=/storage/v1 to match the prefix Caddy strips. Corrected the credential guidance in .env.example and docker-compose.yml, and added two guards to scripts/test_selfhost_supabase_seam.py.","tags":["storage","supabase","s3","sigv4","compose","config","issue-1282"]}

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[coderabbitai]

Warning

Review limit reached

Next included review available in 58 minutes.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: bae91d0b-a482-42fe-b5c6-401f8e15a68b

📥 Commits

Reviewing files that changed from the base of the PR and between 5818cd2 and d23392c.

📒 Files selected for processing (6)

• .env.example
• apps/edge-api/internal/rag/chat_handler_test.go
• deploy/docker/docker-compose.enterprise.yml
• deploy/docker/docker-compose.yml
• scripts/install.sh
• scripts/test_selfhost_supabase_seam.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[sakibsadmanshajib]

Correction to the box state note in the description: the temporary deploy/docker/docker-compose.fix1282.yml override is still present on the box and is being left there deliberately until this merges and deploy-demo-box.yml recreates the services from the tracked compose file. It is inert unless passed with an explicit -f, and the deploy's compose flags do not reference it. Delete it after the post merge deploy is confirmed green: rm ~/hive/deploy/docker/docker-compose.fix1282.yml. Deleting it earlier would leave the running containers correct but the fix un-reproducible if anything recreated supabase-storage before the merge.

[sakibsadmanshajib]

Pre-merge review

Two streams. One ran, one refused.

Antigravity (gemini-3.1-pro-high, high effort): RAN

Six points put to it: the shared-variable choice, the blast radius of the :? required form, prefix correctness against the gateway and the other storage consumers, the security posture change, whether the new guards can actually fail, and what would still leave a fresh deployment broken.

Finding 1, fresh EnterpriseEdge install still broken. Accepted, fixed in ea8451e. scripts/install.sh prompted for S3_ACCESS_KEY and S3_SECRET_KEY as required free text, and labelled the endpoint with a hosted project URL, while that installer starts the enterprise profile and therefore runs supabase-storage in the stack. So the prompt asked for a value that does not exist on any hosted project, which is exactly what steered operators to the service_role key. Both are now generated with openssl the way the installer already generates its other secrets, and the endpoint defaults to the gateway form.

One part of the finding was wrong on the mechanism and worth recording so nobody re-derives it: it claimed _write_env_var's sed could not match because the lines are commented out. .env.example carries an uncommented S3_ACCESS_KEY= at line 394; the commented block at 1025 is the enterprise example beneath it. The write path was never broken. The prompt was.

Finding 2, the :? form fails any invocation that loads this file without the variables set. Correct, and deliberate. Compose interpolates the whole file before profile filtering, so this is not scoped to services that start. The previous behaviour was a silent 500 on every upload, and this repo's own rule is that a loud failure beats a quiet absence. Named in the description so an operator upgrading a box reads it before running the deploy.

Finding 3, the guards asserted text rather than the property. Partly accepted, fixed in ea8451e. The variable check matched a bare prefix, so ${S3_ACCESS_KEY_ID} would have satisfied an assertion about ${S3_ACCESS_KEY}, which is the drift the guard exists to catch. It now requires a name terminator, confirmed red against that exact substitution. The second half, that assert prefix == "/storage/v1" merely pins the literal, I am rebutting: the cross-file assertions derive from the parsed value rather than the literal, so they verify agreement between the compose file, the Caddy route and the documented endpoint whatever the value becomes. The equality line is one extra pin on top of that, not the whole check.

Findings 4, 5 and 6: nothing wrong. It independently confirmed the security reading (the service_role JWT was travelling in the clear in the SigV4 Authorization header on every storage request, and a random access key id ends that), the prefix reasoning against handle_path, and that presigned URLs, multipart upload, artifacts and the audit archive all sign the full path through S3_ENDPOINT and so are fixed rather than broken by a non-empty prefix.

CodeRabbit CLI: SKIPPED, not a pass

✗ Review limit reached
You've used all 3 included reviews currently available.
Usage-based reviews are enabled, but this Git provider account isn't linked to an assigned seat.
Error: Rate limit exceeded

Recorded as SKIPPED rather than clean. Nothing from this stream was read, so its absence is not evidence of anything.

[sakibsadmanshajib]

CI note. The Go tests (edge-api) leg went red on the first run with TestHandleChat_SuppressedChunkUsageNotAccounted, in a package this change does not touch. It is a flaky assertion, not a regression: the test asserted the response body contains no 999, and every frame the RAG chat handler emits carries a generated ragchat-<uuid> id, so a hex uuid containing that substring fails it at random. The id in run 33238715016 was ragchat-39990550-3e90-48db-b254-c54af332a90d.

Fixed in d23392c rather than re-run and ignored: the check now names the usage fields instead of the bare number, and the SPURIOUS-POST-FINISH-MARKER assertion beside it is unchanged and covers the same frame, so no coverage is lost. A flake inside a regression guard is worse than no guard, because the next red on it gets assumed to be the flake.