Skip to content

feat(objectstore): add Rask.ObjectStore, an S3 and Azure Blob client with no cloud SDK - #653

Merged
pal-tamas merged 2 commits into
mainfrom
worktree-object-store
Aug 8, 2026
Merged

feat(objectstore): add Rask.ObjectStore, an S3 and Azure Blob client with no cloud SDK#653
pal-tamas merged 2 commits into
mainfrom
worktree-object-store

Conversation

@pal-tamas

Copy link
Copy Markdown
Owner

Item 2 of #642.

Why a client rather than an SDK

The AWS and Azure SDKs are large, reflection-heavy, and not usable from a browser — which rules them out for the place this is most needed: a WASM app talking to a bucket with no backend in between. Signing SigV4 is a few dozen lines of HMAC and an Azure SAS needs no signing at all, so Rask.ObjectStore does both itself and runs unchanged server-side and in the browser.

It is standalone (Microsoft.Extensions.* only, no Rask.Core), so it is also immediately useful outside #642: Rask.SQLite.Snapshots currently has only a DirectorySnapshotStore, and this is what a cloud one would be built on.

IObjectStore

Deliberately the small intersection every store agrees on, covering S3, Cloudflare R2, Google Cloud Storage (via its S3 interop keys), MinIO, Backblaze B2, DigitalOcean Spaces and Azure Blob.

  • Ranged reads, streamed writes. Object storage charges per byte moved, so GetRangeAsync asks for a range rather than an object. PutAsync(key, Stream, length) uploads without buffering, keeping object size and memory use unrelated.
  • A missing object returns null; a range past the end returns a short read. Kept distinguishable on purpose — anything walking an append-only log must tell "gone" from "nothing new yet", and collapsing them is how a sync client silently concludes its peers vanished.
  • TryCreateAsync is mutual exclusion without a lock service — atomic compare-and-create via If-None-Match: *, which S3, Azure Blob and GCS all support. Chosen over an Azure blob lease, which exists on one provider, needs renewal, and strands the resource if the holder disappears.
  • Credentials are asked for per request, so an expiring STS session or SAS refreshes without rebuilding the store.

How SigV4 is verified

This is the part worth reviewing. A signature that is wrong by one byte is rejected exactly like no signature, and the service never says which part was wrong — so it is either exactly right or silently useless.

The expected Authorization headers in the tests were produced by a separate implementation of the algorithm written in Python from the AWS specification, not by running this code and recording its output. A golden file captured from the implementation under test would only prove it hasn't changed.

That Python reference self-checks against a published vector before its output is used — and that self-check failed on the first run, which was the point of having it. The cause turned out to be a misremembered constant rather than a wrong algorithm, so I pulled the specification and confirmed the derivation and every encoding rule against it. The remaining tests then assert the rules the spec names individually — %20 rather than +, no double-encoding, slashes preserved in a key name, query parameters sorted after encoding, which headers must be signed — so a shared misreading is caught as well as a coding mistake.

Two judgement calls that differ from a minimal reading of the spec:

  • Range and If-None-Match are signed, though only host and x-amz-* are required. They are what say which bytes are being read and whether an existing object may be overwritten; unsigned, anything in the middle could change the meaning of the request. They are safe to sign because this client sets them itself, unlike hop-by-hop headers.
  • TryCreateAsync treats 409 as well as 412 as "someone else won", because S3-compatible stores disagree about which they return.

A limitation found by a failing test

I had asserted that x/y/z and x%2Fy%2Fz sign differently. They don't: System.Uri normalises %2F back to a real separator while parsing, so a key whose name contains an encoded slash is indistinguishable from a path separator before any signing code can see it. Such keys are legal in S3 and unreachable through this client.

That is now documented on IObjectStore and pinned by a test, so if the platform ever changes the behaviour it says so rather than a signature quietly starting to differ.

Clock skew

SigV4 rejects any request more than 15 minutes off the service's clock, and device clocks are genuinely wrong often enough to matter — a user a day out would otherwise get a 403 that says nothing about time. The service's own Date is read from the rejected response and subsequent requests sign against corrected time, so a wrong clock costs one round trip. The streaming upload path probes for the offset first, because a forward-only stream cannot be rewound for a retry.

Testing

  • 58 unit tests: three cross-implementation SigV4 vectors, the individually-named encoding rules, and behaviour for both stores (URL shape, path vs virtual-host addressing, range arithmetic, 404/416 handling, conditional-create outcomes, list pagination for both S3 continuation tokens and Azure markers, credential resolution and DI precedence).
  • Full local unit gate green, dotnet format --verify-no-changes clean, solution builds -warnaserror clean on both net10.0 and net10.0-browser.
  • Browser E2E green (GuideCatalog.cs is under samples/).
  • PackageDependencyTests green — which is what forced the pack steps in release.yml and nightly.yml and the NUGET.md entry. Without those the package would have been built, tested and documented while existing on no feed.

Docs

New docs/object-storage.md (provider table, credentials, the CORS and clock-skew traps, and the stated limits), its own NUGET.md, a guide-catalog entry, llms.txt, and the CHANGELOG.

Not included

Multipart upload, presigned URL generation, bucket administration and server-side copy. Presigned URLs are worth adding to the snapshot-upload path later — they fit "one method, one key, one expiry" — but they do not fit a sync client minting new keys continuously and needing LIST, which would need a backend to mint them and is the thing #642 exists to avoid.

…with no cloud SDK

The AWS and Azure SDKs are large, reflection-heavy, and not usable from a browser, which rules
them out for the place this is most needed: a WASM app talking to a bucket with no backend in
between. Signing SigV4 is a few dozen lines of HMAC and an Azure SAS needs no signing at all,
so the client does both itself and runs unchanged server-side and in the browser.

One IObjectStore covers S3, Cloudflare R2, Google Cloud Storage (via its S3 interop keys),
MinIO, Backblaze B2, DigitalOcean Spaces and Azure Blob. It is standalone -- Microsoft.Extensions.*
only, no Rask.Core -- so Rask.SQLite.Snapshots, which today has only a DirectorySnapshotStore,
can be given a cloud store built on it.

- Ranged reads, streamed writes. Object storage charges per byte moved, so GetRangeAsync asks
  for a range rather than an object, and PutAsync(key, Stream, length) uploads without
  buffering, keeping object size and memory use unrelated.
- A missing object returns null; a range past the end returns a short read. Those stay
  distinguishable on purpose: anything walking an append-only log has to tell "gone" from
  "nothing new yet", and collapsing them is how a sync client silently decides its peers
  vanished.
- TryCreateAsync is mutual exclusion without a lock service -- an atomic compare-and-create
  (If-None-Match: *) that S3, Azure Blob and GCS all support. Preferred over an Azure blob
  lease, which exists on one provider, needs renewal, and strands the resource if the holder
  disappears.
- Credentials are asked for per request, so an expiring STS session or SAS refreshes without
  rebuilding the store. InMemoryObjectStoreCredentials, the browser case, holds one for the
  life of the process and offers no persistence option: a credential that survives a reload is
  one any later script injection can read back, so getting there has to be deliberate.
- Clock skew is handled rather than assumed away. SigV4 rejects a request more than 15 minutes
  off the service's clock and device clocks are genuinely wrong; the service's own Date is read
  from the rejected response and later requests sign against corrected time.

On verifying the signer: the expected Authorization headers in the tests come from a separate
implementation of the algorithm written from the AWS specification, not from recording this
code's own output. That reference self-checks against a published vector first, and the check
failed on its first run -- a misremembered constant rather than a wrong algorithm -- so the
derivation and every encoding rule were then confirmed against the specification directly. The
remaining tests assert the rules it names individually: %20 rather than +, no double-encoding,
slashes preserved in a key, query parameters sorted after encoding.

Range and If-None-Match are signed although only host and x-amz-* are required. They say which
bytes are read and whether an existing object may be overwritten, so unsigned they would let
anything in the middle change the meaning of the request; they are safe to sign because this
client sets them itself, unlike hop-by-hop headers.

One limitation, found by a test that failed: System.Uri normalises %2F back to a real separator
while parsing, so a key whose name contains an encoded slash cannot be addressed. Such keys are
legal in S3 and unreachable here. Documented and pinned.

Item 2 of #642.
…ver re-runs

Follow-up to #652, which cleared only obj/Release/net10.0-browser. With the no-native build left in
bin/Release/net10.0-browser the publish treats the compile as up to date and never re-runs the
scoped-asset bake -- and the staged copy under obj/ has just been deleted, so nothing reaches
publish/wwwroot/_rask at all. Clearing obj alone is worse than clearing neither.

Measured on one worktree: obj only -> 0 files under publish/wwwroot/_rask; obj + bin -> 6.

This is why the gate still failed after #652 merged, with the same symptom it was meant to fix: a
permanently disabled Run button and a 30s click timeout that names nothing. It also cannot be caught by
running the gate twice back to back -- both runs clear obj and leave the same stale bin, so both are
wrong in the same way.

Refs #650.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant