feat(objectstore): add Rask.ObjectStore, an S3 and Azure Blob client with no cloud SDK - #653
Merged
Conversation
…with no cloud SDK The AWS and Azure SDKs are large, reflection-heavy, and not usable from a browser, which rules them out for the place this is most needed: a WASM app talking to a bucket with no backend in between. Signing SigV4 is a few dozen lines of HMAC and an Azure SAS needs no signing at all, so the client does both itself and runs unchanged server-side and in the browser. One IObjectStore covers S3, Cloudflare R2, Google Cloud Storage (via its S3 interop keys), MinIO, Backblaze B2, DigitalOcean Spaces and Azure Blob. It is standalone -- Microsoft.Extensions.* only, no Rask.Core -- so Rask.SQLite.Snapshots, which today has only a DirectorySnapshotStore, can be given a cloud store built on it. - Ranged reads, streamed writes. Object storage charges per byte moved, so GetRangeAsync asks for a range rather than an object, and PutAsync(key, Stream, length) uploads without buffering, keeping object size and memory use unrelated. - A missing object returns null; a range past the end returns a short read. Those stay distinguishable on purpose: anything walking an append-only log has to tell "gone" from "nothing new yet", and collapsing them is how a sync client silently decides its peers vanished. - TryCreateAsync is mutual exclusion without a lock service -- an atomic compare-and-create (If-None-Match: *) that S3, Azure Blob and GCS all support. Preferred over an Azure blob lease, which exists on one provider, needs renewal, and strands the resource if the holder disappears. - Credentials are asked for per request, so an expiring STS session or SAS refreshes without rebuilding the store. InMemoryObjectStoreCredentials, the browser case, holds one for the life of the process and offers no persistence option: a credential that survives a reload is one any later script injection can read back, so getting there has to be deliberate. - Clock skew is handled rather than assumed away. SigV4 rejects a request more than 15 minutes off the service's clock and device clocks are genuinely wrong; the service's own Date is read from the rejected response and later requests sign against corrected time. On verifying the signer: the expected Authorization headers in the tests come from a separate implementation of the algorithm written from the AWS specification, not from recording this code's own output. That reference self-checks against a published vector first, and the check failed on its first run -- a misremembered constant rather than a wrong algorithm -- so the derivation and every encoding rule were then confirmed against the specification directly. The remaining tests assert the rules it names individually: %20 rather than +, no double-encoding, slashes preserved in a key, query parameters sorted after encoding. Range and If-None-Match are signed although only host and x-amz-* are required. They say which bytes are read and whether an existing object may be overwritten, so unsigned they would let anything in the middle change the meaning of the request; they are safe to sign because this client sets them itself, unlike hop-by-hop headers. One limitation, found by a test that failed: System.Uri normalises %2F back to a real separator while parsing, so a key whose name contains an encoded slash cannot be addressed. Such keys are legal in S3 and unreachable here. Documented and pinned. Item 2 of #642.
…ver re-runs Follow-up to #652, which cleared only obj/Release/net10.0-browser. With the no-native build left in bin/Release/net10.0-browser the publish treats the compile as up to date and never re-runs the scoped-asset bake -- and the staged copy under obj/ has just been deleted, so nothing reaches publish/wwwroot/_rask at all. Clearing obj alone is worse than clearing neither. Measured on one worktree: obj only -> 0 files under publish/wwwroot/_rask; obj + bin -> 6. This is why the gate still failed after #652 merged, with the same symptom it was meant to fix: a permanently disabled Run button and a 30s click timeout that names nothing. It also cannot be caught by running the gate twice back to back -- both runs clear obj and leave the same stale bin, so both are wrong in the same way. Refs #650.
pal-tamas
force-pushed
the
worktree-object-store
branch
from
August 8, 2026 09:12
68549ce to
94f392f
Compare
This was referenced Aug 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Item 2 of #642.
Why a client rather than an SDK
The AWS and Azure SDKs are large, reflection-heavy, and not usable from a browser — which rules them out for the place this is most needed: a WASM app talking to a bucket with no backend in between. Signing SigV4 is a few dozen lines of HMAC and an Azure SAS needs no signing at all, so
Rask.ObjectStoredoes both itself and runs unchanged server-side and in the browser.It is standalone (
Microsoft.Extensions.*only, noRask.Core), so it is also immediately useful outside #642:Rask.SQLite.Snapshotscurrently has only aDirectorySnapshotStore, and this is what a cloud one would be built on.IObjectStoreDeliberately the small intersection every store agrees on, covering S3, Cloudflare R2, Google Cloud Storage (via its S3 interop keys), MinIO, Backblaze B2, DigitalOcean Spaces and Azure Blob.
GetRangeAsyncasks for a range rather than an object.PutAsync(key, Stream, length)uploads without buffering, keeping object size and memory use unrelated.null; a range past the end returns a short read. Kept distinguishable on purpose — anything walking an append-only log must tell "gone" from "nothing new yet", and collapsing them is how a sync client silently concludes its peers vanished.TryCreateAsyncis mutual exclusion without a lock service — atomic compare-and-create viaIf-None-Match: *, which S3, Azure Blob and GCS all support. Chosen over an Azure blob lease, which exists on one provider, needs renewal, and strands the resource if the holder disappears.How SigV4 is verified
This is the part worth reviewing. A signature that is wrong by one byte is rejected exactly like no signature, and the service never says which part was wrong — so it is either exactly right or silently useless.
The expected
Authorizationheaders in the tests were produced by a separate implementation of the algorithm written in Python from the AWS specification, not by running this code and recording its output. A golden file captured from the implementation under test would only prove it hasn't changed.That Python reference self-checks against a published vector before its output is used — and that self-check failed on the first run, which was the point of having it. The cause turned out to be a misremembered constant rather than a wrong algorithm, so I pulled the specification and confirmed the derivation and every encoding rule against it. The remaining tests then assert the rules the spec names individually —
%20rather than+, no double-encoding, slashes preserved in a key name, query parameters sorted after encoding, which headers must be signed — so a shared misreading is caught as well as a coding mistake.Two judgement calls that differ from a minimal reading of the spec:
RangeandIf-None-Matchare signed, though onlyhostandx-amz-*are required. They are what say which bytes are being read and whether an existing object may be overwritten; unsigned, anything in the middle could change the meaning of the request. They are safe to sign because this client sets them itself, unlike hop-by-hop headers.TryCreateAsynctreats409as well as412as "someone else won", because S3-compatible stores disagree about which they return.A limitation found by a failing test
I had asserted that
x/y/zandx%2Fy%2Fzsign differently. They don't:System.Urinormalises%2Fback to a real separator while parsing, so a key whose name contains an encoded slash is indistinguishable from a path separator before any signing code can see it. Such keys are legal in S3 and unreachable through this client.That is now documented on
IObjectStoreand pinned by a test, so if the platform ever changes the behaviour it says so rather than a signature quietly starting to differ.Clock skew
SigV4 rejects any request more than 15 minutes off the service's clock, and device clocks are genuinely wrong often enough to matter — a user a day out would otherwise get a 403 that says nothing about time. The service's own
Dateis read from the rejected response and subsequent requests sign against corrected time, so a wrong clock costs one round trip. The streaming upload path probes for the offset first, because a forward-only stream cannot be rewound for a retry.Testing
dotnet format --verify-no-changesclean, solution builds-warnaserrorclean on bothnet10.0andnet10.0-browser.GuideCatalog.csis undersamples/).PackageDependencyTestsgreen — which is what forced the pack steps inrelease.ymlandnightly.ymland theNUGET.mdentry. Without those the package would have been built, tested and documented while existing on no feed.Docs
New
docs/object-storage.md(provider table, credentials, the CORS and clock-skew traps, and the stated limits), its ownNUGET.md, a guide-catalog entry,llms.txt, and the CHANGELOG.Not included
Multipart upload, presigned URL generation, bucket administration and server-side copy. Presigned URLs are worth adding to the snapshot-upload path later — they fit "one method, one key, one expiry" — but they do not fit a sync client minting new keys continuously and needing
LIST, which would need a backend to mint them and is the thing #642 exists to avoid.