feat: migrate an instance's storage between swift and s3 - #4855
Open
Crash-- wants to merge 23 commits into
Open
Conversation
Adds a workflow that builds the production Dockerfile and pushes the
resulting image to ghcr.io/<repo>:s3-test on every push to the
feat/s3-vfs-backend branch, plus a workflow_dispatch trigger that
accepts a custom tag.
Also publishes a ${tag}-<short-sha> tag for traceability.
scripts/build.sh runs `git describe` / `git rev-parse` to derive the build version string. With recent git versions the COPY'd working tree trips the "detected dubious ownership" safety check inside the container (the host user that owned the source no longer owns the files), causing the build step to exit 128 before producing the binary. Whitelisting /app as safe restores the previous behaviour.
scripts/build.sh derives the version from git describe / git rev-parse on the COPY'd working tree, which fails inside the buildx container (exit 128 with no captured output). Inline `go build` with a build-arg version string sidesteps the whole bash + git chain. The workflow passes VERSION_STRING=<tag>-<sha> so the running binary reports a recognizable version.
Implement a complete vfs.VFS backend for S3-compatible object storage (OVH, MinIO, Scaleway, ...) using minio-go/v7, mirroring the Swift V3 implementation. - config: add SchemeS3 and an S3 connection singleton, configured via the fs.url query params (access_key, secret_key, region, bucket_prefix, use_ssl) - model/vfs/vfss3: full VFS implementation with a bucket per orgId, a key prefix per instance, streaming CreateFile via io.Pipe, and multipart uploads with bounded memory - wire SchemeS3 into every storage dispatch point: VFS, avatars, thumbnails, apps copier and file servers, archiver, cache, dynamic assets and capabilities - pkg/s3util: shared S3 helpers (EnsureBucket, DeleteObjects, DeletePrefixObjects, IsNotFound, WrapNotFound) - pkg/appfs and pkg/assets/dynamic: S3 app installation and dynamic assets storage - security hardening: path-traversal protection, S3 error sanitization, MD5 integrity verification and bounded reads, plus the code-review fixes - tests: add the S3 backend to the VFS integration suite (MinIO testcontainer) - docs: document the S3 storage backend architecture and configuration
The model/vfs package now exercises three backends (afero, swift, s3) plus a second MinIO container for S3UploadErrorPropagation, pushing the package past the previous 5-minute per-package limit on GH runners.
WriteContentAt puts bytes at the object key for a given (docID, internalID) pair without creating or touching any CouchDB document. It is the S3-side primitive used by Swift-to-S3 storage migration, which moves object bytes while preserving the shared CouchDB index.
The Swift->S3 migration needs to copy a user-uploaded avatar as-is (it is authoritative, not regenerable). OpenAvatar exposes a generic read-side (content reader + content-type) across all three vfs backends, returning os.ErrNotExist when no avatar is stored so the migration can treat "no avatar" as a skip.
…ollback Mirrors Task 3's vfss3.WriteContentAt so the Swift v3 backend can also act as a copy target for storagemigration.CopyContent, enabling full S3->Swift re-migration (rollback). Also corrects the storagemigration package doc comment, which previously oversold "Swift<->S3" before Swift-as-target was actually implemented.
Add Verify, which re-enumerates the same files/versions/avatar that CopyContent copies and confirms each object exists on the target with a matching byte size, without touching CouchDB. Backed by a new StatContentAt primitive on both the S3 and Swift v3 VFS backends, since either can be the copy target. The orchestrator (a later task) will only flip an instance's backend flag once Verify passes.
Add Migrate, which ties CopyContent/Verify together into the full migration flow: guard the target scheme and instance state, block the instance, copy and verify the content, then flip FsScheme only once Verify passes (DryRun and a failed Verify always leave FsScheme unchanged). FlagOnly supports rollback onto an already-populated backend and requires Force, since writes since cutover would be lost. PurgeSource best-effort deletes the source objects after a successful flip (S3-source only for now; swift-source purge reports "not implemented" rather than silently no-op'ing). Also add config.HasS3Client/HasSwiftConnection, non-panicking readiness checks needed to guard Migrate against a target backend that was never initialized.
FlagOnly previously flipped FsScheme after buildTarget merely ensured the target bucket/container existed, with no check that it actually held the source's content (buildTarget's EnsureBucket/ContainerCreate create an empty target if none exists). Add sourceReport, which computes the expected file/version counts and avatar presence from the source's CouchDB docs, and run Verify against it before flipping; a failed Verify now returns without touching FsScheme. purgeSource also returned "not implemented" for a swift source after the flip had already persisted, making a successful migration look like a failure. Reuse the same per-instance container Delete() that instance destroy/reset already use to actually purge the swift source. Adds coverage for the flag-only flip (populated vs. empty target) and an end-to-end swift-source purge test using the in-memory swifttest server.
Documents the fs.migration_target config key that lets an instance be migrated to a new storage backend (e.g. S3) without changing the global fs.url, and adds a step-by-step Swift-to-S3 migration guide covering the migrate-storage command, rollback, and source purge. Also regenerates the CLI reference page for the new instances migrate-storage command.
…ag-only dry-run Migrate rejected opts.To == the instance's current scheme even when PurgeSource was set, making the documented deferred reclaim step (docs/s3.md step 4, run --purge-source after the flip) and any retry of a failed inline purge impossible. Migrate now detects this case and runs a purge-only mode: it deletes the other backend's leftover data for the instance without copying, verifying, or flipping anything, and without blocking the instance. Also fix --flag-only --force --dry-run mutating state: the FlagOnly branch now checks opts.DryRun after Verify and returns without flipping FsScheme when set. Finally, guard model/stack Start against fs.migration_target pointing at S3 while fs.url is already an s3 scheme, which previously silently overwrote the global S3 client with the migration endpoint.
shepilov
force-pushed
the
feat/s3-vfs-backend
branch
from
July 30, 2026 15:13
52655f5 to
4c7b7a9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
fs_schemeoverride so an instance can use a different storage backend than the stack default; an empty value keeps the globalfs.urlscheme.fs.migration_targetconfig that initializes an S3 connection alongside the default one while the global scheme stays Swift.WriteContentAtandStatContentAton the S3 and Swift v3 backends, andOpenAvataron theAvatarerinterface.model/instance/storagemigration: copy an instance's files, file versions and avatar between backends at the object-storage level without touching the CouchDB index, verify the target, then flip the instance's backend inside a read-only window.--dry-run, an instant--flag-onlyrollback to the retained source, and a deferred purge-only reclaim of the previous backend.POST /instances/:domain/migrate-storage, the admin client method, and thecozy-stack instances migrate-storagecommand.fs.migration_targetand the migration workflow in docs/config.md and docs/s3.md.