feat(parquet): RFC0013 red — store module skeleton + §5 ignored stubs

crates/ourios-parquet/Cargo.toml

@@ -21,6 +21,12 @@ ourios-core = { path = "../ourios-core" }
 arrow-array = { version = "55", default-features = false }
 arrow-schema = { version = "55", default-features = false }
 parquet = { version = "55", default-features = false, features = ["arrow", "zstd"] }
+# Object-storage backend (RFC 0013) — one trait over LocalFileSystem (dev/
+# test) and S3-compatible stores. Already in the workspace tree via
+# DataFusion (the querier); declared here directly as the storage seam's
+# home. `0.13` unifies with that transitive version. The `aws` feature (for
+# the AmazonS3 backend) is added with the `green` S3 implementation.
+object_store = { version = "0.13" }
 # UUIDv7 file naming per RFC 0005 §3.4 — timestamp-in-high-bits
 # means lexicographic sort = creation-order.
 uuid = { version = "1", default-features = false, features = ["v7", "std"] }

crates/ourios-parquet/src/lib.rs

@@ -34,6 +34,7 @@ pub mod manifest;
 pub mod partition;
 pub mod reader;
 pub mod record_batch;
+pub mod store;
 pub mod writer;
 
 pub use audit_reader::{AuditReader, AuditReaderError};
@@ -51,6 +52,7 @@ pub use partition::{
 };
 pub use reader::{Reader, ReaderError};
 pub use record_batch::{BatchError, mined_records_to_batch};
+pub use store::{S3Config, Store, StoreError};
 pub use writer::{DEFAULT_ZSTD_LEVEL, ROW_GROUP_FLUSH_BYTES, Writer, WriterError, WrittenFile};
 
 use std::sync::Arc;

crates/ourios-parquet/src/store.rs

@@ -0,0 +1,142 @@
+//! Object-storage backend (RFC 0013) — the seam behind the writer, reader,
+//! compaction, and audit sink so the RFC 0005 data + audit Parquet and the
+//! RFC 0009 manifest live on local disk (dev/test) or an S3-compatible
+//! bucket (production), without changing the on-disk layout.
+//!
+//! **Status: `red` (RFC 0013).** This is the skeleton the `green` work fills:
+//! the [`Store`] type and its constructors exist and the `LocalFileSystem`
+//! backend is wired, but the S3 backend, the conditional-PUT atomic publish
+//! (RFC0013.3/.4), and the migration of the writer/reader/compaction/audit
+//! consumers from `bucket_root: &Path` onto [`Store`] are not done. The §5
+//! acceptance scenarios are encoded as `#[ignore]`d stubs in
+//! `tests/rfc0013_object_store.rs` and turn green as the backend lands.
+//!
+//! Per RFC 0013 §3.7 the backend is a **module here in `ourios-parquet`**
+//! (not a new crate): `ourios-querier`, `-ingester`, and `-server` already
+//! depend on this crate, so the type is visible to every storage consumer.
+
+use std::sync::Arc;
+
+use object_store::ObjectStore;
+use object_store::local::LocalFileSystem;
+use object_store::path::Path as ObjectPath;
+
+/// A handle to the object store backing a tenant store's Parquet + manifest
+/// objects, addressed by key under `prefix`. Wraps an [`ObjectStore`] so the
+/// same code path targets `LocalFileSystem` or `AmazonS3` / S3-compatible.
+///
+/// **`red` caveat:** `prefix` is reserved and currently always empty, and
+/// [`Store::object_store`] returns the raw backend with **no prefix
+/// scoping**. Per-tenant/prefix isolation (RFC0013.5) is wired at `green` —
+/// do **not** assume this type enforces isolation yet.
+#[derive(Clone)]
+pub struct Store {
+    inner: Arc<dyn ObjectStore>,
+    /// Reserved key prefix (the store root). Always empty at `red`; honoured
+    /// once the consumers migrate onto [`Store`] at `green`.
+    prefix: ObjectPath,
+}
+
+/// Configuration for the S3 / S3-compatible backend (RFC0013.7). Populated
+/// from RFC 0004 config at `green`; a placeholder here so the `red`
+/// constructor signature is stable.
+///
+/// `Default` is a `red` placeholder only — it yields an **empty `bucket`**,
+/// which is not valid; callers must set a non-empty `bucket` (the `green`
+/// `s3()` will reject an empty one).
+#[derive(Debug, Clone, Default)]
+#[non_exhaustive]
+pub struct S3Config {
+    /// Bucket name (required; the empty `Default` is a `red` placeholder).
+    pub bucket: String,
+    /// Optional endpoint override for S3-compatible stores (`MinIO`, R2, …).
+    pub endpoint: Option<String>,
+    /// Region (AWS) — ignored by some S3-compatible stores.
+    pub region: Option<String>,
+    /// Key prefix within the bucket (the store root).
+    pub prefix: Option<String>,
+}
+
+/// Errors from constructing or addressing a [`Store`].
+#[derive(Debug)]
+#[non_exhaustive]
+pub enum StoreError {
+    /// Backend construction failed (bad root, credentials, endpoint, …).
+    Backend(object_store::Error),
+    /// A backend constructor not yet implemented at this RFC 0013 stage.
+    /// Returned (rather than panicking) so an accidental call fails
+    /// gracefully while `red`.
+    Unimplemented(&'static str),
+}
+
+impl std::fmt::Display for StoreError {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match self {
+            Self::Backend(e) => write!(f, "object-store backend: {e}"),
+            Self::Unimplemented(what) => write!(f, "not implemented (RFC 0013 red): {what}"),
+        }
+    }
+}
+
+impl std::error::Error for StoreError {
+    fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {
+        match self {
+            Self::Backend(e) => Some(e),
+            Self::Unimplemented(_) => None,
+        }
+    }
+}
+
+impl Store {
+    /// Local-filesystem backend rooted at `root` (dev / test / CI). Preserves
+    /// today's on-disk layout — the RFC 0005 Hive keys become paths under
+    /// `root`.
+    ///
+    /// # Errors
+    /// [`StoreError::Backend`] if `root` cannot be opened as an
+    /// `object_store` `LocalFileSystem` (e.g. it does not exist).
+    pub fn local(root: impl AsRef<std::path::Path>) -> Result<Self, StoreError> {
+        let fs = LocalFileSystem::new_with_prefix(root).map_err(StoreError::Backend)?;
+        Ok(Self {
+            inner: Arc::new(fs),
+            prefix: ObjectPath::default(),
+        })
+    }
+
+    /// S3 / S3-compatible backend (RFC0013.1/.4/.7).
+    ///
+    /// `red`: not yet built — the `green` implementation constructs an
+    /// `object_store::aws::AmazonS3` (behind the `aws` feature) from `cfg`
+    /// and the RFC 0004 credentials.
+    ///
+    /// # Errors
+    /// At `red`, always [`StoreError::Unimplemented`] — returned rather than
+    /// panicking so an accidental call (e.g. from another workspace crate)
+    /// fails gracefully. At `green` this becomes [`StoreError::Backend`] if
+    /// the `object_store` `AmazonS3` backend cannot be constructed (bad
+    /// endpoint, credentials, or bucket).
+    // `red` stub: `cfg` is unused until the `green` AmazonS3 impl consumes
+    // it (`needless_pass_by_value` fires because we never read it). The
+    // signature is fixed now so consumers can be written against it.
+    #[allow(clippy::needless_pass_by_value, unused_variables)]
+    pub fn s3(cfg: S3Config) -> Result<Self, StoreError> {
+        // RFC0013 green: build AmazonS3 from cfg + RFC 0004 creds.
+        Err(StoreError::Unimplemented(
+            "RFC0013 green: AmazonS3 / S3-compatible backend",
+        ))
+    }
+
+    /// The underlying [`ObjectStore`], for handing to `DataFusion`'s table
+    /// providers on the read path (RFC 0013 §2.2 — the querier registers the
+    /// same store rather than local file paths).
+    #[must_use]
+    pub fn object_store(&self) -> Arc<dyn ObjectStore> {
+        Arc::clone(&self.inner)
+    }
+
+    /// The store's root key prefix.
+    #[must_use]
+    pub fn prefix(&self) -> &ObjectPath {
+        &self.prefix
+    }
+}

crates/ourios-parquet/tests/rfc0013_object_store.rs

@@ -0,0 +1,89 @@
+//! RFC 0013 — object-storage backend acceptance scenarios (§5).
+//!
+//! **Status: `red`.** These are the failing stubs that drive the `green`
+//! implementation: each encodes one RFC0013.§5 scenario and currently
+//! `todo!()`s. They are `#[ignore]`d so the default `cargo test` (and CI)
+//! stays green while the backend is built — `green` replaces each body with
+//! a real assertion and removes the `#[ignore]`. The S3-backed scenarios run
+//! against a MinIO/localstack container (`testcontainers`); the local-backend
+//! scenarios re-run the RFC 0005 / 0009 contract through `Store`.
+//!
+//! See `docs/rfcs/0013-object-storage.md` §5/§6.
+
+// Anchor the stubs to the public surface so they fail to compile if the
+// `Store` seam is removed out from under them.
+use ourios_parquet::{S3Config, Store};
+
+/// Scenario RFC0013.1 — a `MinedRecord` batch written and read through the `AmazonS3`
+/// backend recovers byte-for-byte against the local backend.
+/// See `docs/rfcs/0013-object-storage.md` §5.
+#[test]
+#[ignore = "RFC0013.1 — red until the S3 backend + writer/reader migration land"]
+fn rfc0013_1_round_trip_through_s3_backend() {
+    let _ = Store::s3;
+    todo!("RFC0013.1: S3 round-trip == local round-trip (MinIO testcontainer)")
+}
+
+/// Scenario RFC0013.2 — the existing RFC 0005 / 0009 suites pass unchanged against the
+/// `LocalFileSystem` backend after the seam refactor.
+/// See `docs/rfcs/0013-object-storage.md` §5.
+#[test]
+#[ignore = "RFC0013.2 — red until the consumers run through Store"]
+fn rfc0013_2_local_backend_regresses_nothing() {
+    todo!("RFC0013.2: RFC0005/0009 suites green via the LocalFileSystem Store")
+}
+
+/// Scenario RFC0013.3 — two `compact_partition` runs racing on one partition: exactly
+/// one manifest generation wins; no torn / doubled / missing rows.
+/// See `docs/rfcs/0013-object-storage.md` §5.
+#[test]
+#[ignore = "RFC0013.3 — red until conditional-PUT atomic publish lands"]
+fn rfc0013_3_atomic_publish_under_contention() {
+    todo!("RFC0013.3: exactly-one-wins under concurrent publishers")
+}
+
+/// Scenario RFC0013.4 — generation publish uses conditional PUT (`PutMode::Create` /
+/// `Update{ETag}`) with no `rename` dependency.
+/// See `docs/rfcs/0013-object-storage.md` §5.
+#[test]
+#[ignore = "RFC0013.4 — red until the manifest swap uses conditional PUT"]
+fn rfc0013_4_manifest_swap_via_conditional_put() {
+    todo!("RFC0013.4: publish path uses PutMode::Create/Update, never rename")
+}
+
+/// Scenario RFC0013.5 — operations in tenant X's context address only X's key
+/// sub-prefix; no read/write touches tenant Y's keys (`CLAUDE.md` §3.7).
+/// See `docs/rfcs/0013-object-storage.md` §5.
+#[test]
+#[ignore = "RFC0013.5 — red until tenant key-prefix scoping is wired"]
+fn rfc0013_5_tenant_isolation_across_prefix() {
+    todo!("RFC0013.5: no cross-tenant key access")
+}
+
+/// Scenario RFC0013.6 — with an object-storage backend, only data/audit/manifest
+/// objects reach the store; the WAL stays on local disk (`CLAUDE.md` §3.4).
+/// See `docs/rfcs/0013-object-storage.md` §5.
+#[test]
+#[ignore = "RFC0013.6 — red until the server wires the object-store backend"]
+fn rfc0013_6_wal_stays_local() {
+    let _ = S3Config::default();
+    todo!("RFC0013.6: WAL frames local; only Parquet/manifest in the store")
+}
+
+/// Scenario RFC0013.7 — an S3-compatible store (`MinIO`) configured via an endpoint
+/// override (RFC 0004) reads and writes exactly as AWS S3.
+/// See `docs/rfcs/0013-object-storage.md` §5.
+#[test]
+#[ignore = "RFC0013.7 — red until the S3 backend honours endpoint overrides"]
+fn rfc0013_7_s3_compatible_endpoint_via_override() {
+    todo!("RFC0013.7: MinIO endpoint override works like AWS S3")
+}
+
+/// Scenario RFC0013.8 — the RFC 0005 §3.9 reader forward-compat contract (absent
+/// columns default, unknown columns ignored) holds over the object store.
+/// See `docs/rfcs/0013-object-storage.md` §5.
+#[test]
+#[ignore = "RFC0013.8 — red until the reader resolves objects through Store"]
+fn rfc0013_8_reader_forward_compat_over_store() {
+    todo!("RFC0013.8: RFC0005 §3.9 holds reading through the object store")
+}

docs/rfcs/0013-object-storage.md

@@ -1,7 +1,7 @@
 ---
 rfc: 0013
 title: Object-storage backend (S3-compatible) for the Parquet store
-status: specified
+status: red
 author: Jens Holdgaard Pedersen <jens@holdgaard.org>
 drafting-assistance: Claude
 created: 2026-06-15
@@ -11,7 +11,7 @@ superseded-by: —
 
 # RFC 0013 — Object-storage backend (S3-compatible) for the Parquet store
 
-> **Status note.** **`specified`** (2026-06-15). The first shipping-milestone
+> **Status note.** **`red`** (2026-06-15). The first shipping-milestone
 > spine: today the writer/reader/compactor/audit-sink address a single local
 > filesystem `bucket_root: &Path`, but `CLAUDE.md` §3.6 declares object
 > storage the source of truth. This RFC abstracts the storage seam behind
@@ -20,15 +20,20 @@ superseded-by: —
 > S3-compatible bucket in production and on local disk in dev/test —
 > **without changing the on-disk layout or a single stored row**.
 >
-> §5 has eight Given/When/Then/And scenarios (RFC0013.1–.8) covering every
-> invariant/hazard the RFC touches — §3.6 (object-storage truth +
-> WAL-stays-local), §3.7 (tenant isolation), the RFC 0009 atomic-publish, and
-> the RFC 0005 §3.9 forward-compat contract — each mapped to a test technique
-> in §6 and testable in principle. The §7 open questions (crate shape, lease
-> mechanism, conditional-PUT portability, multipart threshold, credentials,
-> read cache, migration) are `red`/`accepted`-stage **design + implementation**
-> choices, not `specified`-blocking. Next: `specified → red` (failing test
-> stubs) on the maintainer's go.
+> `red` scaffold landed: the `store` module in `ourios-parquet`
+> (`object_store` now a direct dep) with the `Store` type + the
+> `LocalFileSystem`-backed `local()` constructor wired and the `s3()` /
+> consumer-migration paths stubbed; the eight §5 scenarios are encoded as
+> `#[ignore]`d stubs in `tests/rfc0013_object_store.rs` (so CI stays green
+> while the backend is built). The **crate-shape** open question is resolved
+> — a module, not a new crate (§3.7).
+>
+> **`green` work:** implement the S3 backend (`object_store` `aws` feature) +
+> conditional-PUT atomic publish (RFC0013.3/.4); migrate the
+> writer/reader/compaction/audit consumers from `bucket_root: &Path` onto
+> `Store`; un-`#[ignore]` the §5 stubs one by one. The remaining §7 questions
+> (lease mechanism, conditional-PUT portability, multipart threshold,
+> credentials, read cache, migration) are decided across the `green` PRs.
 
 ## 1. Summary
 
@@ -158,13 +163,17 @@ No change to the RFC 0005 schema, the Parquet bytes, the partition layout, or
 the reader's §3.9 forward-compat contract. This RFC is purely about *where*
 the bytes are stored; an operator's existing data semantics are untouched.
 
-### 3.7 Crate shape
+### 3.7 Crate shape — resolved: a module in `ourios-parquet`
 
-The backend lives behind a `Store` type in `ourios-parquet` (it owns the
-writer/reader), exposed to `ourios-server` and `ourios-querier`. Whether this
-warrants a dedicated `ourios-store` crate (a `CLAUDE.md` §7 architectural
-commitment) or a module is §7's first open question. Configuration (endpoint,
-region, bucket, prefix, credentials) flows through RFC 0004.
+The backend is a `store` **module in `ourios-parquet`** (not a new crate),
+exposing a `Store` type. Resolved at `red` against the less-committing
+option (`CLAUDE.md` §7: a new crate is an architectural commitment): the
+dependency graph confirms it — `ourios-querier`, `-ingester`, and `-server`
+already depend on `ourios-parquet`, so the type is visible to every storage
+consumer without a new crate. If a future consumer needs the store *without*
+the Parquet writer/reader, extracting an `ourios-store` crate is a
+mechanical follow-up. Configuration (endpoint, region, bucket, prefix,
+credentials) flows through RFC 0004.
 
 ## 4. Alternatives considered
 
@@ -276,8 +285,8 @@ The S3 integration lane is `#[ignore]` / feature-gated, so the default
 
 ## 7. Open questions
 
-- [ ] **Crate shape** — a dedicated `ourios-store` crate (§7 architectural
-  commitment) or a module in `ourios-parquet`?
+- [x] **Crate shape** — resolved at `red`: a `store` module in
+  `ourios-parquet` (no new crate; dep-graph-confirmed, §3.7).
 - [ ] **Single-writer lease** — is conditional-PUT contention on the manifest
   sufficient, or is a separate lease object needed (RFC 0009 §7)?
 - [ ] **Conditional-PUT portability** — do `If-None-Match`/`If-Match` cover

docs/roadmap.md

@@ -3,7 +3,8 @@
 > Living document. Refreshed at phase boundaries (§4) and whenever
 > a merged PR materially changes the *current state* in §3.
 > Last updated: **2026-06-15** — RFC 0013 (object storage, S3-compatible)
-> drafted → `specified` (first shipping-milestone spine); RFC 0009 (background
+> drafted → `specified` → `red` (first shipping-milestone spine; `store`
+> module skeleton + §5 stubs landed); RFC 0009 (background
 > compaction) flipped to `validated` (RFC0009.7 D2/D3/B2-post measured on
 > `baseline-8vcpu-32gib`, §9.7); RFC 0005 (Parquet storage) and RFC 0010
 > (audit-stream / drift
@@ -115,7 +116,7 @@ captured by B1/B2 (see `benchmarks.md` §2 / §7).
 | 0009 | Background compaction | **`validated`** — §5 RFC0009.1–.6 pass; RFC0009.7 D2/D3/B2-post measured authoritatively on `baseline-8vcpu-32gib` (§9.7: D3 in 256 MiB–2 GiB band, D2 166.8 MiB/s, B2-post ≈6.1×) |
 | 0010 | Audit-stream / drift queries | **`green`** — all 8 §5 scenarios pass (`crates/ourios-querier/tests/drift.rs`); discharges RFC 0001 H5.3; §9 items are `accepted`-gating; general audit aggregation deferred (§3.2) |
 | 0011 | A1 re-scope | **`accepted`** |
-| 0013 | Object storage (S3-compatible) | `specified` — first shipping-milestone spine; `object_store` behind the `&Path` seam, conditional-PUT atomic publish; §5 has 8 Given/When/Then scenarios; §7 design questions are `red`/`accepted`-stage |
+| 0013 | Object storage (S3-compatible) | `red` — first shipping-milestone spine; `store` module skeleton in `ourios-parquet` (`object_store` direct dep, `local()` wired) + 8 `#[ignore]`d §5 stubs; crate-shape resolved (module, not a crate). `green` = S3 backend + conditional-PUT publish + consumer migration |
 
 **Crates — all ten product crates are implemented** (`ourios-core`,
 `-miner`, `-wal`, `-parquet`, `-ingester`, `-querier`, `-server`,