feat(miner): implement tokenize() — RFC0001.3 (first red → green step)

[jensholdgaard]

Summary

First Red → Green step for RFC 0001. Implements the tokenizer per §6.2 step 1 and turns the RFC0001.3 stub from todo!() into a real test.

This is the simplest possible Red-to-Green move — pure function, zero state, no dependencies. Goal of the PR is to validate the workflow end-to-end before tackling stubs that need miner state.

What lands

File	Change
crates/ourios-miner/src/lib.rs	Add pub mod tokenize;
crates/ourios-miner/src/tokenize.rs	New — Tokenized<'a> struct + tokenize(line: &str) -> Tokenized<'_> per §6.2 step 1
crates/ourios-miner/tests/rfc_internal.rs	Drop #[ignore] on RFC0001.3; replace todo!() with assertions matching the §5 scenario verbatim

Implementation

Borrowed Tokenized<'a>:

pub struct Tokenized<'a> {
    pub tokens: Vec<&'a str>,
    pub separators: Vec<&'a str>,
}

No allocation per token / separator (matters at the §6.2 hot-path budget). tokenize walks char_indices() once in O(n), splitting on every codepoint matching char::is_whitespace (ASCII space / tab / CR / LF plus the broader Unicode whitespace classes — U+0085, U+00A0, U+1680, U+2000–U+200A, U+2028, U+2029, U+202F, U+205F, U+3000). Punctuation (=, :, ,, ;, [, ], (, )) stays inside the token per RFC 0001 §6.2's masking-layer-handles-structured-separators contract.

The separators contract: separators.len() == tokens.len() + 1 always. separators[0] is leading whitespace, separators[i + 1] is the run between token i and token i + 1 (empty after the last token; or the full input when no tokens exist).

Test

Asserts exactly what the RFC0001.3 §5 scenario asserts — no more (per docs/verification.md §2.4 "Acceptance criteria are the normative tests an RFC promises will exist"):

• The exact key=value, other=42 example from the scenario body
• The separators contract (len == tokens.len() + 1)
• Each punctuation char from the "And no token boundary at..." clause stays inside a single token
• Sanity: ASCII whitespace splits, U+00A0 (non-breaking space) splits

Edge cases (empty string, all-whitespace, multi-codepoint runs) are valuable but aren't part of RFC0001.3's promise — they belong to a follow-up regression test, not the §5 scenario.

Lifecycle (per docs/verification.md §3 two-loop spec)

• Outer loop (cargo test --all-features): 1 passed, 28 ignored — RFC0001.3 joins the default run; the other 28 stubs stay ignored. CI green.
• Inner loop (cargo test --no-fail-fast -- --ignored): 28 failed (12 + 9 + 7) — count drops by one as expected. RFC0001.3 is "filtered out" of the ignored run because it's no longer ignored.

RFC 0001 stays at status: red until the last #[ignore] comes off. This is 1 of 29.

Verification (CLAUDE.md §6.6)

• cargo fmt --all --check ✓
• cargo clippy --all-targets --all-features -- -D warnings ✓ (pedantic clean — #[must_use] on tokenize, full doc-comments on the public surface)
• cargo test --all-features → 1 passed, 28 ignored ✓
• cargo test --all-features --no-fail-fast -- --ignored → 28 failed (one fewer than before; one stub successfully promoted) ✓
• mdbook build ✓

History note

Force-pushed once on this branch to drop a Cargo.lock change that snuck into the original commit. The lockfile update is correct policy (Ourios is a binary, .gitignore says it should be tracked) but belongs in its own focused PR rather than tagging along with this feature work. Cargo.lock is now back to untracked locally and will land in a separate cleanup PR.

Checklist

• cargo fmt clean
• cargo clippy clean (no new warnings)
• Tests added/updated (RFC0001.3 stub → real test)
• Docs / CHANGELOG.md updated (RFC frontmatter unchanged at red; git-cliff regenerates CHANGELOG.md at release)
• RFC linked: this PR is the first Red → Green promotion for RFC 0001 (§5 scenario RFC0001.3)

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Implements the RFC 0001 tokenizer step (§6.2 step 1) in ourios-miner, promotes the RFC0001.3 test stub from ignored todo!() to a real test, and adds the workspace lockfile for reproducible builds.

Changes:

• Add tokenize module with Tokenized<'a> and tokenize(&str) (Unicode-whitespace splitting, punctuation preserved in tokens).
• Update RFC0001.3 test to run by default and assert expected tokenization behavior.
• Add Cargo.lock to the repo (consistent with the project’s reproducible-build policy).

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
crates/ourios-miner/src/lib.rs	Exposes the new tokenize module publicly.
crates/ourios-miner/src/tokenize.rs	Adds the tokenizer implementation and its public result type.
crates/ourios-miner/tests/rfc_internal.rs	Converts RFC0001.3 from ignored stub into an asserting test.
Cargo.lock	Introduces tracked lockfile for the workspace.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.