diff --git a/docs/backlog/P2/B-0001-example-schema-self-reference.md b/docs/backlog/P2/B-0001-example-schema-self-reference.md new file mode 100644 index 00000000..8b1d779c --- /dev/null +++ b/docs/backlog/P2/B-0001-example-schema-self-reference.md @@ -0,0 +1,57 @@ +--- +id: B-0001 +priority: P2 +status: open +title: Example row — self-reference demonstrating the per-row-file schema +tier: research-grade +effort: S +directive: maintainer Otto-181 (BACKLOG split Phase 1a) +created: 2026-04-24 +last_updated: 2026-04-24 +composes_with: [] +tags: [backlog-schema, example, phase-1a] +--- + +# Example row — self-reference demonstrating the per-row-file schema + +This is a placeholder row that exists to: + +1. Exercise the `tools/backlog/generate-index.sh` generator + against a non-empty `docs/backlog/` tree, so drift-CI and + manual `--check` runs have something to validate. +2. Show contributors what the file shape looks like end-to- + end — frontmatter + body. +3. Serve as the first B-NNNN so Phase-2 content migration + starts numbering from B-0002. + +## What this row claims + +Nothing substantive. It's self-referential: it exists +because the generator needs at least one row file to +demonstrate the sort + index emission, and a zero-row +directory would make the new infrastructure harder to +verify. + +When Phase 2 migrates the real `docs/BACKLOG.md` content +into per-row files, this example either stays as the +schema-documentation-example or gets retired (and +recovered via `git log --diff-filter=D` if needed). + +## Future path + +- Phase 1b: when `backlog-index-integrity.yml` workflow + lands, this row confirms the CI drift-check passes on + a non-trivial input. +- Phase 2: migrate existing BACKLOG.md rows starting at + B-0002. +- Phase 3: remove this example when the schema-demo role + is filled by real content, per CLAUDE.md "retire by + deletion" discipline. + +## Cross-references + +- `tools/backlog/README.md` — schema spec. +- `tools/backlog/generate-index.sh` — the generator this + file exercises. +- `docs/research/backlog-split-design-otto-181.md` — full + design spec. diff --git a/docs/backlog/README.md b/docs/backlog/README.md new file mode 100644 index 00000000..a3fd7d75 --- /dev/null +++ b/docs/backlog/README.md @@ -0,0 +1,37 @@ +# docs/backlog/ — per-row backlog files + +Source of truth for individual backlog rows. Each row is one +markdown file with YAML frontmatter. The top-level +`docs/BACKLOG.md` is auto-generated from this directory. + +See `tools/backlog/README.md` for the full schema, scaffolder, +generator, and phase plan. + +## Quick reference + +- **Add a row:** `tools/backlog/new-row.sh --priority P2 --slug your-slug` + (Phase 1b; manual file creation works in the interim). +- **Regenerate index:** `tools/backlog/generate-index.sh`. +- **Check for drift:** `tools/backlog/generate-index.sh --check`. + +## Directory layout + +```text +docs/backlog/ + README.md ← this file + P0/B--.md ← critical / blocking rows + P1/B--.md ← within 2-3 rounds + P2/B--.md ← research-grade + P3/B--.md ← convenience / deferred +``` + +## Current state — Phase 1a + +Tooling + schema landed. One placeholder row (`B-0001`) +exists to exercise the generator against non-empty input; +it is not substantive backlog content. Phase 2 will migrate +the existing single-file `docs/BACKLOG.md` content into per-row +files starting at `B-0002`. Until Phase 2 lands, the single- +file `docs/BACKLOG.md` remains the authoritative source of +substantive backlog rows; this directory + its generator +exist to provide the target structure + schema demonstration. diff --git a/tools/backlog/README.md b/tools/backlog/README.md new file mode 100644 index 00000000..e782afd1 --- /dev/null +++ b/tools/backlog/README.md @@ -0,0 +1,157 @@ +# Backlog tooling — per-row files + generated index + +Companion to `docs/backlog/` (per-row YAML-frontmatter files) +and the generated `docs/BACKLOG.md` index. + +Origin: maintainer Otto-181 directive to split `docs/BACKLOG.md` +to eliminate the positional-append conflict cascade +documented in Otto-171 queue-saturation memory. Design spec: +`docs/research/backlog-split-design-otto-181.md`. + +## Structure + +```text +docs/ + BACKLOG.md ← generated index (DO NOT EDIT) + backlog/ + README.md ← schema + how-to + P0/B--.md ← one file per row + P1/B--.md + P2/B--.md + P3/B--.md +tools/ + backlog/ + README.md ← this file + generate-index.sh ← regenerates docs/BACKLOG.md + new-row.sh ← scaffolds a new row file (Phase 1b) +``` + +## Per-row file schema + +Each row is one markdown file with YAML frontmatter: + +```markdown +--- +id: B-0042 +priority: P2 +status: open +title: Server Meshing and SpacetimeDB deep research +tier: research-grade +effort: L +directive: maintainer Otto-180 +created: 2026-04-24 +last_updated: 2026-04-24 +composes_with: + - B-0031 + - B-0038 +tags: [game-industry, sharding, multi-node] +--- + +# Server Meshing + SpacetimeDB — deep research on cross-shard communication patterns + +...full row content as markdown... +``` + +## Frontmatter fields + +| Field | Required | Type | Notes | +|----------------|----------|--------------|-------| +| `id` | yes | `B-NNNN` | Zero-padded 4 digits, sequential. Factory-wide unique. | +| `priority` | yes | `P0..P3` | Directory must match (`P2` row → `docs/backlog/P2/`). | +| `status` | yes | enum | `open` / `closed` / `superseded-by-B-NNNN` / `deferred` | +| `title` | yes | string | Short index-display title. | +| `tier` | no | string | Free-form; e.g. `research-grade`, `active-substrate`. | +| `effort` | no | `S` / `M` / `L` | Size estimate. | +| `directive` | no | string | Origin reference; e.g. `maintainer Otto-180`, `Amara 18th ferry #4`. | +| `created` | yes | YYYY-MM-DD | First-landing date. | +| `last_updated` | yes | YYYY-MM-DD | Updated on every content edit. | +| `composes_with`| no | list of `B-NNNN` | Cross-references; strict-lint-candidate Phase-2+. | +| `tags` | no | list of string | Free-form. Examples: `multi-node`, `dst`, `ui-rename`. | + +## Adding a new row + +Phase 1a (current): create the file manually at +`docs/backlog/P/B-NNNN-.md` with the frontmatter +below. Phase 1b will ship a `new-row.sh` scaffolder that +auto-assigns `NNNN` and pre-fills the frontmatter template; +this README is forward-referencing that scaffolder but +neither the script nor its invocation is available until +Phase 1b lands. + +Phase 1b target usage (not functional yet): + +```bash +tools/backlog/new-row.sh --priority P2 --slug server-meshing-research +``` + +Will create `docs/backlog/P2/B-NNNN-server-meshing-research.md` +with pre-filled frontmatter. `NNNN` auto-assigned as the +next unused integer across all priorities. + +Edit the file to add your content + fill optional +frontmatter. Commit the new file. The generator +regenerates `docs/BACKLOG.md` via +`tools/backlog/generate-index.sh` manually until Phase 1b +adds the pre-commit hook. + +## Regenerating the index + +```bash +tools/backlog/generate-index.sh +``` + +Walks `docs/backlog/**/*.md`, parses frontmatter via an +inline awk parser (no external `yq` dependency), emits +`docs/BACKLOG.md` sorted by (priority ascending, id +ascending). Phase 1a uses pure awk to minimize toolchain +surface; if `yq`-style nested-key queries become necessary, +that's a Phase 1b upgrade. + +## CI drift check + +`.github/workflows/backlog-index-integrity.yml` (Phase 1b) +will fail if the committed `docs/BACKLOG.md` doesn't +match the output of `generate-index.sh` run against the +committed row files. Same pattern as +`memory-index-integrity.yml`. + +## Retirement + +Per `CLAUDE.md` "honor those that came before — retired +SKILL.md files retire by plain deletion, recoverable +from git history" discipline: retired rows delete the +file. `git log --diff-filter=D -- docs/backlog/` surfaces +deleted rows for recovery. The `status: superseded-by-B-NNNN` +frontmatter is for rows that are retired-but-still- +referenced; once no live row references the retired ID, +delete the file. + +## Phase status + +- **Phase 1a (this PR):** generator + schema + placeholder + directory. No content migration yet. +- **Phase 1b:** CI drift workflow + `new-row.sh` + scaffolder. +- **Phase 2:** content split mega-PR — reads current + `docs/BACKLOG.md`, generates per-row files, regenerates + index. One-time conflict cascade cost. Recommended to + drain queue to <10 BACKLOG-touching PRs first. +- **Phase 3:** convention updates in `CONTRIBUTING.md` / + `AGENTS.md`. + +## Cross-references + +- `docs/research/backlog-split-design-otto-181.md` — full + design spec + 6 open questions the maintainer's call on (some + answered by reasonable defaults in this phase). +- Hot-file-detector tooling (unmerged at the time of + this Phase-1a PR; recovery path: `git log + --diff-filter=A --all -- tools/hygiene/` if it lands + later) — the detector flagged `docs/BACKLOG.md` as + the repo's top hotspot and named "BACKLOG-per-swim- + lane split" as a remediation option. The design + rationale for this PR does not depend on that + script being present in tree; the driver was + maintainer Otto-181 directly. +- `.github/workflows/memory-index-integrity.yml` — + precedent for the drift-CI pattern. diff --git a/tools/backlog/generate-index.sh b/tools/backlog/generate-index.sh new file mode 100755 index 00000000..8c41b0f8 --- /dev/null +++ b/tools/backlog/generate-index.sh @@ -0,0 +1,196 @@ +#!/usr/bin/env bash +# tools/backlog/generate-index.sh +# +# Regenerates docs/BACKLOG.md from per-row files at +# docs/backlog/P/B--.md. Walks those files, +# parses YAML frontmatter, emits a short-pointer index +# sorted by (priority, id). +# +# Origin: maintainer Otto-181 directive to split BACKLOG.md. +# See docs/research/backlog-split-design-otto-181.md for the +# design spec. +# +# Usage: +# tools/backlog/generate-index.sh # writes docs/BACKLOG.md +# tools/backlog/generate-index.sh --check # exits 2 if drift vs committed +# tools/backlog/generate-index.sh --stdout # prints to stdout, no write +# +# Exit codes: +# 0 success +# 1 environment / dependency error +# 2 drift detected (--check mode only) +# +# Dependencies: bash 4+, POSIX awk, sort, diff, find, mktemp. +# No external `yq` required; inline awk parser handles the +# flat frontmatter schema. `yq` integration is a Phase 1b +# upgrade path if nested frontmatter queries become needed. + +set -euo pipefail + +REPO_ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)" +BACKLOG_DIR="${REPO_ROOT}/docs/backlog" +INDEX_PATH="${REPO_ROOT}/docs/BACKLOG.md" + +mode="write" +while (( $# > 0 )); do + case "$1" in + --check) mode="check"; shift ;; + --stdout) mode="stdout"; shift ;; + *) echo "unknown arg: $1" >&2; exit 1 ;; + esac +done + +# Extract a frontmatter field from a markdown file using +# awk. Simple: reads the first YAML block (between two +# `---` lines) and grabs `field: value`. Doesn't handle +# nested structures — that's fine, our schema is flat. +extract_field() { + local file="$1" + local field="$2" + awk -v field="$field" ' + BEGIN { state = 0; found = "" } # 0=before, 1=inside, 2=after + /^---$/ { + if (state == 0) { state = 1; next } + if (state == 1) { state = 2; next } # Codex P1: stop after closing + } + state == 1 && $1 == field ":" { + sub(/^[^:]+:[[:space:]]*/, "") + # Copilot P1: POSIX-awk compatibility — use octal \047 (single-quote) + # rather than hex \x27 which is not portable across POSIX awk + # implementations (notably macOS awk). + gsub(/^"|"$|^[[:space:]]*\047|\047[[:space:]]*$/, "") # Codex P1: handle both " and '\'' + # Codex P2: trim trailing whitespace so `status: closed ` still + # matches the `closed)` case in generate(). + sub(/[[:space:]]+$/, "") + found = $0 + } + END { print found } + ' "$file" +} + +# Build index content. +# Output pattern per priority section: +# +# ## P2 — research-grade +# +# - [ ] **[B-0042](backlog/P2/B-0042-slug.md)** Title from frontmatter +# - ... + +generate() { + cat <<'HEADER' +# Backlog Index + + + +_Each entry below is a link to a per-row file under +`docs/backlog/`. Entries with `- [ ]` are open; `- [x]` +are closed (status: closed in frontmatter)._ + +HEADER + + # Stable section order + for tier in P0 P1 P2 P3; do + section_dir="${BACKLOG_DIR}/${tier}" + [ -d "$section_dir" ] || continue + # Codex P2: iterate via NUL-delimited find+sort | while read to avoid + # shell word-splitting on whitespace in filenames. Handles paths + # containing spaces without breaking into multiple tokens. + has_files=0 + section_label="" + case "$tier" in + P0) section_label="## P0 — critical / blocking" ;; + P1) section_label="## P1 — within 2-3 rounds" ;; + P2) section_label="## P2 — research-grade" ;; + P3) section_label="## P3 — convenience / deferred" ;; + esac + + while IFS= read -r -d '' file; do + if [ "$has_files" -eq 0 ]; then + echo "" + echo "$section_label" + echo "" + has_files=1 + fi + + link_path="backlog/${tier}/$(basename "$file")" + + id=$(extract_field "$file" "id") + status=$(extract_field "$file" "status") + title=$(extract_field "$file" "title") + + case "$status" in + closed) checkbox="[x]" ;; + superseded-by-*) checkbox="[x]" ;; + *) checkbox="[ ]" ;; + esac + + printf -- "- %s **[%s](%s)** %s\n" \ + "$checkbox" "$id" "$link_path" "$title" + done < <(find "$section_dir" -maxdepth 1 -name 'B-*.md' -type f -print0 2>/dev/null | sort -z) + done + + cat <<'FOOTER' + + +FOOTER +} + +# Generate into a temp file in the same directory as the +# target so mv is a same-filesystem atomic rename. Copilot +# P1: /tmp may be a different filesystem → mv falls back to +# copy+unlink, not atomic. +tmpout="$(mktemp "${INDEX_PATH}.tmp.XXXXXX")" +trap 'rm -f "$tmpout"' EXIT +generate > "$tmpout" + +case "$mode" in + stdout) + cat "$tmpout" + ;; + check) + if [ ! -f "$INDEX_PATH" ]; then + echo "drift: $INDEX_PATH does not exist" >&2 + exit 2 + fi + if ! diff -q "$tmpout" "$INDEX_PATH" >/dev/null; then + echo "drift: $INDEX_PATH differs from generator output" >&2 + echo "diff:" >&2 + diff "$tmpout" "$INDEX_PATH" >&2 || true + exit 2 + fi + echo "ok: $INDEX_PATH matches generator output" + ;; + write) + # Phase-1a safety guard: refuse to overwrite an + # existing BACKLOG.md that has substantial content + # (i.e. the pre-split monolithic backlog that + # Phase 2 will migrate). Until Phase 2 migrates + # content into per-row files, generator --write + # would destroy the real backlog. + # + # Override with BACKLOG_WRITE_FORCE=1 when Phase 2 + # migration PR intentionally regenerates the index. + if [ -f "$INDEX_PATH" ] && [ "${BACKLOG_WRITE_FORCE:-0}" != "1" ]; then + existing_lines=$(wc -l < "$INDEX_PATH" | tr -d ' ') + if [ "$existing_lines" -gt 50 ]; then + cat >&2 <<'GUARDMSG' +generate-index.sh: refusing to overwrite existing +docs/BACKLOG.md — file has substantial content +(Phase-1a guard). Phase 2 content-migration PR should +set BACKLOG_WRITE_FORCE=1 to authorize the overwrite +once per-row files have been populated. + +Use --stdout to preview, --check to compare against +committed, or set BACKLOG_WRITE_FORCE=1 to force. +GUARDMSG + exit 1 + fi + fi + mv "$tmpout" "$INDEX_PATH" + trap - EXIT + echo "wrote $INDEX_PATH" + ;; +esac