diff --git a/AGENTS.md b/AGENTS.md index 5b371eb0..bfd3f27e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,170 +1,87 @@ -# AGENTS.md +# AGENTS.md - Consumer Repository Context -This file is the “mission briefing” for coding agents working in this repository. It is project-specific. For workflow/consumer-repo rules, also read **CLAUDE.md**. +> Read this before changing workflows, prompts, or synced automation files. -## What this project is +## This Is A Consumer Repo -Counter_Risk replaces a spreadsheet-based monthly counterparty risk process. The end state is: +Most workflow logic for this repository lives in `stranske/Workflows`. The consumer repo should only carry repo-specific configuration unless it has an explicitly documented exception. -- A maintainable program that ingests monthly inputs and produces the same outputs the spreadsheet produced (plus new output types later). -- A button-driven “Runner” so **non-technical operators** can run the process without a CLI and without repo access. -- A deterministic, testable pipeline with parity checks against “golden” reference artifacts. +## Source Of Truth -## Non-technical operator workflow (no CLI) +For infrastructure work, follow this order: -Operators should be able to: +1. `stranske/Workflows` root docs: `README.md`, `docs/WORKFLOW_GUIDE.md`, `docs/ci/WORKFLOWS.md` +2. `stranske/Workflows/docs/INTEGRATION_GUIDE.md` and `docs/ops/CONSUMER_REPO_MAINTENANCE.md` +3. The consumer sync source in `stranske/Workflows/templates/consumer-repo/` +4. This repo's local repo-specific files -1. Pick an **as-of date** (and optionally a variant: All Programs / Ex Trend / Trend). -2. Click **Run**. -3. Receive an output folder containing: - - Updated historical workbooks - - Updated monthly PPT (and optionally a static distribution PPT/PDF) - - A manifest and a short data quality summary +If a file is synced from Workflows, fix it in Workflows first. -Operators never need to see GitHub, Python, or a terminal. +## Current Consumer Defaults -## Maintainer workflow (agent-driven) +- First-party consumers currently reference reusable workflows with `@main`. Match that unless you are intentionally pinning to an exact commit SHA for a controlled reason. +- `ci.yml` and `autofix-versions.env` are repo-specific. +- `pr-00-gate.yml` is a create-only standard file. Keep it aligned with the standard gate unless this repo has a documented reason to diverge. +- Synced workflows, prompts, scripts, and consumer docs are managed through `.github/sync-manifest.yml` in Workflows. -Maintainers primarily work through issues and PRs: +## Commonly Managed Files -- Issues should follow `docs/AGENT_ISSUE_FORMAT.md` -- Agent automation is driven by labels (see `WORKFLOW_USER_GUIDE.md`) -- Keepalive expects PR bodies to contain task checkboxes and acceptance criteria checkboxes +Usually edit locally only when the file is repo-specific: -When writing issues, prefer: -- Small scope (one PR per issue) -- Objective acceptance criteria (range-level comparisons, file existence checks, deterministic metrics) -- Explicit file/module names when known +| File | Default owner | Notes | +|------|---------------|-------| +| `ci.yml` | Consumer repo | Repo-specific CI wiring | +| `autofix-versions.env` | Consumer repo | Repo-specific dependency pins | +| `pr-00-gate.yml` | Consumer repo, but should match Workflows standard by default | Create-only standard file | +| `agents-*.yml` | Workflows | Fix in Workflows, not here | +| `autofix.yml` | Workflows | Fix in Workflows, not here | +| `.github/codex/` prompts | Workflows | Fix in Workflows, not here | +| synced scripts/docs | Workflows | Fix in Workflows, not here | -## Source artifacts and parity strategy +## Current Workflow Surfaces -The pipeline must reproduce current outputs using the provided reference artifacts (fixtures). The project should maintain: +The current consumer default automation surface is centered on: -- A `tests/fixtures/` directory for **sanitized** reference files (Excel/PPT) used in regression tests. -- Range-level comparisons for key tables and time-series append logic. -- Structural checks for PPT (slide count, expected shapes present). -- A manifest file in each run output folder recording: - - input hashes - - output paths - - warnings and reconciliation gaps - - top exposures / top movers summaries +- `agents-issue-intake.yml` +- `agents-80-pr-event-hub.yml` +- `agents-81-gate-followups.yml` +- `agents-verifier.yml` +- `autofix.yml` +- `ci.yml` +- `pr-00-gate.yml` -Never silently drop exposures. If a new counterparty appears and there is no matching historical series header, the run must warn or fail based on policy. +Legacy compatibility workflows may still exist during migrations. Do not assume an older filename is canonical without checking the Workflows docs first. -## Key technical constraints and “gotchas” +## Cross-Repo Policy -1. Date semantics must be explicit: - - `as_of_date` is the effective reporting date used on historical workbook x-axes. - - `run_date` is when the pipeline was executed. +Before editing local workflow infrastructure, ask: -2. Name normalization must be deterministic: - - Trim/canonicalize whitespace. - - Apply alias mapping registry first, then fallback mapping. - - Treat Excel/PPT header series names as brittle (spaces and punctuation matter). +**Does this work belong in `stranske/Workflows` instead?** -3. PowerPoint is hybrid: - - Some slides are pasted screenshots (replace pictures). - - Other slides are linked charts (OLE links) that may require refresh (COM automation preferred + fallback instruction file). +The answer is usually yes if the change affects any of these: -4. Workflows are synced from stranske/Workflows: - - Do not edit `.github/workflows/**` unless explicitly operating under a high-privilege environment and the task requires it. - - If workflow changes are needed, fix them in **stranske/Workflows** then sync. +- reusable workflows +- agent prompts or routing +- keepalive/autofix/verifier behavior +- synced workflow files +- synced scripts or docs -## CI test situation (what runs when) +If yes: -This repo is tuned so PR Gate stays reasonably fast by skipping the most expensive suites. +1. Make the source-of-truth change in `stranske/Workflows` +2. Update the sync manifest if a consumer-facing file changed +3. Sync or manually align this repo afterward -### PR Gate +## Useful References -PR Gate uses `.github/workflows/pr-00-gate.yml`: +- `stranske/Workflows/README.md` +- `stranske/Workflows/docs/WORKFLOW_GUIDE.md` +- `stranske/Workflows/docs/ci/WORKFLOWS.md` +- `stranske/Workflows/docs/INTEGRATION_GUIDE.md` +- `stranske/Workflows/docs/ops/CONSUMER_REPO_MAINTENANCE.md` +- `stranske/Workflows/docs/keepalive/Agents.md` +- `stranske/Travel-Plan-Permission` as a reference consumer -- Runs pytest **in parallel** with xdist (`-n auto --dist load`). -- Runs pytest **without coverage**. -- Skips `tests/integration/` and `tests/integrations/`. -- Skips `release`- and `slow`-marked tests (`pytest_markers: "not release and not slow"`). +## Agent-Specific Note -### Main CI (push to `main`) - -Main CI uses `.github/workflows/ci.yml`: - -- Runs pytest **with coverage** and enforces `coverage-min`. -- Runs pytest **in parallel** with xdist (`--dist loadscope`). -- Runs **all** tests including `slow`-marked ones. -- This is where heavyweight pipeline and spec-fixture tests are validated. - -### Slow tests - -Tests that take a long time because they parse real Excel workbooks or run -full pipeline orchestration are tagged `@pytest.mark.slow`. The Gate -excludes them so PR feedback stays under five minutes; Main CI runs them on -every merge. - -Currently slow-marked tests: - -| File | Why it's slow | How it's marked | -|------|---------------|-----------------| -| `tests/pipeline/test_run_pipeline.py` | Each `test_run_pipeline_*` calls `run_pipeline()` with real Excel fixtures (100-170 s per test) | Auto-marked by `tests/pipeline/conftest.py` hook | -| `tests/spec/test_macro_spec_fixtures.py` | Session-scoped fixture parses three NISA workbooks (~85 s) | Module-level `pytestmark = pytest.mark.slow` | - -#### When to mark a test slow - -Mark a new test `@pytest.mark.slow` if it: - -- Calls `run_pipeline()` or `run_pipeline_with_config()` with real fixture files. -- Relies on a session-scoped fixture that parses large workbooks. -- Takes more than ~30 s on CI in isolation. - -You can apply the marker with a decorator (`@pytest.mark.slow`) or add the -test pattern to the `pytest_collection_modifyitems` hook in the appropriate -`conftest.py`. - -#### Running slow tests locally - -```bash -# Run only slow tests -pytest -m slow - -# Run everything (same as Main CI) -pytest - -# Run without slow tests (same as Gate) -pytest -m "not slow" -``` - -### Release tests (nightly or label) - -The slowest release/packaging checks are isolated behind the `release` marker and can be triggered via: - -- Nightly schedule (`.github/workflows/release-e2e.yml`) -- PR label: `run-release` - -### Agent guidance - -- When a PR touches release/packaging mechanics (e.g., `release.spec`, `pyinstaller_runtime_hook.py`, templates/config bundling), make sure `release` tests run by applying the `run-release` label. -- When adding tests that parse real workbooks or run full pipeline orchestration, mark them `@pytest.mark.slow` so the Gate stays fast. Do **not** add heavyweight tests without the marker — it will regress Gate duration for every PR. - -## Agent guardrails (must follow) - -- Also read: `.github/codex/AGENT_INSTRUCTIONS.md` -- Do not modify protected workflow/security files unless explicitly allowed. -- Do not introduce secrets into logs/files. -- Keep changes scoped to the PR’s tasks. - -## Definition of done (per milestone) - -Milestone 1 (Parity using MOSERS-formatted inputs): -- Parse MOSERS-format workbooks into canonical tables. -- Update historical workbooks by appending a new row. -- Update monthly PPT (screenshots replaced + links refreshed or static distribution output). -- Produce run manifest + data quality summary. - -Milestone 2 (Remove VBA/manual prep): -- Ingest raw NISA monthly inputs and generate MOSERS-format structures without running macros. -- Automate cash ingestion (structured source preferred; PDF fallback + overrides). -- Add prior-month futures delta/sign-flip logic for Trend. - -## Where to look for keepalive system docs - -- `WORKFLOW_USER_GUIDE.md` (labels, triggers, operator usage) -- `docs/KEEPALIVE_TROUBLESHOOTING.md` (deep debugging and known failure modes) -- `CLAUDE.md` (consumer repo boundaries and “what not to edit”) +This file is the agent-generic contract. Keep it materially aligned with `CLAUDE.md`; differences between the two should only be agent-specific execution notes, not different repository rules. diff --git a/CLAUDE.md b/CLAUDE.md index 685aac40..40def435 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,171 +1,87 @@ # CLAUDE.md - Consumer Repository Context -> **READ THIS FIRST** before making workflow changes. +> Read this before changing workflows, prompts, or synced automation files. -## This is a Consumer Repo +## This Is A Consumer Repo -This repository uses the **stranske/Workflows** workflow library. Most workflow logic lives there, not here. +Most workflow logic for this repository lives in `stranske/Workflows`. The consumer repo should only carry repo-specific configuration unless it has an explicitly documented exception. -**DO NOT** modify agent workflow files directly - they are synced from Workflows and will be overwritten. +## Source Of Truth -## Architecture +For infrastructure work, follow this order: -``` -stranske/Workflows (central library) - │ - │ reusable workflows called via: - │ uses: stranske/Workflows/.github/workflows/reusable-*.yml@v1 - │ - ▼ -This Repo (consumer) - .github/workflows/ - ├── agents-*.yml → SYNCED from Workflows (don't edit) - ├── autofix.yml → SYNCED from Workflows (don't edit) - ├── pr-00-gate.yml → SYNCED but customizable - ├── ci.yml → REPO-SPECIFIC (can edit) - └── autofix-versions.env → REPO-SPECIFIC (can edit) -``` +1. `stranske/Workflows` root docs: `README.md`, `docs/WORKFLOW_GUIDE.md`, `docs/ci/WORKFLOWS.md` +2. `stranske/Workflows/docs/INTEGRATION_GUIDE.md` and `docs/ops/CONSUMER_REPO_MAINTENANCE.md` +3. The consumer sync source in `stranske/Workflows/templates/consumer-repo/` +4. This repo's local repo-specific files -## Which Files Can Be Edited +If a file is synced from Workflows, fix it in Workflows first. -| File | Editable? | Notes | -|------|-----------|-------| -| `ci.yml` | ✅ Yes | Repo-specific CI configuration | -| `autofix-versions.env` | ✅ Yes | Repo-specific dependency versions | -| `pr-00-gate.yml` | ⚠️ Careful | Synced, but can customize if needed | -| `agents-*.yml` | ❌ No | Synced from Workflows | -| `autofix.yml` | ❌ No | Synced from Workflows | +## Current Consumer Defaults -## Keepalive System +- First-party consumers currently reference reusable workflows with `@main`. Match that unless you are intentionally pinning to an exact commit SHA for a controlled reason. +- `ci.yml` and `autofix-versions.env` are repo-specific. +- `pr-00-gate.yml` is a create-only standard file. Keep it aligned with the standard gate unless this repo has a documented reason to diverge. +- Synced workflows, prompts, scripts, and consumer docs are managed through `.github/sync-manifest.yml` in Workflows. -When an issue is labeled `agent:codex`: -1. `agents-63-issue-intake.yml` creates a PR with bootstrap file -2. `agents-keepalive-loop.yml` runs Codex in iterations -3. Codex works through tasks in PR body until all complete +## Commonly Managed Files -**Key prompts** (in `.github/codex/prompts/`): -- `keepalive_next_task.md` - Normal work instructions -- `fix_ci_failures.md` - CI fix instructions +Usually edit locally only when the file is repo-specific: -## Common Issues +| File | Default owner | Notes | +|------|---------------|-------| +| `ci.yml` | Consumer repo | Repo-specific CI wiring | +| `autofix-versions.env` | Consumer repo | Repo-specific dependency pins | +| `pr-00-gate.yml` | Consumer repo, but should match Workflows standard by default | Create-only standard file | +| `agents-*.yml` | Workflows | Fix in Workflows, not here | +| `autofix.yml` | Workflows | Fix in Workflows, not here | +| `.github/codex/` prompts | Workflows | Fix in Workflows, not here | +| synced scripts/docs | Workflows | Fix in Workflows, not here | -### Workflow fails with "workflow file issue" -- A reusable workflow is being called that doesn't exist -- Check Workflows repo has the required `reusable-*.yml` file -- Consumer workflows call INTO Workflows repo, not local files +## Current Workflow Surfaces -### Workflow startup_failure (zero jobs) +The current consumer default automation surface is centered on: -**MANDATORY**: Before theorizing, find what changed: +- `agents-issue-intake.yml` +- `agents-80-pr-event-hub.yml` +- `agents-81-gate-followups.yml` +- `agents-verifier.yml` +- `autofix.yml` +- `ci.yml` +- `pr-00-gate.yml` -```bash -# 1. Find boundary between success and failure -gh run list --repo owner/repo --workflow="workflow.yml" --limit 100 --json databaseId,conclusion,createdAt \ - --jq '[.[] | select(.conclusion == "success" or .conclusion == "startup_failure")] | group_by(.conclusion) | .[] | {conclusion: .[0].conclusion, first: .[-1].createdAt, last: .[0].createdAt}' +Legacy compatibility workflows may still exist during migrations. Do not assume an older filename is canonical without checking the Workflows docs first. -# 2. Find commits between last success and first failure -git log --oneline --since="LAST_SUCCESS_DATE" --until="FIRST_FAILURE_DATE" -- path/to/workflow.yml +## Cross-Repo Policy -# 3. Fix ONLY what that diff changed -git show COMMIT_SHA -- path/to/workflow.yml -``` +Before editing local workflow infrastructure, ask: -Common causes: -- **Top-level `permissions:` on hybrid workflows** (workflow_call + workflow_dispatch) -- Invalid YAML syntax -- Invalid permission scopes +**Does this work belong in `stranske/Workflows` instead?** -See `docs/INTEGRATION_GUIDE.md` in Workflows repo. +The answer is usually yes if the change affects any of these: -### Agent not picking up changes -- Check PR has `agent:codex` label -- Check Gate workflow passed (green checkmark) -- Check PR body has unchecked tasks +- reusable workflows +- agent prompts or routing +- keepalive/autofix/verifier behavior +- synced workflow files +- synced scripts or docs -### Need to update agent workflows -- DON'T edit locally - changes will be overwritten -- Fix in Workflows repo → sync will propagate here -- Or request manual sync: `gh workflow run maint-68-sync-consumer-repos.yml --repo stranske/Workflows` +If yes: -## Reference Implementation +1. Make the source-of-truth change in `stranske/Workflows` +2. Update the sync manifest if a consumer-facing file changed +3. Sync or manually align this repo afterward -**Travel-Plan-Permission** is the reference consumer repo. When debugging: -1. Check if it works there first -2. Compare this repo's `.github/` with Travel-Plan-Permission -3. Look for missing files or differences +## Useful References -## Workflows Documentation +- `stranske/Workflows/README.md` +- `stranske/Workflows/docs/WORKFLOW_GUIDE.md` +- `stranske/Workflows/docs/ci/WORKFLOWS.md` +- `stranske/Workflows/docs/INTEGRATION_GUIDE.md` +- `stranske/Workflows/docs/ops/CONSUMER_REPO_MAINTENANCE.md` +- `stranske/Workflows/docs/keepalive/Agents.md` +- `stranske/Travel-Plan-Permission` as a reference consumer -For detailed docs, see **stranske/Workflows**: -- `docs/INTEGRATION_GUIDE.md` - How consumer repos work -- `docs/keepalive/GoalsAndPlumbing.md` - Keepalive system design -- `docs/keepalive/SETUP_CHECKLIST.md` - Required files and secrets +## Claude-Specific Note -## Quick Debug Commands - -```bash -# Compare workflows with reference repo -diff .github/workflows/autofix.yml \ - <(gh api repos/stranske/Travel-Plan-Permission/contents/.github/workflows/autofix.yml --jq '.content' | base64 -d) - -# Check for missing files -gh api repos/stranske/Travel-Plan-Permission/contents/.github/workflows --jq '.[].name' | sort > /tmp/tpp.txt -ls .github/workflows/ | sort > /tmp/here.txt -diff /tmp/tpp.txt /tmp/here.txt - -# Trigger workflow sync from Workflows repo -gh workflow run maint-68-sync-consumer-repos.yml --repo stranske/Workflows -``` - ---- - -## �� POLICY: Cross-Repo Work - -> **CRITICAL**: Read this before ANY work that might affect the Workflows repo. - -### Policy Checkpoint Trigger - -When creating a todo list, ask: - -**"Does this work need changes in stranske/Workflows?"** - -Signs that you need Workflows changes: -- Adding a new agent capability -- Modifying how keepalive/autofix/verifier works -- Needing a new Codex prompt -- Bug in a reusable workflow - -### If YES → Work in Workflows First - -1. Clone/checkout stranske/Workflows -2. Make changes there (following Workflows CLAUDE.md policy) -3. Ensure sync manifest is updated -4. Trigger sync to propagate to this repo -5. Then verify in this repo - -**DO NOT** try to fix Workflows issues by editing local files - they will be overwritten on next sync. - -### Add Policy Verification Todo - -When your todo list involves cross-repo coordination, add as **FINAL** item: - -``` -✅ Verify cross-repo policy compliance: - - [ ] Changes made in correct repo (Workflows vs Consumer) - - [ ] Sync triggered if needed - - [ ] Both repos have passing CI -``` - -### Quick Commands - -```bash -# Check if a file is synced (compare to template) -diff .github/workflows/agents-keepalive-loop.yml \ - <(gh api repos/stranske/Workflows/contents/templates/consumer-repo/.github/workflows/agents-keepalive-loop.yml --jq '.content' | base64 -d) - -# Trigger sync from Workflows -gh workflow run maint-68-sync-consumer-repos.yml --repo stranske/Workflows -f repos="${{ github.repository }}" - -# Check sync manifest for what SHOULD be here -gh api repos/stranske/Workflows/contents/.github/sync-manifest.yml --jq '.content' | base64 -d -``` +Keep this file materially aligned with `AGENTS.md`. Differences between the two should only be agent-specific execution notes, not different repository rules. diff --git a/WORKFLOW_USER_GUIDE.md b/WORKFLOW_USER_GUIDE.md index 64982f7a..2ebf1931 100644 --- a/WORKFLOW_USER_GUIDE.md +++ b/WORKFLOW_USER_GUIDE.md @@ -754,17 +754,17 @@ The Workflows repository includes maintenance workflows that handle sync, update --- -### `maint-61-create-floating-v1-tag.yml` - Update Floating Tag -**Purpose:** Updates `v1` tag to latest `v1.x.x` release +### `maint-73-refresh-reusable-tags.yml` - Legacy Tag Refresh Notice +**Purpose:** Historical maintenance workflow for floating-tag management **Trigger:** After new release created **What It Does:** -- Finds latest v1 series release -- Updates `v1` tag to point to it -- Enables consumers to use `@v1` for latest +- Records that first-party consumers now standardize on `@main` +- Leaves any historical floating-tag maintenance to explicit migration work +- Does not change the current first-party consumer default -**Use When:** After any v1.x.x release +**Use When:** Only when you are auditing historical versioning behavior --- diff --git a/tools/requirements-llm.txt b/tools/requirements-llm.txt index 17b7feb5..dbd18ba6 100644 --- a/tools/requirements-llm.txt +++ b/tools/requirements-llm.txt @@ -6,7 +6,7 @@ # change is required when bumping this file. # - Use strict X.Y.Z pins to keep workflow installs reproducible. langchain==1.2.10 -langchain-core==1.2.16 +langchain-core==1.2.17 langchain-community==0.4.1 langchain-openai==1.1.10 langchain-anthropic==1.3.4