Skip to content

chore: sync workflow templates #324

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
stranske wants to merge 1 commit into from
Closed

chore: sync workflow templates #324

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
203 changes: 60 additions & 143 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,170 +1,87 @@
# AGENTS.md
# AGENTS.md - Consumer Repository Context

This file is the “mission briefing” for coding agents working in this repository. It is project-specific. For workflow/consumer-repo rules, also read **CLAUDE.md**.
> Read this before changing workflows, prompts, or synced automation files.

## What this project is
## This Is A Consumer Repo

Counter_Risk replaces a spreadsheet-based monthly counterparty risk process. The end state is:
Most workflow logic for this repository lives in `stranske/Workflows`. The consumer repo should only carry repo-specific configuration unless it has an explicitly documented exception.

- A maintainable program that ingests monthly inputs and produces the same outputs the spreadsheet produced (plus new output types later).
- A button-driven “Runner” so **non-technical operators** can run the process without a CLI and without repo access.
- A deterministic, testable pipeline with parity checks against “golden” reference artifacts.
## Source Of Truth

## Non-technical operator workflow (no CLI)
For infrastructure work, follow this order:

Operators should be able to:
1. `stranske/Workflows` root docs: `README.md`, `docs/WORKFLOW_GUIDE.md`, `docs/ci/WORKFLOWS.md`
2. `stranske/Workflows/docs/INTEGRATION_GUIDE.md` and `docs/ops/CONSUMER_REPO_MAINTENANCE.md`
3. The consumer sync source in `stranske/Workflows/templates/consumer-repo/`
4. This repo's local repo-specific files

1. Pick an **as-of date** (and optionally a variant: All Programs / Ex Trend / Trend).
2. Click **Run**.
3. Receive an output folder containing:
- Updated historical workbooks
- Updated monthly PPT (and optionally a static distribution PPT/PDF)
- A manifest and a short data quality summary
If a file is synced from Workflows, fix it in Workflows first.

Operators never need to see GitHub, Python, or a terminal.
## Current Consumer Defaults

## Maintainer workflow (agent-driven)
- First-party consumers currently reference reusable workflows with `@main`. Match that unless you are intentionally pinning to an exact commit SHA for a controlled reason.
- `ci.yml` and `autofix-versions.env` are repo-specific.
- `pr-00-gate.yml` is a create-only standard file. Keep it aligned with the standard gate unless this repo has a documented reason to diverge.
- Synced workflows, prompts, scripts, and consumer docs are managed through `.github/sync-manifest.yml` in Workflows.

Maintainers primarily work through issues and PRs:
## Commonly Managed Files

- Issues should follow `docs/AGENT_ISSUE_FORMAT.md`
- Agent automation is driven by labels (see `WORKFLOW_USER_GUIDE.md`)
- Keepalive expects PR bodies to contain task checkboxes and acceptance criteria checkboxes
Usually edit locally only when the file is repo-specific:

When writing issues, prefer:
- Small scope (one PR per issue)
- Objective acceptance criteria (range-level comparisons, file existence checks, deterministic metrics)
- Explicit file/module names when known
| File | Default owner | Notes |
|------|---------------|-------|
| `ci.yml` | Consumer repo | Repo-specific CI wiring |
| `autofix-versions.env` | Consumer repo | Repo-specific dependency pins |
| `pr-00-gate.yml` | Consumer repo, but should match Workflows standard by default | Create-only standard file |
| `agents-*.yml` | Workflows | Fix in Workflows, not here |
| `autofix.yml` | Workflows | Fix in Workflows, not here |
| `.github/codex/` prompts | Workflows | Fix in Workflows, not here |
| synced scripts/docs | Workflows | Fix in Workflows, not here |

## Source artifacts and parity strategy
## Current Workflow Surfaces

The pipeline must reproduce current outputs using the provided reference artifacts (fixtures). The project should maintain:
The current consumer default automation surface is centered on:

- A `tests/fixtures/` directory for **sanitized** reference files (Excel/PPT) used in regression tests.
- Range-level comparisons for key tables and time-series append logic.
- Structural checks for PPT (slide count, expected shapes present).
- A manifest file in each run output folder recording:
- input hashes
- output paths
- warnings and reconciliation gaps
- top exposures / top movers summaries
- `agents-issue-intake.yml`
- `agents-80-pr-event-hub.yml`
- `agents-81-gate-followups.yml`
- `agents-verifier.yml`
- `autofix.yml`
- `ci.yml`
- `pr-00-gate.yml`

Never silently drop exposures. If a new counterparty appears and there is no matching historical series header, the run must warn or fail based on policy.
Legacy compatibility workflows may still exist during migrations. Do not assume an older filename is canonical without checking the Workflows docs first.

## Key technical constraints and “gotchas”
## Cross-Repo Policy

1. Date semantics must be explicit:
- `as_of_date` is the effective reporting date used on historical workbook x-axes.
- `run_date` is when the pipeline was executed.
Before editing local workflow infrastructure, ask:

2. Name normalization must be deterministic:
- Trim/canonicalize whitespace.
- Apply alias mapping registry first, then fallback mapping.
- Treat Excel/PPT header series names as brittle (spaces and punctuation matter).
**Does this work belong in `stranske/Workflows` instead?**

3. PowerPoint is hybrid:
- Some slides are pasted screenshots (replace pictures).
- Other slides are linked charts (OLE links) that may require refresh (COM automation preferred + fallback instruction file).
The answer is usually yes if the change affects any of these:

4. Workflows are synced from stranske/Workflows:
- Do not edit `.github/workflows/**` unless explicitly operating under a high-privilege environment and the task requires it.
- If workflow changes are needed, fix them in **stranske/Workflows** then sync.
- reusable workflows
- agent prompts or routing
- keepalive/autofix/verifier behavior
- synced workflow files
- synced scripts or docs

## CI test situation (what runs when)
If yes:

This repo is tuned so PR Gate stays reasonably fast by skipping the most expensive suites.
1. Make the source-of-truth change in `stranske/Workflows`
2. Update the sync manifest if a consumer-facing file changed
3. Sync or manually align this repo afterward

### PR Gate
## Useful References

PR Gate uses `.github/workflows/pr-00-gate.yml`:
- `stranske/Workflows/README.md`
- `stranske/Workflows/docs/WORKFLOW_GUIDE.md`
- `stranske/Workflows/docs/ci/WORKFLOWS.md`
- `stranske/Workflows/docs/INTEGRATION_GUIDE.md`
- `stranske/Workflows/docs/ops/CONSUMER_REPO_MAINTENANCE.md`
- `stranske/Workflows/docs/keepalive/Agents.md`
- `stranske/Travel-Plan-Permission` as a reference consumer

- Runs pytest **in parallel** with xdist (`-n auto --dist load`).
- Runs pytest **without coverage**.
- Skips `tests/integration/` and `tests/integrations/`.
- Skips `release`- and `slow`-marked tests (`pytest_markers: "not release and not slow"`).
## Agent-Specific Note

### Main CI (push to `main`)

Main CI uses `.github/workflows/ci.yml`:

- Runs pytest **with coverage** and enforces `coverage-min`.
- Runs pytest **in parallel** with xdist (`--dist loadscope`).
- Runs **all** tests including `slow`-marked ones.
- This is where heavyweight pipeline and spec-fixture tests are validated.

### Slow tests

Tests that take a long time because they parse real Excel workbooks or run
full pipeline orchestration are tagged `@pytest.mark.slow`. The Gate
excludes them so PR feedback stays under five minutes; Main CI runs them on
every merge.

Currently slow-marked tests:

| File | Why it's slow | How it's marked |
|------|---------------|-----------------|
| `tests/pipeline/test_run_pipeline.py` | Each `test_run_pipeline_*` calls `run_pipeline()` with real Excel fixtures (100-170 s per test) | Auto-marked by `tests/pipeline/conftest.py` hook |
| `tests/spec/test_macro_spec_fixtures.py` | Session-scoped fixture parses three NISA workbooks (~85 s) | Module-level `pytestmark = pytest.mark.slow` |

#### When to mark a test slow

Mark a new test `@pytest.mark.slow` if it:

- Calls `run_pipeline()` or `run_pipeline_with_config()` with real fixture files.
- Relies on a session-scoped fixture that parses large workbooks.
- Takes more than ~30 s on CI in isolation.

You can apply the marker with a decorator (`@pytest.mark.slow`) or add the
test pattern to the `pytest_collection_modifyitems` hook in the appropriate
`conftest.py`.

#### Running slow tests locally

```bash
# Run only slow tests
pytest -m slow

# Run everything (same as Main CI)
pytest

# Run without slow tests (same as Gate)
pytest -m "not slow"
```

### Release tests (nightly or label)

The slowest release/packaging checks are isolated behind the `release` marker and can be triggered via:

- Nightly schedule (`.github/workflows/release-e2e.yml`)
- PR label: `run-release`

### Agent guidance

- When a PR touches release/packaging mechanics (e.g., `release.spec`, `pyinstaller_runtime_hook.py`, templates/config bundling), make sure `release` tests run by applying the `run-release` label.
- When adding tests that parse real workbooks or run full pipeline orchestration, mark them `@pytest.mark.slow` so the Gate stays fast. Do **not** add heavyweight tests without the marker — it will regress Gate duration for every PR.

## Agent guardrails (must follow)

- Also read: `.github/codex/AGENT_INSTRUCTIONS.md`
- Do not modify protected workflow/security files unless explicitly allowed.
- Do not introduce secrets into logs/files.
- Keep changes scoped to the PR’s tasks.

## Definition of done (per milestone)

Milestone 1 (Parity using MOSERS-formatted inputs):
- Parse MOSERS-format workbooks into canonical tables.
- Update historical workbooks by appending a new row.
- Update monthly PPT (screenshots replaced + links refreshed or static distribution output).
- Produce run manifest + data quality summary.

Milestone 2 (Remove VBA/manual prep):
- Ingest raw NISA monthly inputs and generate MOSERS-format structures without running macros.
- Automate cash ingestion (structured source preferred; PDF fallback + overrides).
- Add prior-month futures delta/sign-flip logic for Trend.

## Where to look for keepalive system docs

- `WORKFLOW_USER_GUIDE.md` (labels, triggers, operator usage)
- `docs/KEEPALIVE_TROUBLESHOOTING.md` (deep debugging and known failure modes)
- `CLAUDE.md` (consumer repo boundaries and “what not to edit”)
This file is the agent-generic contract. Keep it materially aligned with `CLAUDE.md`; differences between the two should only be agent-specific execution notes, not different repository rules.
Loading
Loading