-
Notifications
You must be signed in to change notification settings - Fork 10
docs: agent operations roadmap + telemetry plan #26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
e936e8b
docs: add agent operations roadmap and telemetry plan
nutt-adam ad6f468
feat(automation): add Codex SDLC orchestration framework and helper s…
nutt-adam 501d8e4
ci: add workflow_dispatch orchestrator for Tutti SDLC smoke/auto runs
nutt-adam a395cf7
docs: specialize Codex SDLC topology into focused independent agents
nutt-adam c48b1ac
docs(workflow): add Claude planner lane and no-commit guard to SDLC c…
nutt-adam 935237b
fix(workflow): remove missing plan-file dependency from SDLC cycle
nutt-adam c3dc86f
fix: always collect CodeRabbit feedback and claim selected issues
nutt-adam 916ab11
chore: address PR review actionables for roadmap and workflow
nutt-adam 4090d6a
fix: harden sdlc orchestration flow for reruns and worktree alignment
nutt-adam 59eec6f
fix: remove duplicate issue preselection and wire issue_label through…
nutt-adam 4465a31
refactor: extract SDLC inline python blocks into automation scripts
nutt-adam 6ff9e88
fix: harden coderabbit wait/collect script error handling
nutt-adam c53d7db
fix(automation): include inline CodeRabbit comments and strict succes…
wren-nutthouse 1fa8fd3
Merge branch 'main' into roadmap/agent-ops-telemetry
wren-nutthouse File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| name: Tutti SDLC Orchestrator | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
| inputs: | ||
| mode: | ||
| description: "Workflow mode to execute" | ||
| required: true | ||
| default: "smoke" | ||
| type: choice | ||
| options: | ||
| - smoke | ||
| - auto | ||
| issue_label: | ||
| description: "GitHub issue label to select (for automation scripts)" | ||
| required: false | ||
| default: "agent-ops" | ||
|
|
||
| jobs: | ||
| orchestrate: | ||
| runs-on: ubuntu-latest | ||
| concurrency: | ||
| group: sdlc-orchestrator-${{ github.repository }} | ||
| cancel-in-progress: false | ||
| permissions: | ||
| contents: write | ||
| pull-requests: write | ||
| issues: write | ||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| fetch-depth: 0 | ||
|
|
||
| - name: Setup Rust | ||
| uses: dtolnay/rust-toolchain@stable | ||
|
|
||
| - name: Setup Git identity | ||
| run: | | ||
| git config user.name "github-actions[bot]" | ||
| git config user.email "41898282+github-actions[bot]@users.noreply.github.com" | ||
|
|
||
| - name: Create automation config | ||
| run: | | ||
| cp docs/examples/tutti-codex-sdlc.toml tutti.toml | ||
|
|
||
| - name: Verify gh auth | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} | ||
| run: | | ||
| gh auth status | ||
|
|
||
| - name: Preflight tests | ||
| run: cargo test -q | ||
|
|
||
| - name: Run smoke workflow | ||
| if: ${{ inputs.mode == 'smoke' }} | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} | ||
| ISSUE_LABEL: ${{ inputs.issue_label }} | ||
| run: | | ||
| cargo run --quiet -- run sdlc-smoke --strict | ||
|
|
||
| - name: Verify codex CLI is available | ||
| if: ${{ inputs.mode == 'auto' }} | ||
| run: | | ||
| command -v codex >/dev/null || { echo "codex CLI not found on runner"; exit 1; } | ||
|
|
||
| - name: Run full auto workflow | ||
| if: ${{ inputs.mode == 'auto' }} | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} | ||
| ISSUE_LABEL: ${{ inputs.issue_label }} | ||
| run: | | ||
| cargo run --quiet -- run sdlc-auto --strict | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,99 @@ | ||
| # Tutti Agent Operations Roadmap (Execution Plan) | ||
|
|
||
| ## Goal | ||
| Position Tutti as the agent operations layer (not just a coding assistant) by closing reliability, governance, observability, and adoption gaps. | ||
|
|
||
| ## Priorities | ||
|
|
||
| ### P0 — Foundation (must ship first) | ||
| 1. **Identity + Auth + RBAC** | ||
| - User/org/workspace identities | ||
| - API keys and role-based permissions | ||
| - Audit trail for sensitive actions | ||
|
|
||
| 2. **Production Control Plane Hardening** | ||
| - Stable API contracts | ||
| - Durable event stream semantics | ||
| - Idempotency for mutating endpoints | ||
| - Queryable run/event history | ||
|
|
||
| 3. **Reliability Primitives** | ||
| - Retry policies + backoff | ||
| - Checkpoint/resume guarantees | ||
| - Dead-letter handling for failed actions | ||
| - Circuit breakers / provider failover | ||
|
|
||
| 4. **Approval and Guardrail Workflows** | ||
| - Human approval gates for risky actions | ||
| - Policy-as-code enforcement | ||
| - Exception handling with traceability | ||
|
|
||
| 5. **Operator Observability** | ||
| - Per-run timeline and failure stages | ||
| - Root cause attribution (routing/tool/model/policy) | ||
| - SLO dashboards and alerting hooks | ||
|
|
||
| ### P1 — Scale and Governance | ||
| 6. **Cost Governance** | ||
| - Workspace/agent budgets | ||
| - Forecasting and anomaly detection | ||
| - Hard-stop + escalation behavior | ||
|
|
||
| 7. **Deployment and Operations Story** | ||
| - Managed and self-hosted deployment paths | ||
| - Secrets/backup/migration tooling | ||
| - Upgrade safety and rollback docs | ||
|
|
||
| 8. **Ecosystem + Templates** | ||
| - Connector starter set (CRM, ticketing, docs, messaging) | ||
| - Template arrangements/workflows by use case | ||
| - Under 30 min time-to-first-value path | ||
|
|
||
| 9. **Trust and Compliance Pack** | ||
| - Security posture docs | ||
| - Data retention/export controls | ||
| - Auditability and incident response basics | ||
|
|
||
| 10. **Positioning + Onboarding Narrative** | ||
| - Replace generic “agent framework” messaging | ||
| - “Run reliable agent operations in one day” onboarding flow | ||
| - Product-led proof of value in first session | ||
|
|
||
| ## Telemetry Plan (parallel to all milestones) | ||
|
|
||
| ### North Star | ||
| - **Activation rate:** installs that reach first successful workflow run within 24h | ||
|
|
||
| ### Event instrumentation | ||
| - Install: `tutti_install_started`, `tutti_install_completed` | ||
| - Activation: `first_workspace_created`, `first_up_success`, `first_send`, `first_workflow_run` | ||
| - Habit: DAW (daily active workspaces), returning workspace cohorts (D1/D7/D30), runs per workspace | ||
| - Outcome: run success/failure, failure type (`routing/tool/model/policy`), time-to-success | ||
| - Commercial: conversion funnel install → first up → first successful run → D7 return | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
|
|
||
| **PII boundaries (mandatory):** | ||
| - Allowed fields: timestamps, boolean flags, counters, duration buckets, workflow/run status, failure category. | ||
| - Identifier handling: workspace/user identifiers must be one-way pseudonymized (salted hash). | ||
| - Explicitly excluded: workspace names, user emails, full prompts/messages, raw payload bodies. | ||
| - Retention: raw event payloads 30 days; aggregated cohort and DAW metrics 12 months. | ||
| - Enforcement: apply and verify via the "Redaction/privacy defaults" deliverable before dashboard rollout. | ||
|
|
||
| ### Telemetry deliverables | ||
| - Event schema + versioning | ||
| - Redaction/privacy defaults | ||
| - Dashboard for activation, reliability, retention | ||
| - Alerting on activation regression and failure spikes | ||
|
|
||
| ## Suggested sequencing (8-week sketch) | ||
| - **Weeks 1-2:** P0.1 + P0.2 + telemetry schema foundation | ||
| - **Weeks 3-4:** P0.3 + P0.4 + failure attribution dashboards | ||
| - **Weeks 5-6:** P0.5 + P1.6 (observability + cost governance) | ||
| - **Weeks 7-8:** P1.7 + P1.8 + onboarding narrative improvements | ||
| - Ongoing: P1.9 + P1.10 hardening and GTM tuning | ||
|
|
||
| ## Success criteria | ||
| - Activation: >= 60% of installs reach first successful workflow run within 24h | ||
| - Reliability: run failure rate < 5% and MTTR < 2 hours | ||
| - Operator trust: approval/audit/cost controls are production-usable and exercised in at least one documented runbook | ||
| - Adoption: >= 3 production-ready templates with documented use cases | ||
| - Retention: D7 returning-workspace cohort >= 40% | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| # Codex SDLC Orchestration (Tutti-for-Tutti) | ||
|
|
||
| This framework automates the SDLC loop for Tutti using Codex agents: | ||
|
|
||
| 1. Select issue from GitHub | ||
| 2. Create issue branch | ||
| 3. Implement with specialized agents (implementation, testing, docs/release) | ||
| 4. Validate locally | ||
| 5. Open PR | ||
| 6. Wait for CodeRabbit review | ||
| 7. Apply review fixes | ||
| 8. Re-validate and update PR | ||
|
|
||
| ## Specialized agent topology (recommended) | ||
|
|
||
| Use 6 focused agents: | ||
| - `planner` (Claude) — issue decomposition, risk/test/release planning | ||
| - `conductor` (Codex) — orchestration/handoffs only | ||
| - `implementer` (Codex) — code changes in `src/**` | ||
| - `tester` (Codex) — tests and validation ownership | ||
| - `docs-release` (Codex) — docs/changelog/version responsibilities | ||
| - `reviewer` (Codex) — strict release-readiness review | ||
|
|
||
| This keeps each agent independent and accountable to one concern while preserving Codex-heavy execution. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - `gh` authenticated with repo access | ||
| - `codex` CLI authenticated and available in PATH | ||
| - `git`, `python3` available | ||
| - Repo has labels for issue intake (default `agent-ops`) | ||
|
|
||
| ## Test run first (required) | ||
|
|
||
| Before unattended automation, run a smoke workflow that: | ||
|
|
||
| - selects issue (`select_issue.sh`) | ||
| - creates branch (`create_issue_branch.sh`) | ||
| - runs validation (`cargo test --quiet`) | ||
| - does **not** open PR or push | ||
|
|
||
| Only after successful smoke, run full cycle. | ||
|
|
||
| ## Example workflow file | ||
|
|
||
| Use `docs/examples/tutti-codex-sdlc.toml` as a starting point. | ||
|
|
||
| ## Core scripts | ||
|
|
||
| - `scripts/automation/select_issue.sh` | ||
| - `scripts/automation/create_issue_branch.sh` | ||
| - `scripts/automation/wait_coderabbit.sh` | ||
| - `scripts/automation/collect_coderabbit_feedback.sh` | ||
|
|
||
| ## Operational notes | ||
|
|
||
| - Keep branch naming deterministic: `auto/issue-<num>-<timestamp>` | ||
| - Always include issue reference in commit and PR body | ||
| - Enforce docs/version updates in implementation prompt | ||
| - Require test pass before PR open and before merge/land | ||
| - If CodeRabbit fails, gather feedback and route to Codex fix step | ||
|
|
||
| ## Suggested runbook | ||
|
|
||
| 1. `tt run sdlc-smoke --strict` | ||
| 2. Inspect logs/output artifacts under `.tutti/state/auto/` | ||
| 3. `tt run sdlc-auto --strict` | ||
| 4. Monitor with `tt watch` / `tt logs` | ||
| 5. Land only after checks + review pass | ||
|
|
||
| ## Safety | ||
|
|
||
| - Start with `fail_mode = "closed"` on command steps | ||
| - Keep PR creation and land as explicit steps (no hidden auto-merge) | ||
| - Add approval gate before land for now |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.