diff --git a/.agents/skills/ado-commit/SKILL.md b/.agents/skills/ado-commit/SKILL.md deleted file mode 100644 index 4bfc99bd2..000000000 --- a/.agents/skills/ado-commit/SKILL.md +++ /dev/null @@ -1,94 +0,0 @@ ---- -name: ado-commit -description: Create well-formatted conventional commits in a repository hosted on Azure DevOps (ADO / Azure Repos). Use this whenever the user asks to commit changes and the project is on Azure DevOps — dev.azure.com, visualstudio.com, or explicit mentions of ADO, Azure Repos, or work item IDs like `AB#1234`. Automatically appends `AB#` work-item trailers when the branch name or staged changes reference one, and attributes AI-assisted authorship. -metadata: - provider: atomic ---- - -# ADO Commit - -Create a conventional commit on an Azure DevOps-hosted repository: $ARGUMENTS - -## Current state - -- Git status: !`git status --porcelain` -- Current branch: !`git branch --show-current` -- Staged diff (stat): !`git diff --cached --stat` -- Unstaged diff (stat): !`git diff --stat` -- Recent commits: !`git log --oneline -5` - -## Workflow - -The only ADO-specific bits are (a) work-item trailers and (b) the conventions this repo has adopted for talking to reviewers who open PRs in Azure DevOps. - -1. **Stage.** If nothing is staged, stage all modified and new files with `git add -A`. If specific files are already staged, commit only those. -2. **Diff.** Run `git diff --cached` to understand the actual change. Read the diff — don't just trust the path names — because the message needs to describe *what changed and why*, not *which files changed*. -3. **Split if needed.** If the staged diff contains multiple unrelated logical changes, propose splitting into separate commits. One commit = one reason to change. -4. **Write the message** in Conventional Commits format (see below), then commit via `git commit --message "" [--trailer ...]`. Pass trailers with `--trailer` so git formats them correctly; don't cat-heredoc them into the body. -5. **Don't skip pre-commit hooks.** If `.pre-commit-config.yaml` exists, hooks run automatically and their failures are signal, not noise. Never pass `--no-verify`. - -## Conventional Commits — quick reference - -``` -(optional scope): - - - - -``` - -Common types: `feat`, `fix`, `docs`, `style`, `refactor`, `perf`, `test`, `build`, `ci`, `chore`, `revert`. Append `!` after type/scope for breaking changes (e.g. `feat(api)!: change response format`). Keep the subject under 72 characters, imperative mood, no trailing period. - -**Examples:** - -``` -feat(auth): add JWT refresh endpoint -fix(ui): resolve layout shift on mobile nav -refactor(db): migrate from raw SQL to query builder -chore(deps): bump TypeScript to 5.5 -feat(api)!: change pagination response shape -``` - -## Work-item trailers (ADO-specific) - -Azure DevOps auto-links commits to work items when the message contains `AB#`. Include one whenever you can identify the target work item, because it keeps the board in sync without anyone clicking around. - -**Where to find the ID:** - -- **Branch name** — patterns like `feature/1234-...`, `bug/AB1234-...`, `user/name/1234-...` usually encode the work item ID. -- **User input** — if the user mentions "work item 1234" or "this closes 1234", use that. -- **Prior commits on the branch** — run `git log --oneline origin/main..HEAD` and check if earlier commits reference an ID. -- **ADO MCP** — if the project has the `azure-devops` MCP server configured and you're still unsure, call `wit_my_work_items` (or `search_workitem` with a keyword from the change) to surface likely candidates. Ask the user to confirm rather than guessing. - -**How to add it** — as a trailer, not in the subject: - -```bash -git commit \ - --message "feat(auth): add JWT refresh endpoint" \ - --trailer "AB#1234" \ - --trailer "Assistant-model: Claude Code" -``` - -If you genuinely can't find a work-item ID, skip the trailer rather than inventing one. A missing trailer is recoverable; a wrong one pollutes the board. - -## AI authorship trailer - -ADO code reviews often surface in audit contexts, so mark AI-assisted commits honestly. Use an `Assistant-model` trailer rather than `Co-authored-by` — most git tooling validates the latter as an email, and we want to distinguish *assistance* from *authorship*: - -``` -Assistant-model: Claude Code -``` - -Add it every time you commit on the user's behalf. - -## Putting it together - -```bash -git add -A -git diff --cached --stat # sanity check -git commit \ - --message "fix(parser): handle nested escape sequences" \ - --trailer "AB#5678" \ - --trailer "Assistant-model: Claude Code" -git log -1 # show the user the result -``` diff --git a/.agents/skills/ado-create-pr/SKILL.md b/.agents/skills/ado-create-pr/SKILL.md deleted file mode 100644 index c8a6f62cb..000000000 --- a/.agents/skills/ado-create-pr/SKILL.md +++ /dev/null @@ -1,211 +0,0 @@ ---- -name: ado-create-pr -description: Commit, push, and open a pull request in Azure DevOps. Use whenever the user wants to open, update, or draft a PR and the project is hosted on Azure DevOps (`dev.azure.com`, `visualstudio.com`, or explicit mentions of ADO, Azure Repos, or work item IDs like `AB#1234`). Links work items to the PR, sets reviewers, and supports draft-by-default. -metadata: - provider: atomic ---- - -# ADO Create Pull Request - -Commit changes, push the branch, and open or update an Azure DevOps pull request with a conventional-commit-style title and a complete description: $ARGUMENTS - -## Current state - -- Git status: !`git status --porcelain` -- Current branch: !`git branch --show-current` -- Default branch: !`git rev-parse --abbrev-ref origin/HEAD 2>/dev/null | sed 's|origin/||' || echo main` -- Staged diff (stat): !`git diff --cached --stat` -- Unstaged diff (stat): !`git diff --stat` -- Recent commits on this branch: !`git log --oneline -10` -- Commits ahead of default: !`git log --oneline origin/$(git rev-parse --abbrev-ref origin/HEAD 2>/dev/null | sed 's|origin/||' || echo main)..HEAD 2>/dev/null | head -20` -- Remote URL (to confirm ADO host): !`git remote get-url origin 2>/dev/null || echo "no-remote"` - -## Use the Azure DevOps MCP tools - -All ADO operations in this workflow go through the Azure DevOps MCP tools — never `az` / `az devops`. When you see tool names like `repo_create_pull_request` or `wit_link_work_item_to_pull_request` below, call the matching Azure DevOps tool from your tool list. If no Azure DevOps MCP tools are loaded in this session, stop and ask the user if they want to fallback to the `az` CLI. - -## Workflow - -### 1. Stage and commit - -Follow the **ado-commit** skill for the commit step — same conventional-commit format, same AI-authorship trailer, same `AB#` work-item trailer rules. Split into multiple commits if the staged diff covers unrelated concerns. - -If the user is currently on the default branch (`main` / `master`), switch to a feature branch *before* committing. A reasonable default name is `user/` or `feature/`; if a work-item ID is known, prefix it: `feature/1234-`. - -### 2. Push - -```bash -git push -u origin "$(git branch --show-current)" -``` - -`-u` sets upstream tracking so subsequent pushes don't need arguments. - -### 3. Gather context for the PR - -Read the *full* diff against the base branch, not just the last commit — a PR title needs to summarize the whole branch, not one step of it. - -```bash -git diff origin/...HEAD -``` - -Open the files that changed significantly so you can describe the *why* accurately. If there's an existing PR for this branch, fetch it first (see step 5) and edit rather than replace — a human may already have curated the title or description. - -### 4. Identify the repo, project, and work items - -The MCP tools need identifiers: - -- **Project and repository name** — parse from the `origin` remote URL. ADO URLs follow `https://dev.azure.com///_git/` or `https://.visualstudio.com//_git/`. -- **Repository ID** — call `repo_get_repo_by_name_or_id` with `{ project, repositoryNameOrId: }`. Use the returned `id` for subsequent calls. -- **Work item IDs** — scan the branch name and every commit subject/body on the branch (`git log origin/..HEAD`) for `AB#`, `#`, or numeric prefixes like `feature/1234-...`. If the user mentioned a work item in the prompt, trust that. - -If projects or repos aren't obvious, `core_list_projects` and `repo_list_repos_by_project` let you browse. - -### 5. Check for an existing PR - -``` -repo_list_pull_requests_by_repo_or_project { - repositoryId: , - status: "active", - sourceRefName: "refs/heads/" -} -``` - -If a result comes back, you're in *update* mode — keep the existing PR's ID and edit in place in step 7. Otherwise you're in *create* mode. - -### 6. Generate title and description - -**Title** — Conventional Commits, under 72 chars. For a single-commit PR the commit subject works; for a multi-commit PR synthesize a higher-level subject that captures the whole branch. - -``` -feat(auth): add JWT token refresh endpoint -fix(ui): resolve layout shift on mobile nav -refactor(db): migrate from raw SQL to query builder -feat(api)!: change pagination response shape -``` - -**Description** — use this template, omitting sections that don't apply: - -```markdown -## Summary - -[1–2 sentences on what this PR does and why] - -## Changes - -- [Key change 1] -- [Key change 2] - -## Breaking Changes - -[What breaks and the migration step — delete this section if none] - -## Test Plan - -- [How this was verified — commands, manual checks, screenshots] - -## Work Items - -AB#1234 -``` - -Keep the `AB#` references in the description — ADO parses them and shows the linked work items alongside the PR. You'll *also* link them via MCP in step 8 so the links are first-class, not just string-matched. - -### 7. Create or update the PR - -**Create (default to draft):** - -``` -repo_create_pull_request { - repositoryId: , - sourceRefName: "refs/heads/", - targetRefName: "refs/heads/", - title: "", - description: "", - isDraft: true -} -``` - -`sourceRefName` and `targetRefName` need the full `refs/heads/` prefix — a common mistake is passing the bare branch name and getting a cryptic 400. - -**Update (existing PR):** - -``` -repo_update_pull_request { - repositoryId: , - pullRequestId: , - title: "", - description: "" -} -``` - -Respect the existing title/description if they're already meaningful — enhance rather than overwrite. If the existing title already follows conventional commits and is accurate, leave it alone. - -### 8. Link work items - -Even if the description contains `AB#`, explicitly link each work item so it shows up as a structured PR-WorkItem relationship: - -``` -wit_link_work_item_to_pull_request { - projectId: , - repositoryId: , - pullRequestId: , - workItemId: -} -``` - -Call once per work item ID. - -### 9. Reviewers (optional) - -If the user named reviewers, resolve them to identity IDs and attach: - -``` -core_get_identity_ids { searchFilter: "" } -repo_update_pull_request_reviewers { - repositoryId: , - pullRequestId: , - reviewerIds: [], - action: "add" -} -``` - -Don't auto-assign reviewers the user didn't mention — ADO default reviewer policies usually handle that, and guessing people's IDs is a good way to ping the wrong person. - -### 10. Report back - -Print the PR's web URL (returned in the create/update response as `url` or `_links.web.href`) so the user can click through. Summarize: branch → target, draft status, work items linked, reviewers added. - -## Guidelines - -- **Draft by default.** Pass `isDraft: true` unless the user says otherwise. It's easier to mark ready than to walk back a premature review request. -- **Never skip pre-commit hooks.** They run locally during commits created in step 1. A hook failure is the hook earning its keep. -- **Always attribute AI assistance** via the `Assistant-model` trailer on every commit (see the ado-commit skill). -- **Respect existing content.** If updating an existing PR, keep what's already curated; only replace sections that are stale or wrong. -- **Holistic title.** The PR title is one line describing the whole branch. Don't concatenate commit subjects. - -## Related Azure DevOps tools - -The 10-step workflow above names the tools you need on the happy path. Reach for the ones below when the situation calls for it — don't run them by default. Grouped by the sub-task they unlock. - -| Sub-task | Tool | When to reach for it | -| ---------------------- | --------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | -| **Locate the repo** | `core_list_projects` | Project name isn't obvious from the remote URL | -| | `repo_list_repos_by_project` | Multiple repos in the project and you need to pick | -| **Inspect branches** | `repo_list_branches_by_repo` | Confirm the default / target branch exists before creating the PR | -| | `repo_get_branch_by_name` | Pull the latest commit or branch policies on the source branch | -| | `repo_create_branch` | Branch off server-side when the user isn't working locally | -| **Pick a work item** | `wit_my_work_items` | User didn't name one — surface their active items so they can confirm | -| | `search_workitem` | Keyword search when the work item ID is uncertain | -| | `wit_get_work_item` / `wit_get_work_items_batch_by_ids` | Fetch title/state to enrich the PR description (e.g. "Closes AB#1234 — add JWT refresh") | -| **Inspect the change** | `repo_get_pull_request_changes` | Programmatic diff on an existing PR when local `git diff` isn't enough | -| | `repo_search_commits` | Verify specific commits landed on the source branch | -| | `repo_get_file_content` | Re-read a file at a specific commit to describe it accurately | -| **Fetch the PR** | `repo_get_pull_request_by_id` | Reload the PR after create/update (web URL, status, policy state) | -| **Extra linking** | `wit_add_artifact_link` | Link a commit or build to a work item (non-PR relationship) | -| | `wit_add_work_item_comment` | Post "PR #N opened" on the work item so watchers see it async | -| **Comments & votes** | `repo_list_pull_request_threads` / `repo_list_pull_request_thread_comments` | Read existing review threads before editing the PR | -| | `repo_create_pull_request_thread` | Seed a context comment on the new PR (e.g. testing notes) | -| | `repo_reply_to_comment` | Respond to a reviewer inline | -| | `repo_vote_pull_request` | Approve / wait-for-author / reject on the user's behalf — only when explicitly asked | -| **CI signal** | `pipelines_get_build_status` | Check whether the branch's CI is green before un-drafting | -| | `pipelines_get_build_log` | Pull logs when CI is red and you're helping diagnose | \ No newline at end of file diff --git a/.agents/skills/advanced-evaluation/references/bias-mitigation.md b/.agents/skills/advanced-evaluation/references/bias-mitigation.md deleted file mode 100644 index b6595aa18..000000000 --- a/.agents/skills/advanced-evaluation/references/bias-mitigation.md +++ /dev/null @@ -1,288 +0,0 @@ -# Bias Mitigation Techniques for LLM Evaluation - -This reference details specific techniques for mitigating known biases in LLM-as-a-Judge systems. - -## Position Bias - -### The Problem - -In pairwise comparison, LLMs systematically prefer responses in certain positions. Research shows: -- GPT has mild first-position bias (~55% preference for first position in ties) -- Claude shows similar patterns -- Smaller models often show stronger bias - -### Mitigation: Position Swapping Protocol - -```python -async def position_swap_comparison(response_a, response_b, prompt, criteria): - # Pass 1: Original order - result_ab = await compare(response_a, response_b, prompt, criteria) - - # Pass 2: Swapped order - result_ba = await compare(response_b, response_a, prompt, criteria) - - # Map second result (A in second position → B in first) - result_ba_mapped = { - 'winner': {'A': 'B', 'B': 'A', 'TIE': 'TIE'}[result_ba['winner']], - 'confidence': result_ba['confidence'] - } - - # Consistency check - if result_ab['winner'] == result_ba_mapped['winner']: - return { - 'winner': result_ab['winner'], - 'confidence': (result_ab['confidence'] + result_ba_mapped['confidence']) / 2, - 'position_consistent': True - } - else: - # Disagreement indicates position bias was a factor - return { - 'winner': 'TIE', - 'confidence': 0.5, - 'position_consistent': False, - 'bias_detected': True - } -``` - -### Alternative: Multiple Shuffles - -For higher reliability, use multiple position orderings: - -```python -async def multi_shuffle_comparison(response_a, response_b, prompt, criteria, n_shuffles=3): - results = [] - for i in range(n_shuffles): - if i % 2 == 0: - r = await compare(response_a, response_b, prompt, criteria) - else: - r = await compare(response_b, response_a, prompt, criteria) - r['winner'] = {'A': 'B', 'B': 'A', 'TIE': 'TIE'}[r['winner']] - results.append(r) - - # Majority vote - winners = [r['winner'] for r in results] - final_winner = max(set(winners), key=winners.count) - agreement = winners.count(final_winner) / len(winners) - - return { - 'winner': final_winner, - 'confidence': agreement, - 'n_shuffles': n_shuffles - } -``` - -## Length Bias - -### The Problem - -LLMs tend to rate longer responses higher, regardless of quality. This manifests as: -- Verbose responses receiving inflated scores -- Concise but complete responses penalized -- Padding and repetition being rewarded - -### Mitigation: Explicit Prompting - -Include anti-length-bias instructions in the prompt: - -``` -CRITICAL EVALUATION GUIDELINES: -- Do NOT prefer responses because they are longer -- Concise, complete answers are as valuable as detailed ones -- Penalize unnecessary verbosity or repetition -- Focus on information density, not word count -``` - -### Mitigation: Length-Normalized Scoring - -```python -def length_normalized_score(score, response_length, target_length=500): - """Adjust score based on response length.""" - length_ratio = response_length / target_length - - if length_ratio > 2.0: - # Penalize excessively long responses - penalty = (length_ratio - 2.0) * 0.1 - return max(score - penalty, 1) - elif length_ratio < 0.3: - # Penalize excessively short responses - penalty = (0.3 - length_ratio) * 0.5 - return max(score - penalty, 1) - else: - return score -``` - -### Mitigation: Separate Length Criterion - -Make length a separate, explicit criterion so it's not implicitly rewarded: - -```python -criteria = [ - {"name": "Accuracy", "description": "Factual correctness", "weight": 0.4}, - {"name": "Completeness", "description": "Covers key points", "weight": 0.3}, - {"name": "Conciseness", "description": "No unnecessary content", "weight": 0.3} # Explicit -] -``` - -## Self-Enhancement Bias - -### The Problem - -Models rate outputs generated by themselves (or similar models) higher than outputs from different models. - -### Mitigation: Cross-Model Evaluation - -Use a different model family for evaluation than generation: - -```python -def get_evaluator_model(generator_model): - """Select evaluator to avoid self-enhancement bias.""" - if 'gpt' in generator_model.lower(): - return 'claude-4-5-sonnet' - elif 'claude' in generator_model.lower(): - return 'gpt-5.2' - else: - return 'gpt-5.2' # Default -``` - -### Mitigation: Blind Evaluation - -Remove model attribution from responses before evaluation: - -```python -def anonymize_response(response, model_name): - """Remove model-identifying patterns.""" - patterns = [ - f"As {model_name}", - "I am an AI", - "I don't have personal opinions", - # Model-specific patterns - ] - anonymized = response - for pattern in patterns: - anonymized = anonymized.replace(pattern, "[REDACTED]") - return anonymized -``` - -## Verbosity Bias - -### The Problem - -Detailed explanations receive higher scores even when the extra detail is irrelevant or incorrect. - -### Mitigation: Relevance-Weighted Scoring - -```python -async def relevance_weighted_evaluation(response, prompt, criteria): - # First, assess relevance of each segment - relevance_scores = await assess_relevance(response, prompt) - - # Weight evaluation by relevance - segments = split_into_segments(response) - weighted_scores = [] - for segment, relevance in zip(segments, relevance_scores): - if relevance > 0.5: # Only count relevant segments - score = await evaluate_segment(segment, prompt, criteria) - weighted_scores.append(score * relevance) - - return sum(weighted_scores) / len(weighted_scores) -``` - -### Mitigation: Rubric with Verbosity Penalty - -Include explicit verbosity penalties in rubrics: - -```python -rubric_levels = [ - { - "score": 5, - "description": "Complete and concise. All necessary information, nothing extraneous.", - "characteristics": ["Every sentence adds value", "No repetition", "Appropriately scoped"] - }, - { - "score": 3, - "description": "Complete but verbose. Contains unnecessary detail or repetition.", - "characteristics": ["Main points covered", "Some tangents", "Could be more concise"] - }, - # ... etc -] -``` - -## Authority Bias - -### The Problem - -Confident, authoritative tone is rated higher regardless of accuracy. - -### Mitigation: Evidence Requirement - -Require explicit evidence for claims: - -``` -For each claim in the response: -1. Identify whether it's a factual claim -2. Note if evidence or sources are provided -3. Score based on verifiability, not confidence - -IMPORTANT: Confident claims without evidence should NOT receive higher scores than -hedged claims with evidence. -``` - -### Mitigation: Fact-Checking Layer - -Add a fact-checking step before scoring: - -```python -async def fact_checked_evaluation(response, prompt, criteria): - # Extract claims - claims = await extract_claims(response) - - # Fact-check each claim - fact_check_results = await asyncio.gather(*[ - verify_claim(claim) for claim in claims - ]) - - # Adjust score based on fact-check results - accuracy_factor = sum(r['verified'] for r in fact_check_results) / len(fact_check_results) - - base_score = await evaluate(response, prompt, criteria) - return base_score * (0.7 + 0.3 * accuracy_factor) # At least 70% of score -``` - -## Aggregate Bias Detection - -Monitor for systematic biases in production: - -```python -class BiasMonitor: - def __init__(self): - self.evaluations = [] - - def record(self, evaluation): - self.evaluations.append(evaluation) - - def detect_position_bias(self): - """Detect if first position wins more often than expected.""" - first_wins = sum(1 for e in self.evaluations if e['first_position_winner']) - expected = len(self.evaluations) * 0.5 - z_score = (first_wins - expected) / (expected * 0.5) ** 0.5 - return {'bias_detected': abs(z_score) > 2, 'z_score': z_score} - - def detect_length_bias(self): - """Detect if longer responses score higher.""" - from scipy.stats import spearmanr - lengths = [e['response_length'] for e in self.evaluations] - scores = [e['score'] for e in self.evaluations] - corr, p_value = spearmanr(lengths, scores) - return {'bias_detected': corr > 0.3 and p_value < 0.05, 'correlation': corr} -``` - -## Summary Table - -| Bias | Primary Mitigation | Secondary Mitigation | Detection Method | -|------|-------------------|---------------------|------------------| -| Position | Position swapping | Multiple shuffles | Consistency check | -| Length | Explicit prompting | Length normalization | Length-score correlation | -| Self-enhancement | Cross-model evaluation | Anonymization | Model comparison study | -| Verbosity | Relevance weighting | Rubric penalties | Relevance scoring | -| Authority | Evidence requirement | Fact-checking layer | Confidence-accuracy correlation | - diff --git a/.agents/skills/advanced-evaluation/references/evaluation-pipeline.md b/.agents/skills/advanced-evaluation/references/evaluation-pipeline.md deleted file mode 100644 index 72dc4e328..000000000 --- a/.agents/skills/advanced-evaluation/references/evaluation-pipeline.md +++ /dev/null @@ -1,43 +0,0 @@ -# Evaluation Pipeline Diagram - -Visual layout of a production evaluation pipeline. - -``` -┌─────────────────────────────────────────────────┐ -│ Evaluation Pipeline │ -├─────────────────────────────────────────────────┤ -│ │ -│ Input: Response + Prompt + Context │ -│ │ │ -│ ▼ │ -│ ┌─────────────────────┐ │ -│ │ Criteria Loader │ ◄── Rubrics, weights │ -│ └──────────┬──────────┘ │ -│ │ │ -│ ▼ │ -│ ┌─────────────────────┐ │ -│ │ Primary Scorer │ ◄── Direct or Pairwise │ -│ └──────────┬──────────┘ │ -│ │ │ -│ ▼ │ -│ ┌─────────────────────┐ │ -│ │ Bias Mitigation │ ◄── Position swap, etc. │ -│ └──────────┬──────────┘ │ -│ │ │ -│ ▼ │ -│ ┌─────────────────────┐ │ -│ │ Confidence Scoring │ ◄── Calibration │ -│ └──────────┬──────────┘ │ -│ │ │ -│ ▼ │ -│ Output: Scores + Justifications + Confidence │ -│ │ -└─────────────────────────────────────────────────┘ -``` - -## Pipeline Stages - -1. **Criteria Loader**: Loads rubrics and criterion weights from configuration -2. **Primary Scorer**: Applies direct scoring or pairwise comparison -3. **Bias Mitigation**: Runs position swaps, length normalization, and other debiasing -4. **Confidence Scoring**: Calibrates confidence based on position consistency and evidence strength diff --git a/.agents/skills/advanced-evaluation/references/implementation-patterns.md b/.agents/skills/advanced-evaluation/references/implementation-patterns.md deleted file mode 100644 index 088d5b01f..000000000 --- a/.agents/skills/advanced-evaluation/references/implementation-patterns.md +++ /dev/null @@ -1,315 +0,0 @@ -# LLM-as-Judge Implementation Patterns - -This reference provides detailed implementation patterns for building production-grade LLM evaluation systems. - -## Pattern 1: Structured Evaluation Pipeline - -The most reliable evaluation systems follow a structured pipeline that separates concerns: - -``` -Input Validation → Criteria Loading → Scoring → Bias Mitigation → Output Formatting -``` - -### Input Validation Layer - -Before evaluation begins, validate: - -1. **Response presence**: Non-empty response to evaluate -2. **Prompt presence**: Original prompt for context -3. **Criteria validity**: At least one criterion with name and description -4. **Weight normalization**: Weights sum to 1.0 (or normalize them) - -```python -def validate_input(response, prompt, criteria): - if not response or not response.strip(): - raise ValueError("Response cannot be empty") - if not prompt or not prompt.strip(): - raise ValueError("Prompt cannot be empty") - if not criteria or len(criteria) == 0: - raise ValueError("At least one criterion required") - - # Normalize weights - total_weight = sum(c.get('weight', 1) for c in criteria) - for c in criteria: - c['weight'] = c.get('weight', 1) / total_weight -``` - -### Criteria Loading Layer - -Criteria should be loaded from configuration, not hardcoded: - -```python -class CriteriaLoader: - def __init__(self, rubric_path=None): - self.rubrics = self._load_rubrics(rubric_path) - - def get_criteria(self, task_type): - return self.rubrics.get(task_type, self.default_criteria) - - def get_rubric(self, criterion_name): - return self.rubrics.get(criterion_name, {}).get('levels', []) -``` - -### Scoring Layer - -The scoring layer handles the actual LLM call: - -```python -async def score_response(response, prompt, criteria, rubric, model): - system_prompt = build_system_prompt(criteria, rubric) - user_prompt = build_user_prompt(response, prompt, criteria) - - result = await generate_text( - model=model, - system=system_prompt, - prompt=user_prompt, - temperature=0.3 # Lower temperature for consistency - ) - - return parse_scores(result.text) -``` - -### Bias Mitigation Layer - -For pairwise comparison, always include position swapping: - -```python -async def compare_with_bias_mitigation(response_a, response_b, prompt, criteria, model): - # First pass: A first - pass1 = await compare_pair(response_a, response_b, prompt, criteria, model) - - # Second pass: B first - pass2 = await compare_pair(response_b, response_a, prompt, criteria, model) - - # Map pass2 winner back - pass2_mapped = map_winner(pass2.winner) # A→B, B→A, TIE→TIE - - # Check consistency - if pass1.winner == pass2_mapped: - return { - 'winner': pass1.winner, - 'confidence': (pass1.confidence + pass2.confidence) / 2, - 'consistent': True - } - else: - return { - 'winner': 'TIE', - 'confidence': 0.5, - 'consistent': False - } -``` - -## Pattern 2: Hierarchical Evaluation - -For complex evaluations, use a hierarchical approach: - -``` -Quick Screen (cheap model) → Detailed Evaluation (expensive model) → Human Review (edge cases) -``` - -### Quick Screen Implementation - -```python -async def quick_screen(response, prompt, threshold=0.7): - """Fast, cheap screening for obvious passes/fails.""" - result = await generate_text( - model='gpt-5.2', # Cheaper model - prompt=f"Rate 0-1 if this response adequately addresses the prompt:\n\nPrompt: {prompt}\n\nResponse: {response}", - temperature=0 - ) - score = float(result.text.strip()) - return score, score > threshold -``` - -### Detailed Evaluation - -```python -async def detailed_evaluation(response, prompt, criteria): - """Full evaluation for borderline or important cases.""" - result = await generate_text( - model='gpt-5.2', # More capable model - system=DETAILED_EVALUATION_PROMPT, - prompt=build_detailed_prompt(response, prompt, criteria), - temperature=0.3 - ) - return parse_detailed_scores(result.text) -``` - -## Pattern 3: Panel of LLM Judges (PoLL) - -For high-stakes evaluation, use multiple models: - -```python -async def poll_evaluation(response, prompt, criteria, models): - """Aggregate judgments from multiple LLM judges.""" - results = await asyncio.gather(*[ - score_with_model(response, prompt, criteria, model) - for model in models - ]) - - # Aggregate scores - aggregated = aggregate_scores(results) - - # Calculate agreement - agreement = calculate_agreement(results) - - return { - 'scores': aggregated, - 'agreement': agreement, - 'individual_results': results - } - -def aggregate_scores(results): - """Aggregate scores using median (robust to outliers).""" - scores = {} - for criterion in results[0]['scores'].keys(): - criterion_scores = [r['scores'][criterion] for r in results] - scores[criterion] = { - 'score': statistics.median(criterion_scores), - 'std': statistics.stdev(criterion_scores) if len(criterion_scores) > 1 else 0 - } - return scores -``` - -## Pattern 4: Confidence Calibration - -Confidence scores should be calibrated to actual reliability: - -```python -def calibrate_confidence(raw_confidence, position_consistent, evidence_count): - """Calibrate confidence based on multiple signals.""" - - # Base confidence from model output - calibrated = raw_confidence - - # Position consistency is a strong signal - if not position_consistent: - calibrated *= 0.6 # Significant reduction - - # More evidence = higher confidence - evidence_factor = min(evidence_count / 3, 1.0) # Cap at 3 pieces - calibrated *= (0.7 + 0.3 * evidence_factor) - - return min(calibrated, 0.99) # Never 100% confident -``` - -## Pattern 5: Output Formatting - -Always return structured outputs with consistent schemas: - -```python -@dataclass -class ScoreResult: - criterion: str - score: float - max_score: float - justification: str - evidence: List[str] - improvement: str - -@dataclass -class EvaluationResult: - success: bool - scores: List[ScoreResult] - overall_score: float - weighted_score: float - summary: Dict[str, Any] - metadata: Dict[str, Any] - -def format_output(scores, metadata) -> EvaluationResult: - """Format evaluation results consistently.""" - return EvaluationResult( - success=True, - scores=scores, - overall_score=sum(s.score for s in scores) / len(scores), - weighted_score=calculate_weighted_score(scores), - summary=generate_summary(scores), - metadata=metadata - ) -``` - -## Error Handling Patterns - -### Graceful Degradation - -```python -async def evaluate_with_fallback(response, prompt, criteria): - try: - return await full_evaluation(response, prompt, criteria) - except RateLimitError: - # Fall back to simpler evaluation - return await simple_evaluation(response, prompt, criteria) - except ParseError as e: - # Return partial results with error flag - return { - 'success': False, - 'partial_results': e.partial_data, - 'error': str(e) - } -``` - -### Retry Logic - -```python -async def evaluate_with_retry(response, prompt, criteria, max_retries=3): - for attempt in range(max_retries): - try: - result = await evaluate(response, prompt, criteria) - if is_valid_result(result): - return result - except TransientError: - await asyncio.sleep(2 ** attempt) # Exponential backoff - - raise EvaluationError("Max retries exceeded") -``` - -## Testing Patterns - -### Unit Tests for Parsing - -```python -def test_score_parsing(): - raw_output = '{"scores": [{"criterion": "Accuracy", "score": 4}]}' - result = parse_scores(raw_output) - assert result.scores[0].criterion == "Accuracy" - assert result.scores[0].score == 4 - -def test_malformed_output(): - raw_output = 'Invalid JSON' - with pytest.raises(ParseError): - parse_scores(raw_output) -``` - -### Integration Tests with Real API - -```python -@pytest.mark.integration -async def test_full_evaluation_pipeline(): - result = await evaluate( - response="Water boils at 100°C at sea level.", - prompt="At what temperature does water boil?", - criteria=[{"name": "Accuracy", "description": "Factual correctness", "weight": 1}] - ) - - assert result.success - assert len(result.scores) == 1 - assert result.scores[0].score >= 4 # Should score high for accurate response -``` - -### Bias Detection Tests - -```python -async def test_position_bias_mitigation(): - # Same response in both positions should tie - result = await compare( - response_a="Same response", - response_b="Same response", - prompt="Test prompt", - criteria=["quality"], - swap_positions=True - ) - - assert result.winner == "TIE" - assert result.consistent == True -``` - diff --git a/.agents/skills/advanced-evaluation/references/metrics-guide.md b/.agents/skills/advanced-evaluation/references/metrics-guide.md deleted file mode 100644 index 829fa0e2c..000000000 --- a/.agents/skills/advanced-evaluation/references/metrics-guide.md +++ /dev/null @@ -1,331 +0,0 @@ -# Metric Selection Guide for LLM Evaluation - -This reference provides guidance on selecting appropriate metrics for different evaluation scenarios. - -## Metric Categories - -### Classification Metrics - -Use for binary or multi-class evaluation tasks (pass/fail, correct/incorrect). - -#### Precision - -``` -Precision = True Positives / (True Positives + False Positives) -``` - -**Interpretation**: Of all responses the judge said were good, what fraction were actually good? - -**Use when**: False positives are costly (e.g., approving unsafe content) - -```python -def precision(predictions, ground_truth): - true_positives = sum(1 for p, g in zip(predictions, ground_truth) if p == 1 and g == 1) - predicted_positives = sum(predictions) - return true_positives / predicted_positives if predicted_positives > 0 else 0 -``` - -#### Recall - -``` -Recall = True Positives / (True Positives + False Negatives) -``` - -**Interpretation**: Of all actually good responses, what fraction did the judge identify? - -**Use when**: False negatives are costly (e.g., missing good content in filtering) - -```python -def recall(predictions, ground_truth): - true_positives = sum(1 for p, g in zip(predictions, ground_truth) if p == 1 and g == 1) - actual_positives = sum(ground_truth) - return true_positives / actual_positives if actual_positives > 0 else 0 -``` - -#### F1 Score - -``` -F1 = 2 * (Precision * Recall) / (Precision + Recall) -``` - -**Interpretation**: Harmonic mean of precision and recall - -**Use when**: You need a single number balancing both concerns - -```python -def f1_score(predictions, ground_truth): - p = precision(predictions, ground_truth) - r = recall(predictions, ground_truth) - return 2 * p * r / (p + r) if (p + r) > 0 else 0 -``` - -### Agreement Metrics - -Use for comparing automated evaluation with human judgment. - -#### Cohen's Kappa (κ) - -``` -κ = (Observed Agreement - Expected Agreement) / (1 - Expected Agreement) -``` - -**Interpretation**: Agreement adjusted for chance -- κ > 0.8: Almost perfect agreement -- κ 0.6-0.8: Substantial agreement -- κ 0.4-0.6: Moderate agreement -- κ < 0.4: Fair to poor agreement - -**Use for**: Binary or categorical judgments - -```python -def cohens_kappa(judge1, judge2): - from sklearn.metrics import cohen_kappa_score - return cohen_kappa_score(judge1, judge2) -``` - -#### Weighted Kappa - -For ordinal scales where disagreement severity matters: - -```python -def weighted_kappa(judge1, judge2): - from sklearn.metrics import cohen_kappa_score - return cohen_kappa_score(judge1, judge2, weights='quadratic') -``` - -**Interpretation**: Penalizes large disagreements more than small ones - -### Correlation Metrics - -Use for ordinal/continuous scores. - -#### Spearman's Rank Correlation (ρ) - -**Interpretation**: Correlation between rankings, not absolute values -- ρ > 0.9: Very strong correlation -- ρ 0.7-0.9: Strong correlation -- ρ 0.5-0.7: Moderate correlation -- ρ < 0.5: Weak correlation - -**Use when**: Order matters more than exact values - -```python -def spearmans_rho(scores1, scores2): - from scipy.stats import spearmanr - rho, p_value = spearmanr(scores1, scores2) - return {'rho': rho, 'p_value': p_value} -``` - -#### Kendall's Tau (τ) - -**Interpretation**: Similar to Spearman but based on pairwise concordance - -**Use when**: You have many tied values - -```python -def kendalls_tau(scores1, scores2): - from scipy.stats import kendalltau - tau, p_value = kendalltau(scores1, scores2) - return {'tau': tau, 'p_value': p_value} -``` - -#### Pearson Correlation (r) - -**Interpretation**: Linear correlation between scores - -**Use when**: Exact score values matter, not just order - -```python -def pearsons_r(scores1, scores2): - from scipy.stats import pearsonr - r, p_value = pearsonr(scores1, scores2) - return {'r': r, 'p_value': p_value} -``` - -### Pairwise Comparison Metrics - -#### Agreement Rate - -``` -Agreement = (Matching Decisions) / (Total Comparisons) -``` - -**Interpretation**: Simple percentage of agreement - -```python -def pairwise_agreement(decisions1, decisions2): - matches = sum(1 for d1, d2 in zip(decisions1, decisions2) if d1 == d2) - return matches / len(decisions1) -``` - -#### Position Consistency - -``` -Consistency = (Consistent across position swaps) / (Total comparisons) -``` - -**Interpretation**: How often does swapping position change the decision? - -```python -def position_consistency(results): - consistent = sum(1 for r in results if r['position_consistent']) - return consistent / len(results) -``` - -## Selection Decision Tree - -``` -What type of evaluation task? -│ -├── Binary classification (pass/fail) -│ └── Use: Precision, Recall, F1, Cohen's κ -│ -├── Ordinal scale (1-5 rating) -│ ├── Comparing to human judgments? -│ │ └── Use: Spearman's ρ, Weighted κ -│ └── Comparing two automated judges? -│ └── Use: Kendall's τ, Spearman's ρ -│ -├── Pairwise preference -│ └── Use: Agreement rate, Position consistency -│ -└── Multi-label classification - └── Use: Macro-F1, Micro-F1, Per-label metrics -``` - -## Metric Selection by Use Case - -### Use Case 1: Validating Automated Evaluation - -**Goal**: Ensure automated evaluation correlates with human judgment - -**Recommended Metrics**: -1. Primary: Spearman's ρ (for ordinal scales) or Cohen's κ (for categorical) -2. Secondary: Per-criterion agreement -3. Diagnostic: Confusion matrix for systematic errors - -```python -def validate_automated_eval(automated_scores, human_scores, criteria): - results = {} - - # Overall correlation - results['overall_spearman'] = spearmans_rho(automated_scores, human_scores) - - # Per-criterion agreement - for criterion in criteria: - auto_crit = [s[criterion] for s in automated_scores] - human_crit = [s[criterion] for s in human_scores] - results[f'{criterion}_spearman'] = spearmans_rho(auto_crit, human_crit) - - return results -``` - -### Use Case 2: Comparing Two Models - -**Goal**: Determine which model produces better outputs - -**Recommended Metrics**: -1. Primary: Win rate (from pairwise comparison) -2. Secondary: Position consistency (bias check) -3. Diagnostic: Per-criterion breakdown - -```python -def compare_models(model_a_outputs, model_b_outputs, prompts): - results = [] - for a, b, p in zip(model_a_outputs, model_b_outputs, prompts): - comparison = await compare_with_position_swap(a, b, p) - results.append(comparison) - - return { - 'a_wins': sum(1 for r in results if r['winner'] == 'A'), - 'b_wins': sum(1 for r in results if r['winner'] == 'B'), - 'ties': sum(1 for r in results if r['winner'] == 'TIE'), - 'position_consistency': position_consistency(results) - } -``` - -### Use Case 3: Quality Monitoring - -**Goal**: Track evaluation quality over time - -**Recommended Metrics**: -1. Primary: Rolling agreement with human spot-checks -2. Secondary: Score distribution stability -3. Diagnostic: Bias indicators (position, length) - -```python -class QualityMonitor: - def __init__(self, window_size=100): - self.window = deque(maxlen=window_size) - - def add_evaluation(self, automated, human_spot_check=None): - self.window.append({ - 'automated': automated, - 'human': human_spot_check, - 'length': len(automated['response']) - }) - - def get_metrics(self): - # Filter to evaluations with human spot-checks - with_human = [e for e in self.window if e['human'] is not None] - - if len(with_human) < 10: - return {'insufficient_data': True} - - auto_scores = [e['automated']['score'] for e in with_human] - human_scores = [e['human']['score'] for e in with_human] - - return { - 'correlation': spearmans_rho(auto_scores, human_scores), - 'mean_difference': np.mean([a - h for a, h in zip(auto_scores, human_scores)]), - 'length_correlation': spearmans_rho( - [e['length'] for e in self.window], - [e['automated']['score'] for e in self.window] - ) - } -``` - -## Interpreting Metric Results - -### Good Evaluation System Indicators - -| Metric | Good | Acceptable | Concerning | -|--------|------|------------|------------| -| Spearman's ρ | > 0.8 | 0.6-0.8 | < 0.6 | -| Cohen's κ | > 0.7 | 0.5-0.7 | < 0.5 | -| Position consistency | > 0.9 | 0.8-0.9 | < 0.8 | -| Length correlation | < 0.2 | 0.2-0.4 | > 0.4 | - -### Warning Signs - -1. **High agreement but low correlation**: May indicate calibration issues -2. **Low position consistency**: Position bias affecting results -3. **High length correlation**: Length bias inflating scores -4. **Per-criterion variance**: Some criteria may be poorly defined - -## Reporting Template - -```markdown -## Evaluation System Metrics Report - -### Human Agreement -- Spearman's ρ: 0.82 (p < 0.001) -- Cohen's κ: 0.74 -- Sample size: 500 evaluations - -### Bias Indicators -- Position consistency: 91% -- Length-score correlation: 0.12 - -### Per-Criterion Performance -| Criterion | Spearman's ρ | κ | -|-----------|--------------|---| -| Accuracy | 0.88 | 0.79 | -| Clarity | 0.76 | 0.68 | -| Completeness | 0.81 | 0.72 | - -### Recommendations -- All metrics within acceptable ranges -- Monitor "Clarity" criterion - lower agreement may indicate need for rubric refinement -``` - diff --git a/.agents/skills/advanced-evaluation/scripts/evaluation_example.py b/.agents/skills/advanced-evaluation/scripts/evaluation_example.py deleted file mode 100644 index 3839487df..000000000 --- a/.agents/skills/advanced-evaluation/scripts/evaluation_example.py +++ /dev/null @@ -1,392 +0,0 @@ -"""Advanced Evaluation Example - -Use when: building LLM-as-judge evaluation pipelines, comparing model outputs -with position-bias mitigation, or generating domain-specific scoring rubrics. - -This module demonstrates the three core evaluation patterns from the -advanced-evaluation skill: direct scoring, pairwise comparison with position -swapping, and rubric generation. All functions use pseudocode-style examples -that work across Python environments without specific dependencies. -""" - -from __future__ import annotations - -from typing import Any - -__all__ = [ - "direct_scoring_example", - "pairwise_comparison_example", - "rubric_generation_example", -] - - -# ============================================================================= -# DIRECT SCORING EXAMPLE -# ============================================================================= - - -def direct_scoring_example() -> dict[str, Any]: - """Rate a single response against defined criteria using direct scoring. - - Use when: evaluating objective criteria like factual accuracy, instruction - following, or toxicity where a clear ground truth or rubric exists. - - Returns: - Dictionary containing per-criterion scores, evidence, justifications, - and a weighted summary. - """ - - # Input - prompt: str = "Explain quantum entanglement to a high school student" - response: str = ( - "Quantum entanglement is like having two magical coins that are connected. " - "When you flip one and it lands on heads, the other instantly shows tails, " - 'no matter how far apart they are. Scientists call this "spooky action at a distance."' - ) - - criteria: list[dict[str, Any]] = [ - {"name": "Accuracy", "description": "Scientific correctness", "weight": 0.4}, - {"name": "Clarity", "description": "Understandable for audience", "weight": 0.3}, - {"name": "Engagement", "description": "Interesting and memorable", "weight": 0.3}, - ] - - # System prompt for the evaluator - system_prompt: str = ( - "You are an expert evaluator. Assess the response against each criterion.\n\n" - "For each criterion:\n" - "1. Find specific evidence in the response\n" - "2. Score according to the rubric (1-5 scale)\n" - "3. Justify your score with evidence\n" - "4. Suggest one specific improvement\n\n" - "Be objective and consistent. Base scores on explicit evidence." - ) - - # User prompt structure - user_prompt: str = f"""## Original Prompt -{prompt} - -## Response to Evaluate -{response} - -## Criteria -1. **Accuracy** (weight: 0.4): Scientific correctness -2. **Clarity** (weight: 0.3): Understandable for audience -3. **Engagement** (weight: 0.3): Interesting and memorable - -## Output Format -Respond with valid JSON: -{{ - "scores": [ - {{ - "criterion": "Accuracy", - "score": 4, - "evidence": ["quote or observation"], - "justification": "why this score", - "improvement": "specific suggestion" - }} - ], - "summary": {{ - "assessment": "overall quality summary", - "strengths": ["strength 1"], - "weaknesses": ["weakness 1"] - }} -}}""" - - # Expected output structure - expected_output: dict[str, Any] = { - "scores": [ - { - "criterion": "Accuracy", - "score": 4, - "evidence": ["Correctly uses analogy", "Mentions spooky action at a distance"], - "justification": "Core concept is correct, analogy is appropriate", - "improvement": "Could mention it's a quantum mechanical phenomenon", - }, - { - "criterion": "Clarity", - "score": 5, - "evidence": ["Simple coin analogy", "No jargon"], - "justification": "Appropriate for high school level", - "improvement": "None needed", - }, - { - "criterion": "Engagement", - "score": 4, - "evidence": ["Magical coins", "Spooky action quote"], - "justification": "Memorable imagery and Einstein quote", - "improvement": "Could add a real-world application", - }, - ], - "summary": { - "assessment": "Good explanation suitable for the target audience", - "strengths": ["Clear analogy", "Age-appropriate language"], - "weaknesses": ["Could be more comprehensive"], - }, - } - - # Calculate weighted score - total_weight: float = sum(c["weight"] for c in criteria) - weighted_score: float = sum( - s["score"] * next(c["weight"] for c in criteria if c["name"] == s["criterion"]) - for s in expected_output["scores"] - ) / total_weight - - print(f"Weighted Score: {weighted_score:.2f}/5") - return expected_output - - -# ============================================================================= -# PAIRWISE COMPARISON WITH POSITION BIAS MITIGATION -# ============================================================================= - - -def pairwise_comparison_example() -> dict[str, Any]: - """Compare two responses with position-swapped bias mitigation. - - Use when: evaluating subjective preferences like tone, style, or - persuasiveness where pairwise comparison achieves higher human-judge - agreement than direct scoring. - - Returns: - Dictionary containing the winner, confidence score, and whether - position consistency was achieved across both passes. - """ - - prompt: str = "Explain machine learning to a beginner" - - response_a: str = ( - "Machine learning is a subset of artificial intelligence that enables " - "systems to learn and improve from experience without being explicitly " - "programmed. It uses statistical techniques to give computers the ability " - "to identify patterns in data." - ) - - response_b: str = ( - "Imagine teaching a dog a new trick. You show the dog what to do, give " - "treats when it's right, and eventually it learns. Machine learning works " - "similarly - we show computers lots of examples, tell them when they're " - "right, and they learn to recognize patterns on their own." - ) - - criteria: list[str] = ["clarity", "accessibility", "accuracy"] - - # System prompt emphasizing bias awareness - system_prompt: str = ( - "You are an expert evaluator comparing two AI responses.\n\n" - "CRITICAL INSTRUCTIONS:\n" - "- Do NOT prefer responses because they are longer\n" - "- Do NOT prefer responses based on position (first vs second)\n" - "- Focus ONLY on quality according to the specified criteria\n" - "- Ties are acceptable when responses are genuinely equivalent" - ) - - # Build evaluation prompt for a given ordering - def evaluate_pass( - first_response: str, - second_response: str, - first_label: str, - second_label: str, - ) -> str: - """Build evaluation prompt for one pass of position-swapped comparison. - - Use when: constructing the prompt for a single evaluation pass before - swapping response positions for bias mitigation. - """ - return f"""## Original Prompt -{prompt} - -## Response {first_label} -{first_response} - -## Response {second_label} -{second_response} - -## Comparison Criteria -{', '.join(criteria)} - -## Output Format -{{ - "comparison": [ - {{"criterion": "clarity", "winner": "A|B|TIE", "reasoning": "..."}} - ], - "result": {{ - "winner": "A|B|TIE", - "confidence": 0.0-1.0, - "reasoning": "overall reasoning" - }} -}}""" - - # Position bias mitigation protocol - print("Pass 1: A in first position") - pass1_result: dict[str, Any] = {"winner": "B", "confidence": 0.8} - - print("Pass 2: B in first position (swapped)") - pass2_result: dict[str, Any] = {"winner": "A", "confidence": 0.75} # A because B was first - - # Map pass2 result back (swap labels) - def map_winner(winner: str) -> str: - """Map winner label after position swap.""" - return {"A": "B", "B": "A", "TIE": "TIE"}[winner] - - pass2_mapped: str = map_winner(pass2_result["winner"]) - print(f"Pass 2 mapped winner: {pass2_mapped}") - - # Check consistency - consistent: bool = pass1_result["winner"] == pass2_mapped - - final_result: dict[str, Any] - if consistent: - final_result = { - "winner": pass1_result["winner"], - "confidence": (pass1_result["confidence"] + pass2_result["confidence"]) / 2, - "position_consistent": True, - } - else: - final_result = { - "winner": "TIE", - "confidence": 0.5, - "position_consistent": False, - "bias_detected": True, - } - - print(f"\nFinal Result: {final_result}") - return final_result - - -# ============================================================================= -# RUBRIC GENERATION -# ============================================================================= - - -def rubric_generation_example() -> dict[str, Any]: - """Generate a domain-specific scoring rubric for consistent evaluation. - - Use when: establishing evaluation standards for a new criterion, reducing - scoring variance (rubrics cut variance by 40-60%), or onboarding new - evaluators to an existing evaluation pipeline. - - Returns: - Dictionary containing score levels, characteristics, examples, - scoring guidelines, and edge case handling. - """ - - criterion_name: str = "Code Readability" - criterion_description: str = "How easy the code is to understand and maintain" - domain: str = "software engineering" - scale: str = "1-5" - strictness: str = "balanced" - - system_prompt: str = ( - f"You are an expert in creating evaluation rubrics.\n" - f"Create clear, actionable rubrics with distinct boundaries between levels.\n\n" - f"Strictness: {strictness}\n" - f"- lenient: Lower bar for passing scores\n" - f"- balanced: Fair, typical expectations\n" - f"- strict: High standards, critical evaluation" - ) - - user_prompt: str = f"""Create a scoring rubric for: - -**Criterion**: {criterion_name} -**Description**: {criterion_description} -**Scale**: {scale} -**Domain**: {domain} - -Generate: -1. Clear descriptions for each score level -2. Specific characteristics that define each level -3. Brief example text for each level -4. General scoring guidelines -5. Edge cases with guidance""" - - # Expected rubric structure - rubric: dict[str, Any] = { - "criterion": criterion_name, - "scale": {"min": 1, "max": 5}, - "levels": [ - { - "score": 1, - "label": "Poor", - "description": "Code is difficult to understand without significant effort", - "characteristics": [ - "No meaningful variable or function names", - "No comments or documentation", - "Deeply nested or convoluted logic", - ], - "example": "def f(x): return x[0]*x[1]+x[2]", - }, - { - "score": 3, - "label": "Adequate", - "description": "Code is understandable with some effort", - "characteristics": [ - "Most variables have meaningful names", - "Basic comments for complex sections", - "Logic is followable but could be cleaner", - ], - "example": ( - "def calc_total(items): # calculate sum\n" - " total = 0\n" - " for i in items: total += i\n" - " return total" - ), - }, - { - "score": 5, - "label": "Excellent", - "description": "Code is immediately clear and maintainable", - "characteristics": [ - "All names are descriptive and consistent", - "Comprehensive documentation", - "Clean, modular structure", - ], - "example": ( - "def calculate_total_price(items: List[Item]) -> Decimal:\n" - " '''Calculate the total price of all items.'''\n" - " return sum(item.price for item in items)" - ), - }, - ], - "scoring_guidelines": [ - "Focus on readability, not cleverness", - "Consider the intended audience (team skill level)", - "Consistency matters more than style preference", - ], - "edge_cases": [ - { - "situation": "Code uses domain-specific abbreviations", - "guidance": "Score based on readability for domain experts, not general audience", - }, - { - "situation": "Code is auto-generated", - "guidance": "Apply same standards but note in evaluation", - }, - ], - } - - print("Generated Rubric:") - for level in rubric["levels"]: - print(f" {level['score']}: {level['label']} - {level['description']}") - - return rubric - - -# ============================================================================= -# MAIN -# ============================================================================= - -if __name__ == "__main__": - print("=" * 60) - print("DIRECT SCORING EXAMPLE") - print("=" * 60) - direct_scoring_example() - - print("\n" + "=" * 60) - print("PAIRWISE COMPARISON EXAMPLE") - print("=" * 60) - pairwise_comparison_example() - - print("\n" + "=" * 60) - print("RUBRIC GENERATION EXAMPLE") - print("=" * 60) - rubric_generation_example() diff --git a/.agents/skills/ast-grep/SKILL.md b/.agents/skills/ast-grep/SKILL.md deleted file mode 100644 index c5f65300d..000000000 --- a/.agents/skills/ast-grep/SKILL.md +++ /dev/null @@ -1,325 +0,0 @@ ---- -name: ast-grep -description: Guide for writing ast-grep rules to perform structural code search and analysis. Use when users need to search codebases using Abstract Syntax Tree (AST) patterns, find specific code structures, or perform complex code queries that go beyond simple text search. This skill should be used when users ask to search for code patterns, find specific language constructs, or locate code with particular structural characteristics. -metadata: - provider: atomic ---- - -# ast-grep Code Search - -## Overview - -This skill helps translate natural language queries into ast-grep rules for structural code search. ast-grep uses Abstract Syntax Tree (AST) patterns to match code based on its structure rather than just text, enabling powerful and precise code search across large codebases. - -## When to Use This Skill - -Use this skill when users: -- Need to search for code patterns using structural matching (e.g., "find all async functions that don't have error handling") -- Want to locate specific language constructs (e.g., "find all function calls with specific parameters") -- Request searches that require understanding code structure rather than just text -- Ask to search for code with particular AST characteristics -- Need to perform complex code queries that traditional text search cannot handle - -## General Workflow - -Follow this process to help users write effective ast-grep rules: - -### Step 1: Understand the Query - -Clearly understand what the user wants to find. Ask clarifying questions if needed: -- What specific code pattern or structure are they looking for? -- Which programming language? -- Are there specific edge cases or variations to consider? -- What should be included or excluded from matches? - -### Step 2: Create Example Code - -Write a simple code snippet that represents what the user wants to match. Save this to a temporary file for testing. - -**Example:** -If searching for "async functions that use await", create a test file: - -```javascript -// test_example.js -async function example() { - const result = await fetchData(); - return result; -} -``` - -### Step 3: Write the ast-grep Rule - -Translate the pattern into an ast-grep rule. Start simple and add complexity as needed. - -**Key principles:** -- Always use `stopBy: end` for relational rules (`inside`, `has`) to ensure search goes to the end of the direction -- Use `pattern` for simple structures -- Use `kind` with `has`/`inside` for complex structures -- Break complex queries into smaller sub-rules using `all`, `any`, or `not` - -**Example rule file (test_rule.yml):** -```yaml -id: async-with-await -language: javascript -rule: - kind: function_declaration - has: - pattern: await $EXPR - stopBy: end -``` - -See `references/rule_reference.md` for comprehensive rule documentation. - -### Step 4: Test the Rule - -Use ast-grep CLI to verify the rule matches the example code. There are two main approaches: - -**Option A: Test with inline rules (for quick iterations)** -```bash -echo "async function test() { await fetch(); }" | ast-grep scan --inline-rules "id: test -language: javascript -rule: - kind: function_declaration - has: - pattern: await \$EXPR - stopBy: end" --stdin -``` - -**Option B: Test with rule files (recommended for complex rules)** -```bash -ast-grep scan --rule test_rule.yml test_example.js -``` - -**Debugging if no matches:** -1. Simplify the rule (remove sub-rules) -2. Add `stopBy: end` to relational rules if not present -3. Use `--debug-query` to understand the AST structure (see below) -4. Check if `kind` values are correct for the language - -### Step 5: Search the Codebase - -Once the rule matches the example code correctly, search the actual codebase: - -**For simple pattern searches:** -```bash -ast-grep run --pattern 'console.log($ARG)' --lang javascript /path/to/project -``` - -**For complex rule-based searches:** -```bash -ast-grep scan --rule my_rule.yml /path/to/project -``` - -**For inline rules (without creating files):** -```bash -ast-grep scan --inline-rules "id: my-rule -language: javascript -rule: - pattern: \$PATTERN" /path/to/project -``` - -## ast-grep CLI Commands - -### Inspect Code Structure (--debug-query) - -Dump the AST structure to understand how code is parsed: - -```bash -ast-grep run --pattern 'async function example() { await fetch(); }' \ - --lang javascript \ - --debug-query=cst -``` - -**Available formats:** -- `cst`: Concrete Syntax Tree (shows all nodes including punctuation) -- `ast`: Abstract Syntax Tree (shows only named nodes) -- `pattern`: Shows how ast-grep interprets your pattern - -**Use this to:** -- Find the correct `kind` values for nodes -- Understand the structure of code you want to match -- Debug why patterns aren't matching - -**Example:** -```bash -# See the structure of your target code -ast-grep run --pattern 'class User { constructor() {} }' \ - --lang javascript \ - --debug-query=cst - -# See how ast-grep interprets your pattern -ast-grep run --pattern 'class $NAME { $$$BODY }' \ - --lang javascript \ - --debug-query=pattern -``` - -### Test Rules (scan with --stdin) - -Test a rule against code snippet without creating files: - -```bash -echo "const x = await fetch();" | ast-grep scan --inline-rules "id: test -language: javascript -rule: - pattern: await \$EXPR" --stdin -``` - -**Add --json for structured output:** -```bash -echo "const x = await fetch();" | ast-grep scan --inline-rules "..." --stdin --json -``` - -### Search with Patterns (run) - -Simple pattern-based search for single AST node matches: - -```bash -# Basic pattern search -ast-grep run --pattern 'console.log($ARG)' --lang javascript . - -# Search specific files -ast-grep run --pattern 'class $NAME' --lang python /path/to/project - -# JSON output for programmatic use -ast-grep run --pattern 'function $NAME($$$)' --lang javascript --json . -``` - -**When to use:** -- Simple, single-node matches -- Quick searches without complex logic -- When you don't need relational rules (inside/has) - -### Search with Rules (scan) - -YAML rule-based search for complex structural queries: - -```bash -# With rule file -ast-grep scan --rule my_rule.yml /path/to/project - -# With inline rules -ast-grep scan --inline-rules "id: find-async -language: javascript -rule: - kind: function_declaration - has: - pattern: await \$EXPR - stopBy: end" /path/to/project - -# JSON output -ast-grep scan --rule my_rule.yml --json /path/to/project -``` - -**When to use:** -- Complex structural searches -- Relational rules (inside, has, precedes, follows) -- Composite logic (all, any, not) -- When you need the power of full YAML rules - -**Tip:** For relational rules (inside/has), always add `stopBy: end` to ensure complete traversal. - -## Tips for Writing Effective Rules - -### Always Use stopBy: end - -For relational rules, always use `stopBy: end` unless there's a specific reason not to: - -```yaml -has: - pattern: await $EXPR - stopBy: end -``` - -This ensures the search traverses the entire subtree rather than stopping at the first non-matching node. - -### Start Simple, Then Add Complexity - -Begin with the simplest rule that could work: -1. Try a `pattern` first -2. If that doesn't work, try `kind` to match the node type -3. Add relational rules (`has`, `inside`) as needed -4. Combine with composite rules (`all`, `any`, `not`) for complex logic - -### Use the Right Rule Type - -- **Pattern**: For simple, direct code matching (e.g., `console.log($ARG)`) -- **Kind + Relational**: For complex structures (e.g., "function containing await") -- **Composite**: For logical combinations (e.g., "function with await but not in try-catch") - -### Debug with AST Inspection - -When rules don't match: -1. Use `--debug-query=cst` to see the actual AST structure -2. Check if metavariables are being detected correctly -3. Verify the node `kind` matches what you expect -4. Ensure relational rules are searching in the right direction - -### Escaping in Inline Rules - -When using `--inline-rules`, escape metavariables in shell commands: -- Use `\$VAR` instead of `$VAR` (shell interprets `$` as variable) -- Or use single quotes: `'$VAR'` works in most shells - -**Example:** -```bash -# Correct: escaped $ -ast-grep scan --inline-rules "rule: {pattern: 'console.log(\$ARG)'}" . - -# Or use single quotes -ast-grep scan --inline-rules 'rule: {pattern: "console.log($ARG)"}' . -``` - -## Common Use Cases - -### Find Functions with Specific Content - -Find async functions that use await: -```bash -ast-grep scan --inline-rules "id: async-await -language: javascript -rule: - all: - - kind: function_declaration - - has: - pattern: await \$EXPR - stopBy: end" /path/to/project -``` - -### Find Code Inside Specific Contexts - -Find console.log inside class methods: -```bash -ast-grep scan --inline-rules "id: console-in-class -language: javascript -rule: - pattern: console.log(\$\$\$) - inside: - kind: method_definition - stopBy: end" /path/to/project -``` - -### Find Code Missing Expected Patterns - -Find async functions without try-catch: -```bash -ast-grep scan --inline-rules "id: async-no-trycatch -language: javascript -rule: - all: - - kind: function_declaration - - has: - pattern: await \$EXPR - stopBy: end - - not: - has: - pattern: try { \$\$\$ } catch (\$E) { \$\$\$ } - stopBy: end" /path/to/project -``` - -## Resources - -### references/ -Contains detailed documentation for ast-grep rule syntax: -- `rule_reference.md`: Comprehensive ast-grep rule documentation covering atomic rules, relational rules, composite rules, and metavariables - -Load these references when detailed rule syntax information is needed. diff --git a/.agents/skills/ast-grep/references/rule_reference.md b/.agents/skills/ast-grep/references/rule_reference.md deleted file mode 100644 index 9821e35ff..000000000 --- a/.agents/skills/ast-grep/references/rule_reference.md +++ /dev/null @@ -1,297 +0,0 @@ -# ast-grep Rule Reference - -This document provides comprehensive documentation for ast-grep rule syntax, covering all rule types and metavariables. - -## Introduction to ast-grep Rules - -ast-grep rules are declarative specifications for matching and filtering Abstract Syntax Tree (AST) nodes. They enable structural code search and analysis by defining conditions an AST node must meet to be matched. - -### Rule Categories - -ast-grep rules are categorized into three types: - -* **Atomic Rules**: Match individual AST nodes based on intrinsic properties like code patterns (`pattern`), node type (`kind`), or text content (`regex`). -* **Relational Rules**: Define conditions based on a target node's position or relationship to other nodes (e.g., `inside`, `has`, `precedes`, `follows`). -* **Composite Rules**: Combine other rules using logical operations (AND, OR, NOT) to form complex matching criteria (e.g., `all`, `any`, `not`, `matches`). - -## Anatomy of an ast-grep Rule Object - -The ast-grep rule object is the core configuration unit defining how ast-grep identifies and filters AST nodes. It's typically written in YAML format. - -### General Structure - -Every field within an ast-grep Rule Object is optional, but at least one "positive" key (e.g., `kind`, `pattern`) must be present. - -A node matches a rule if it satisfies all fields defined within that rule object, implying an implicit logical AND operation. - -For rules using metavariables that depend on prior matching, explicit `all` composite rules are recommended to guarantee execution order. - -### Rule Object Properties - -| Property | Type | Category | Purpose | Example | -| :--- | :--- | :--- | :--- | :--- | -| `pattern` | String or Object | Atomic | Matches AST node by code pattern. | `pattern: console.log($ARG)` | -| `kind` | String | Atomic | Matches AST node by its kind name. | `kind: call_expression` | -| `regex` | String | Atomic | Matches node's text by Rust regex. | `regex: ^[a-z]+$` | -| `nthChild` | number, string, Object | Atomic | Matches nodes by their index within parent's children. | `nthChild: 1` | -| `range` | RangeObject | Atomic | Matches node by character-based start/end positions. | `range: { start: { line: 0, column: 0 }, end: { line: 0, column: 10 } }` | -| `inside` | Object | Relational | Target node must be inside node matching sub-rule. | `inside: { pattern: class $C { $$$ }, stopBy: end }` | -| `has` | Object | Relational | Target node must have descendant matching sub-rule. | `has: { pattern: await $EXPR, stopBy: end }` | -| `precedes` | Object | Relational | Target node must appear before node matching sub-rule. | `precedes: { pattern: return $VAL }` | -| `follows` | Object | Relational | Target node must appear after node matching sub-rule. | `follows: { pattern: import $M from '$P' }` | -| `all` | Array | Composite | Matches if all sub-rules match. | `all: [ { kind: call_expression }, { pattern: foo($A) } ]` | -| `any` | Array | Composite | Matches if any sub-rules match. | `any: [ { pattern: foo() }, { pattern: bar() } ]` | -| `not` | Object | Composite | Matches if sub-rule does not match. | `not: { pattern: console.log($ARG) }` | -| `matches` | String | Composite | Matches if predefined utility rule matches. | `matches: my-utility-rule-id` | - -## Atomic Rules - -Atomic rules match individual AST nodes based on their intrinsic properties. - -### pattern: String and Object Forms - -The `pattern` rule matches a single AST node based on a code pattern. - -**String Pattern**: Directly matches using ast-grep's pattern syntax with metavariables. - -```yaml -pattern: console.log($ARG) -``` - -**Object Pattern**: Offers granular control for ambiguous patterns or specific contexts. - -* `selector`: Pinpoints a specific part of the parsed pattern to match. - ```yaml - pattern: - selector: field_definition - context: class { $F } - ``` - -* `context`: Provides surrounding code context for correct parsing. - -* `strictness`: Modifies the pattern's matching algorithm (`cst`, `smart`, `ast`, `relaxed`, `signature`). - ```yaml - pattern: - context: foo($BAR) - strictness: relaxed - ``` - -### kind: Matching by Node Type - -The `kind` rule matches an AST node by its `tree_sitter_node_kind` name, derived from the language's Tree-sitter grammar. Useful for targeting constructs like `call_expression` or `function_declaration`. - -```yaml -kind: call_expression -``` - -### regex: Text-Based Node Matching - -The `regex` rule matches the entire text content of an AST node using a Rust regular expression. It's not a "positive" rule, meaning it matches any node whose text satisfies the regex, regardless of its structural kind. - -### nthChild: Positional Node Matching - -The `nthChild` rule finds nodes by their 1-based index within their parent's children list, counting only named nodes by default. - -* `number`: Matches the exact nth child. Example: `nthChild: 1` -* `string`: Matches positions using An+B formula. Example: `2n+1` -* `Object`: Provides granular control: - * `position`: `number` or An+B string. - * `reverse`: `true` to count from the end. - * `ofRule`: An ast-grep rule to filter the sibling list before counting. - -### range: Position-Based Node Matching - -The `range` rule matches an AST node based on its character-based start and end positions. A `RangeObject` defines `start` and `end` fields, each with 0-based `line` and `column`. `start` is inclusive, `end` is exclusive. - -## Relational Rules - -Relational rules filter targets based on their position relative to other AST nodes. They can include `stopBy` and `field` options. - -### inside: Matching Within a Parent Node - -Requires the target node to be inside another node matching the `inside` sub-rule. - -```yaml -inside: - pattern: class $C { $$$ } - stopBy: end -``` - -### has: Matching with a Descendant Node - -Requires the target node to have a descendant node matching the `has` sub-rule. - -```yaml -has: - pattern: await $EXPR - stopBy: end -``` - -### precedes and follows: Sequential Node Matching - -* `precedes`: Target node must appear before a node matching the `precedes` sub-rule. -* `follows`: Target node must appear after a node matching the `follows` sub-rule. - -Both include `stopBy` but not `field`. - -### stopBy and field: Refining Relational Searches - -**stopBy**: Controls search termination for relational rules. - -* `"neighbor"` (default): Stops when immediate surrounding node doesn't match. -* `"end"`: Searches to the end of the direction (root for `inside`, leaf for `has`). -* `Rule object`: Stops when a surrounding node matches the provided rule (inclusive). - -**field**: Specifies a sub-node within the target node that should match the relational rule. Only for `inside` and `has`. - -**Best Practice**: When unsure, always use `stopBy: end` to ensure the search goes to the end of the direction. - -## Composite Rules - -Composite rules combine atomic and relational rules using logical operations. - -### all: Conjunction (AND) of Rules - -Matches a node only if all sub-rules in the list match. Guarantees order of rule matching, important for metavariables. - -```yaml -all: - - kind: call_expression - - pattern: console.log($ARG) -``` - -### any: Disjunction (OR) of Rules - -Matches a node if any sub-rules in the list match. - -```yaml -any: - - pattern: console.log($ARG) - - pattern: console.warn($ARG) - - pattern: console.error($ARG) -``` - -### not: Negation (NOT) of a Rule - -Matches a node if the single sub-rule does not match. - -```yaml -not: - pattern: console.log($ARG) -``` - -### matches: Rule Reuse and Utility Rules - -Takes a rule-id string, matching if the referenced utility rule matches. Enables rule reuse and recursive rules. - -## Metavariables - -Metavariables are placeholders in patterns to match dynamic content in the AST. - -### $VAR: Single Named Node Capture - -Captures a single named node in the AST. - -* **Valid**: `$META`, `$META_VAR`, `$_` -* **Invalid**: `$invalid`, `$123`, `$KEBAB-CASE` -* **Example**: `console.log($GREETING)` matches `console.log('Hello World')`. -* **Reuse**: `$A == $A` matches `a == a` but not `a == b`. - -### $$VAR: Single Unnamed Node Capture - -Captures a single unnamed node (e.g., operators, punctuation). - -**Example**: To match the operator in `a + b`, use `$$OP`. - -```yaml -rule: - kind: binary_expression - has: - field: operator - pattern: $$OP -``` - -### $$$MULTI_META_VARIABLE: Multi-Node Capture - -Matches zero or more AST nodes (non-greedy). Useful for variable numbers of arguments or statements. - -* **Example**: `console.log($$$)` matches `console.log()`, `console.log('hello')`, and `console.log('debug:', key, value)`. -* **Example**: `function $FUNC($$$ARGS) { $$$ }` matches functions with varying parameters/statements. - -### Non-Capturing Metavariables (_VAR) - -Metavariables starting with an underscore (`_`) are not captured. They can match different content even if named identically, optimizing performance. - -* **Example**: `$_FUNC($_FUNC)` matches `test(a)` and `testFunc(1 + 1)`. - -### Important Considerations for Metavariable Detection - -* **Syntax Matching**: Only exact metavariable syntax (e.g., `$A`, `$$B`, `$$$C`) is recognized. -* **Exclusive Content**: Metavariable text must be the only text within an AST node. -* **Non-working**: `obj.on$EVENT`, `"Hello $WORLD"`, `a $OP b`, `$jq`. - -The ast-grep playground is useful for debugging patterns and visualizing metavariables. - -## Common Patterns and Examples - -### Finding Functions with Specific Content - -Find functions that contain await expressions: - -```yaml -rule: - kind: function_declaration - has: - pattern: await $EXPR - stopBy: end -``` - -### Finding Code Inside Specific Contexts - -Find console.log calls inside class methods: - -```yaml -rule: - pattern: console.log($$$) - inside: - kind: method_definition - stopBy: end -``` - -### Combining Multiple Conditions - -Find async functions that use await but don't have try-catch: - -```yaml -rule: - all: - - kind: function_declaration - - has: - pattern: await $EXPR - stopBy: end - - not: - has: - pattern: try { $$$ } catch ($E) { $$$ } - stopBy: end -``` - -### Matching Multiple Alternatives - -Find any type of console method call: - -```yaml -rule: - any: - - pattern: console.log($$$) - - pattern: console.warn($$$) - - pattern: console.error($$$) - - pattern: console.debug($$$) -``` - -## Troubleshooting Tips - -1. **Rule doesn't match**: Use `dump_syntax_tree` to see the actual AST structure -2. **Relational rule issues**: Ensure `stopBy: end` is set for deep searches -3. **Wrong node kind**: Check the language's Tree-sitter grammar for correct kind names -4. **Metavariable not working**: Ensure it's the only content in its AST node -5. **Pattern too complex**: Break it down into simpler sub-rules using `all` diff --git a/.agents/skills/atomic/SKILL.md b/.agents/skills/atomic/SKILL.md deleted file mode 100644 index 0fb605d3d..000000000 --- a/.agents/skills/atomic/SKILL.md +++ /dev/null @@ -1,645 +0,0 @@ ---- -name: atomic -description: | - The Atomic guide. Activate whenever the user asks how Atomic works, when to use - which workflow or skill, how to chain research → spec → implementation, how to - create custom workflows, how to refine a prompt, how to see available workflows, - or any "how do I" / "when do I" / decision question about using Atomic. Also - handles `/atomic what's new` for recent releases, `/atomic example` for a - spec-driven dev walkthrough, and `/atomic workflows` for the workflow primer - and custom workflow creation guide. -metadata: - provider: atomic ---- - -You are the **Atomic guide**. Users come to you with questions about using Atomic -— its workflows, skills, subagents, or which to reach for. Answer using the -canonical content blocks below. For questions outside the canonical set, read -Atomic's source. Read like good documentation: state what the user can run, show -the command, move on. No hype, no marketing. - -## Argument routing - -The user invoked you with `$ARGUMENTS` after `/atomic`. Branch on it: - -- **Empty / no args** → render the **help menu** block. -- **"overview"** → render the **30-second overview** block. -- **"example"** → render the **/atomic example** block (spec-driven dev with built-ins). -- **"workflows"** → render the **/atomic workflows** block (primer + custom workflows). -- **"what's new" / "whats new" / "news" / "updates" / "changelog"** → run the **What's New** flow. -- **Anything else** → treat as a question. First match against the **canonical Q&A blocks**. If none match, fall back to **source-reading**. - -Every output ends with the standard **cross-nudge close**. - -## Detect the calling agent - -The skill runs inside one of three coding agents. Detect which by reading env -vars in order, then substitute the matching values throughout user-facing -examples. Never list multiple agents' values side-by-side — show only the -detected one. - -| Detected | Env signal | `-a` flag | Display name | Agents directory | -|---|---|---|---|---| -| Claude Code | `CLAUDECODE=1` | `-a claude` | Claude Code | `.claude/agents/` | -| GitHub Copilot CLI | `COPILOT_AGENT_ID` or `COPILOT_ALLOW_ALL` | `-a copilot` | GitHub Copilot CLI | `.github/agents/` | -| OpenCode | `OPENCODE_CLIENT` or `OPENCODE_CONFIG*` | `-a opencode` | OpenCode | `.opencode/agents/` | - -Probe with: `printenv | grep -E '^(CLAUDECODE|COPILOT_|OPENCODE_)' | head -20`. If -none match, default to `-a claude` and `Claude Code` as the display name. - -Throughout this skill, any token like `` or `` refers to the -row matching the detected agent. Substitute before showing the user — never leak -placeholders or alternate-agent values into user-facing prose. - -## Formatting: command highlighting - -The templates below already wrap `>`-prefixed command lines in markdown inline -code (single backticks) so they render as monospace pills, e.g. -`` `> /atomic overview` ``. Render the templates verbatim — the backticks are -intentional. - -All templates are written as **markdown-native** content — paragraph text, -headings, lists, and tables — never as fenced ``` ``` ``` blocks or 4+ space -indented code blocks. Keep them that way; backticks don't render as inline -code inside a code block. - -If you ever need to render a new `>`-prefixed line (e.g., in a source-reading -fallback answer), wrap the `>` and command text in single backticks the same -way. Don't wrap lines that begin with `$`, `#`, or a bare command, and don't -wrap `>` characters that appear mid-sentence (e.g., in arrows like -`research → spec → implementation` or comparisons like `>=`). - -## Cross-nudge close - -Every block **except the help menu** — overview, example, workflows, Q&A -answer, what's new — ends with a short "where to next" pointer to two -relevant other modes plus `/atomic `. This makes the surface -recursively discoverable: a user who runs one /atomic command always learns -about the others. The `/atomic ` escape valve is always included as -the third item. - -**The help menu (`/atomic` with no args) does NOT get a cross-nudge close** — -the menu itself already lists every entry point, so a "where to next" footer -would be redundant. - -Format (render as plain paragraph text — **not** inside a fenced code block — -so the backticks render as inline-code pills): - - ───────────────────────────────────────────────────────────────── - - Where to next: - - `> /atomic ` - `> /atomic ` - `> /atomic ` always available — ask anything - -Pick the two non-Q&A pointers based on what's most useful next: - -| User just ran | Suggest as next | -|---|---| -| `/atomic overview` | `/atomic example`, `/atomic workflows` | -| `/atomic example` | `/atomic workflows`, `/atomic overview` | -| `/atomic workflows` | `/atomic example`, `/atomic overview` | -| `/atomic ` | `/atomic example`, `/atomic workflows` | -| `/atomic what's new` | `/atomic example`, `/atomic overview` | - ---- - -## Block: help menu (`/atomic` no args) - -Rendered when the user invokes `/atomic` with no arguments. Short -table-of-contents pointing at the longer blocks. - -Render verbatim. **Do not** enclose this block in a fenced code block — the -backticks below render as inline-code monospace pills only when this is -plain paragraph text. - -Atomic. Select where to start: - - `> /atomic overview` 30-second overview of workflows, skills, subagents - `> /atomic example` spec-driven development walkthrough on this repo - `> /atomic workflows` reliably automate complex engineering work - `> /atomic ` ask anything ("when do I use X?", "how do I…") - ---- - -## Block: 30-second overview (`/atomic overview`) - -Use this when the user invokes `/atomic overview`. It's a single-screen overview — workflows and skills together. Deliver this in **one turn**. - -**Formatting rules for this block** — readability matters more than density here. Follow these exactly: -- Use the `##` subheadings shown below (they render larger, with extra vertical space). -- Insert a blank line between every paragraph, list, and code block — never let two blocks touch. -- Insert a `---` horizontal rule between the two major sections (Workflows / Skills) for a clean visual break. -- Keep paragraphs to **one or two sentences max**. Break run-on prose into bullets. - -Output the content below verbatim (substitute `` with the detected agent): - ---- - -## ✦ Workflows - -Deterministic multi-stage pipelines that wrap your coding agent. Three built-ins: - -| Workflow | What it does | -|---|---| -| **`deep-research-codebase`** | Crawls the full repo and writes a grounded research file for one big question (auth flow, migration planning, end-to-end traces) | -| **`ralph`** | Plan → orchestrate → review → simplify code — the loop prevents context and code drift, which is what lets long-running tasks finish reliably | -| **`open-claude-design`** | Discover design system → generate → refine → export; produces high-fidelity designs that follow your existing design system | - -**Three ways to invoke a workflow:** - -| Mode | Example | Best for | -|---|---|---| -| Natural language | `atomic chat -a ` → `> run deep-research-codebase on how payments retries work end-to-end` | Day-to-day — what you'll use most | -| Picker | `atomic workflow -a ` | Browsing what's available | -| Long form | `atomic workflow -n ralph -a "harden the retry path with idempotency keys"` | CI, cron, shell scripts | - -**Canonical path for repo-wide work:** `deep-research-codebase` → `ralph`. Use this for heavy research across the whole repo — full migrations, end-to-end audits, cross-cutting refactors. The first stage crawls the repo and writes a grounded research file; the second implements against it with a bounded plan → orchestrate → review → simplify loop. - -*Inside `atomic chat -a `:* - - `> run deep-research-codebase on how our payments service handles retries end-to-end` - - `> use ralph with research/2026-05-08-payments-retries.md to harden the retry path` - -For work scoped to a portion of the repo, use the **`/research-codebase`** skill instead — see Skills below. - -**Write your own.** Describe a workflow in plain English to **`/workflow-creator`** — it generates a `defineWorkflow().run().compile()` TypeScript file. Be specific about stages, models, and outputs. - ---- - -## ✦ Skills - -Scoped expertise your coding agent summons mid-conversation with `/skill-name`. - -**Research a slice of the repo.** When you're working on a portion of the codebase rather than the whole thing, use **`/research-codebase`** instead of the `deep-research-codebase` workflow — scoped, cheaper, faster. Optional middle step: **`/create-spec`** turns the research into a precise spec when requirements are fuzzy. Then `ralph` implements: - -*Inside `atomic chat -a `:* - - `> /research-codebase how the rate limiter works in src/middleware/` - - `> /create-spec from research/2026-05-08-rate-limiter.md` *(optional)* - - `> use ralph with specs/2026-05-08-rate-limiter.md to add a per-user budget tier` - -**Sharpen a prompt before you ship it.** Use **`/prompt-engineer`** to refine a vague ask into a precise, well-structured prompt before handing it to a workflow or agent — especially worthwhile for `ralph`, `deep-research-codebase`, or any long-running run where a fuzzy prompt costs you a full loop. - -Run **`/find-skills `** to discover the rest of the catalog on demand. - ---- - -Then render the standard cross-nudge close as plain paragraph text (not inside -a fenced block — the backticks must render as inline-code pills): - - ───────────────────────────────────────────────────────────────── - - Where to next: - - `> /atomic example` see this used end-to-end (spec-driven development) - `> /atomic workflows` reliably automate complex engineering work - `> /atomic ` always available — ask anything - ---- - -## Block: /atomic example (spec-driven dev with built-ins) - -When the user invokes `/atomic example`, render the block below verbatim. -Substitute today's date (YYYY-MM-DD) wherever the placeholder `` -appears in example file paths so the dates feel current rather than frozen. - -**Spec-driven development with Atomic** — three steps. Only step 1 changes by scope. - -## 1. Research - -Default — for any portion of the codebase, even a large one spanning many files, folders, or a whole subsystem: - - `> /research-codebase how the rate limiter works in src/middleware/` - -Escalate to whole-repo only when you genuinely need every corner in scope (cross-cutting audit, full migration, end-to-end trace across services): - - `> run deep-research-codebase on how payments retries work end-to-end` - -→ writes `research/-.md` - -## 2. Spec *(optional — skip if your prompt is already tight)* - - `> /create-spec from research/-.md` - -→ writes `specs/-.md` - -## 3. Implement - - `> use ralph with to ` - -Then render the cross-nudge close as plain paragraph text (not inside a fence): - - ───────────────────────────────────────────────────────────────── - - Where to next: - - `> /atomic workflows` reliably automate complex engineering work - `> /atomic overview` quick refresh on the catalog - `> /atomic ` always available — ask anything - ---- - -## Block: /atomic workflows (primer + custom workflow creation) - -When the user invokes `/atomic workflows`, render the block below verbatim. -Substitute `` with the detected agent's `-a` flag value (`claude`, -`copilot`, or `opencode`) consistently throughout. Do **not** reference -`@bastani/atomic-sdk` in user-facing output. - -**Workflows in Atomic** - -A workflow is a deterministic, multi-stage pipeline — defined as a TypeScript file using `defineWorkflow().run().compile()` — that wraps your coding agent so the same complex job runs the same way every time. - -For example: a workflow that takes your open GitHub issues, generates a PR for each, runs an automated code review pass, and surfaces the results for an engineer to approve before merge. - -## Three built-in workflows - -| Workflow | What it does | -|---|---| -| **`deep-research-codebase`** | Crawls the full repo and writes a grounded research file for one big question (migrations, audits, traces). | -| **`ralph`** | Plan → orchestrate → review → simplify code — the loop prevents context and code drift, which is what lets long-running tasks finish reliably | -| **`open-claude-design`** | Discover design system → generate → refine → export; produces high-fidelity designs that follow your existing design system. | - -## Three ways to invoke any workflow - -| Mode | Example | Best for | -|---|---|---| -| Natural language | `> run deep-research-codebase on ` (inside `atomic chat -a `) | Day-to-day — what you'll use most | -| Picker | `atomic workflow -a ` | Browsing what's available | -| Long form | `atomic workflow -n ralph -a ""` | CI, cron, shell scripts | - ---- - -## Writing your own workflow - -Use **`/workflow-creator`**. It takes a plain-English description and generates the TypeScript file for you. The single biggest factor in output quality is prompt specificity. A good prompt names: - -- **The trigger** — what kicks it off (event, file pattern, CLI arg?) -- **The stages** — sequential? parallel fan-out? -- **The model per stage** — opus 4.7 xhigh? haiku for cheap fan-out? -- **The final artifact** — PR comment? research file? JSON report? -- **Failure handling** — skip? retry? abort? - -An example that works today: - - `> use the workflow-creator to create a code-review workflow that goes through GitHub and reviews all PRs tagged "review needed" — first pass using opus 4.7 xhigh, second pass using gpt 5.5 xhigh to reduce false negatives, then aggregates a single review comment on each PR with the merged feedback` - -Once `/workflow-creator` generates the code-review workflow, run it via the picker (`atomic workflow -a `) or from chat: - - `> run our code-review workflow for all the PRs in our backlog` - -Then render the cross-nudge close as plain paragraph text (not inside a fence): - - ───────────────────────────────────────────────────────────────── - - Where to next: - - `> /atomic example` see workflows used end-to-end with skills - `> /atomic overview` quick refresh on the full catalog - `> /atomic ` always available — ask anything - ---- - -## Canonical Q&A blocks - -When the user asks a free-form `/atomic `, match the question against -the canonical answers below. If matched, render the answer block verbatim -(lightly adapted to mirror the user's phrasing in the lead-in line, but keep -the body unchanged). Always end with the standard cross-nudge close. - -Substitute `` with today's date (YYYY-MM-DD) and `` with the -detected agent's `-a` flag value (`claude`, `copilot`, `opencode`). - -### Q1 — When deep-research-codebase vs /research-codebase? - -**Match phrasings:** "when do I use deep-research-codebase", "deep-research vs research-codebase", "research workflow vs research skill", "scope of research", "should I run deep-research-codebase or /research-codebase", "which research command", "research-codebase or deep-research". - -**Render:** - -**Start with `/research-codebase`. Escalate only when you need the whole repo.** - -- **Default — `/research-codebase`** *(skill)* — works for any portion of the codebase: one file, many files, a folder, or a whole subsystem. e.g. *"how does the rate limiter work in `src/middleware/`"* -- **Escalate — `deep-research-codebase`** *(workflow)* — only when you genuinely need every corner in scope: cross-cutting audit, full migration, end-to-end trace across services. e.g. *"how does our auth flow work end-to-end across all services"* - -**Then decide on spec.** - -- Requirements are tight (named files, concrete acceptance criteria) → skip `/create-spec`, go straight to `ralph` with the research file. -- Requirements are fuzzy (vague verbs, open edges in the research) → `/create-spec` from the research file, answer its questions, then hand the spec to `ralph`. - -Final command in either case: - - `> use ralph with to ` - -Rule of thumb: the skill handles almost everything. Reach for the workflow only when the answer truly requires the whole repo — it's heavier and writes a durable team artifact. - -Then render the cross-nudge close as plain paragraph text (not inside a fence): - - ───────────────────────────────────────────────────────────────── - - Where to next: - - `> /atomic example` see both paths used end-to-end - `> /atomic workflows` reliably automate complex engineering work - `> /atomic ` always available — ask anything - -### Q2 — When should I run /create-spec, and when should I skip? - -**Match phrasings:** "when do I use /create-spec", "when to skip create-spec", "do I need create-spec", "create-spec or skip", "should I run /create-spec", "is /create-spec necessary". - -**Render:** - -**`/create-spec`** turns a research file into a precise spec by interviewing you on the parts that aren't clear yet. Skip it when your prompt is already tight; reach for it when requirements are fuzzy. - -**Skip `/create-spec` when:** - -- You can name the files and symbols you'll touch -- Acceptance criteria are concrete (e.g., *"add per-user budget tier with hourly reset, return 429 above threshold"*) -- The research file already answers your open questions - -**Reach for `/create-spec` when:** - -- Verbs in your prompt are vague (*"improve"*, *"fix"*, *"make better"*) -- The research surfaced edges you don't have answers for -- Multiple stakeholders will touch the same code and you want them aligned before `ralph` starts - -Use it like: - - `> /create-spec from research/-.md` - -↳ writes `specs/-.md` after the interview. - -Then hand the spec to ralph: - - `> use ralph with specs/-.md to ` - -Then render the cross-nudge close as plain paragraph text (not inside a fence): - - ───────────────────────────────────────────────────────────────── - - Where to next: - - `> /atomic example` see /create-spec in the spec-driven dev path - `> /atomic workflows` reliably automate complex engineering work - `> /atomic ` always available — ask anything - -### Q3 — How do I refine a prompt? (/prompt-engineer) - -**Match phrasings:** "how to refine my prompt", "how to improve my prompt", "/prompt-engineer", "make my prompt better", "tighten my prompt", "before running a workflow", "prompt is vague". - -**Render:** - -**`/prompt-engineer`** sharpens a fuzzy prompt before a long-running job. Small upfront investment, big quality lift. - -**Reach for it when:** - -- Your prompt has vague verbs (*"improve"*, *"fix"*, *"make better"*) -- You're about to spend real tokens on a workflow (`deep-research-codebase`, `ralph`, `open-claude-design`, or a custom one) -- Output quality has been inconsistent on similar prompts -- You're handing a prompt to `ralph` or `deep-research-codebase` and want one more pass before committing - -Skip it when your prompt already names files, symbols, or concrete acceptance criteria. - -Use it like: - - `> /prompt-engineer rewrite this for clarity: "make the auth flow better and add tests"` - -↳ returns a tightened version you can paste into your next command. - -Then render the cross-nudge close as plain paragraph text (not inside a fence): - - ───────────────────────────────────────────────────────────────── - - Where to next: - - `> /atomic example` spec-driven development end-to-end - `> /atomic workflows` reliably automate complex engineering work - `> /atomic ` always available — ask anything - -### Q4 — How do I see what workflows are available? - -**Match phrasings:** "how to see workflows", "list workflows", "what workflows are available", "show workflows", "browse workflows", "available atomic workflows", "find workflows". - -**Render:** - -**Three ways to see what workflows are available:** - -1. **Picker** — interactive list with descriptions: `atomic workflow -a ` -2. **From inside chat** — the agent answers from its loaded skill list: - - `> what atomic workflows are available` - -3. **Read the source directly** — `ls .atomic/workflows/` (project-local) or `ls ~/.atomic/workflows/` (user-global). Each subdirectory is one workflow. - -**Three built-ins ship with every install:** - -| Workflow | What it does | -|---|---| -| **`deep-research-codebase`** | Whole-repo crawl → grounded research file for one big question | -| **`ralph`** | Plan → orchestrate → review → simplify code — the loop prevents context and code drift, which is what lets long-running tasks finish reliably | -| **`open-claude-design`** | High-fidelity designs that follow your existing design system | - -Then render the cross-nudge close as plain paragraph text (not inside a fence): - - ───────────────────────────────────────────────────────────────── - - Where to next: - - `> /atomic workflows` reliably automate complex engineering work - `> /atomic example` see workflows used end-to-end - `> /atomic ` always available — ask anything - -### Q5 — How do I create my own workflow? - -**Match phrasings:** "create custom workflow", "make my own workflow", "build a workflow", "/workflow-creator", "write a workflow", "design a workflow", "custom workflow", "how do I write a workflow". - -**Render:** - -Use **`/workflow-creator`**. It takes a plain-English description and generates a TypeScript workflow file you can run today. - -The single biggest factor in output quality is prompt specificity. A good prompt names: - -- **The trigger** — what kicks it off (event, file pattern, CLI arg?) -- **The stages** — sequential? parallel fan-out? -- **The model per stage** — opus 4.7 xhigh? haiku for cheap fan-out? -- **The final artifact** — PR comment? research file? JSON report? -- **Failure handling** — skip? retry? abort? - -An example that works today: - - `> use /workflow-creator to create a code-review workflow that goes through GitHub and reviews all PRs tagged "review needed" — first pass using opus 4.7 xhigh, second pass using gpt 5.5 xhigh to reduce false negatives, then aggregates a single review comment on each PR with the merged feedback` - -Once the file is generated, register and run it: - - `atomic workflow refresh` — picks up the new workflow - - `atomic workflow -n -a ` — run it - -Then render the cross-nudge close as plain paragraph text (not inside a fence): - - ───────────────────────────────────────────────────────────────── - - Where to next: - - `> /atomic workflows` reliably automate complex engineering work - `> /atomic example` spec-driven dev end-to-end with built-ins - `> /atomic ` always available — ask anything - ---- - -## Source-reading fallback - -When no canonical Q&A block matches, read Atomic's source rather than improvising -from training data. Your goal is a focused, verifiable answer — not a doc dump. - -### Procedure - -1. Try to match against the canonical Q&A blocks above first. -2. If no match, identify the topic from the user's question and read the - relevant files using the topic→path routing table below. -3. Cite file paths in your answer (e.g., `packages/atomic-sdk/src/runtime/runner.ts:42`) - so users can verify what you said. -4. Keep the answer focused on the user's question. Don't paste large file - sections — summarize and cite. -5. Always end with the standard cross-nudge close (see the format below). - -### Topic → path routing - -| Topic the user asked about | Where to look | -|---|---| -| Workflow runtime / dispatch / stages | `packages/atomic-sdk/src/runtime/`, `packages/atomic-sdk/src/workflows/` | -| Agent SDK adapters (Claude / Copilot / OpenCode) | `packages/atomic-sdk/src/providers/` | -| Skill loading & discovery | `.agents/skills/` | -| CLI entry and commands | `packages/atomic/src/cli.ts`, `packages/atomic/src/commands/` — also run `atomic --help` and `atomic --help` | -| Built-in workflow definitions | `packages/atomic-sdk/src/workflows/builtin/` | -| User-custom workflow definitions | `.atomic/workflows/` (project-local), `~/.atomic/workflows/` (user-global) | -| Subagent definitions | `.claude/agents/` (Claude Code), `.github/agents/` (Copilot), `.opencode/agents/` (OpenCode) — match the detected agent | -| Releases, versions, what's new | `CHANGELOG.md` (or the **What's New** flow below) | -| Architecture, conceptual docs | `docs/` | -| Tests / behavior contracts | `packages/atomic-sdk/**/*.test.ts`, `packages/atomic/**/*.test.ts` | -| Settings / config | `settings.json` in the project root, `~/.atomic/settings.json` for user-global | - -### Cross-nudge close for fallback answers - -After answering, append the same close used by canonical Q&A blocks (render -as plain paragraph text — not inside a fence): - - ───────────────────────────────────────────────────────────────── - - Where to next: - - `> /atomic example` spec-driven development end-to-end - `> /atomic workflows` reliably automate complex engineering work - `> /atomic ` always available — ask anything - -### When you can't answer with confidence - -If after reading the source you still can't answer the question well: - -- Say so plainly. Don't fabricate. -- Point at the relevant subdirectory or `docs/` page the user can explore. -- Suggest running `atomic --help` or `atomic --help` for CLI questions. -- Suggest `/find-skills ` if the question is about whether a skill exists. - ---- - -## What's New flow (`/atomic what's new`) - -Say *"Let me grab the latest releases for you…"*, then read **`CHANGELOG.md`** -as the source of truth. Never hardcode a GitHub repo slug — if you ever need -the canonical repo URL, parse it from `package.json#repository.url`. - -### Resolve the CHANGELOG path - -Try in this order; use the first that exists: - -1. `CHANGELOG.md` at the project root (when running inside the Atomic repo itself). -2. `node_modules/@bastani/atomic-sdk/CHANGELOG.md` (installed as a dep). -3. `node_modules/@bastani/atomic/CHANGELOG.md` (legacy install path). - -If none exist, fall back to `gh release list --repo /` against the -slug parsed from `package.json#repository.url` — extract owner/repo from a URL -like `git+https://github.com//.git`. **Do not hardcode -`flora131/atomic` or any other slug.** - -### Parse the CHANGELOG - -The file follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) format: - -``` -## [] — - -### Added -- ... - -### Fixed -- ... - -### Breaking Changes -- ... -``` - -For each version section: - -- **Skip pre-releases** — versions matching `\d+\.\d+\.\d+-(alpha|beta|rc)\.\d+`. -- Take items from `### Added`, `### Fixed`, and `### Breaking Changes` (priority order). -- Drop items from `### Internal`, `### Refactored`, `### Tests`, `### Docs`, - `### Chore`, or any non-user-facing section. - -Take the **3 most recent stable versions**. For each, pick the **top 1–3 -highest-signal items**: favor new commands, new workflows, new flags, fixed -crashes, and breaking changes — over internal plumbing. - -### Render - -``` -✦ What's New in Atomic ✦ - -▸ v — - • - • - -▸ v — - • <…> - -▸ v — - • <…> -``` - -Rewrite each bullet in plain language — strip leading `**Name:**` prefixes, -remove implementation jargon, keep it to one sentence. Surface breaking changes -clearly with a `⚠ Breaking:` prefix on the bullet. - -End with: *"Want the full changelog? Open `CHANGELOG.md`."* - -Atomic releases multiple times per day during active periods, so **don't cache** -— always re-read CHANGELOG.md on each invocation. - -### Cross-nudge close - -After the release block, render the cross-nudge close as plain paragraph text -(not inside a fence): - - ───────────────────────────────────────────────────────────────── - - Where to next: - - `> /atomic example` spec-driven dev end-to-end - `> /atomic overview` quick refresh on the catalog - `> /atomic ` always available — ask anything - ---- - -## Important behaviors - -- **Never run `atomic workflow ...` yourself.** Show the command, let the user run it. -- **No tour. No ASCII frames. No mascot.** Read like documentation. -- **Always end with the cross-nudge close** so users discover the other modes. -- **For canonical answers, render the block verbatim** (lightly adapted to the user's exact phrasing). Don't paraphrase — these blocks are tuned for clarity and consistency. -- **For source-reading, cite file paths** (e.g., `packages/atomic/src/workflow/runner.ts:42`) so users can verify. -- **Substitute the detected agent's `-a` flag, display name, and agents directory** consistently throughout examples. Never list multiple agents side-by-side. -- **Don't pile on routes.** When a question matches a canonical block, render that one. Don't append "and you might also like…" suggestions beyond the cross-nudge close. diff --git a/.agents/skills/bdi-mental-states/references/bdi-ontology-core.md b/.agents/skills/bdi-mental-states/references/bdi-ontology-core.md deleted file mode 100644 index 2b37974f3..000000000 --- a/.agents/skills/bdi-mental-states/references/bdi-ontology-core.md +++ /dev/null @@ -1,207 +0,0 @@ -# BDI Ontology Core Patterns - -Core ontology design patterns for Belief-Desire-Intention mental state modeling. - -## Class Hierarchy - -### Mental Entities (Endurants) - -``` -bdi:MentalEntity -├── bdi:Belief # Informational dimension -├── bdi:Desire # Motivational dimension -├── bdi:Intention # Deliberative dimension -├── bdi:Goal # Description of desired end state -└── bdi:Plan # Structured action sequence -``` - -### Mental Processes (Perdurants) - -``` -bdi:MentalProcess -├── bdi:BeliefProcess # Forms/updates beliefs from perception -├── bdi:DesireProcess # Generates desires from beliefs -├── bdi:IntentionProcess # Commits to desires as intentions -├── bdi:Planning # Transforms intentions into plans -└── bdi:PlanExecution # Executes plan actions -``` - -### Supporting Entities - -``` -bdi:WorldState # Configuration of environment -bdi:Justification # Evidential basis for mental states -bdi:Task # Atomic unit of planned action -bdi:Action # Execution of a task -bdi:TimeInterval # Temporal validity bounds -bdi:TimeInstant # Point in time reference -``` - -## Object Properties - -### Motivational Relations - -| Property | Domain | Range | Description | -|----------|--------|-------|-------------| -| `motivates` | Belief | Desire | Belief provides reason for desire | -| `isMotivatedBy` | Desire | Belief | Inverse of motivates | -| `fulfils` | Intention | Desire | Intention commits to achieving desire | -| `isFulfilledBy` | Desire | Intention | Inverse of fulfils | -| `isSupportedBy` | Intention | Belief | Beliefs supporting intention viability | - -### Generative Relations - -| Property | Domain | Range | Description | -|----------|--------|-------|-------------| -| `generates` | MentalProcess | MentalEntity | Process creates mental state | -| `isGeneratedBy` | MentalEntity | MentalProcess | Inverse of generates | -| `modifies` | MentalProcess | MentalEntity | Process updates existing state | -| `suppresses` | MentalProcess | MentalEntity | Process deactivates state | -| `isTriggeredBy` | MentalProcess | MentalEntity | State initiates process | - -### Referential Relations - -| Property | Domain | Range | Description | -|----------|--------|-------|-------------| -| `refersTo` | MentalEntity | WorldState | Mental state about world | -| `perceives` | Agent | WorldState | Agent observes world | -| `bringsAbout` | Action | WorldState | Action causes world change | -| `reasonsUpon` | MentalProcess | MentalEntity | Input to reasoning | - -### Structural Relations - -| Property | Domain | Range | Description | -|----------|--------|-------|-------------| -| `hasPart` | MentalEntity | MentalEntity | Meronymic composition | -| `specifies` | Intention | Plan | Intention defines plan | -| `addresses` | Plan | Goal | Plan achieves goal | -| `hasComponent` | Plan | Task | Plan contains tasks | -| `precedes` | Task | Task | Task ordering | - -### Temporal Relations - -| Property | Domain | Range | Description | -|----------|--------|-------|-------------| -| `atTime` | Entity | TimeInstant | Point occurrence | -| `hasValidity` | MentalEntity | TimeInterval | Persistence bounds | -| `hasStartTime` | TimeInterval | TimeInstant | Interval start | -| `hasEndTime` | TimeInterval | TimeInstant | Interval end | - -### Justification Relations - -| Property | Domain | Range | Description | -|----------|--------|-------|-------------| -| `isJustifiedBy` | MentalEntity | Justification | Evidential support | -| `justifies` | Justification | MentalEntity | Inverse relation | - -## Ontological Restrictions - -### Belief Restrictions - -```turtle -bdi:Belief rdfs:subClassOf [ - a owl:Restriction ; - owl:onProperty bdi:refersTo ; - owl:someValuesFrom bdi:WorldState -] . - -bdi:Belief rdfs:subClassOf [ - a owl:Restriction ; - owl:onProperty bdi:hasValidity ; - owl:maxCardinality 1 -] . -``` - -### Desire Restrictions - -```turtle -bdi:Desire rdfs:subClassOf [ - a owl:Restriction ; - owl:onProperty bdi:isMotivatedBy ; - owl:someValuesFrom bdi:Belief -] . -``` - -### Intention Restrictions - -```turtle -bdi:Intention rdfs:subClassOf [ - a owl:Restriction ; - owl:onProperty bdi:fulfils ; - owl:cardinality 1 -] . - -bdi:Intention rdfs:subClassOf [ - a owl:Restriction ; - owl:onProperty bdi:isSupportedBy ; - owl:someValuesFrom bdi:Belief -] . -``` - -### Mental Process Restrictions - -```turtle -bdi:BeliefProcess rdfs:subClassOf [ - a owl:Restriction ; - owl:onProperty bdi:generates ; - owl:allValuesFrom bdi:Belief -] . - -bdi:DesireProcess rdfs:subClassOf [ - a owl:Restriction ; - owl:onProperty bdi:generates ; - owl:allValuesFrom bdi:Desire -] . - -bdi:IntentionProcess rdfs:subClassOf [ - a owl:Restriction ; - owl:onProperty bdi:generates ; - owl:allValuesFrom bdi:Intention -] . -``` - -## DOLCE Alignment - -The BDI ontology aligns with DOLCE Ultra Lite (DUL) foundational ontology: - -| BDI Class | DUL Superclass | Rationale | -|-----------|----------------|-----------| -| `Agent` | `dul:Agent` | Intentional entity capable of action | -| `Belief` | `dul:InformationObject` | Information-bearing entity | -| `Desire` | `dul:Description` | Describes desired state | -| `Intention` | `dul:Description` | Describes committed course | -| `Goal` | `dul:Goal` | Desired end state description | -| `Plan` | `dul:Plan` | Organized action sequence | -| `WorldState` | `dul:Situation` | Configuration of entities | -| `MentalProcess` | `dul:Event` | Temporally extended occurrence | -| `Task` | `dul:Task` | Unit of planned work | -| `Action` | `dul:Action` | Performed task instance | - -## Reused Ontology Design Patterns - -### EventCore Pattern -Used for mental processes with temporal aspects and participant roles. - -### Situation Pattern -Used for world state configurations that mental states reference. - -### TimeIndexedSituation Pattern -Used for associating mental states with validity intervals. - -### BasicPlan Pattern -Used for goal-plan-task structures linking intentions to actions. - -### Provenance Pattern -Used for justification tracking and evidential chains. - -## Namespace Declarations - -```turtle -@prefix bdi: . -@prefix dul: . -@prefix owl: . -@prefix rdf: . -@prefix rdfs: . -@prefix xsd: . -``` - diff --git a/.agents/skills/bdi-mental-states/references/framework-integration.md b/.agents/skills/bdi-mental-states/references/framework-integration.md deleted file mode 100644 index 693af5a4b..000000000 --- a/.agents/skills/bdi-mental-states/references/framework-integration.md +++ /dev/null @@ -1,582 +0,0 @@ -# BDI Framework Integration Patterns - -Integration patterns for connecting BDI ontology with executable agent frameworks. - -## SEMAS Rule Translation - -Map BDI ontology constructs to SEMAS production rules. - -### Ontology-to-Rule Mapping - -| BDI Construct | SEMAS Element | Example | -|---------------|---------------|---------| -| Belief | HEAD fact | `belief(agent_a, store_open)` | -| Supporting beliefs | CONDITIONALS | `[CONDITIONALS: time(weekday)]` | -| Desire generation | TAIL action | `generate_desire(agent, goal)` | -| Intention commitment | TAIL action | `commit_intention(agent, goal)` | -| Plan specification | TAIL action | `create_plan(agent, plan_id)` | - -### Rule Templates - -**Belief triggers desire formation:** -```prolog -[HEAD: belief(Agent, Fact)] / -[CONDITIONALS: context_condition(Agent, Context)] » -[TAIL: generate_desire(Agent, DesiredState)]. -``` - -**Desire triggers intention commitment:** -```prolog -[HEAD: desire(Agent, Goal)] / -[CONDITIONALS: belief(Agent, SupportingFact1), - belief(Agent, SupportingFact2)] » -[TAIL: commit_intention(Agent, Goal)]. -``` - -**Intention triggers planning:** -```prolog -[HEAD: intention(Agent, Goal)] / -[CONDITIONALS: goal(GoalSpec)] » -[TAIL: create_plan(Agent, PlanId)]. -``` - -**Plan triggers execution:** -```prolog -[HEAD: plan(Agent, PlanId)] / -[CONDITIONALS: ready_to_execute(Agent)] » -[TAIL: execute_plan(Agent, PlanId)]. -``` - -### Complete SEMAS Example - -```prolog -% ============================================================ -% GROCERY SHOPPING SCENARIO -% ============================================================ - -% Phase 1: Belief formation from world state -[HEAD: perceive(agent_a, store_open)] / -[CONDITIONALS: time(weekday_afternoon)] » -[TAIL: add_belief(agent_a, store_open)]. - -% Phase 2: Desire generation from belief -[HEAD: belief(agent_a, store_open)] / -[CONDITIONALS: belief(agent_a, needs_groceries)] » -[TAIL: generate_desire(agent_a, buy_groceries)]. - -% Phase 3: Intention commitment from desire -[HEAD: desire(agent_a, buy_groceries)] / -[CONDITIONALS: belief(agent_a, has_shopping_list), - belief(agent_a, store_open), - belief(agent_a, has_transportation)] » -[TAIL: commit_intention(agent_a, buy_groceries)]. - -% Phase 4: Plan creation from intention -[HEAD: intention(agent_a, buy_groceries)] / -[CONDITIONALS: goal(complete_shopping)] » -[TAIL: create_plan(agent_a, shopping_plan)]. - -% Phase 5: Plan execution -[HEAD: plan(agent_a, shopping_plan)] / -[CONDITIONALS: preconditions_met(shopping_plan)] » -[TAIL: execute_task(agent_a, drive_to_store), - execute_task(agent_a, select_items), - execute_task(agent_a, checkout), - execute_task(agent_a, return_home)]. - -% Phase 6: World state update -[HEAD: task_complete(agent_a, checkout)] / -[CONDITIONALS: items_purchased(agent_a)] » -[TAIL: update_world_state(has_groceries), - remove_desire(agent_a, buy_groceries), - remove_intention(agent_a, buy_groceries)]. -``` - -### Python Translation Layer - -```python -from rdflib import Graph, Namespace, RDF - -BDI = Namespace("https://w3id.org/fossr/ontology/bdi/") - -def ontology_to_semas_rules(bdi_graph: Graph) -> list[str]: - """ - Translate BDI ontology instances to SEMAS production rules. - """ - rules = [] - - # Extract belief-desire-intention chains - for intention in bdi_graph.subjects(RDF.type, BDI.Intention): - # Get supporting beliefs - supporting_beliefs = list(bdi_graph.objects(intention, BDI.isSupportedBy)) - - # Get fulfilled desire - fulfilled_desires = list(bdi_graph.objects(intention, BDI.fulfils)) - - # Get specified plan - specified_plans = list(bdi_graph.objects(intention, BDI.specifies)) - - if fulfilled_desires and supporting_beliefs: - desire = fulfilled_desires[0] - beliefs_str = ", ".join([format_belief(b, bdi_graph) for b in supporting_beliefs]) - - rule = ( - f"[HEAD: {format_desire(desire, bdi_graph)}] / " - f"[CONDITIONALS: {beliefs_str}] » " - f"[TAIL: commit_intention({format_intention(intention, bdi_graph)})]" - ) - rules.append(rule) - - if specified_plans: - plan = specified_plans[0] - rule = ( - f"[HEAD: {format_intention(intention, bdi_graph)}] / " - f"[CONDITIONALS: ready_to_plan] » " - f"[TAIL: create_plan({format_plan(plan, bdi_graph)})]" - ) - rules.append(rule) - - return rules - -def format_belief(belief_uri, graph): - label = graph.value(belief_uri, RDFS.label) - return f"belief({label or belief_uri.split('/')[-1]})" - -def format_desire(desire_uri, graph): - label = graph.value(desire_uri, RDFS.label) - return f"desire({label or desire_uri.split('/')[-1]})" - -def format_intention(intention_uri, graph): - label = graph.value(intention_uri, RDFS.label) - return f"intention({label or intention_uri.split('/')[-1]})" - -def format_plan(plan_uri, graph): - label = graph.value(plan_uri, RDFS.label) - return f"plan({label or plan_uri.split('/')[-1]})" -``` - -## Logic Augmented Generation (LAG) - -Augment LLM outputs with BDI ontological constraints. - -### LAG Pipeline Architecture - -``` -┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ -│ User Query │────▶│ Ontology │────▶│ Augmented │ -│ │ │ Injection │ │ Prompt │ -└─────────────────┘ └─────────────────┘ └─────────────────┘ - │ - ▼ -┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ -│ Validated │◀────│ Ontology │◀────│ LLM Response │ -│ RDF Triples │ │ Validation │ │ (Triples) │ -└─────────────────┘ └─────────────────┘ └─────────────────┘ -``` - -### LAG Implementation - -```python -from rdflib import Graph, Namespace -from rdflib.plugins.parsers.notation3 import BadSyntax - -BDI = Namespace("https://w3id.org/fossr/ontology/bdi/") - -class BDILogicAugmentedGenerator: - def __init__(self, ontology_path: str, llm_client): - self.ontology = Graph() - self.ontology.parse(ontology_path, format='turtle') - self.llm = llm_client - - def generate_mental_states(self, context: str) -> Graph: - """ - Generate BDI mental states from context using LAG. - """ - # Phase 1: Inject ontology into prompt - ontology_turtle = self.ontology.serialize(format='turtle') - augmented_prompt = self._build_augmented_prompt(context, ontology_turtle) - - # Phase 2: Generate with LLM - response = self.llm.generate(augmented_prompt) - - # Phase 3: Extract and validate triples - triples = self._extract_triples(response) - validated = self._validate_against_ontology(triples) - - if not validated['is_consistent']: - # Retry with feedback - return self._retry_with_feedback(context, validated['errors']) - - return validated['graph'] - - def _build_augmented_prompt(self, context: str, ontology: str) -> str: - return f""" -You are a BDI mental state modeler. Given the following context, generate -RDF triples representing the agent's beliefs, desires, and intentions. - -## BDI Ontology (use these classes and properties): -{ontology} - -## Context to Model: -{context} - -## Instructions: -1. Identify world states from the context -2. Generate beliefs that refer to those world states -3. Generate desires motivated by those beliefs -4. Generate intentions that fulfill desires and are supported by beliefs -5. Include justifications for each mental state -6. Include temporal validity intervals - -Output valid Turtle RDF triples only. -""" - - def _extract_triples(self, response: str) -> str: - """Extract Turtle content from LLM response.""" - # Find turtle block in response - if "```turtle" in response: - start = response.find("```turtle") + 9 - end = response.find("```", start) - return response[start:end].strip() - return response - - def _validate_against_ontology(self, triples: str) -> dict: - """Validate generated triples against BDI ontology.""" - result = {'is_consistent': True, 'errors': [], 'graph': None} - - try: - generated = Graph() - generated.parse(data=triples, format='turtle') - result['graph'] = generated - - # Validate constraints - errors = [] - - # Check: Every intention must fulfill a desire - for intention in generated.subjects(RDF.type, BDI.Intention): - if not list(generated.objects(intention, BDI.fulfils)): - errors.append(f"Intention {intention} does not fulfill any desire") - - # Check: Every belief should reference a world state - for belief in generated.subjects(RDF.type, BDI.Belief): - if not list(generated.objects(belief, BDI.refersTo)): - errors.append(f"Belief {belief} does not reference a world state") - - # Check: Desires should be motivated by beliefs - for desire in generated.subjects(RDF.type, BDI.Desire): - if not list(generated.objects(desire, BDI.isMotivatedBy)): - errors.append(f"Desire {desire} has no motivating belief") - - if errors: - result['is_consistent'] = False - result['errors'] = errors - - except BadSyntax as e: - result['is_consistent'] = False - result['errors'] = [f"Invalid Turtle syntax: {e}"] - - return result - - def _retry_with_feedback(self, context: str, errors: list) -> Graph: - """Retry generation with error feedback.""" - feedback_prompt = f""" -Previous generation had errors: -{chr(10).join(errors)} - -Please regenerate the mental states fixing these issues. - -Context: {context} -""" - response = self.llm.generate(feedback_prompt) - triples = self._extract_triples(response) - result = self._validate_against_ontology(triples) - - if result['is_consistent']: - return result['graph'] - else: - raise ValueError(f"Failed to generate valid mental states: {result['errors']}") -``` - -### Inconsistency Detection Example - -```python -def detect_location_inconsistency(graph: Graph) -> list[str]: - """ - Detect inconsistencies where agent cannot be in two places. - """ - inconsistencies = [] - - # Query for location beliefs - query = """ - PREFIX bdi: - - SELECT ?agent ?belief1 ?belief2 ?loc1 ?loc2 WHERE { - ?agent bdi:hasBelief ?belief1 , ?belief2 . - ?belief1 bdi:refersTo ?ws1 . - ?belief2 bdi:refersTo ?ws2 . - ?ws1 bdi:hasLocation ?loc1 . - ?ws2 bdi:hasLocation ?loc2 . - FILTER(?belief1 != ?belief2 && ?loc1 != ?loc2) - - # Check temporal overlap - ?belief1 bdi:hasValidity ?interval1 . - ?belief2 bdi:hasValidity ?interval2 . - ?interval1 bdi:hasStartTime ?start1 ; bdi:hasEndTime ?end1 . - ?interval2 bdi:hasStartTime ?start2 ; bdi:hasEndTime ?end2 . - FILTER(?start1 < ?end2 && ?start2 < ?end1) - } - """ - - for row in graph.query(query): - inconsistencies.append( - f"Agent {row.agent} has conflicting location beliefs: " - f"{row.loc1} and {row.loc2} at overlapping times" - ) - - return inconsistencies -``` - -## JADE/JADEX Integration - -Map BDI ontology to JADE/JADEX agent platform structures. - -### JADE Agent Structure - -```java -public class BDIAgent extends Agent { - // Mental state storage (maps to ontology individuals) - private Set beliefs = new HashSet<>(); - private Set desires = new HashSet<>(); - private Set intentions = new HashSet<>(); - - // Ontology-backed mental state management - private Graph mentalStateGraph; - - public void addBelief(Belief belief) { - beliefs.add(belief); - - // Add to RDF graph - Resource beliefResource = mentalStateGraph.createResource(belief.getUri()); - beliefResource.addProperty(RDF.type, BDI.Belief); - beliefResource.addProperty(BDI.refersTo, belief.getWorldState().getUri()); - beliefResource.addProperty(BDI.hasValidity, createInterval(belief.getValidity())); - - // Trigger desire formation - triggerDesireProcess(belief); - } - - public void commitIntention(Intention intention) { - intentions.add(intention); - - Resource intentionResource = mentalStateGraph.createResource(intention.getUri()); - intentionResource.addProperty(RDF.type, BDI.Intention); - intentionResource.addProperty(BDI.fulfils, intention.getDesire().getUri()); - - for (Belief support : intention.getSupportingBeliefs()) { - intentionResource.addProperty(BDI.isSupportedBy, support.getUri()); - } - - // Trigger planning - triggerPlanning(intention); - } - - // Export mental states as RDF - public String exportMentalStates() { - return mentalStateGraph.serialize(Format.TURTLE); - } - - // Import mental states from RDF - public void importMentalStates(String turtle) { - Graph imported = new Graph(); - imported.parse(turtle, Format.TURTLE); - - // Reconstruct Java objects from RDF - for (Resource belief : imported.listSubjectsWithProperty(RDF.type, BDI.Belief)) { - Belief b = reconstructBelief(belief); - beliefs.add(b); - } - // ... similar for desires and intentions - } -} -``` - -### JADEX Goal Mapping - -```java -// Map BDI ontology goals to JADEX goals -@Goal -public class OntologyBackedGoal { - @GoalParameter - protected String goalUri; - - @GoalParameter - protected Graph ontologyGraph; - - public OntologyBackedGoal(Resource goalResource, Graph graph) { - this.goalUri = goalResource.getURI(); - this.ontologyGraph = graph; - } - - @GoalTargetCondition - public boolean isAchieved() { - // Query ontology for goal achievement - String query = """ - PREFIX bdi: - ASK { - ?execution bdi:addresses <%s> ; - bdi:bringsAbout ?worldState . - } - """.formatted(goalUri); - - return ontologyGraph.ask(query); - } - - @GoalDropCondition - public boolean shouldDrop() { - // Check if supporting beliefs are invalidated - String query = """ - PREFIX bdi: - ASK { - ?intention bdi:specifies ?plan . - ?plan bdi:addresses <%s> . - ?intention bdi:isSupportedBy ?belief . - ?belief bdi:hasValidity ?interval . - ?interval bdi:hasEndTime ?end . - FILTER(?end < NOW()) - } - """.formatted(goalUri); - - return ontologyGraph.ask(query); - } -} -``` - -## RDF Triple Store Integration - -### Triple Store Configuration - -```python -from rdflib import Graph -from rdflib.plugins.stores.sparqlstore import SPARQLUpdateStore - -class BDIMentalStateStore: - def __init__(self, endpoint: str): - self.store = SPARQLUpdateStore() - self.store.open((endpoint + "/query", endpoint + "/update")) - self.graph = Graph(store=self.store, identifier="http://example.org/bdi") - - def add_belief(self, agent_uri: str, belief_data: dict): - """Add belief to triple store.""" - belief_uri = f"{agent_uri}/belief/{belief_data['id']}" - - self.graph.add((URIRef(belief_uri), RDF.type, BDI.Belief)) - self.graph.add((URIRef(belief_uri), RDFS.label, Literal(belief_data['label']))) - self.graph.add((URIRef(belief_uri), BDI.refersTo, URIRef(belief_data['world_state']))) - self.graph.add((URIRef(agent_uri), BDI.hasMentalState, URIRef(belief_uri))) - - # Add temporal validity - interval_uri = f"{belief_uri}/validity" - self.graph.add((URIRef(belief_uri), BDI.hasValidity, URIRef(interval_uri))) - self.graph.add((URIRef(interval_uri), BDI.hasStartTime, - Literal(belief_data['start_time'], datatype=XSD.dateTime))) - self.graph.add((URIRef(interval_uri), BDI.hasEndTime, - Literal(belief_data['end_time'], datatype=XSD.dateTime))) - - def get_active_beliefs(self, agent_uri: str, at_time: datetime) -> list: - """Query beliefs active at specific time.""" - query = """ - PREFIX bdi: - PREFIX xsd: - - SELECT ?belief ?label WHERE { - <%s> bdi:hasMentalState ?belief . - ?belief a bdi:Belief ; - rdfs:label ?label ; - bdi:hasValidity ?interval . - ?interval bdi:hasStartTime ?start ; - bdi:hasEndTime ?end . - FILTER(?start <= "%s"^^xsd:dateTime && ?end >= "%s"^^xsd:dateTime) - } - """ % (agent_uri, at_time.isoformat(), at_time.isoformat()) - - return list(self.graph.query(query)) - - def get_cognitive_chain(self, intention_uri: str) -> dict: - """Trace complete cognitive chain for an intention.""" - query = """ - PREFIX bdi: - - SELECT ?intention ?desire ?belief ?worldState ?plan WHERE { - <%s> a bdi:Intention ; - bdi:fulfils ?desire ; - bdi:isSupportedBy ?belief . - OPTIONAL { <%s> bdi:specifies ?plan } - ?desire bdi:isMotivatedBy ?belief . - ?belief bdi:refersTo ?worldState . - } - """ % (intention_uri, intention_uri) - - results = list(self.graph.query(query)) - if results: - row = results[0] - return { - 'intention': str(row.intention), - 'desire': str(row.desire), - 'belief': str(row.belief), - 'world_state': str(row.worldState), - 'plan': str(row.plan) if row.plan else None - } - return None -``` - -## FIPA ACL Integration - -Map BDI mental states to FIPA Agent Communication Language. - -```python -from fipa_acl import ACLMessage, Performative - -class BDICommunicator: - def __init__(self, agent_id: str, mental_state_store: BDIMentalStateStore): - self.agent_id = agent_id - self.store = mental_state_store - - def share_belief(self, belief_uri: str, receiver: str) -> ACLMessage: - """Create INFORM message to share belief.""" - belief_triples = self.store.get_belief_as_turtle(belief_uri) - - message = ACLMessage() - message.performative = Performative.INFORM - message.sender = self.agent_id - message.receiver = receiver - message.content = belief_triples - message.ontology = "https://w3id.org/fossr/ontology/bdi/" - message.language = "turtle" - - return message - - def request_belief_confirmation(self, belief_uri: str, receiver: str) -> ACLMessage: - """Create QUERY-IF message to confirm shared belief.""" - message = ACLMessage() - message.performative = Performative.QUERY_IF - message.sender = self.agent_id - message.receiver = receiver - message.content = f"ASK {{ <{belief_uri}> a bdi:Belief }}" - message.language = "sparql" - - return message - - def propose_intention(self, intention_uri: str, receiver: str) -> ACLMessage: - """Create PROPOSE message for coordinated intention.""" - intention_triples = self.store.get_intention_as_turtle(intention_uri) - - message = ACLMessage() - message.performative = Performative.PROPOSE - message.sender = self.agent_id - message.receiver = receiver - message.content = intention_triples - message.ontology = "https://w3id.org/fossr/ontology/bdi/" - - return message -``` - diff --git a/.agents/skills/bdi-mental-states/references/rdf-examples.md b/.agents/skills/bdi-mental-states/references/rdf-examples.md deleted file mode 100644 index fd79c2b54..000000000 --- a/.agents/skills/bdi-mental-states/references/rdf-examples.md +++ /dev/null @@ -1,315 +0,0 @@ -# BDI RDF Examples - -Complete RDF/Turtle examples for BDI mental state modeling. - -## Complete Cognitive Workflow - -```turtle -@prefix bdi: . -@prefix ex: . -@prefix xsd: . -@prefix rdfs: . - -# ============================================================ -# PHASE 1: World State Perception -# ============================================================ - -ex:WorldState_traffic a bdi:WorldState ; - rdfs:comment "Heavy traffic on Route 101" ; - bdi:atTime "2026-01-04T08:30:00"^^xsd:dateTime ; - bdi:isPerceivedBy ex:Agent_commuter ; - bdi:triggers ex:BeliefProcess_assess_traffic . - -# ============================================================ -# PHASE 2: Belief Formation -# ============================================================ - -ex:BeliefProcess_assess_traffic a bdi:BeliefProcess ; - bdi:generates ex:Belief_traffic_delay ; - bdi:reasonsUpon ex:WorldState_traffic ; - bdi:isProcessedBy ex:Agent_commuter ; - bdi:atTime "2026-01-04T08:31:00"^^xsd:dateTime . - -ex:Belief_traffic_delay a bdi:Belief ; - rdfs:label "Traffic will cause 30-minute delay" ; - bdi:refersTo ex:WorldState_traffic ; - bdi:hasValidity ex:TimeInterval_morning_commute ; - bdi:hasPart ex:Belief_route_congested , ex:Belief_delay_duration ; - bdi:isJustifiedBy ex:Justification_traffic_report ; - bdi:motivates ex:Desire_arrive_on_time . - -ex:Belief_route_congested a bdi:Belief ; - rdfs:comment "Route 101 is congested" . - -ex:Belief_delay_duration a bdi:Belief ; - rdfs:comment "Delay estimated at 30 minutes" . - -ex:Justification_traffic_report a bdi:Justification ; - rdfs:label "Real-time traffic data from navigation system" ; - bdi:justifies ex:Belief_traffic_delay . - -# ============================================================ -# PHASE 3: Desire Formation -# ============================================================ - -ex:DesireProcess_plan_arrival a bdi:DesireProcess ; - bdi:generates ex:Desire_arrive_on_time ; - bdi:reasonsUpon ex:Belief_traffic_delay ; - bdi:isProcessedBy ex:Agent_commuter . - -ex:Desire_arrive_on_time a bdi:Desire ; - rdfs:label "I desire to arrive at work on time" ; - bdi:isMotivatedBy ex:Belief_traffic_delay ; - bdi:refersTo ex:WorldState_on_time_arrival . - -# ============================================================ -# PHASE 4: Intention Commitment -# ============================================================ - -ex:IntentionProcess_commit_route a bdi:IntentionProcess ; - bdi:generates ex:Intention_take_alternate_route ; - bdi:reasonsUpon ex:Desire_arrive_on_time ; - bdi:isProcessedBy ex:Agent_commuter . - -ex:Intention_take_alternate_route a bdi:Intention ; - rdfs:label "I will take alternate route via Highway 280" ; - bdi:fulfils ex:Desire_arrive_on_time ; - bdi:isSupportedBy ex:Belief_traffic_delay ; - bdi:specifies ex:Plan_alternate_commute ; - bdi:isJustifiedBy ex:Justification_time_optimization . - -ex:Justification_time_optimization a bdi:Justification ; - rdfs:label "Alternate route saves 20 minutes based on current conditions" ; - bdi:justifies ex:Intention_take_alternate_route . - -# ============================================================ -# PHASE 5: Planning -# ============================================================ - -ex:Planning_route_selection a bdi:Planning ; - bdi:reasonsUpon ex:Intention_take_alternate_route ; - bdi:defines ex:Plan_alternate_commute ; - bdi:atTime ex:TimeInterval_planning_phase . - -ex:Plan_alternate_commute a bdi:Plan ; - rdfs:label "Alternate commute via Highway 280" ; - bdi:addresses ex:Goal_arrive_by_9am ; - bdi:beginsWith ex:Task_exit_Route101 ; - bdi:endsWith ex:Task_arrive_parking ; - bdi:hasComponent ex:Task_exit_Route101 , ex:Task_merge_280 , - ex:Task_navigate_280 , ex:Task_arrive_parking . - -ex:Task_exit_Route101 a bdi:Task ; - rdfs:label "Exit Route 101 at Whipple Ave" ; - bdi:precedes ex:Task_merge_280 . - -ex:Task_merge_280 a bdi:Task ; - rdfs:label "Merge onto Highway 280 North" ; - bdi:precedes ex:Task_navigate_280 . - -ex:Task_navigate_280 a bdi:Task ; - rdfs:label "Continue on Highway 280 for 8 miles" ; - bdi:precedes ex:Task_arrive_parking . - -ex:Task_arrive_parking a bdi:Task ; - rdfs:label "Arrive at office parking garage" . - -ex:Goal_arrive_by_9am a bdi:Goal ; - rdfs:label "Arrive at work by 9:00 AM" . - -# ============================================================ -# PHASE 6: Plan Execution -# ============================================================ - -ex:PlanExecution_commute a bdi:PlanExecution ; - bdi:satisfies ex:Plan_alternate_commute ; - bdi:addresses ex:Goal_arrive_by_9am ; - bdi:isExecutedBy ex:Agent_commuter ; - bdi:hasComponent ex:Action_exit , ex:Action_merge , - ex:Action_drive_280 , ex:Action_park ; - bdi:atTime ex:TimeInterval_execution ; - bdi:bringsAbout ex:WorldState_arrived_on_time . - -ex:Action_exit a bdi:Action ; - bdi:isExecutionOf ex:Task_exit_Route101 ; - bdi:isPerformedBy ex:Agent_commuter ; - bdi:atTime "2026-01-04T08:35:00"^^xsd:dateTime . - -ex:Action_merge a bdi:Action ; - bdi:isExecutionOf ex:Task_merge_280 ; - bdi:isPerformedBy ex:Agent_commuter ; - bdi:atTime "2026-01-04T08:37:00"^^xsd:dateTime . - -ex:Action_drive_280 a bdi:Action ; - bdi:isExecutionOf ex:Task_navigate_280 ; - bdi:isPerformedBy ex:Agent_commuter ; - bdi:atTime "2026-01-04T08:40:00"^^xsd:dateTime . - -ex:Action_park a bdi:Action ; - bdi:isExecutionOf ex:Task_arrive_parking ; - bdi:isPerformedBy ex:Agent_commuter ; - bdi:bringsAbout ex:WorldState_arrived_on_time ; - bdi:atTime "2026-01-04T08:52:00"^^xsd:dateTime . - -# ============================================================ -# PHASE 7: Resulting World State -# ============================================================ - -ex:WorldState_arrived_on_time a bdi:WorldState ; - rdfs:comment "Agent arrived at work at 8:52 AM" ; - bdi:atTime "2026-01-04T08:52:00"^^xsd:dateTime . - -# ============================================================ -# TEMPORAL INTERVALS -# ============================================================ - -ex:TimeInterval_morning_commute a bdi:TimeInterval ; - bdi:hasStartTime "2026-01-04T08:30:00"^^xsd:dateTime ; - bdi:hasEndTime "2026-01-04T09:00:00"^^xsd:dateTime . - -ex:TimeInterval_planning_phase a bdi:TimeInterval ; - bdi:hasStartTime "2026-01-04T08:31:00"^^xsd:dateTime ; - bdi:hasEndTime "2026-01-04T08:34:00"^^xsd:dateTime . - -ex:TimeInterval_execution a bdi:TimeInterval ; - bdi:hasStartTime "2026-01-04T08:35:00"^^xsd:dateTime ; - bdi:hasEndTime "2026-01-04T08:52:00"^^xsd:dateTime . -``` - -## Multi-Agent Coordination Example - -```turtle -@prefix bdi: . -@prefix ex: . -@prefix fipa: . - -# Shared belief about project deadline -ex:Agent_developer a bdi:Agent ; - bdi:hasMentalState ex:Belief_deadline_friday . - -ex:Agent_manager a bdi:Agent ; - bdi:hasMentalState ex:Belief_deadline_friday . - -ex:Belief_deadline_friday a bdi:Belief ; - rdfs:label "Project deadline is Friday 5 PM" ; - bdi:refersTo ex:WorldState_deadline ; - bdi:hasValidity ex:TimeInterval_project_week . - -ex:WorldState_deadline a bdi:WorldState ; - rdfs:comment "Project XYZ must be delivered by 2026-01-10T17:00:00" . - -# Agent-specific mental states -ex:Agent_developer - bdi:hasDesire ex:Desire_complete_coding ; - bdi:hasIntention ex:Intention_implement_features . - -ex:Desire_complete_coding a bdi:Desire ; - rdfs:label "Complete feature implementation" ; - bdi:isMotivatedBy ex:Belief_deadline_friday . - -ex:Intention_implement_features a bdi:Intention ; - rdfs:label "Implement features A, B, and C" ; - bdi:fulfils ex:Desire_complete_coding ; - bdi:specifies ex:Plan_development . - -ex:Agent_manager - bdi:hasDesire ex:Desire_ensure_delivery ; - bdi:hasIntention ex:Intention_coordinate_team . - -ex:Desire_ensure_delivery a bdi:Desire ; - rdfs:label "Ensure on-time project delivery" ; - bdi:isMotivatedBy ex:Belief_deadline_friday . - -ex:Intention_coordinate_team a bdi:Intention ; - rdfs:label "Coordinate team activities" ; - bdi:fulfils ex:Desire_ensure_delivery ; - bdi:specifies ex:Plan_project_management . - -# FIPA communication -ex:Message_M1 a fipa:ACLMessage ; - fipa:sender ex:Agent_manager ; - fipa:receiver ex:Agent_developer ; - fipa:content ex:Belief_deadline_friday ; - fipa:performative fipa:inform . -``` - -## Conflict Resolution Example - -```turtle -@prefix bdi: . -@prefix ex: . - -# Conflicting location beliefs -ex:Belief_at_home a bdi:Belief ; - bdi:refersTo ex:WorldState_home ; - rdfs:comment "Agent is currently at home" . - -ex:Belief_at_office a bdi:Belief ; - bdi:refersTo ex:WorldState_office ; - rdfs:comment "Agent is at office" . - -# Conflicting intentions -ex:Intention_work_from_home a bdi:Intention ; - bdi:isSupportedBy ex:Belief_at_home ; - rdfs:label "Work from home today" . - -ex:Intention_attend_meeting a bdi:Intention ; - bdi:isSupportedBy ex:Belief_at_office ; - rdfs:label "Attend in-person meeting" . - -# Justification for conflict resolution -ex:Justification_location_conflict a bdi:Justification ; - rdfs:comment "Cannot simultaneously be at home and office" ; - bdi:justifies ex:Intention_resolution . - -# Resolved intention -ex:Intention_resolution a bdi:Intention ; - rdfs:label "Attend meeting via video call from home" ; - bdi:fulfils ex:Desire_meeting_participation ; - bdi:isSupportedBy ex:Belief_at_home ; - bdi:isJustifiedBy ex:Justification_location_conflict . -``` - -## T2B2T Payment Processing Example - -```turtle -@prefix bdi: . -@prefix ex: . -@prefix xsd: . - -# PHASE 1: Triples-to-Beliefs (External RDF → Internal Mental State) - -ex:WorldState_notification a bdi:WorldState ; - rdfs:comment "Push notification: Ghadeh requested $250 via Zelle" ; - bdi:atTime "2025-10-27T10:15:00"^^xsd:dateTime ; - bdi:triggers ex:BeliefProcess_BP1 . - -ex:BeliefProcess_BP1 a bdi:BeliefProcess ; - bdi:generates ex:Belief_payment_request ; - bdi:isProcessedBy ex:Agent_A . - -ex:Belief_payment_request a bdi:Belief ; - rdfs:label "Ghadeh requested $250" ; - bdi:refersTo ex:WorldState_notification ; - bdi:motivates ex:Desire_pay_Ghadeh . - -ex:Desire_pay_Ghadeh a bdi:Desire ; - rdfs:label "Pay Ghadeh $250" ; - bdi:isMotivatedBy ex:Belief_payment_request . - -ex:Intention_I1 a bdi:Intention ; - rdfs:label "Pay Ghadeh $250" ; - bdi:fulfils ex:Desire_pay_Ghadeh ; - bdi:specifies ex:Plan_payment . - -# PHASE 2: Beliefs-to-Triples (Mental State → External RDF) - -ex:PlanExecution_PE1 a bdi:PlanExecution ; - bdi:satisfies ex:Plan_payment ; - bdi:bringsAbout ex:WorldState_payment_complete . - -ex:WorldState_payment_complete a bdi:WorldState ; - rdfs:comment "Payment of $250 sent to Ghadeh via Zelle" ; - bdi:atTime "2025-10-27T10:20:00"^^xsd:dateTime . -``` - diff --git a/.agents/skills/bdi-mental-states/references/sparql-competency.md b/.agents/skills/bdi-mental-states/references/sparql-competency.md deleted file mode 100644 index 782386605..000000000 --- a/.agents/skills/bdi-mental-states/references/sparql-competency.md +++ /dev/null @@ -1,420 +0,0 @@ -# SPARQL Competency Queries - -Validation queries for BDI ontology implementations based on competency questions. - -## Mental Entity Queries - -### CQ1: What are all mental entities? - -```sparql -PREFIX bdi: -PREFIX rdf: -PREFIX rdfs: - -SELECT DISTINCT ?entity ?type WHERE { - ?entity rdf:type ?type . - ?type rdfs:subClassOf* bdi:MentalEntity . -} -``` - -### CQ2: What beliefs does an agent hold? - -```sparql -PREFIX bdi: - -SELECT ?belief ?label WHERE { - ?agent bdi:hasMentalState ?belief . - ?belief a bdi:Belief . - OPTIONAL { ?belief rdfs:label ?label } -} -``` - -### CQ3: What desires does an agent have? - -```sparql -PREFIX bdi: - -SELECT ?desire ?label WHERE { - ?agent bdi:hasDesire ?desire . - ?desire a bdi:Desire . - OPTIONAL { ?desire rdfs:label ?label } -} -``` - -### CQ4: What intentions has an agent committed to? - -```sparql -PREFIX bdi: - -SELECT ?intention ?label WHERE { - ?agent bdi:hasIntention ?intention . - ?intention a bdi:Intention . - OPTIONAL { ?intention rdfs:label ?label } -} -``` - -## Motivational Chain Queries - -### CQ5: What beliefs motivated formation of a given desire? - -```sparql -PREFIX bdi: - -SELECT ?belief ?beliefLabel WHERE { - ?desire bdi:isMotivatedBy ?belief . - ?belief a bdi:Belief . - OPTIONAL { ?belief rdfs:label ?beliefLabel } -} -``` - -### CQ6: Which desire does a particular intention fulfill? - -```sparql -PREFIX bdi: - -SELECT ?desire ?desireLabel WHERE { - ?intention bdi:fulfils ?desire . - ?desire a bdi:Desire . - OPTIONAL { ?desire rdfs:label ?desireLabel } -} -``` - -### CQ7: What beliefs support a given intention? - -```sparql -PREFIX bdi: - -SELECT ?belief ?label WHERE { - ?intention bdi:isSupportedBy ?belief . - ?belief a bdi:Belief . - OPTIONAL { ?belief rdfs:label ?label } -} -``` - -### CQ8: Trace complete cognitive chain for an intention - -```sparql -PREFIX bdi: - -SELECT ?intention ?desire ?belief ?worldState WHERE { - ?intention a bdi:Intention ; - bdi:fulfils ?desire ; - bdi:isSupportedBy ?belief . - ?desire bdi:isMotivatedBy ?belief . - ?belief bdi:refersTo ?worldState . -} -``` - -## Mental Process Queries - -### CQ9: Which mental process generated a belief? - -```sparql -PREFIX bdi: - -SELECT ?process ?processType WHERE { - ?process bdi:generates ?belief . - ?belief a bdi:Belief . - ?process a ?processType . - FILTER(?processType != owl:NamedIndividual) -} -``` - -### CQ10: What triggered a mental process? - -```sparql -PREFIX bdi: - -SELECT ?process ?trigger ?triggerType WHERE { - ?process a bdi:MentalProcess ; - bdi:isTriggeredBy ?trigger . - ?trigger a ?triggerType . -} -``` - -### CQ11: What did a mental process reason upon? - -```sparql -PREFIX bdi: - -SELECT ?process ?input WHERE { - ?process a bdi:MentalProcess ; - bdi:reasonsUpon ?input . -} -``` - -## Plan and Goal Queries - -### CQ12: What plan does an intention specify? - -```sparql -PREFIX bdi: - -SELECT ?intention ?plan ?goal WHERE { - ?intention bdi:specifies ?plan . - ?plan a bdi:Plan ; - bdi:addresses ?goal . -} -``` - -### CQ13: What is the ordered sequence of tasks in a plan? - -```sparql -PREFIX bdi: - -SELECT ?plan ?task ?nextTask WHERE { - ?plan a bdi:Plan ; - bdi:hasComponent ?task . - OPTIONAL { ?task bdi:precedes ?nextTask } -} -ORDER BY ?task -``` - -### CQ14: What is the first and last task of a plan? - -```sparql -PREFIX bdi: - -SELECT ?plan ?firstTask ?lastTask WHERE { - ?plan a bdi:Plan ; - bdi:beginsWith ?firstTask ; - bdi:endsWith ?lastTask . -} -``` - -### CQ15: Which actions executed which tasks? - -```sparql -PREFIX bdi: - -SELECT ?action ?task ?time WHERE { - ?action bdi:isExecutionOf ?task ; - bdi:atTime ?time . -} -ORDER BY ?time -``` - -## Temporal Queries - -### CQ16: What mental states are valid at a specific time? - -```sparql -PREFIX bdi: -PREFIX xsd: - -SELECT ?mentalState ?type WHERE { - ?mentalState bdi:hasValidity ?interval . - ?interval bdi:hasStartTime ?start ; - bdi:hasEndTime ?end . - ?mentalState a ?type . - FILTER(?start <= "2026-01-04T10:00:00"^^xsd:dateTime && - ?end >= "2026-01-04T10:00:00"^^xsd:dateTime) -} -``` - -### CQ17: When was a belief formed? - -```sparql -PREFIX bdi: - -SELECT ?belief ?formationTime WHERE { - ?process bdi:generates ?belief ; - bdi:atTime ?formationTime . - ?belief a bdi:Belief . -} -``` - -### CQ18: What is the temporal validity of an intention? - -```sparql -PREFIX bdi: - -SELECT ?intention ?start ?end WHERE { - ?intention a bdi:Intention ; - bdi:hasValidity ?interval . - ?interval bdi:hasStartTime ?start ; - bdi:hasEndTime ?end . -} -``` - -## Justification Queries - -### CQ19: What justifies a belief? - -```sparql -PREFIX bdi: - -SELECT ?belief ?justification ?justLabel WHERE { - ?belief a bdi:Belief ; - bdi:isJustifiedBy ?justification . - OPTIONAL { ?justification rdfs:label ?justLabel } -} -``` - -### CQ20: What justifies an intention? - -```sparql -PREFIX bdi: - -SELECT ?intention ?justification ?justLabel WHERE { - ?intention a bdi:Intention ; - bdi:isJustifiedBy ?justification . - OPTIONAL { ?justification rdfs:label ?justLabel } -} -``` - -## Compositional Queries - -### CQ21: What parts comprise a complex belief? - -```sparql -PREFIX bdi: - -SELECT ?belief ?part ?partLabel WHERE { - ?belief a bdi:Belief ; - bdi:hasPart ?part . - OPTIONAL { ?part rdfs:label ?partLabel } -} -``` - -### CQ22: Find composite mental entities - -```sparql -PREFIX bdi: - -SELECT ?composite (COUNT(?part) AS ?partCount) WHERE { - ?composite bdi:hasPart ?part . -} -GROUP BY ?composite -HAVING (COUNT(?part) > 1) -``` - -## World State Queries - -### CQ23: What world state does a belief refer to? - -```sparql -PREFIX bdi: - -SELECT ?belief ?worldState ?wsComment WHERE { - ?belief a bdi:Belief ; - bdi:refersTo ?worldState . - OPTIONAL { ?worldState rdfs:comment ?wsComment } -} -``` - -### CQ24: What actions brought about a world state? - -```sparql -PREFIX bdi: - -SELECT ?action ?worldState WHERE { - ?action bdi:bringsAbout ?worldState . - ?worldState a bdi:WorldState . -} -``` - -### CQ25: What world states has an agent perceived? - -```sparql -PREFIX bdi: - -SELECT ?agent ?worldState ?time WHERE { - ?agent bdi:perceives ?worldState . - OPTIONAL { ?worldState bdi:atTime ?time } -} -``` - -## Validation Queries (OWLUnit Style) - -### V1: Every intention must fulfill exactly one desire - -```sparql -PREFIX bdi: - -SELECT ?intention WHERE { - ?intention a bdi:Intention . - FILTER NOT EXISTS { ?intention bdi:fulfils ?desire } -} -# Expected: Empty result set -``` - -### V2: Every belief must reference a world state - -```sparql -PREFIX bdi: - -SELECT ?belief WHERE { - ?belief a bdi:Belief . - FILTER NOT EXISTS { ?belief bdi:refersTo ?worldState } -} -# Expected: Empty result set (or only abstract beliefs) -``` - -### V3: Mental processes must reason upon something - -```sparql -PREFIX bdi: - -SELECT ?process WHERE { - ?process a bdi:MentalProcess . - FILTER NOT EXISTS { ?process bdi:reasonsUpon ?input } -} -# Expected: Empty result set -``` - -### V4: BeliefProcess must generate only Beliefs - -```sparql -PREFIX bdi: - -SELECT ?process ?generated WHERE { - ?process a bdi:BeliefProcess ; - bdi:generates ?generated . - FILTER NOT EXISTS { ?generated a bdi:Belief } -} -# Expected: Empty result set -``` - -### V5: Plans must have begin and end tasks - -```sparql -PREFIX bdi: - -SELECT ?plan WHERE { - ?plan a bdi:Plan . - FILTER NOT EXISTS { - ?plan bdi:beginsWith ?first ; - bdi:endsWith ?last - } -} -# Expected: Empty result set -``` - -## Multi-Agent Queries - -### CQ26: What beliefs are shared across agents? - -```sparql -PREFIX bdi: - -SELECT ?belief (COUNT(DISTINCT ?agent) AS ?agentCount) WHERE { - ?agent bdi:hasMentalState ?belief . - ?belief a bdi:Belief . -} -GROUP BY ?belief -HAVING (COUNT(DISTINCT ?agent) > 1) -``` - -### CQ27: Which agents share the same desire? - -```sparql -PREFIX bdi: - -SELECT ?desire ?agent1 ?agent2 WHERE { - ?agent1 bdi:hasDesire ?desire . - ?agent2 bdi:hasDesire ?desire . - FILTER(?agent1 != ?agent2) -} -``` - diff --git a/.agents/skills/bun/SKILL.md b/.agents/skills/bun/SKILL.md index ba9ba4d0b..2fc17fa0d 100644 --- a/.agents/skills/bun/SKILL.md +++ b/.agents/skills/bun/SKILL.md @@ -1,11 +1,10 @@ --- -name: Bun +name: bun description: Use when building, testing, and deploying JavaScript/TypeScript applications. Reach for Bun when you need to run scripts, manage dependencies, bundle code, or test applications with a single unified tool. metadata: - provider: atomic - mintlify-proj: bun - version: "1.0" - internal: true + mintlify-proj: bun + version: "1.0" + internal: true --- # Bun Skill Reference @@ -17,6 +16,7 @@ Bun is a unified JavaScript runtime, package manager, bundler, and test runner w ## When to Use Use Bun when: + - **Running scripts**: Execute TypeScript/JavaScript files directly without compilation steps (`bun run file.ts`) - **Managing dependencies**: Install, add, remove, or update packages faster than npm/yarn/pnpm (`bun install`, `bun add`) - **Bundling code**: Build JavaScript/TypeScript for browser or server targets with `bun build` @@ -118,26 +118,33 @@ Set in `bunfig.toml`: `linker = "isolated"` or via CLI: `bun install --linker is ## Workflow ### 1. Initialize a Project + ```bash bun init my-app cd my-app ``` + Choose template: Blank, React, or Library. Creates `package.json`, `tsconfig.json`, `.gitignore`. ### 2. Install Dependencies + ```bash bun install ``` + Reads `package.json`, downloads packages, creates `bun.lock`. Much faster than npm. ### 3. Add Packages + ```bash bun add react bun add -d @types/react typescript ``` + Updates `package.json` and `bun.lock` automatically. ### 4. Write and Run Code + ```bash # Create index.ts echo "console.log('Hello Bun!')" > index.ts @@ -145,9 +152,11 @@ echo "console.log('Hello Bun!')" > index.ts # Run it bun run index.ts ``` + Bun transpiles TypeScript on-the-fly; no build step needed. ### 5. Create HTTP Server + ```typescript // server.ts const server = Bun.serve({ @@ -158,11 +167,13 @@ const server = Bun.serve({ }); console.log(`Listening on ${server.url}`); ``` + ```bash bun run server.ts ``` ### 6. Write Tests + ```typescript // math.test.ts import { test, expect } from "bun:test"; @@ -171,22 +182,28 @@ test("2 + 2 = 4", () => { expect(2 + 2).toBe(4); }); ``` + ```bash bun test ``` + Finds and runs all `*.test.ts` files automatically. ### 7. Bundle for Production + ```bash bun build ./src/index.ts --outdir ./dist --minify ``` + Outputs optimized bundle to `dist/`. Use `--target browser|node|bun` to control output format. ### 8. Create Standalone Executable + ```bash bun build ./cli.ts --outfile mycli --compile ./mycli ``` + Bundles code + Bun runtime into single executable; no dependencies needed. ## Common Gotchas @@ -222,6 +239,7 @@ Before submitting work with Bun: **Comprehensive navigation**: https://bun.com/docs/llms.txt — Page-by-page listing of all Bun documentation. **Critical pages**: + 1. [Bun Runtime](https://bun.com/docs/runtime) — Execute files, scripts, and manage the runtime 2. [Package Manager](https://bun.com/docs/pm/cli/install) — Install, add, remove packages and manage dependencies 3. [Bundler](https://bun.com/docs/bundler) — Bundle JavaScript/TypeScript for production @@ -230,4 +248,4 @@ Before submitting work with Bun: --- -> For additional documentation and navigation, see: https://bun.com/docs/llms.txt \ No newline at end of file +> For additional documentation and navigation, see: https://bun.com/docs/llms.txt diff --git a/.agents/skills/context-compression/references/evaluation-framework.md b/.agents/skills/context-compression/references/evaluation-framework.md deleted file mode 100644 index eac3e0572..000000000 --- a/.agents/skills/context-compression/references/evaluation-framework.md +++ /dev/null @@ -1,213 +0,0 @@ -# Context Compression Evaluation Framework - -This document provides the complete evaluation framework for measuring context compression quality, including probe types, scoring rubrics, and LLM judge configuration. - -## Probe Types - -### Recall Probes - -Test factual retention of specific details from conversation history. - -**Structure:** -``` -Question: [Ask for specific fact from truncated history] -Expected: [Exact detail that should be preserved] -Scoring: Match accuracy of technical details -``` - -**Examples:** -- "What was the original error message that started this debugging session?" -- "What version of the dependency did we decide to use?" -- "What was the exact command that failed?" - -### Artifact Probes - -Test file tracking and modification awareness. - -**Structure:** -``` -Question: [Ask about files created, modified, or examined] -Expected: [Complete list with change descriptions] -Scoring: Completeness of file list and accuracy of change descriptions -``` - -**Examples:** -- "Which files have we modified? Describe what changed in each." -- "What new files did we create in this session?" -- "Which configuration files did we examine but not change?" - -### Continuation Probes - -Test ability to continue work without re-fetching context. - -**Structure:** -``` -Question: [Ask about next steps or current state] -Expected: [Actionable next steps based on session history] -Scoring: Ability to continue without requesting re-read of files -``` - -**Examples:** -- "What should we do next?" -- "What tests are still failing and why?" -- "What was left incomplete from our last step?" - -### Decision Probes - -Test retention of reasoning chains and decision rationale. - -**Structure:** -``` -Question: [Ask about why a decision was made] -Expected: [Reasoning that led to the decision] -Scoring: Preservation of decision context and alternatives considered -``` - -**Examples:** -- "We discussed options for the Redis issue. What did we decide and why?" -- "Why did we choose connection pooling over per-request connections?" -- "What alternatives did we consider for the authentication fix?" - -## Scoring Rubrics - -### Accuracy Dimension - -| Criterion | Question | Score 0 | Score 3 | Score 5 | -|-----------|----------|---------|---------|---------| -| accuracy_factual | Are facts, file paths, and technical details correct? | Completely incorrect or fabricated | Mostly accurate with minor errors | Perfectly accurate | -| accuracy_technical | Are code references and technical concepts correct? | Major technical errors | Generally correct with minor issues | Technically precise | - -### Context Awareness Dimension - -| Criterion | Question | Score 0 | Score 3 | Score 5 | -|-----------|----------|---------|---------|---------| -| context_conversation_state | Does the response reflect current conversation state? | No awareness of prior context | General awareness with gaps | Full awareness of conversation history | -| context_artifact_state | Does the response reflect which files/artifacts were accessed? | No awareness of artifacts | Partial artifact awareness | Complete artifact state awareness | - -### Artifact Trail Dimension - -| Criterion | Question | Score 0 | Score 3 | Score 5 | -|-----------|----------|---------|---------|---------| -| artifact_files_created | Does the agent know which files were created? | No knowledge | Knows most files | Perfect knowledge | -| artifact_files_modified | Does the agent know which files were modified and what changed? | No knowledge | Good knowledge of most modifications | Perfect knowledge of all modifications | -| artifact_key_details | Does the agent remember function names, variable names, error messages? | No recall | Recalls most key details | Perfect recall | - -### Completeness Dimension - -| Criterion | Question | Score 0 | Score 3 | Score 5 | -|-----------|----------|---------|---------|---------| -| completeness_coverage | Does the response address all parts of the question? | Ignores most parts | Addresses most parts | Addresses all parts thoroughly | -| completeness_depth | Is sufficient detail provided? | Superficial or missing detail | Adequate detail | Comprehensive detail | - -### Continuity Dimension - -| Criterion | Question | Score 0 | Score 3 | Score 5 | -|-----------|----------|---------|---------|---------| -| continuity_work_state | Can the agent continue without re-fetching previously accessed information? | Cannot continue without re-fetching all context | Can continue with minimal re-fetching | Can continue seamlessly | -| continuity_todo_state | Does the agent maintain awareness of pending tasks? | Lost track of all TODOs | Good awareness with some gaps | Perfect task awareness | -| continuity_reasoning | Does the agent retain rationale behind previous decisions? | No memory of reasoning | Generally remembers reasoning | Excellent retention | - -### Instruction Following Dimension - -| Criterion | Question | Score 0 | Score 3 | Score 5 | -|-----------|----------|---------|---------|---------| -| instruction_format | Does the response follow the requested format? | Ignores format | Generally follows format | Perfectly follows format | -| instruction_constraints | Does the response respect stated constraints? | Ignores constraints | Mostly respects constraints | Fully respects all constraints | - -## LLM Judge Configuration - -### System Prompt - -``` -You are an expert evaluator assessing AI assistant responses in software development conversations. - -Your task is to grade responses against specific rubric criteria. For each criterion: -1. Read the criterion question carefully -2. Examine the response for evidence -3. Assign a score from 0-5 based on the scoring guide -4. Provide brief reasoning for your score - -Be objective and consistent. Focus on what is present in the response, not what could have been included. -``` - -### Judge Input Format - -```json -{ - "probe_question": "What was the original error message?", - "model_response": "[Response to evaluate]", - "compacted_context": "[The compressed context that was provided]", - "ground_truth": "[Optional: known correct answer]", - "rubric_criteria": ["accuracy_factual", "accuracy_technical", "context_conversation_state"] -} -``` - -### Judge Output Format - -```json -{ - "criterionResults": [ - { - "criterionId": "accuracy_factual", - "score": 5, - "reasoning": "Response correctly identifies the 401 error, specific endpoint, and root cause." - } - ], - "aggregateScore": 4.8, - "dimensionScores": { - "accuracy": 4.9, - "context_awareness": 4.5, - "artifact_trail": 3.2, - "completeness": 5.0, - "continuity": 4.8, - "instruction_following": 5.0 - } -} -``` - -## Benchmark Results Reference - -Performance across compression methods (based on 36,000+ messages): - -| Method | Overall | Accuracy | Context | Artifact | Complete | Continuity | Instruction | -|--------|---------|----------|---------|----------|----------|------------|-------------| -| Anchored Iterative | 3.70 | 4.04 | 4.01 | 2.45 | 4.44 | 3.80 | 4.99 | -| Regenerative | 3.44 | 3.74 | 3.56 | 2.33 | 4.37 | 3.67 | 4.95 | -| Opaque | 3.35 | 3.43 | 3.64 | 2.19 | 4.37 | 3.77 | 4.92 | - -**Key Findings:** - -1. **Accuracy gap**: 0.61 points between best and worst methods -2. **Context awareness gap**: 0.45 points, favoring anchored iterative -3. **Artifact trail**: Universally weak (2.19-2.45), needs specialized handling -4. **Completeness and instruction following**: Minimal differentiation - -## Statistical Considerations - -- Differences of 0.26-0.35 points are consistent across task types and session lengths -- Pattern holds for both short and long sessions -- Pattern holds across debugging, feature implementation, and code review tasks -- Sample size: 36,611 messages across hundreds of compression points - -## Implementation Notes - -### Probe Generation - -Generate probes at each compression point based on truncated history: -1. Extract factual claims for recall probes -2. Extract file operations for artifact probes -3. Extract incomplete tasks for continuation probes -4. Extract decision points for decision probes - -### Grading Process - -1. Feed probe question + model response + compressed context to judge -2. Evaluate against each criterion in rubric -3. Output structured JSON with scores and reasoning -4. Compute dimension scores as weighted averages -5. Compute overall score as unweighted average of dimensions - -### Blinding - -The judge should not know which compression method produced the response being evaluated. This prevents bias toward known methods. - diff --git a/.agents/skills/context-compression/scripts/compression_evaluator.py b/.agents/skills/context-compression/scripts/compression_evaluator.py deleted file mode 100644 index f8ce4f4b6..000000000 --- a/.agents/skills/context-compression/scripts/compression_evaluator.py +++ /dev/null @@ -1,862 +0,0 @@ -""" -Context Compression Evaluation - -Public API for evaluating context compression quality using probe-based -assessment. This module provides three composable components: - -- **ProbeGenerator**: Extracts factual claims, file operations, and decisions - from conversation history, then generates typed probes for evaluation. - Use when: building a compression evaluation pipeline and needing to - automatically derive test questions from raw conversation history. - -- **CompressionEvaluator**: Scores probe responses against a multi-dimensional - rubric (accuracy, context awareness, artifact trail, completeness, - continuity, instruction following). Use when: comparing compression methods - or validating that a compression strategy preserves critical information. - -- **StructuredSummarizer**: Implements anchored iterative summarization with - explicit sections for session intent, file tracking, decisions, and next - steps. Use when: compressing long-running coding sessions where file - tracking and decision rationale must survive compression. - -Top-level convenience function: -- **evaluate_compression_quality**: End-to-end pipeline that generates probes, - collects model responses, evaluates them, and returns a scored summary with - recommendations. Use when: running a one-shot compression quality check - without wiring up individual components. - -PRODUCTION NOTES: -- The LLM judge calls are stubbed for demonstration. Production systems - should implement actual API calls to a frontier model. -- Token estimation uses simplified heuristics. Production systems should - use model-specific tokenizers. -- Ground truth extraction uses pattern matching. Production systems may - benefit from more sophisticated fact extraction. -""" - -from dataclasses import dataclass, field -from typing import List, Dict, Optional, Callable -from enum import Enum -import json -import re - -__all__ = [ - "ProbeType", - "Probe", - "CriterionResult", - "EvaluationResult", - "RUBRIC_CRITERIA", - "ProbeGenerator", - "CompressionEvaluator", - "StructuredSummarizer", - "evaluate_compression_quality", -] - - -class ProbeType(Enum): - """Types of evaluation probes for compression quality assessment.""" - RECALL = "recall" - ARTIFACT = "artifact" - CONTINUATION = "continuation" - DECISION = "decision" - - -@dataclass -class Probe: - """A probe question for evaluating compression quality. - - Use when: constructing evaluation inputs for CompressionEvaluator. - Each probe targets a specific information category that compression - may have lost. - """ - probe_type: ProbeType - question: str - ground_truth: Optional[str] = None - context_reference: Optional[str] = None - - -@dataclass -class CriterionResult: - """Result for a single evaluation criterion.""" - criterion_id: str - score: float - reasoning: str - - -@dataclass -class EvaluationResult: - """Complete evaluation result for a probe response. - - Contains per-criterion scores, per-dimension aggregates, and an - overall aggregate score. - """ - probe: Probe - response: str - criterion_results: List[CriterionResult] - aggregate_score: float - dimension_scores: Dict[str, float] = field(default_factory=dict) - - -# Evaluation Rubrics - -RUBRIC_CRITERIA: Dict[str, List[Dict]] = { - "accuracy": [ - { - "id": "accuracy_factual", - "question": "Are facts, file paths, and technical details correct?", - "weight": 0.6 - }, - { - "id": "accuracy_technical", - "question": "Are code references and technical concepts correct?", - "weight": 0.4 - } - ], - "context_awareness": [ - { - "id": "context_conversation_state", - "question": "Does the response reflect current conversation state?", - "weight": 0.5 - }, - { - "id": "context_artifact_state", - "question": "Does the response reflect which files/artifacts were accessed?", - "weight": 0.5 - } - ], - "artifact_trail": [ - { - "id": "artifact_files_created", - "question": "Does the agent know which files were created?", - "weight": 0.3 - }, - { - "id": "artifact_files_modified", - "question": "Does the agent know which files were modified?", - "weight": 0.4 - }, - { - "id": "artifact_key_details", - "question": "Does the agent remember function names, variable names, error messages?", - "weight": 0.3 - } - ], - "completeness": [ - { - "id": "completeness_coverage", - "question": "Does the response address all parts of the question?", - "weight": 0.6 - }, - { - "id": "completeness_depth", - "question": "Is sufficient detail provided?", - "weight": 0.4 - } - ], - "continuity": [ - { - "id": "continuity_work_state", - "question": "Can the agent continue without re-fetching information?", - "weight": 0.4 - }, - { - "id": "continuity_todo_state", - "question": "Does the agent maintain awareness of pending tasks?", - "weight": 0.3 - }, - { - "id": "continuity_reasoning", - "question": "Does the agent retain rationale behind previous decisions?", - "weight": 0.3 - } - ], - "instruction_following": [ - { - "id": "instruction_format", - "question": "Does the response follow the requested format?", - "weight": 0.5 - }, - { - "id": "instruction_constraints", - "question": "Does the response respect stated constraints?", - "weight": 0.5 - } - ] -} - - -class ProbeGenerator: - """Generate typed probes from conversation history. - - Use when: automatically deriving evaluation questions from raw - conversation history at compression points. Extracts facts, file - operations, and decisions via pattern matching, then produces - one probe per category. - - For production systems, replace the regex-based extraction with - an LLM-based extractor for higher recall. - """ - - def __init__(self, conversation_history: str) -> None: - self.history = conversation_history - self.extracted_facts = self._extract_facts() - self.extracted_files = self._extract_files() - self.extracted_decisions = self._extract_decisions() - - def generate_probes(self) -> List[Probe]: - """Generate all probe types for evaluation. - - Use when: preparing evaluation inputs at a compression point. - Returns one probe per category (recall, artifact, continuation, - decision) based on extractable content from the history. - """ - probes: List[Probe] = [] - - # Recall probes - if self.extracted_facts: - probes.append(Probe( - probe_type=ProbeType.RECALL, - question="What was the original error or issue that started this session?", - ground_truth=self.extracted_facts.get("original_error"), - context_reference="session_start" - )) - - # Artifact probes - if self.extracted_files: - probes.append(Probe( - probe_type=ProbeType.ARTIFACT, - question="Which files have we modified? Describe what changed in each.", - ground_truth=json.dumps(self.extracted_files), - context_reference="file_operations" - )) - - # Continuation probes - probes.append(Probe( - probe_type=ProbeType.CONTINUATION, - question="What should we do next?", - ground_truth=self.extracted_facts.get("next_steps"), - context_reference="task_state" - )) - - # Decision probes - if self.extracted_decisions: - probes.append(Probe( - probe_type=ProbeType.DECISION, - question="What key decisions did we make and why?", - ground_truth=json.dumps(self.extracted_decisions), - context_reference="decision_points" - )) - - return probes - - def _extract_facts(self) -> Dict[str, str]: - """Extract factual claims from history.""" - facts: Dict[str, str] = {} - - # Extract error patterns - error_patterns = [ - r"error[:\s]+(.+?)(?:\n|$)", - r"(\d{3})\s+(Unauthorized|Not Found|Internal Server Error)", - r"exception[:\s]+(.+?)(?:\n|$)" - ] - - for pattern in error_patterns: - match = re.search(pattern, self.history, re.IGNORECASE) - if match: - facts["original_error"] = match.group(0).strip() - break - - # Extract next steps - next_step_patterns = [ - r"next[:\s]+(.+?)(?:\n|$)", - r"TODO[:\s]+(.+?)(?:\n|$)", - r"remaining[:\s]+(.+?)(?:\n|$)" - ] - - for pattern in next_step_patterns: - match = re.search(pattern, self.history, re.IGNORECASE) - if match: - facts["next_steps"] = match.group(0).strip() - break - - return facts - - def _extract_files(self) -> List[Dict[str, str]]: - """Extract file operations from history.""" - files: List[Dict[str, str]] = [] - - # Common file patterns - file_patterns = [ - r"(?:modified|changed|updated|edited)\s+([^\s]+\.[a-z]+)", - r"(?:created|added)\s+([^\s]+\.[a-z]+)", - r"(?:read|examined|opened)\s+([^\s]+\.[a-z]+)" - ] - - for pattern in file_patterns: - matches = re.findall(pattern, self.history, re.IGNORECASE) - for match in matches: - if match not in [f["path"] for f in files]: - files.append({ - "path": match, - "operation": "modified" if "modif" in pattern else "created" if "creat" in pattern else "read" - }) - - return files - - def _extract_decisions(self) -> List[Dict[str, str]]: - """Extract decision points from history.""" - decisions: List[Dict[str, str]] = [] - - decision_patterns = [ - r"decided to\s+(.+?)(?:\n|$)", - r"chose\s+(.+?)(?:\n|$)", - r"going with\s+(.+?)(?:\n|$)", - r"will use\s+(.+?)(?:\n|$)" - ] - - for pattern in decision_patterns: - matches = re.findall(pattern, self.history, re.IGNORECASE) - for match in matches: - decisions.append({ - "decision": match.strip(), - "context": pattern.split("\\s+")[0] - }) - - return decisions[:5] # Limit to 5 decisions - - -class CompressionEvaluator: - """Evaluate compression quality using probes and LLM judge. - - Use when: comparing compression methods or validating that a specific - compression pass preserved critical information. Scores responses - across six dimensions (accuracy, context awareness, artifact trail, - completeness, continuity, instruction following) and produces an - aggregate quality score. - - The evaluate() method is the primary entry point. Call it once per - probe, then call get_summary() to retrieve aggregated results. - """ - - def __init__(self, model: str = "gpt-5.2") -> None: - self.model = model - self.results: List[EvaluationResult] = [] - - def evaluate(self, - probe: Probe, - response: str, - compressed_context: str) -> EvaluationResult: - """Evaluate a single probe response against the rubric. - - Use when: scoring how well a model's response (given compressed - context) answers a probe question. Returns per-criterion scores, - per-dimension aggregates, and an overall score. - - Args: - probe: The probe question with expected ground truth. - response: The model's response to evaluate. - compressed_context: The compressed context that was provided - to the model when generating the response. - - Returns: - EvaluationResult with scores and reasoning across all - applicable dimensions. - """ - # Get relevant criteria based on probe type - criteria = self._get_criteria_for_probe(probe.probe_type) - - # Evaluate each criterion - criterion_results: List[CriterionResult] = [] - for criterion in criteria: - result = self._evaluate_criterion( - criterion, - probe, - response, - compressed_context - ) - criterion_results.append(result) - - # Calculate dimension scores - dimension_scores = self._calculate_dimension_scores(criterion_results) - - # Calculate aggregate score - aggregate_score = sum(dimension_scores.values()) / len(dimension_scores) if dimension_scores else 0.0 - - result = EvaluationResult( - probe=probe, - response=response, - criterion_results=criterion_results, - aggregate_score=aggregate_score, - dimension_scores=dimension_scores - ) - - self.results.append(result) - return result - - def get_summary(self) -> Dict: - """Get summary of all evaluation results. - - Use when: all probes have been evaluated and an aggregate - report is needed to compare methods or make a go/no-go - decision on a compression strategy. - - Returns: - Dictionary with total evaluations, average score, - per-dimension averages, and weakest/strongest dimensions. - """ - if not self.results: - return {"error": "No evaluations performed"} - - avg_score = sum(r.aggregate_score for r in self.results) / len(self.results) - - # Average dimension scores - dimension_totals: Dict[str, float] = {} - dimension_counts: Dict[str, int] = {} - - for result in self.results: - for dim, score in result.dimension_scores.items(): - dimension_totals[dim] = dimension_totals.get(dim, 0) + score - dimension_counts[dim] = dimension_counts.get(dim, 0) + 1 - - avg_dimensions = { - dim: dimension_totals[dim] / dimension_counts[dim] - for dim in dimension_totals - } - - return { - "total_evaluations": len(self.results), - "average_score": avg_score, - "dimension_averages": avg_dimensions, - "weakest_dimension": min(avg_dimensions, key=avg_dimensions.get) if avg_dimensions else None, - "strongest_dimension": max(avg_dimensions, key=avg_dimensions.get) if avg_dimensions else None, - } - - def _get_criteria_for_probe(self, probe_type: ProbeType) -> List[Dict]: - """Get relevant criteria for probe type.""" - criteria: List[Dict] = [] - - # All probes get accuracy and completeness - criteria.extend(RUBRIC_CRITERIA["accuracy"]) - criteria.extend(RUBRIC_CRITERIA["completeness"]) - - # Add type-specific criteria - if probe_type == ProbeType.ARTIFACT: - criteria.extend(RUBRIC_CRITERIA["artifact_trail"]) - elif probe_type == ProbeType.CONTINUATION: - criteria.extend(RUBRIC_CRITERIA["continuity"]) - elif probe_type == ProbeType.RECALL: - criteria.extend(RUBRIC_CRITERIA["context_awareness"]) - elif probe_type == ProbeType.DECISION: - criteria.extend(RUBRIC_CRITERIA["context_awareness"]) - criteria.extend(RUBRIC_CRITERIA["continuity"]) - - criteria.extend(RUBRIC_CRITERIA["instruction_following"]) - - return criteria - - def _evaluate_criterion(self, - criterion: Dict, - probe: Probe, - response: str, - context: str) -> CriterionResult: - """ - Evaluate a single criterion using LLM judge. - - PRODUCTION NOTE: This is a stub implementation. - Production systems should call the actual LLM API: - - ```python - result = openai.chat.completions.create( - model="gpt-5.2", - messages=[ - {"role": "system", "content": JUDGE_SYSTEM_PROMPT}, - {"role": "user", "content": self._format_judge_input(criterion, probe, response, context)} - ] - ) - return self._parse_judge_output(result) - ``` - """ - # Stub implementation - in production, call LLM judge - score = self._heuristic_score(criterion, response, probe.ground_truth) - reasoning = f"Evaluated {criterion['id']} based on response content." - - return CriterionResult( - criterion_id=criterion["id"], - score=score, - reasoning=reasoning - ) - - def _heuristic_score(self, - criterion: Dict, - response: str, - ground_truth: Optional[str]) -> float: - """ - Heuristic scoring for demonstration. - - Production systems should use LLM judge instead. - """ - score = 3.0 # Base score - - # Adjust based on response length and content - if len(response) < 50: - score -= 1.0 # Too short - elif len(response) > 500: - score += 0.5 # Detailed - - # Check for technical content - if any(ext in response for ext in [".ts", ".py", ".js", ".md"]): - score += 0.5 # Contains file references - - overlap_ratio = self._ground_truth_overlap_ratio(response, ground_truth) - if overlap_ratio >= 0.75: - score += 1.0 - elif overlap_ratio >= 0.4: - score += 0.5 - elif ground_truth: - score -= 0.5 - - return min(5.0, max(0.0, score)) - - def _ground_truth_overlap_ratio(self, - response: str, - ground_truth: Optional[str]) -> float: - if not ground_truth: - return 0.0 - - terms = self._extract_ground_truth_terms(ground_truth) - if not terms: - return 1.0 if ground_truth.lower() in response.lower() else 0.0 - - response_lower = response.lower() - matches = sum(1 for term in terms if term in response_lower) - return matches / len(terms) - - def _extract_ground_truth_terms(self, ground_truth: str) -> List[str]: - try: - parsed = json.loads(ground_truth) - except json.JSONDecodeError: - return [ground_truth.lower()] if ground_truth.strip() else [] - - terms: List[str] = [] - - def collect(value) -> None: - if isinstance(value, str): - normalized = value.strip().lower() - if normalized: - terms.append(normalized) - elif isinstance(value, dict): - for nested in value.values(): - collect(nested) - elif isinstance(value, list): - for nested in value: - collect(nested) - - collect(parsed) - return list(dict.fromkeys(terms)) - - def _calculate_dimension_scores(self, - criterion_results: List[CriterionResult]) -> Dict[str, float]: - """Calculate dimension scores from criterion results.""" - dimension_scores: Dict[str, float] = {} - - for dimension, criteria in RUBRIC_CRITERIA.items(): - criterion_ids = [c["id"] for c in criteria] - relevant_results = [ - r for r in criterion_results - if r.criterion_id in criterion_ids - ] - - if relevant_results: - # Weighted average - total_weight = sum( - c["weight"] for c in criteria - if c["id"] in [r.criterion_id for r in relevant_results] - ) - weighted_sum = sum( - r.score * next(c["weight"] for c in criteria if c["id"] == r.criterion_id) - for r in relevant_results - ) - dimension_scores[dimension] = weighted_sum / total_weight if total_weight > 0 else 0.0 - - return dimension_scores - - -class StructuredSummarizer: - """Generate structured summaries with explicit sections. - - Use when: implementing anchored iterative summarization for - long-running coding sessions. Maintains a persistent summary - with dedicated sections for session intent, file modifications, - decisions, current state, and next steps. - - Call update_from_span() each time a new content span is truncated. - The summarizer merges new information into existing sections rather - than regenerating, preventing cumulative detail loss. - """ - - TEMPLATE = """## Session Intent -{intent} - -## Files Modified -{files_modified} - -## Files Read (Not Modified) -{files_read} - -## Decisions Made -{decisions} - -## Current State -{current_state} - -## Next Steps -{next_steps} -""" - - def __init__(self) -> None: - self.sections: Dict = { - "intent": "", - "files_modified": [], - "files_read": [], - "decisions": [], - "current_state": "", - "next_steps": [] - } - - def update_from_span(self, new_content: str) -> str: - """Update summary from newly truncated content span. - - Use when: a compression trigger fires and a portion of - conversation history is about to be discarded. Pass the - content that will be truncated; the summarizer extracts - structured information and merges it with prior state. - - Args: - new_content: The conversation span being truncated. - - Returns: - Formatted summary string with all sections populated. - """ - # Extract information from new content - new_info = self._extract_from_content(new_content) - - # Merge with existing sections - self._merge_sections(new_info) - - # Generate formatted summary - return self._format_summary() - - def _extract_from_content(self, content: str) -> Dict: - """Extract structured information from content.""" - extracted: Dict = { - "intent": "", - "files_modified": [], - "files_read": [], - "decisions": [], - "current_state": "", - "next_steps": [] - } - - # Extract file modifications - mod_pattern = r"(?:modified|changed|updated|fixed)\s+([^\s]+\.[a-z]+)[:\s]*(.+?)(?:\n|$)" - for match in re.finditer(mod_pattern, content, re.IGNORECASE): - extracted["files_modified"].append({ - "path": match.group(1), - "change": match.group(2).strip()[:100] - }) - - # Extract file reads - read_pattern = r"(?:read|examined|opened|checked)\s+([^\s]+\.[a-z]+)" - for match in re.finditer(read_pattern, content, re.IGNORECASE): - file_path = match.group(1) - if file_path not in [f["path"] for f in extracted["files_modified"]]: - extracted["files_read"].append(file_path) - - # Extract decisions - decision_pattern = r"(?:decided|chose|going with|will use)\s+(.+?)(?:\n|$)" - for match in re.finditer(decision_pattern, content, re.IGNORECASE): - extracted["decisions"].append(match.group(1).strip()[:150]) - - return extracted - - def _merge_sections(self, new_info: Dict) -> None: - """Merge new information with existing sections.""" - # Update intent if empty - if new_info["intent"] and not self.sections["intent"]: - self.sections["intent"] = new_info["intent"] - - # Merge file lists (deduplicate by path) - existing_mod_paths = [f["path"] for f in self.sections["files_modified"]] - for file_info in new_info["files_modified"]: - if file_info["path"] not in existing_mod_paths: - self.sections["files_modified"].append(file_info) - - # Merge read files - for file_path in new_info["files_read"]: - if file_path not in self.sections["files_read"]: - self.sections["files_read"].append(file_path) - - # Append decisions - self.sections["decisions"].extend(new_info["decisions"]) - - # Update current state (latest wins) - if new_info["current_state"]: - self.sections["current_state"] = new_info["current_state"] - - # Merge next steps - self.sections["next_steps"].extend(new_info["next_steps"]) - - def _format_summary(self) -> str: - """Format sections into summary string.""" - files_modified_str = "\n".join( - f"- {f['path']}: {f['change']}" - for f in self.sections["files_modified"] - ) or "None" - - files_read_str = "\n".join( - f"- {f}" for f in self.sections["files_read"] - ) or "None" - - decisions_str = "\n".join( - f"- {d}" for d in self.sections["decisions"][-5:] # Keep last 5 - ) or "None" - - next_steps_str = "\n".join( - f"{i+1}. {s}" for i, s in enumerate(self.sections["next_steps"][-5:]) - ) or "None" - - return self.TEMPLATE.format( - intent=self.sections["intent"] or "Not specified", - files_modified=files_modified_str, - files_read=files_read_str, - decisions=decisions_str, - current_state=self.sections["current_state"] or "In progress", - next_steps=next_steps_str - ) - - -def evaluate_compression_quality( - original_history: str, - compressed_context: str, - model_response_fn: Callable[[str, str], str], -) -> Dict: - """Evaluate compression quality for a conversation end-to-end. - - Use when: running a one-shot quality check on a compression pass. - Generates probes from original history, collects model responses - using the compressed context, evaluates each response, and returns - a scored summary with actionable recommendations. - - Args: - original_history: The full conversation before compression. - compressed_context: The compressed version to evaluate. - model_response_fn: Callable that takes (compressed_context, question) - and returns the model's response string. - - Returns: - Dictionary with total evaluations, average score, per-dimension - averages, weakest/strongest dimensions, and recommendations list. - """ - # Generate probes - generator = ProbeGenerator(original_history) - probes = generator.generate_probes() - - # Evaluate each probe - evaluator = CompressionEvaluator() - - for probe in probes: - # Get model response using compressed context - response = model_response_fn(compressed_context, probe.question) - - # Evaluate response - evaluator.evaluate(probe, response, compressed_context) - - # Get summary - summary = evaluator.get_summary() - - # Add recommendations - summary["recommendations"] = [] - - if summary.get("weakest_dimension") == "artifact_trail": - summary["recommendations"].append( - "Consider implementing separate artifact tracking outside compression" - ) - - if summary.get("average_score", 0) < 3.5: - summary["recommendations"].append( - "Compression quality is below threshold - consider less aggressive compression" - ) - - return summary - - -if __name__ == "__main__": - # Demo: generate probes and evaluate a sample compression - - sample_history = """ - User reported error: 401 Unauthorized on /api/auth/login endpoint. - Examined auth.controller.ts - JWT generation looks correct. - Examined middleware/cors.ts - no issues found. - Modified config/redis.ts: Fixed connection pooling configuration. - Modified services/session.service.ts: Added retry logic for transient failures. - Decided to use Redis connection pool instead of per-request connections. - Modified tests/auth.test.ts: Updated mock setup for new config. - 14 tests passing, 2 failing (mock setup issues). - Next: Fix remaining test failures in session service mocks. - """ - - sample_compressed = """ - ## Session Intent - Debug 401 Unauthorized on /api/auth/login. - - ## Root Cause - Stale Redis connection in session store. - - ## Files Modified - - config/redis.ts: Fixed connection pooling - - services/session.service.ts: Added retry logic - - tests/auth.test.ts: Updated mock setup - - ## Test Status - 14 passing, 2 failing - - ## Next Steps - 1. Fix remaining test failures - """ - - # Stub model response function - def mock_model_response(context: str, question: str) -> str: - if "error" in question.lower(): - return "The original error was a 401 Unauthorized on /api/auth/login." - if "files" in question.lower(): - return "Modified config/redis.ts, services/session.service.ts, tests/auth.test.ts." - if "next" in question.lower(): - return "Fix remaining test failures in session service mocks." - if "decision" in question.lower(): - return "Decided to use Redis connection pool instead of per-request connections." - return "No specific information available." - - # Run evaluation - result = evaluate_compression_quality( - original_history=sample_history, - compressed_context=sample_compressed, - model_response_fn=mock_model_response, - ) - - print("=== Compression Quality Evaluation ===") - print(f"Total evaluations: {result['total_evaluations']}") - print(f"Average score: {result['average_score']:.2f}") - print() - print("Dimension averages:") - for dim, score in result.get("dimension_averages", {}).items(): - print(f" {dim}: {score:.2f}") - print() - print(f"Weakest dimension: {result.get('weakest_dimension')}") - print(f"Strongest dimension: {result.get('strongest_dimension')}") - print() - if result.get("recommendations"): - print("Recommendations:") - for rec in result["recommendations"]: - print(f" - {rec}") - else: - print("No recommendations - compression quality looks acceptable.") diff --git a/.agents/skills/context-compression/tests/test_compression_evaluator.py b/.agents/skills/context-compression/tests/test_compression_evaluator.py deleted file mode 100644 index 0bdd6fbe9..000000000 --- a/.agents/skills/context-compression/tests/test_compression_evaluator.py +++ /dev/null @@ -1,56 +0,0 @@ -import importlib.util -import unittest -from pathlib import Path - - -MODULE_PATH = ( - Path(__file__).resolve().parents[1] / "scripts" / "compression_evaluator.py" -) -MODULE_SPEC = importlib.util.spec_from_file_location( - "compression_evaluator", MODULE_PATH -) -if MODULE_SPEC is None or MODULE_SPEC.loader is None: - raise RuntimeError(f"Unable to load compression_evaluator.py from {MODULE_PATH}") -COMPRESSION_EVALUATOR = importlib.util.module_from_spec(MODULE_SPEC) -MODULE_SPEC.loader.exec_module(COMPRESSION_EVALUATOR) - - -class CompressionEvaluatorTests(unittest.TestCase): - def test_json_ground_truth_terms_score_when_response_mentions_artifacts( - self, - ) -> None: - evaluator = COMPRESSION_EVALUATOR.CompressionEvaluator() - - rich_score = evaluator._heuristic_score( - {"id": "artifact_files_modified"}, - "We modified src/app.py and updated README.md during the session.", - '[{"path": "src/app.py", "operation": "modified"}, {"path": "README.md", "operation": "updated"}]', - ) - poor_score = evaluator._heuristic_score( - {"id": "artifact_files_modified"}, - "We changed some files but I do not remember which ones.", - '[{"path": "src/app.py", "operation": "modified"}, {"path": "README.md", "operation": "updated"}]', - ) - - self.assertGreater(rich_score, poor_score) - self.assertGreaterEqual(rich_score, 4.0) - - def test_plain_text_ground_truth_still_uses_substring_match(self) -> None: - evaluator = COMPRESSION_EVALUATOR.CompressionEvaluator() - - exact_score = evaluator._heuristic_score( - {"id": "continuity_work_state"}, - "Next: fix the websocket timeout before rerunning tests.", - "fix the websocket timeout", - ) - missing_score = evaluator._heuristic_score( - {"id": "continuity_work_state"}, - "Next: inspect logs again.", - "fix the websocket timeout", - ) - - self.assertGreater(exact_score, missing_score) - - -if __name__ == "__main__": - unittest.main() diff --git a/.agents/skills/context-degradation/references/patterns.md b/.agents/skills/context-degradation/references/patterns.md deleted file mode 100644 index 663a3c32d..000000000 --- a/.agents/skills/context-degradation/references/patterns.md +++ /dev/null @@ -1,314 +0,0 @@ -# Context Degradation Patterns: Technical Reference - -This document provides technical details on diagnosing and measuring context degradation. - -## Attention Distribution Analysis - -### U-Shaped Curve Measurement - -Measure attention distribution across context positions: - -```python -def measure_attention_distribution(model, context_tokens, query): - """ - Measure how attention varies across context positions. - - Returns distribution showing attention weight by position. - """ - attention_by_position = [] - - for position in range(len(context_tokens)): - # Measure model's attention to this position - attention = get_attention_weights(model, context_tokens, query, position) - attention_by_position.append({ - "position": position, - "attention": attention, - "is_beginning": position < len(context_tokens) * 0.1, - "is_end": position > len(context_tokens) * 0.9, - "is_middle": True # Will be overwritten - }) - - # Classify positions - for item in attention_by_position: - if item["is_beginning"] or item["is_end"]: - item["region"] = "attention_favored" - else: - item["region"] = "attention_degraded" - - return attention_by_position -``` - -### Lost-in-Middle Detection - -Detect when critical information falls in degraded attention regions: - -```python -def detect_lost_in_middle(critical_positions, attention_distribution): - """ - Check if critical information is in attention-favored positions. - - Args: - critical_positions: List of positions containing critical info - attention_distribution: Output from measure_attention_distribution - - Returns: - Dictionary with detection results and recommendations - """ - results = { - "at_risk": [], - "safe": [], - "recommendations": [] - } - - for pos in critical_positions: - region = attention_distribution[pos]["region"] - if region == "attention_degraded": - results["at_risk"].append(pos) - else: - results["safe"].append(pos) - - # Generate recommendations - if results["at_risk"]: - results["recommendations"].extend([ - "Move critical information to attention-favored positions", - "Use explicit markers to highlight critical information", - "Consider splitting context to reduce middle section" - ]) - - return results -``` - -## Context Poisoning Detection - -### Hallucination Tracking - -Track potential hallucinations across conversation turns: - -```python -class HallucinationTracker: - def __init__(self): - self.claims = [] - self.verifications = [] - - def add_claims(self, text): - """Extract claims from text for later verification.""" - claims = extract_claims(text) - self.claims.extend([{"text": c, "verified": None} for c in claims]) - - def verify_claims(self, ground_truth): - """Verify claims against ground truth.""" - for claim in self.claims: - if claim["verified"] is None: - claim["verified"] = check_claim(claim["text"], ground_truth) - - def get_poisoning_indicators(self): - """ - Return indicators of potential context poisoning. - - High ratio of unverified claims suggests poisoning risk. - """ - unverified = sum(1 for c in self.claims if not c["verified"]) - verified_false = sum(1 for c in self.claims if c["verified"] == False) - - return { - "unverified_count": unverified, - "false_count": verified_false, - "poisoning_risk": verified_false > 0 or unverified > len(self.claims) * 0.3 - } -``` - -### Error Propagation Analysis - -Track how errors flow through context: - -```python -def analyze_error_propagation(context, error_points): - """ - Analyze how errors at specific points affect downstream context. - - Returns visualization of error spread and impact assessment. - """ - impact_map = {} - - for error_point in error_points: - # Find all references to content after error point - downstream_refs = find_references(context, after=error_point) - - for ref in downstream_refs: - if ref not in impact_map: - impact_map[ref] = [] - impact_map[ref].append({ - "source": error_point, - "type": classify_error_type(context[error_point]) - }) - - # Assess severity - high_impact_areas = [k for k, v in impact_map.items() if len(v) > 3] - - return { - "impact_map": impact_map, - "high_impact_areas": high_impact_areas, - "requires_intervention": len(high_impact_areas) > 0 - } -``` - -## Distraction Metrics - -### Relevance Scoring - -Score relevance of context elements to current task: - -```python -def score_context_relevance(context_elements, task_description): - """ - Score each context element for relevance to current task. - - Returns scores and identifies high-distraction elements. - """ - task_embedding = embed(task_description) - - scored_elements = [] - for i, element in enumerate(context_elements): - element_embedding = embed(element) - relevance = cosine_similarity(task_embedding, element_embedding) - scored_elements.append({ - "index": i, - "content_preview": element[:100], - "relevance_score": relevance - }) - - # Sort by relevance - scored_elements.sort(key=lambda x: x["relevance_score"], reverse=True) - - # Identify potential distractors - threshold = calculate_relevance_threshold(scored_elements) - distractors = [e for e in scored_elements if e["relevance_score"] < threshold] - - return { - "scored_elements": scored_elements, - "distractors": distractors, - "recommendation": f"Consider removing {len(distractors)} low-relevance elements" - } -``` - -## Degradation Monitoring System - -### Context Health Dashboard - -Implement continuous monitoring of context health: - -```python -class ContextHealthMonitor: - def __init__(self, model, context_window_limit): - self.model = model - self.limit = context_window_limit - self.metrics = [] - - def assess_health(self, context, task): - """ - Assess overall context health for current task. - - Returns composite score and component metrics. - """ - metrics = { - "token_count": len(context), - "utilization_ratio": len(context) / self.limit, - "attention_distribution": measure_attention_distribution(self.model, context, task), - "relevance_scores": score_context_relevance(context, task), - "age_tokens": count_recent_tokens(context) - } - - # Calculate composite health score - health_score = self._calculate_composite(metrics) - - result = { - "health_score": health_score, - "metrics": metrics, - "status": self._interpret_score(health_score), - "recommendations": self._generate_recommendations(metrics) - } - - self.metrics.append(result) - return result - - def _calculate_composite(self, metrics): - """Calculate composite health score from components.""" - # Weighted combination of metrics - utilization_penalty = min(metrics["utilization_ratio"] * 0.5, 0.3) - attention_penalty = self._calculate_attention_penalty(metrics["attention_distribution"]) - relevance_penalty = self._calculate_relevance_penalty(metrics["relevance_scores"]) - - base_score = 1.0 - score = base_score - utilization_penalty - attention_penalty - relevance_penalty - return max(0, score) - - def _interpret_score(self, score): - """Interpret health score and return status.""" - if score > 0.8: - return "healthy" - elif score > 0.6: - return "warning" - elif score > 0.4: - return "degraded" - else: - return "critical" -``` - -### Alert Thresholds - -Configure appropriate alert thresholds: - -```python -CONTEXT_ALERTS = { - "utilization_warning": 0.7, # 70% of context limit - "utilization_critical": 0.9, # 90% of context limit - "attention_degraded_ratio": 0.3, # 30% in middle region - "relevance_threshold": 0.3, # Below 30% relevance - "consecutive_warnings": 3 # Three warnings triggers alert -} -``` - -## Recovery Procedures - -### Context Truncation Strategy - -When context degrades beyond recovery, truncate strategically: - -```python -def truncate_context_for_recovery(context, preserved_elements, target_size): - """ - Truncate context while preserving critical elements. - - Strategy: - 1. Preserve system prompt and tool definitions - 2. Preserve recent conversation turns - 3. Preserve critical retrieved documents - 4. Summarize older content if needed - 5. Truncate from middle if still over target - """ - truncated = [] - - # Category 1: Critical system elements (preserve always) - system_elements = extract_system_elements(context) - truncated.extend(system_elements) - - # Category 2: Recent conversation (preserve more) - recent_turns = extract_recent_turns(context, num_turns=10) - truncated.extend(recent_turns) - - # Category 3: Critical documents (preserve key ones) - critical_docs = extract_critical_documents(context, preserved_elements) - truncated.extend(critical_docs) - - # Check size and summarize if needed - while len(truncated) > target_size: - # Summarize oldest category 3 elements - truncated = summarize_oldest(truncated, category="documents") - - # If still too large, truncate oldest turns - if len(truncated) > target_size: - truncated = truncate_oldest_turns(truncated, keep_recent=5) - - return truncated -``` - diff --git a/.agents/skills/context-degradation/scripts/degradation_detector.py b/.agents/skills/context-degradation/scripts/degradation_detector.py deleted file mode 100644 index 11c0246ab..000000000 --- a/.agents/skills/context-degradation/scripts/degradation_detector.py +++ /dev/null @@ -1,614 +0,0 @@ -""" -Context Degradation Detection — Public API -============================================ - -Detect, measure, and diagnose context degradation patterns in LLM agent systems. - -Public API: - measure_attention_distribution — Map attention weight across context positions. - detect_lost_in_middle — Flag critical information in degraded-attention regions. - analyze_context_structure — Assess structural degradation risk factors. - PoisoningDetector — Detect context poisoning indicators (error accumulation, - contradictions, hallucination markers). - ContextHealthAnalyzer — Run composite health analysis combining attention, - poisoning, and utilization metrics. - analyze_agent_context — One-call convenience function for agent sessions. - -PRODUCTION NOTES: -- The attention estimation functions simulate U-shaped attention curves for demonstration - purposes. Production systems should extract actual attention weights from model internals - when available (e.g., via TransformerLens or model-specific APIs). -- Token estimation uses simplified heuristics (~1 token per whitespace-split word). - Production systems should use model-specific tokenizers for accurate counts. -- Poisoning and hallucination detection uses pattern matching as a proxy. Production - systems may benefit from fine-tuned classifiers or model-based detection. -""" - -import random -import re -from typing import Dict, List, Optional - -__all__ = [ - "measure_attention_distribution", - "detect_lost_in_middle", - "analyze_context_structure", - "PoisoningDetector", - "ContextHealthAnalyzer", - "analyze_agent_context", -] - - -# --------------------------------------------------------------------------- -# Attention Distribution Analysis -# --------------------------------------------------------------------------- - -def measure_attention_distribution( - context_tokens: List[str], - query: str, -) -> List[Dict[str, object]]: - """Map simulated attention weight to each context position. - - Use when: diagnosing whether critical information sits in the - low-attention middle region of a long context. - - Args: - context_tokens: Whitespace-split tokens (or chunks) of the context. - query: The query or task description the context is meant to support. - - Returns: - List of dicts, one per position, each containing: - position (int), attention (float), region (str), tokens (str | None). - """ - n = len(context_tokens) - attention_by_position: List[Dict[str, object]] = [] - - for position in range(n): - is_beginning = position < n * 0.1 - is_end = position > n * 0.9 - - attention = _estimate_attention(position, n, is_beginning, is_end) - - attention_by_position.append({ - "position": position, - "attention": attention, - "region": "attention_favored" if (is_beginning or is_end) else "attention_degraded", - "tokens": context_tokens[position][:50] if position < 5 or position > n - 5 else None, - }) - - return attention_by_position - - -def _estimate_attention( - position: int, - total: int, - is_beginning: bool, - is_end: bool, -) -> float: - """Estimate attention weight for a single position. - - Simulates the U-shaped attention curve documented in lost-in-middle research: - - Beginning tokens receive high attention (primacy / attention-sink effect). - - End tokens receive high attention (recency effect). - - Middle tokens receive degraded attention. - - IMPORTANT: This is a simulation for demonstration. Production systems should - extract actual attention weights from model forward passes or use - interpretability libraries (e.g., TransformerLens). - """ - if is_beginning: - return 0.8 + random.random() * 0.2 - elif is_end: - return 0.7 + random.random() * 0.3 - else: - middle_progress = (position - total * 0.1) / (total * 0.8) - base_attention = 0.3 * (1 - middle_progress) + 0.1 * middle_progress - return base_attention + random.random() * 0.1 - - -# --------------------------------------------------------------------------- -# Lost-in-Middle Detection -# --------------------------------------------------------------------------- - -def detect_lost_in_middle( - critical_positions: List[int], - attention_distribution: List[Dict[str, object]], -) -> Dict[str, object]: - """Check if critical information sits in attention-degraded positions. - - Use when: context has been assembled and you need to verify that - high-priority content is not buried in the low-attention middle zone. - - Args: - critical_positions: Indices into the context that hold critical info. - attention_distribution: Output of ``measure_attention_distribution``. - - Returns: - Dict with keys: at_risk (list[int]), safe (list[int]), - recommendations (list[str]), degradation_score (float 0-1). - """ - results: Dict[str, object] = { - "at_risk": [], - "safe": [], - "recommendations": [], - "degradation_score": 0.0, - } - - at_risk_count = 0 - total_critical = len(critical_positions) - - for pos in critical_positions: - if pos < len(attention_distribution): - region = attention_distribution[pos]["region"] - if region == "attention_degraded": - results["at_risk"].append(pos) - at_risk_count += 1 - else: - results["safe"].append(pos) - - if total_critical > 0: - results["degradation_score"] = at_risk_count / total_critical - - if results["at_risk"]: - results["recommendations"].extend([ - "Move critical information to attention-favored positions", - "Use explicit markers to highlight critical information", - "Consider splitting context to reduce middle section", - f"{at_risk_count}/{total_critical} critical items are in degraded region", - ]) - - return results - - -# --------------------------------------------------------------------------- -# Context Structure Analysis -# --------------------------------------------------------------------------- - -def analyze_context_structure(context: str) -> Dict[str, object]: - """Assess structural degradation risk factors in a context string. - - Use when: evaluating whether a context layout puts too much content - in the low-attention middle zone before sending it to a model. - - Args: - context: The full context string to analyze. - - Returns: - Dict with total_lines, sections list, middle_content_ratio, - and degradation_risk level (low / medium / high). - """ - lines = context.split("\n") - sections: List[Dict[str, object]] = [] - - current_section: Dict[str, object] = {"start": 0, "type": "unknown", "length": 0} - - for i, line in enumerate(lines): - if line.startswith("#"): - if current_section["length"] > 0: - sections.append(current_section) - current_section = { - "start": i, - "type": "header", - "length": 1, - "header": line.lstrip("#").strip(), - } - else: - current_section["length"] += 1 - - sections.append(current_section) - - n = len(lines) - middle_start = int(n * 0.3) - middle_end = int(n * 0.7) - - middle_content = sum( - s["length"] for s in sections - if s["start"] >= middle_start and s["start"] <= middle_end - ) - - middle_ratio = middle_content / n if n > 0 else 0 - return { - "total_lines": n, - "sections": sections, - "middle_content_ratio": middle_ratio, - "degradation_risk": ( - "high" if middle_ratio > 0.5 - else "medium" if middle_ratio > 0.3 - else "low" - ), - } - - -# --------------------------------------------------------------------------- -# Context Poisoning Detection -# --------------------------------------------------------------------------- - -class PoisoningDetector: - """Detect context poisoning indicators via pattern matching. - - Use when: context quality is suspect — outputs degrade on previously - successful tasks, tool calls misalign, or hallucinations persist - despite corrections. - """ - - def __init__(self) -> None: - self.claims: List[Dict[str, object]] = [] - self.error_patterns: List[str] = [ - r"error", - r"failed", - r"exception", - r"cannot", - r"unable", - r"invalid", - r"not found", - ] - - def extract_claims(self, text: str) -> List[Dict[str, object]]: - """Extract claims from text for verification tracking. - - Use when: building a provenance chain to trace which claims - entered context and whether they have been verified. - - Args: - text: Raw text to extract claims from. - - Returns: - List of claim dicts with id, text, verified status, and - error indicator flag. - """ - sentences = text.split(".") - claims: List[Dict[str, object]] = [] - - for i, sentence in enumerate(sentences): - sentence = sentence.strip() - if len(sentence) < 10: - continue - - claims.append({ - "id": i, - "text": sentence, - "verified": None, - "has_error_indicator": any( - re.search(pattern, sentence, re.IGNORECASE) - for pattern in self.error_patterns - ), - }) - - self.claims.extend(claims) - return claims - - def detect_poisoning(self, context: str) -> Dict[str, object]: - """Detect potential context poisoning indicators. - - Use when: agent output quality has degraded and context - contamination is suspected. Checks for error accumulation, - contradictions, and hallucination markers. - - Args: - context: The full context string to analyze. - - Returns: - Dict with poisoning_risk (bool), indicators (list), - and overall_risk level (low / medium / high). - """ - indicators: List[Dict[str, object]] = [] - - # Check for error accumulation - error_count = sum( - 1 for pattern in self.error_patterns - if re.search(pattern, context, re.IGNORECASE) - ) - - if error_count > 3: - indicators.append({ - "type": "error_accumulation", - "count": error_count, - "severity": "high" if error_count > 5 else "medium", - "message": f"Found {error_count} error indicators in context", - }) - - # Check for contradiction patterns - contradictions = self._detect_contradictions(context) - if contradictions: - indicators.append({ - "type": "contradictions", - "count": len(contradictions), - "examples": contradictions[:3], - "severity": "high", - "message": f"Found {len(contradictions)} potential contradictions", - }) - - # Check for hallucination markers - hallucination_markers = self._detect_hallucination_markers(context) - if hallucination_markers: - indicators.append({ - "type": "hallucination_markers", - "count": len(hallucination_markers), - "severity": "medium", - "message": f"Found {len(hallucination_markers)} phrases associated with uncertain claims", - }) - - return { - "poisoning_risk": len(indicators) > 0, - "indicators": indicators, - "overall_risk": ( - "high" if len(indicators) > 2 - else "medium" if len(indicators) > 0 - else "low" - ), - } - - def _detect_contradictions(self, text: str) -> List[str]: - """Detect potential contradictions in text.""" - contradictions: List[str] = [] - - conflict_patterns = [ - (r"however", r"but"), - (r"on the other hand", r"instead"), - (r"although", r"yet"), - (r"despite", r"nevertheless"), - ] - - for pattern1, pattern2 in conflict_patterns: - if re.search(pattern1, text, re.IGNORECASE) and re.search(pattern2, text, re.IGNORECASE): - sentences = text.split(".") - for sentence in sentences: - if (re.search(pattern1, sentence, re.IGNORECASE) - or re.search(pattern2, sentence, re.IGNORECASE)): - stripped = sentence.strip() - if stripped and len(stripped) < 200: - contradictions.append(stripped[:100]) - - return contradictions[:5] - - def _detect_hallucination_markers(self, text: str) -> List[str]: - """Detect phrases associated with uncertain or hallucinated claims.""" - markers = [ - "may have been", - "might have", - "could potentially", - "possibly", - "apparently", - "reportedly", - "it is said that", - "sources suggest", - "believed to be", - "thought to be", - ] - - return [marker for marker in markers if marker in text.lower()] - - -# --------------------------------------------------------------------------- -# Context Health Analyzer -# --------------------------------------------------------------------------- - -class ContextHealthAnalyzer: - """Run composite health analysis on a context string. - - Use when: performing routine health checks on agent context during - long-running sessions, or when setting up automated monitoring that - triggers compaction or isolation before degradation hits. - - Combines attention distribution, poisoning detection, and utilization - metrics into a single 0-1 health score with status interpretation. - """ - - def __init__(self, context_limit: int = 100_000) -> None: - self.context_limit: int = context_limit - self.metrics_history: List[Dict[str, object]] = [] - - def analyze( - self, - context: str, - critical_positions: Optional[List[int]] = None, - ) -> Dict[str, object]: - """Perform comprehensive context health analysis. - - Use when: a single health-check call is needed that covers - attention, poisoning, and utilization in one pass. - - Args: - context: The full context string to analyze. - critical_positions: Indices of tokens holding critical info. - Defaults to the first 10 positions if not provided. - - Returns: - Dict with health_score (float 0-1), status (str), - metrics (dict), issues (dict), and recommendations (list[str]). - """ - tokens = context.split() - - token_count = len(tokens) - utilization = token_count / self.context_limit - - attention_dist = measure_attention_distribution( - tokens[:1000], # Sample for efficiency - "current_task", - ) - - degradation = detect_lost_in_middle( - critical_positions or list(range(10)), - attention_dist, - ) - - poisoning = PoisoningDetector().detect_poisoning(context) - - health_score = self._calculate_health_score( - utilization=utilization, - degradation=degradation["degradation_score"], - poisoning_risk=1.0 if poisoning["poisoning_risk"] else 0.0, - ) - - result: Dict[str, object] = { - "health_score": health_score, - "status": self._interpret_score(health_score), - "metrics": { - "token_count": token_count, - "utilization": utilization, - "degradation_score": degradation["degradation_score"], - "poisoning_risk": poisoning["overall_risk"], - }, - "issues": { - "lost_in_middle": degradation, - "poisoning": poisoning, - }, - "recommendations": self._generate_recommendations( - utilization, degradation, poisoning - ), - } - - self.metrics_history.append(result) - return result - - def _calculate_health_score( - self, - utilization: float, - degradation: float, - poisoning_risk: float, - ) -> float: - """Calculate composite health score (0-1, higher is healthier).""" - utilization_penalty = min(utilization * 0.5, 0.3) - degradation_penalty = degradation * 0.3 - poisoning_penalty = poisoning_risk * 0.2 - - score = 1.0 - utilization_penalty - degradation_penalty - poisoning_penalty - return max(0.0, min(1.0, score)) - - def _interpret_score(self, score: float) -> str: - """Map numeric score to human-readable status.""" - if score > 0.8: - return "healthy" - elif score > 0.6: - return "warning" - elif score > 0.4: - return "degraded" - else: - return "critical" - - def _generate_recommendations( - self, - utilization: float, - degradation: Dict[str, object], - poisoning: Dict[str, object], - ) -> List[str]: - """Generate actionable recommendations based on analysis.""" - recommendations: List[str] = [] - - if utilization > 0.8: - recommendations.append("Context near limit - consider compaction") - recommendations.append("Implement observation masking for tool outputs") - - if degradation.get("at_risk"): - recommendations.append("Critical information in degraded attention region") - recommendations.append("Move key information to beginning or end of context") - - if poisoning["poisoning_risk"]: - recommendations.append("Context poisoning indicators detected") - recommendations.append("Review and remove potentially erroneous information") - - if not recommendations: - recommendations.append("Context appears healthy - continue monitoring") - - return recommendations - - -# --------------------------------------------------------------------------- -# Convenience Function -# --------------------------------------------------------------------------- - -def analyze_agent_context( - context: str, - context_limit: int = 80_000, - critical_positions: Optional[List[int]] = None, -) -> Dict[str, object]: - """One-call health analysis for an agent session. - - Use when: a quick health check is needed without manually configuring - an analyzer instance. Prints a summary and returns the full result dict. - - Args: - context: The full context string to analyze. - context_limit: Maximum token budget for this agent's context window. - critical_positions: Indices of critical tokens. Defaults to [0..4]. - - Returns: - Full health analysis dict from ``ContextHealthAnalyzer.analyze``. - """ - analyzer = ContextHealthAnalyzer(context_limit=context_limit) - - if critical_positions is None: - critical_positions = list(range(5)) - - result = analyzer.analyze(context, critical_positions) - - print(f"Health Score: {result['health_score']:.2f}") - print(f"Status: {result['status']}") - print("Recommendations:") - for rec in result["recommendations"]: - print(f" - {rec}") - - return result - - -# --------------------------------------------------------------------------- -# CLI Demo -# --------------------------------------------------------------------------- - -if __name__ == "__main__": - # Demonstrate the public API with synthetic context - print("=" * 60) - print("Context Degradation Detector — Demo") - print("=" * 60) - - # Build a synthetic context with identifiable sections - intro = "System prompt: Analyze quarterly revenue data and produce a report. " - middle = "Background information. " * 200 # Filler to simulate long context - conclusion = "Key finding: Revenue increased 15% year-over-year. " - sample_context = intro + middle + conclusion - - print(f"\nSample context length: {len(sample_context.split())} tokens") - - # 1. Structure analysis - print("\n--- Structure Analysis ---") - structure = analyze_context_structure(sample_context) - print(f" Lines: {structure['total_lines']}") - print(f" Middle content ratio: {structure['middle_content_ratio']:.2f}") - print(f" Degradation risk: {structure['degradation_risk']}") - - # 2. Attention distribution (first 50 tokens for brevity) - print("\n--- Attention Distribution (first 50 tokens) ---") - tokens = sample_context.split()[:50] - attention = measure_attention_distribution(tokens, "quarterly revenue") - favored = sum(1 for a in attention if a["region"] == "attention_favored") - degraded = sum(1 for a in attention if a["region"] == "attention_degraded") - print(f" Favored positions: {favored}") - print(f" Degraded positions: {degraded}") - - # 3. Lost-in-middle detection - print("\n--- Lost-in-Middle Detection ---") - critical = [0, 1, 2, 25, 26, 48, 49] # Start, middle, end - lim_result = detect_lost_in_middle(critical, attention) - print(f" At risk: {lim_result['at_risk']}") - print(f" Safe: {lim_result['safe']}") - print(f" Degradation score: {lim_result['degradation_score']:.2f}") - - # 4. Poisoning detection - print("\n--- Poisoning Detection ---") - poisoned_context = ( - "The API returned an error. However, the system reportedly " - "recovered. But the error persisted and the request failed. " - "Unable to parse the response. Sources suggest the endpoint " - "may have been deprecated. Although retries succeeded, yet " - "the invalid token caused an exception." - ) - detector = PoisoningDetector() - poisoning = detector.detect_poisoning(poisoned_context) - print(f" Poisoning risk: {poisoning['poisoning_risk']}") - print(f" Overall risk: {poisoning['overall_risk']}") - for indicator in poisoning["indicators"]: - print(f" [{indicator['severity']}] {indicator['message']}") - - # 5. Full health analysis - print("\n--- Full Health Analysis ---") - result = analyze_agent_context(sample_context) - print(f"\n Full result keys: {list(result.keys())}") diff --git a/.agents/skills/context-fundamentals/references/context-components.md b/.agents/skills/context-fundamentals/references/context-components.md deleted file mode 100644 index 2c0a6d5e0..000000000 --- a/.agents/skills/context-fundamentals/references/context-components.md +++ /dev/null @@ -1,283 +0,0 @@ -# Context Components: Technical Reference - -This document provides detailed technical reference for each context component in agent systems. - -## System Prompt Engineering - -### Section Structure - -Organize system prompts into distinct sections with clear boundaries. A recommended structure: - -``` - -Context about the domain, user preferences, or project-specific details - - - -Core behavioral guidelines and task instructions - - - -When and how to use available tools - - - -Expected output format and quality standards - -``` - -This structure allows agents to locate relevant information quickly and enables selective context loading in advanced implementations. - -### Altitude Calibration - -The "altitude" of instructions refers to the level of abstraction. Consider these examples: - -**Too Low (Brittle):** -``` -If the user asks about pricing, check the pricing table in docs/pricing.md. -If the table shows USD, convert to EUR using the exchange rate in -config/exchange_rates.json. If the user is in the EU, add VAT at the -applicable rate from config/vat_rates.json. Format the response with -the currency symbol, two decimal places, and a note about VAT. -``` - -**Too High (Vague):** -``` -Help users with pricing questions. Be helpful and accurate. -``` - -**Optimal (Heuristic-Driven):** -``` -For pricing inquiries: -1. Retrieve current rates from docs/pricing.md -2. Apply user location adjustments (see config/location_defaults.json) -3. Format with appropriate currency and tax considerations - -Prefer exact figures over estimates. When rates are unavailable, -say so explicitly rather than projecting. -``` - -The optimal altitude provides clear steps while allowing flexibility in execution. - -## Tool Definition Specification - -### Schema Structure - -Each tool should define: - -```python -{ - "name": "tool_function_name", - "description": "Clear description of what the tool does and when to use it", - "parameters": { - "type": "object", - "properties": { - "param_name": { - "type": "string", - "description": "What this parameter controls", - "default": "reasonable_default_value" - } - }, - "required": ["param_name"] - }, - "returns": { - "type": "object", - "description": "What the tool returns and its structure" - } -} -``` - -### Description Engineering - -Tool descriptions should answer: what the tool does, when to use it, and what it produces. Include usage context, examples, and edge cases. - -**Weak Description:** -``` -Search the database for customer information. -``` - -**Strong Description:** -``` -Retrieve customer information by ID or email. - -Use when: -- User asks about a specific customer's details, history, or status -- User provides a customer identifier and needs related information - -Returns customer object with: -- Basic info (name, email, account status) -- Order history summary -- Support ticket count - -Returns null if customer not found. Returns error if database unreachable. -``` - -## Retrieved Document Management - -### Identifier Design - -Design identifiers that convey meaning and enable efficient retrieval: - -**Poor identifiers:** -- `data/file1.json` -- `ref/ref.md` -- `2024/q3/report` - -**Strong identifiers:** -- `customer_pricing_rates.json` -- `engineering_onboarding_checklist.md` -- `2024_q3_revenue_report.pdf` - -Strong identifiers allow agents to locate relevant files even without search tools. - -### Document Chunking Strategy - -For large documents, chunk strategically to preserve semantic coherence: - -```python -# Pseudocode for semantic chunking -def chunk_document(content): - """Split document at natural semantic boundaries.""" - boundaries = find_section_headers(content) - boundaries += find_paragraph_breaks(content) - boundaries += find_logical_breaks(content) - - chunks = [] - for i in range(len(boundaries) - 1): - chunk = content[boundaries[i]:boundaries[i+1]] - if len(chunk) > MIN_CHUNK_SIZE and len(chunk) < MAX_CHUNK_SIZE: - chunks.append(chunk) - - return chunks -``` - -Avoid arbitrary character limits that split mid-sentence or mid-concept. - -## Message History Management - -### Turn Representation - -Structure message history to preserve key information: - -```python -{ - "role": "user" | "assistant" | "tool", - "content": "message text", - "reasoning": "optional chain-of-thought", - "tool_calls": [list if role="assistant"], - "tool_output": "output if role="tool"", - "summary": "compact summary if conversation is long" -} -``` - -### Summary Injection Pattern - -For long conversations, inject summaries at intervals: - -```python -def inject_summaries(messages, summary_interval=20): - """Inject summaries at regular intervals to preserve context.""" - summarized = [] - for i, msg in enumerate(messages): - summarized.append(msg) - if i > 0 and i % summary_interval == 0: - summary = generate_summary(summarized[-summary_interval:]) - summarized.append({ - "role": "system", - "content": f"Conversation summary: {summary}", - "is_summary": True - }) - return summarized -``` - -## Tool Output Optimization - -### Response Formats - -Provide response format options to control token usage: - -```python -def get_customer_response_format(): - return { - "format": "concise | detailed", - "fields": ["id", "name", "email", "status", "history_summary"] - } -``` - -The concise format returns essential fields only; detailed returns complete objects. - -### Observation Masking - -For verbose tool outputs, consider masking patterns: - -```python -def mask_observation(output, max_length=500): - """Replace long observations with compact references.""" - if len(output) <= max_length: - return output - - reference_id = store_observation(output) - return f"[Previous observation elided. Full content stored at reference {reference_id}]" -``` - -This preserves information access while reducing token usage. - -## Context Budget Estimation - -### Token Counting Approximation - -For planning purposes, estimate tokens at approximately 4 characters per token for English text: - -``` -1000 words ≈ 7500 characters ≈ 1800-2000 tokens -``` - -This is a rough approximation; actual tokenization varies by model and content type. - -### Context Budget Allocation - -Allocate context budget across components: - -| Component | Typical Range | Notes | -|-----------|---------------|-------| -| System prompt | 500-2000 tokens | Stable across session | -| Tool definitions | 100-500 per tool | Grows with tool count | -| Retrieved documents | Variable | Often largest consumer | -| Message history | Variable | Grows with conversation | -| Tool outputs | Variable | Can dominate context | - -Monitor actual usage during development to establish baseline allocations. - -## Progressive Disclosure Implementation - -### Skill Activation Pattern - -```python -def activate_skill_context(skill_name, task_description): - """Load skill context when task matches skill description.""" - skill_metadata = load_all_skill_metadata() - - relevant_skills = [] - for skill in skill_metadata: - if skill_matches_task(skill, task_description): - relevant_skills.append(skill) - - # Load full content only for most relevant skills - for skill in relevant_skills[:MAX_CONCURRENT_SKILLS]: - skill_context = load_skill_content(skill) - inject_into_context(skill_context) -``` - -### Reference Loading Pattern - -```python -def get_reference(file_reference): - """Load reference file only when explicitly needed.""" - if not file_reference.is_loaded: - file_reference.content = read_file(file_reference.path) - file_reference.is_loaded = True - return file_reference.content -``` - -This pattern ensures files are loaded once and cached for the session. - diff --git a/.agents/skills/context-fundamentals/scripts/context_manager.py b/.agents/skills/context-fundamentals/scripts/context_manager.py deleted file mode 100644 index d3daa21ad..000000000 --- a/.agents/skills/context-fundamentals/scripts/context_manager.py +++ /dev/null @@ -1,533 +0,0 @@ -""" -Context Management Utilities for Agent Systems. - -Public API ----------- -Functions: - estimate_token_count — Rough token estimate from text (demo only). - estimate_message_tokens — Token estimate for a message list. - count_tokens_by_type — Break down token usage by context component. - truncate_context — Trim a context string to a token budget. - truncate_messages — Trim message history while preserving structure. - validate_context_structure — Detect empty, oversized, or duplicate sections. - build_agent_context — Assemble an optimized context dict from parts. - -Classes: - ContextBuilder — Priority-aware context assembly with budgets. - ProgressiveDisclosureManager — Lazy file loading with caching. - -Usage ------ -Import individual utilities or use `build_agent_context` as the high-level -entry point: - - from context_manager import build_agent_context - result = build_agent_context( - task="Refactor auth module", - system_prompt="You are a senior Python engineer.", - documents=["# Auth module docs ..."], - ) - print(result["usage_report"]) - -Run this module directly (`python context_manager.py`) for an interactive demo -that builds a sample context and prints the usage report. - -Note: Token estimation in this module uses a character-ratio heuristic. For -production systems, replace `estimate_token_count` with a real tokenizer -(tiktoken for OpenAI, Anthropic's token-counting API, etc.). -""" - -from __future__ import annotations - -import hashlib -from typing import Any, Dict, List, Optional - -__all__ = [ - "estimate_token_count", - "estimate_message_tokens", - "count_tokens_by_type", - "truncate_context", - "truncate_messages", - "validate_context_structure", - "build_agent_context", - "ContextBuilder", - "ProgressiveDisclosureManager", -] - - -# --------------------------------------------------------------------------- -# Token estimation -# --------------------------------------------------------------------------- - -def estimate_token_count(text: str) -> int: - """Return a rough token estimate for *text*. - - Uses the ~4 characters-per-token heuristic for English prose. - - Use when: quick budget checks during development or logging. Do NOT rely - on this for hard budget enforcement — code, URLs, and non-English text - tokenize at very different ratios (see module docstring). - - WARNING: Production systems must use a real tokenizer: - - OpenAI models → ``tiktoken`` - - Anthropic → Anthropic token-counting API - - Others → provider-specific tokenizer - """ - return len(text) // 4 - - -def estimate_message_tokens(messages: List[Dict[str, Any]]) -> int: - """Estimate total tokens across a list of chat messages. - - Use when: deciding whether to trigger compaction on message history. - Each message adds ~10 tokens of role/formatting overhead on top of - its content tokens. - """ - total = 0 - for msg in messages: - content = msg.get("content", "") - total += estimate_token_count(content) - total += 10 # Overhead for role/formatting - return total - - -def count_tokens_by_type(context: Dict[str, Any]) -> Dict[str, int]: - """Break down token usage by context component type. - - Use when: profiling where tokens are spent so the highest-cost - component can be targeted for compression first. - - Recognized keys in *context*: ``system``, ``tools`` (list), - ``documents`` (list), ``messages`` (list). - """ - breakdown: Dict[str, int] = { - "system_prompt": 0, - "tool_definitions": 0, - "retrieved_documents": 0, - "message_history": 0, - "tool_outputs": 0, - "other": 0, - } - - if "system" in context: - breakdown["system_prompt"] = estimate_token_count(context["system"]) - - if "tools" in context: - for tool in context["tools"]: - breakdown["tool_definitions"] += estimate_token_count(str(tool)) - - if "documents" in context: - for doc in context["documents"]: - breakdown["retrieved_documents"] += estimate_token_count(doc) - - if "messages" in context: - breakdown["message_history"] = estimate_message_tokens(context["messages"]) - - return breakdown - - -# --------------------------------------------------------------------------- -# Context Builder -# --------------------------------------------------------------------------- - -class ContextBuilder: - """Build context with priority-aware budget management. - - Use when: assembling context from multiple sources (system prompt, - retrieved documents, task description) and enforcing a hard token - ceiling. Higher-priority sections are kept first when the budget is - tight. - - Example:: - - builder = ContextBuilder(context_limit=80_000) - builder.add_section("system", prompt, priority=10) - builder.add_section("task", task_text, priority=9) - built = builder.build() - """ - - def __init__(self, context_limit: int = 100_000) -> None: - self.context_limit: int = context_limit - self.sections: Dict[str, Dict[str, Any]] = {} - self.order: List[str] = [] - - def add_section( - self, - name: str, - content: str, - priority: int = 0, - category: str = "other", - ) -> None: - """Add or replace a named section. - - Higher *priority* values are kept first when the budget is tight. - """ - if name not in self.sections: - self.order.append(name) - - self.sections[name] = { - "content": content, - "priority": priority, - "category": category, - "tokens": estimate_token_count(content), - } - - def build(self, max_tokens: Optional[int] = None) -> str: - """Assemble context string within the token budget. - - Sections are included in descending priority order until the - budget is exhausted. Returns the concatenated text of all - included sections. - """ - limit = max_tokens or self.context_limit - - sorted_sections = sorted( - self.order, - key=lambda n: self.sections[n]["priority"], - reverse=True, - ) - - context_parts: List[str] = [] - current_tokens = 0 - - for name in sorted_sections: - section = self.sections[name] - section_tokens = section["tokens"] - - if current_tokens + section_tokens <= limit: - context_parts.append(section["content"]) - current_tokens += section_tokens - - return "\n\n".join(context_parts) - - def get_usage_report(self) -> Dict[str, Any]: - """Return a summary of current context utilization. - - Use when: logging context composition during development or - deciding whether to trigger compaction. - """ - total = sum(s["tokens"] for s in self.sections.values()) - return { - "total_tokens": total, - "limit": self.context_limit, - "utilization": total / self.context_limit if self.context_limit else 0, - "by_section": { - name: s["tokens"] for name, s in self.sections.items() - }, - "status": self._get_status(total), - } - - def _get_status(self, total: int) -> str: - """Return 'critical', 'warning', or 'healthy' based on utilization.""" - ratio = total / self.context_limit if self.context_limit else 0 - if ratio > 0.9: - return "critical" - elif ratio > 0.7: - return "warning" - else: - return "healthy" - - -# --------------------------------------------------------------------------- -# Context Truncation -# --------------------------------------------------------------------------- - -def truncate_context( - context: str, - max_tokens: int, - preserve_start: bool = True, -) -> str: - """Truncate *context* to approximately *max_tokens*. - - Use when: a single large text block must fit a hard budget and - semantic chunking is not available. - - Set *preserve_start* to ``True`` (default) to keep the beginning - (system prompts, top-of-file content) or ``False`` to keep the end - (most recent information). - """ - tokens = context.split() - if len(tokens) <= max_tokens: - return context - - if preserve_start: - kept = tokens[:max_tokens] - else: - kept = tokens[-max_tokens:] - - return " ".join(kept) - - -def truncate_messages( - messages: List[Dict[str, Any]], - max_tokens: int, -) -> List[Dict[str, Any]]: - """Truncate message history while preserving structural integrity. - - Use when: message history exceeds budget and compaction has not yet - been implemented. Keeps: (1) the system prompt, (2) any existing - summary message, and (3) the most recent messages that fit. - - Strategy: - 1. Always keep the system prompt. - 2. Keep any existing summary message. - 3. Fill remaining budget with the most recent messages. - """ - system_prompt: Optional[Dict[str, Any]] = None - recent_messages: List[Dict[str, Any]] = [] - summary: Optional[Dict[str, Any]] = None - - for msg in messages: - if msg.get("role") == "system": - system_prompt = msg - elif msg.get("is_summary"): - summary = msg - else: - recent_messages.append(msg) - - tokens_for_system = ( - estimate_token_count(system_prompt["content"]) if system_prompt else 0 - ) - tokens_for_summary = ( - estimate_token_count(summary["content"]) if summary else 0 - ) - available = max_tokens - tokens_for_system - tokens_for_summary - - tokens_for_recent = estimate_message_tokens(recent_messages) - if tokens_for_recent > available: - truncated_recent: List[Dict[str, Any]] = [] - current_tokens = 0 - for msg in reversed(recent_messages): - msg_tokens = estimate_token_count(msg.get("content", "")) - if current_tokens + msg_tokens <= available: - truncated_recent.insert(0, msg) - current_tokens += msg_tokens - recent_messages = truncated_recent - - result: List[Dict[str, Any]] = [] - if system_prompt: - result.append(system_prompt) - if summary: - result.append(summary) - result.extend(recent_messages) - return result - - -# --------------------------------------------------------------------------- -# Context Validation -# --------------------------------------------------------------------------- - -def validate_context_structure(context: Dict[str, Any]) -> Dict[str, Any]: - """Validate a context dict for common structural issues. - - Use when: testing context assembly before sending to the model. - Checks for empty sections, excessive length, missing recommended - sections, and potential duplicate content. - - Returns a dict with ``valid`` (bool), ``issues`` (list), and - ``recommendations`` (list). - """ - issues: List[str] = [] - recommendations: List[str] = [] - - # Check for empty sections (skip list-type values like documents - # which are legitimately empty when no documents are retrieved) - for section, content in context.items(): - if content is None or (isinstance(content, str) and not content): - issues.append(f"Empty {section} section") - recommendations.append(f"Remove or populate {section}") - - # Check for excessive length - total_tokens = sum(estimate_token_count(str(c)) for c in context.values()) - if total_tokens > 80_000: - issues.append( - f"Context length ({total_tokens} tokens) exceeds recommended limit" - ) - recommendations.append("Consider context compaction or partitioning") - - # Check for missing sections - recommended_sections = ["system", "task"] - for section in recommended_sections: - if section not in context: - issues.append(f"Missing recommended section: {section}") - recommendations.append( - f"Add {section} section with relevant information" - ) - - # Check for duplicate content (first 1000 chars, hashed for consistency) - seen_content: set[str] = set() - for section, content in context.items(): - content_str = str(content)[:1000] - content_hash = hashlib.md5(content_str.encode()).hexdigest() - if content_hash in seen_content: - issues.append(f"Potential duplicate content in {section}") - seen_content.add(content_hash) - - return { - "valid": len(issues) == 0, - "issues": issues, - "recommendations": recommendations, - } - - -# --------------------------------------------------------------------------- -# Progressive Disclosure -# --------------------------------------------------------------------------- - -class ProgressiveDisclosureManager: - """Lazy loader for progressive disclosure of file-based context. - - Use when: an agent has access to many reference files but should - only pay the token cost for files that the current task actually - needs. Summaries are loaded first; detail files are loaded on demand - and cached for the session. - - Example:: - - pdm = ProgressiveDisclosureManager(base_dir="docs") - overview = pdm.load_summary("docs/api_summary.md") - # ... later, when detail is needed ... - detail = pdm.load_detail("docs/api/endpoints.md") - """ - - def __init__(self, base_dir: str = ".") -> None: - self.base_dir: str = base_dir - self.loaded_files: Dict[str, str] = {} - - def load_summary(self, summary_path: str) -> str: - """Load a summary file, returning cached content if available.""" - if summary_path in self.loaded_files: - return self.loaded_files[summary_path] - try: - with open(summary_path, "r") as f: - content = f.read() - self.loaded_files[summary_path] = content - return content - except FileNotFoundError: - return "" - - def load_detail(self, detail_path: str, force: bool = False) -> str: - """Load a detail file on demand. - - Set *force* to ``True`` to bypass the cache and re-read the file - (useful when the underlying file may have changed). - """ - if not force and detail_path in self.loaded_files: - return self.loaded_files[detail_path] - try: - with open(detail_path, "r") as f: - content = f.read() - self.loaded_files[detail_path] = content - return content - except FileNotFoundError: - return "" - - def get_contextual_info(self, reference: Dict[str, Any]) -> str: - """Return summary or detail based on the reference's flags. - - Use when: a reference dict carries both ``summary_path`` and - ``detail_path`` and the caller sets ``need_detail=True`` only - when full content is required. - """ - summary_path = reference.get("summary_path") - detail_path = reference.get("detail_path") - need_detail = reference.get("need_detail", False) - - if need_detail and detail_path: - return self.load_detail(detail_path) - elif summary_path: - return self.load_summary(summary_path) - else: - return "" - - -# --------------------------------------------------------------------------- -# High-level entry point -# --------------------------------------------------------------------------- - -def build_agent_context( - task: str, - system_prompt: str, - documents: Optional[List[str]] = None, - context_limit: int = 80_000, -) -> Dict[str, Any]: - """Build an optimized, validated context dict for an agent task. - - Use when: assembling context for a single inference call. Combines - system prompt, task description, and optional retrieved documents - into a priority-ordered context string, then validates the result. - - Returns a dict with keys ``context`` (str), ``usage_report`` (dict), - and ``validation`` (dict). - """ - builder = ContextBuilder(context_limit=context_limit) - - # System prompt — highest priority, persists across turns - builder.add_section("system", system_prompt, priority=10, category="system") - - # Task description — second priority - builder.add_section("task", task, priority=9, category="task") - - # Retrieved documents — loaded just-in-time - if documents: - for i, doc in enumerate(documents): - builder.add_section( - f"document_{i}", - doc, - priority=5, - category="retrieved", - ) - - context_dict: Dict[str, Any] = { - "system": system_prompt, - "task": task, - "documents": documents or [], - } - - validation = validate_context_structure(context_dict) - - return { - "context": builder.build(), - "usage_report": builder.get_usage_report(), - "validation": validation, - } - - -# --------------------------------------------------------------------------- -# Demo -# --------------------------------------------------------------------------- - -if __name__ == "__main__": - print("=== Context Manager Demo ===\n") - - sample_prompt = ( - "You are a senior Python engineer. Follow PEP 8, use type hints, " - "and write docstrings for all public functions." - ) - sample_task = "Refactor the authentication module to use OAuth 2.0." - sample_docs = [ - "# OAuth 2.0 Reference\nThe OAuth 2.0 authorization framework...", - "# Current Auth Module\ndef login(user, password): ...", - ] - - result = build_agent_context( - task=sample_task, - system_prompt=sample_prompt, - documents=sample_docs, - ) - - report = result["usage_report"] - print(f"Total tokens : {report['total_tokens']}") - print(f"Utilization : {report['utilization']:.1%}") - print(f"Status : {report['status']}") - print(f"\nBreakdown by section:") - for section, tokens in report["by_section"].items(): - print(f" {section:20s} : {tokens:,} tokens") - - validation = result["validation"] - if validation["valid"]: - print("\nValidation : PASSED") - else: - print(f"\nValidation : FAILED") - for issue in validation["issues"]: - print(f" - {issue}") diff --git a/.agents/skills/context-optimization/references/optimization_techniques.md b/.agents/skills/context-optimization/references/optimization_techniques.md deleted file mode 100644 index a5c60b307..000000000 --- a/.agents/skills/context-optimization/references/optimization_techniques.md +++ /dev/null @@ -1,272 +0,0 @@ -# Context Optimization Reference - -This document provides detailed technical reference for context optimization techniques and strategies. - -## Compaction Strategies - -### Summary-Based Compaction - -Summary-based compaction replaces verbose content with concise summaries while preserving key information. The approach works by identifying sections that can be compressed, generating summaries that capture essential points, and replacing full content with summaries. - -The effectiveness of compaction depends on what information is preserved. Critical decisions, user preferences, and current task state should never be compacted. Intermediate results and supporting evidence can be summarized more aggressively. Boilerplate, repeated information, and exploratory reasoning can often be removed entirely. - -### Token Budget Allocation - -Effective context budgeting requires understanding how different context components consume tokens and allocating budget strategically: - -| Component | Typical Range | Notes | -|-----------|---------------|-------| -| System prompt | 500-2000 tokens | Stable across session | -| Tool definitions | 100-500 per tool | Grows with tool count | -| Retrieved documents | Variable | Often largest consumer | -| Message history | Variable | Grows with conversation | -| Tool outputs | Variable | Can dominate context | - -### Compaction Thresholds - -Trigger compaction at appropriate thresholds to maintain performance: - -- Warning threshold at 70% of effective context limit -- Compaction trigger at 80% of effective context limit -- Aggressive compaction at 90% of effective context limit - -The exact thresholds depend on model behavior and task characteristics. Some models show graceful degradation while others exhibit sharp performance cliffs. - -## Observation Masking Patterns - -### Selective Masking - -Not all observations should be masked equally. Consider masking observations that have served their purpose and are no longer needed for active reasoning. Keep observations that are central to the current task. Keep observations from the most recent turn. Keep observations that may be referenced again. - -### Masking Implementation - -```python -def selective_mask(observations: List[Dict], current_task: Dict) -> List[Dict]: - """ - Selectively mask observations based on relevance. - - Returns observations with mask field indicating masked content. - """ - masked = [] - - for obs in observations: - relevance = calculate_relevance(obs, current_task) - - if relevance < 0.3 and obs["age"] > 3: - # Low relevance and old - mask - masked.append({ - **obs, - "masked": True, - "reference": store_for_reference(obs["content"]), - "summary": summarize_content(obs["content"]) - }) - else: - masked.append({ - **obs, - "masked": False - }) - - return masked -``` - -## KV-Cache Optimization - -### Prefix Stability - -KV-cache hit rates depend on prefix stability. Stable prefixes enable cache reuse across requests. Dynamic prefixes invalidate cache and force recomputation. - -Elements that should remain stable include system prompts, tool definitions, and frequently used templates. Elements that may vary include timestamps, session identifiers, and query-specific content. - -### Cache-Friendly Design - -Design prompts to maximize cache hit rates: - -1. Place stable content at the beginning -2. Use consistent formatting across requests -3. Avoid dynamic content in prompts when possible -4. Use placeholders for dynamic content - -```python -# Cache-unfriendly: Dynamic timestamp in prompt -system_prompt = f""" -Current time: {datetime.now().isoformat()} -You are a helpful assistant. -""" - -# Cache-friendly: Stable prompt with dynamic time as variable -system_prompt = """ -You are a helpful assistant. -Current time is provided separately when relevant. -""" -``` - -## Context Partitioning Strategies - -### Sub-Agent Isolation - -Partition work across sub-agents to prevent any single context from growing too large. Each sub-agent operates with a clean context focused on its subtask. - -### Partition Planning - -```python -def plan_partitioning(task: Dict, context_limit: int) -> Dict: - """ - Plan how to partition a task based on context limits. - - Returns partitioning strategy and subtask definitions. - """ - estimated_context = estimate_task_context(task) - - if estimated_context <= context_limit: - return { - "strategy": "single_agent", - "subtasks": [task] - } - - # Plan multi-agent approach - subtasks = decompose_task(task) - - return { - "strategy": "multi_agent", - "subtasks": subtasks, - "coordination": "hierarchical" - } -``` - -## Optimization Decision Framework - -### When to Optimize - -Consider context optimization when context utilization exceeds 70%, when response quality degrades as conversations extend, when costs increase due to long contexts, or when latency increases with conversation length. - -### What Optimization to Apply - -Choose optimization strategies based on context composition: - -If tool outputs dominate context, apply observation masking. If retrieved documents dominate context, apply summarization or partitioning. If message history dominates context, apply compaction with summarization. If multiple components contribute, combine strategies. - -### Evaluation of Optimization - -After applying optimization, evaluate effectiveness: - -- Measure token reduction achieved -- Measure quality preservation (output quality should not degrade) -- Measure latency improvement -- Measure cost reduction - -Iterate on optimization strategies based on evaluation results. - -## Common Pitfalls - -### Over-Aggressive Compaction - -Compacting too aggressively can remove critical information. Always preserve task goals, user preferences, and recent conversation context. Test compaction at increasing aggressiveness levels to find the optimal balance. - -### Masking Critical Observations - -Masking observations that are still needed can cause errors. Track observation usage and only mask content that is no longer referenced. Consider keeping references to masked content that could be retrieved if needed. - -### Ignoring Attention Distribution - -The lost-in-middle phenomenon means that information placement matters. Place critical information at attention-favored positions (beginning and end of context). Use explicit markers to highlight important content. - -### Premature Optimization - -Not all contexts require optimization. Adding optimization machinery has overhead. Optimize only when context limits actually constrain agent performance. - -## Monitoring and Alerting - -### Key Metrics - -Track these metrics to understand optimization needs: - -- Context token count over time -- Cache hit rates for repeated patterns -- Response quality metrics by context size -- Cost per conversation by context length -- Latency by context size - -### Alert Thresholds - -Set alerts for: - -- Context utilization above 80% -- Cache hit rate below 50% -- Quality score drop of more than 10% -- Cost increase above baseline - -## Integration Patterns - -### Integration with Agent Framework - -Integrate optimization into agent workflow: - -```python -class OptimizingAgent: - def __init__(self, context_limit: int = 80000): - self.context_limit = context_limit - self.optimizer = ContextOptimizer() - - def process(self, user_input: str, context: Dict) -> Dict: - # Check if optimization needed - if self.optimizer.should_compact(context): - context = self.optimizer.compact(context) - - # Process with optimized context - response = self._call_model(user_input, context) - - # Track metrics - self.optimizer.record_metrics(context, response) - - return response -``` - -### Integration with Memory Systems - -Connect optimization with memory systems: - -```python -class MemoryAwareOptimizer: - def __init__(self, memory_system, context_limit: int): - self.memory = memory_system - self.limit = context_limit - - def optimize_context(self, current_context: Dict, task: str) -> Dict: - # Check if information is in memory - relevant_memories = self.memory.retrieve(task) - - # Move information to memory if not needed in context - for mem in relevant_memories: - if mem["importance"] < threshold: - current_context = remove_from_context(current_context, mem) - # Keep reference that memory can be retrieved - - return current_context -``` - -## Performance Benchmarks - -### Compaction Performance - -Compaction should reduce token count while preserving quality. Target: - -- 50-70% token reduction for aggressive compaction -- Less than 5% quality degradation from compaction -- Less than 10% latency increase from compaction overhead - -### Masking Performance - -Observation masking should reduce token count significantly: - -- 60-80% reduction in masked observations -- Less than 2% quality impact from masking -- Near-zero latency overhead - -### Cache Performance - -KV-cache optimization should improve cost and latency: - -- 70%+ cache hit rate for stable workloads -- 50%+ cost reduction from cache hits -- 40%+ latency reduction from cache hits - diff --git a/.agents/skills/context-optimization/scripts/compaction.py b/.agents/skills/context-optimization/scripts/compaction.py deleted file mode 100644 index b38c0aaa3..000000000 --- a/.agents/skills/context-optimization/scripts/compaction.py +++ /dev/null @@ -1,562 +0,0 @@ -""" -Context Optimization Utilities — compaction, masking, budgeting, and cache optimization. - -Public API ----------- -Functions: - estimate_token_count(text) -> int - estimate_message_tokens(messages) -> int - categorize_messages(messages) -> dict - summarize_content(content, category, max_length) -> str - design_stable_prompt(template, dynamic_values) -> str - calculate_cache_metrics(requests, cache) -> dict - -Classes: - ObservationStore — Store and mask verbose tool outputs with retrievable references. - ContextBudget — Token budget allocation and optimization trigger detection. - -PRODUCTION NOTES: -- Token estimation uses simplified heuristics (~4 chars/token for English). - Production systems should use model-specific tokenizers: - - OpenAI: tiktoken library - - Anthropic: anthropic tokenizer - - Local models: HuggingFace tokenizers - -- Summarization functions use simple heuristics for demonstration. - Production systems should use: - - LLM-based summarization for high-quality compression - - Domain-specific summarization models - - Schema-based summarization for structured outputs - -- Cache metrics are illustrative. Production systems should integrate - with actual inference infrastructure metrics. -""" - -from typing import List, Dict, Optional, Tuple -import hashlib -import re -import time - -__all__ = [ - "estimate_token_count", - "estimate_message_tokens", - "categorize_messages", - "summarize_content", - "summarize_tool_output", - "summarize_conversation", - "summarize_document", - "summarize_general", - "ObservationStore", - "ContextBudget", - "design_stable_prompt", - "calculate_cache_metrics", - "generate_cache_recommendations", -] - - -# --------------------------------------------------------------------------- -# Token estimation -# --------------------------------------------------------------------------- - -def estimate_token_count(text: str) -> int: - """ - Estimate token count for text. - - Use when: a quick token budget check is needed and a model-specific - tokenizer is unavailable or too slow for the hot path. - - Uses approximation: ~4 characters per token for English. - - WARNING: This is a rough estimate. Actual tokenization varies by: - - Model (GPT-5.2, Claude 4.5, Gemini 3 have different tokenizers) - - Content type (code typically has higher token density) - - Language (non-English may have 2-3x higher token/char ratio) - - Production usage:: - - import tiktoken - enc = tiktoken.encoding_for_model("gpt-4") - token_count = len(enc.encode(text)) - """ - return len(text) // 4 - - -def estimate_message_tokens(messages: List[Dict[str, str]]) -> int: - """ - Estimate token count for a message list. - - Use when: checking whether the current conversation is approaching - the context budget threshold before deciding to compact or mask. - """ - total = 0 - for msg in messages: - content = msg.get("content", "") - total += estimate_token_count(content) - # Add overhead for role/formatting - total += 10 - return total - - -# --------------------------------------------------------------------------- -# Compaction functions -# --------------------------------------------------------------------------- - -def categorize_messages(messages: List[Dict]) -> Dict[str, List[Dict]]: - """ - Categorize messages for selective compaction. - - Use when: preparing to compact context and needing to apply different - summarization strategies per category (tool outputs first, then old - conversation turns, then retrieved documents — never the system prompt). - - Returns a dict mapping category name to list of messages. - """ - categories: Dict[str, List[Dict]] = { - "system_prompt": [], - "tool_definition": [], - "tool_output": [], - "conversation": [], - "retrieved_document": [], - "other": [], - } - - for msg in messages: - role = msg.get("role", "user") - content = msg.get("content", "") - - if role == "system": - categories["system_prompt"].append({**msg, "category": "system_prompt"}) - elif "tool_use" in msg.get("type", ""): - categories["tool_output"].append({**msg, "category": "tool_output"}) - elif role == "user": - categories["conversation"].append({**msg, "category": "conversation"}) - elif "retrieved" in msg.get("tags", []): - categories["retrieved_document"].append({**msg, "category": "retrieved_document"}) - else: - categories["other"].append({**msg, "category": "other"}) - - return categories - - -def summarize_content(content: str, category: str, max_length: int = 500) -> str: - """ - Summarize content for compaction, dispatching by category. - - Use when: compacting context and needing category-aware summarization - (tool outputs get metric extraction, conversations get decision - extraction, documents get lead-paragraph extraction). - """ - if category == "tool_output": - return summarize_tool_output(content, max_length) - elif category == "conversation": - return summarize_conversation(content, max_length) - elif category == "retrieved_document": - return summarize_document(content, max_length) - else: - return summarize_general(content, max_length) - - -def summarize_tool_output(content: str, max_length: int = 500) -> str: - """ - Summarize tool output by extracting metrics and key findings. - - Use when: a tool output has served its immediate purpose and needs - to be compacted while preserving actionable data points. - """ - # Look for metrics (numbers with context) - metrics = re.findall(r'(\w+):\s*([\d.,]+)', content) - - # Look for key findings (lines with important keywords) - keywords = ["result", "found", "total", "success", "error", "value"] - findings = [] - for line in content.split('\n'): - if any(kw in line.lower() for kw in keywords): - findings.append(line.strip()) - - summary_parts = [] - if metrics: - summary_parts.append(f"Metrics: {', '.join([f'{k}={v}' for k, v in metrics])}") - if findings: - summary_parts.append("Key findings: " + "; ".join(findings[:3])) - - result = " | ".join(summary_parts) if summary_parts else "[Tool output summarized]" - return result[:max_length] - - -def summarize_conversation(content: str, max_length: int = 500) -> str: - """ - Summarize conversational content by extracting decisions and questions. - - Use when: older conversation turns need compaction and the key - decisions/commitments must survive while filler is removed. - """ - decisions = re.findall(r'(?i)(?:decided|decision|chose|chosen)[:\s]+([^.]+)', content) - questions = re.findall(r'(?:\?|question)[:\s]+([^.]+)', content) - - summary_parts = [] - if decisions: - decision_texts = [d.strip() for d in decisions[:5]] - summary_parts.append(f"Decisions: {'; '.join(decision_texts)}") - if questions: - question_texts = [q.strip() for q in questions[:3]] - summary_parts.append(f"Open questions: {'; '.join(question_texts)}") - - if not summary_parts: - # Fallback: extract the first few substantive sentences - sentences = [s.strip() for s in content.split('.') if len(s.strip()) > 20] - if sentences: - summary_parts.append('. '.join(sentences[:3]) + '.') - - result = " | ".join(summary_parts) if summary_parts else "[Conversation summarized]" - return result[:max_length] - - -def summarize_document(content: str, max_length: int = 500) -> str: - """ - Summarize document content using lead-paragraph extraction. - - Use when: a retrieved document has been consumed for reasoning and - only a brief reference needs to remain in context. - """ - paragraphs = content.split('\n\n') - if paragraphs: - first_para = paragraphs[0].strip() - sentences = first_para.split('. ') - if len(sentences) > 2: - first_para = '. '.join(sentences[:2]) + '.' - return first_para[:max_length] - return "[Document summarized]" - - -def summarize_general(content: str, max_length: int = 500) -> str: - """ - General-purpose summarization via truncation. - - Use when: content does not fit a specific category and a simple - truncation with ellipsis is acceptable. - """ - return content[:max_length] + "..." if len(content) > max_length else content - - -# --------------------------------------------------------------------------- -# Observation masking -# --------------------------------------------------------------------------- - -class ObservationStore: - """ - Store and mask verbose tool outputs with retrievable references. - - Use when: tool outputs dominate context (>50% of tokens) and older - observations have already served their reasoning purpose. Stores the - full content externally and replaces it with a compact reference - containing a key-point summary. - - Example:: - - store = ObservationStore(max_size=500) - masked, ref_id = store.mask(long_tool_output, max_length=200) - # masked: "[Obs:a1b2c3d4 elided. Key: ... Full content retrievable.]" - # Later retrieval: - original = store.retrieve(ref_id) - """ - - def __init__(self, max_size: int = 1000) -> None: - self.observations: Dict[str, Dict] = {} - self.order: List[str] = [] - self.max_size = max_size - - def store(self, content: str, metadata: Optional[Dict] = None) -> str: - """Store observation and return reference ID.""" - ref_id = self._generate_ref_id(content) - - self.observations[ref_id] = { - "content": content, - "metadata": metadata or {}, - "stored_at": time.time(), - "last_accessed": time.time(), - } - self.order.append(ref_id) - - # Evict oldest if over limit - if len(self.order) > self.max_size: - oldest = self.order.pop(0) - del self.observations[oldest] - - return ref_id - - def retrieve(self, ref_id: str) -> Optional[str]: - """Retrieve observation by reference ID.""" - if ref_id in self.observations: - self.observations[ref_id]["last_accessed"] = time.time() - return self.observations[ref_id]["content"] - return None - - def mask(self, content: str, max_length: int = 200) -> Tuple[str, Optional[str]]: - """ - Mask observation if longer than max_length. - - Use when: deciding per-observation whether to keep inline or - replace with a compact reference. Returns (masked_content, ref_id) - where ref_id is None if the content was short enough to keep. - """ - if len(content) <= max_length: - return content, None - - ref_id = self.store(content) - key_point = self._extract_key_point(content) - masked = f"[Obs:{ref_id} elided. Key: {key_point}. Full content retrievable.]" - return masked, ref_id - - def _generate_ref_id(self, content: str) -> str: - """Generate unique reference ID.""" - hash_input = f"{content[:100]}{time.time()}" - return hashlib.md5(hash_input.encode()).hexdigest()[:8] - - def _extract_key_point(self, content: str) -> str: - """Extract key point from observation.""" - lines = [line for line in content.split('\n') if len(line) > 20] - if lines: - return lines[0][:50] + "..." - sentences = content.split('. ') - if sentences: - return sentences[0][:50] + "..." - return content[:50] + "..." - - -# --------------------------------------------------------------------------- -# Context budget management -# --------------------------------------------------------------------------- - -class ContextBudget: - """ - Token budget allocation and optimization trigger detection. - - Use when: building an agent loop that needs to monitor context usage - across categories and trigger compaction/masking at the right thresholds - rather than waiting until the window overflows. - - Example:: - - budget = ContextBudget(total_limit=128_000) - budget.allocate("system_prompt", 1500) - budget.allocate("tool_definitions", 3000) - # ... after each agent turn: - should_act, reasons = budget.should_optimize(current_usage) - if should_act: - # apply masking or compaction based on reasons - pass - """ - - def __init__(self, total_limit: int) -> None: - self.total_limit = total_limit - self.allocated: Dict[str, int] = { - "system_prompt": 0, - "tool_definitions": 0, - "retrieved_docs": 0, - "message_history": 0, - "tool_outputs": 0, - "other": 0, - } - self.reserved = 5000 # Reserved buffer - self.reservation_limit = total_limit - self.reserved - - def allocate(self, category: str, amount: int) -> bool: - """ - Allocate budget to category. Returns True on success, False if - the allocation would exceed the reservation limit. - """ - if category not in self.allocated: - category = "other" - - current = sum(self.allocated.values()) - proposed = current + amount - - if proposed > self.reservation_limit: - return False - - self.allocated[category] += amount - return True - - def remaining(self) -> int: - """Get remaining unallocated budget.""" - current = sum(self.allocated.values()) - return self.reservation_limit - current - - def get_usage(self) -> Dict[str, object]: - """ - Get current usage breakdown. - - Use when: logging or displaying context budget state for - monitoring dashboards or debug output. - """ - total = sum(self.allocated.values()) - return { - "total_used": total, - "total_limit": self.total_limit, - "remaining": self.remaining(), - "by_category": dict(self.allocated), - "utilization_ratio": total / self.total_limit, - } - - def should_optimize( - self, current_usage: int, metrics: Optional[Dict[str, float]] = None - ) -> Tuple[bool, List[Tuple[str, object]]]: - """ - Determine if optimization should trigger. - - Use when: called at the end of each agent loop iteration to - decide whether to apply compaction, masking, or both before - the next model call. - - Returns (should_optimize, list_of_reasons). - """ - reasons: List[Tuple[str, object]] = [] - - # Check utilization - utilization = current_usage / self.total_limit - if utilization > 0.8: - reasons.append(("high_utilization", utilization)) - - # Check degradation metrics if provided - if metrics: - if metrics.get("attention_degradation", 0) > 0.3: - reasons.append(("attention_degradation", True)) - if metrics.get("quality_score", 1.0) < 0.8: - reasons.append(("quality_degradation", True)) - - return len(reasons) > 0, reasons - - -# --------------------------------------------------------------------------- -# Cache optimization -# --------------------------------------------------------------------------- - -def design_stable_prompt(template: str, dynamic_values: Optional[Dict] = None) -> str: - """ - Stabilize a prompt template for maximum KV-cache hit rate. - - Use when: constructing system prompts or few-shot prefixes that will - be reused across many requests. Replaces dynamic content (timestamps, - session IDs, counters) with stable placeholders so the prefix hash - remains constant. - """ - result = template - - # Replace timestamps - date_pattern = r'\d{4}-\d{2}-\d{2}' - result = re.sub(date_pattern, '[DATE_STABLE]', result) - - # Replace session IDs - session_pattern = r'Session \d+' - result = re.sub(session_pattern, 'Session [STABLE]', result) - - # Replace counters - counter_pattern = r'\d+/\d+' - result = re.sub(counter_pattern, '[COUNTER_STABLE]', result) - - return result - - -def calculate_cache_metrics( - requests: List[Dict], cache: Dict[str, Dict] -) -> Dict[str, object]: - """ - Calculate KV-cache hit metrics for a request sequence. - - Use when: evaluating whether prompt restructuring improved cache - utilization. Feed in the request log and current cache state to - get hit/miss rates and actionable recommendations. - """ - hits = 0 - misses = 0 - - for req in requests: - prefix = req.get("prefix_hash", "") - token_count = req.get("token_count", 0) - - if prefix in cache: - hits += token_count * cache[prefix].get("hit_ratio", 0) - else: - misses += token_count - - total = hits + misses - - return { - "hit_rate": hits / total if total > 0 else 0, - "cache_hits": hits, - "cache_misses": misses, - "recommendations": generate_cache_recommendations(hits, misses), - } - - -def generate_cache_recommendations(hits: int, misses: int) -> List[str]: - """ - Generate recommendations for cache optimization based on hit/miss ratio. - - Use when: cache metrics indicate sub-optimal hit rates and concrete - next steps are needed. - """ - recommendations: List[str] = [] - - hit_rate = hits / (hits + misses) if (hits + misses) > 0 else 0 - - if hit_rate < 0.5: - recommendations.append("Consider stabilizing system prompts") - recommendations.append("Reduce variation in request prefixes") - - if hit_rate < 0.8: - recommendations.append("Group similar requests together") - recommendations.append("Use consistent formatting across requests") - - return recommendations - - -# --------------------------------------------------------------------------- -# Demo / smoke test -# --------------------------------------------------------------------------- - -if __name__ == "__main__": - print("=== Context Optimization Utilities — Demo ===\n") - - # 1. Token estimation - sample_text = "The quick brown fox jumps over the lazy dog. " * 20 - tokens = estimate_token_count(sample_text) - print(f"1. Token estimate for {len(sample_text)}-char text: ~{tokens} tokens\n") - - # 2. Observation masking - store = ObservationStore(max_size=100) - long_output = ( - "Result: 42 items found\n" - "Total processing time: 3.2s\n" - "Details:\n" + "\n".join([f" Item {i}: value={i*10}" for i in range(20)]) - ) - masked, ref_id = store.mask(long_output, max_length=100) - print(f"2. Masked observation:\n {masked}") - print(f" Ref ID: {ref_id}") - retrieved = store.retrieve(ref_id) - print(f" Retrievable: {retrieved is not None}\n") - - # 3. Context budget - budget = ContextBudget(total_limit=128_000) - budget.allocate("system_prompt", 1500) - budget.allocate("tool_definitions", 3000) - budget.allocate("message_history", 95_000) - usage = budget.get_usage() - print(f"3. Budget utilization: {usage['utilization_ratio']:.1%}") - should_opt, reasons = budget.should_optimize( - current_usage=int(128_000 * 0.85) - ) - print(f" Should optimize: {should_opt}, reasons: {reasons}\n") - - # 4. Cache-stable prompt - raw_prompt = "Session 42 started on 2025-12-20. Progress: 3/10 tasks." - stable = design_stable_prompt(raw_prompt) - print(f"4. Original prompt: {raw_prompt}") - print(f" Stabilized: {stable}\n") - - # 5. Summarization - tool_out = "count: 150\nstatus: success\nFound 3 errors in module A." - summary = summarize_content(tool_out, "tool_output", max_length=200) - print(f"5. Tool output summary: {summary}\n") - - print("=== Demo complete ===") diff --git a/.agents/skills/create-spec/SKILL.md b/.agents/skills/create-spec/SKILL.md deleted file mode 100644 index 392e3325e..000000000 --- a/.agents/skills/create-spec/SKILL.md +++ /dev/null @@ -1,249 +0,0 @@ ---- -name: create-spec -description: Create a detailed execution plan/spec/prd for implementing features or refactors in a codebase by leveraging existing research in the codebase. -metadata: - provider: atomic ---- - -You are tasked with creating a spec for implementing a new feature or system change in the codebase by leveraging existing research in the **$ARGUMENTS** path. If no research path is specified, use the entire `research/` directory. IMPORTANT: Research documents are located in the `research/` directory — do NOT look in the `specs/` directory for research. Follow the template below to produce a comprehensive specification as output in the `specs/` folder using the findings from RELEVANT research documents found in `research/`. The spec file MUST be named using the format `YYYY-MM-DD-topic.md` (e.g., `specs/2026-03-26-my-feature.md`), where the date is the current date and the topic is a kebab-case summary. Tip: It's good practice to use the `codebase-research-locator` and `codebase-research-analyzer` agents to help you find and analyze the research documents in the `research/` directory. It is also HIGHLY recommended to cite relevant research throughout the spec for additional context. - - - -- Please use your AskUserQuestion tool to provide a rich interface to ask the user for their input on a question. -- Please DO NOT implement anything in this stage, just create the comprehensive spec as described below. -- When writing the spec, DO NOT include information about concrete dates/timelines (e.g. # minutes, hours, days, weeks, etc.) and favor explicit phases (e.g. Phase 1, Phase 2, etc.). -- Once the spec is generated ask questions one at a time OR in logical groups: - - Refer to section "## 9. Open Questions / Unresolved Issues", go through each question one by one, and use **contrastive clarification** (presenting 2-3 specific options with concrete tradeoffs) rather than open-ended questions. This means presenting interpretations like "(A) Option X — tradeoff Y" and "(B) Option Z — tradeoff W" instead of asking "what do you think about X?". Update the spec with the user's answers as you walk through the questions. - - Interview the user relentlessly about every aspect of this plan/spec until you reach a shared understanding with them. Walk down each branch of the design tree, resolving dependencies between decisions one-by-one. For each question, provide your recommended answer (i.e., **contrastive clarification**). - - If a question can be answered by exploring the codebase, explore the codebase instead and confirm with the user that this is their inferred intent. -- Finally, once the spec is generated and after open questions are answered, provide an executive summary of the spec to the user including the path to the generated spec document in the `specs/` directory. - - Encourage the user to review the spec for best results and provide feedback or ask any follow-up questions they may have. - - - -# [Project Name] Technical Design Document / RFC - -| Document Metadata | Details | -| ---------------------- | ------------------------------------------------------------------------------ | -| Author(s) | !`git config user.name` | -| Status | Draft (WIP) / In Review (RFC) / Approved / Implemented / Deprecated / Rejected | -| Team / Owner | | -| Created / Last Updated | | - -## 1. Executive Summary - -_Instruction: A "TL;DR" of the document. Assume the reader is a VP or an engineer from another team who has 2 minutes. Summarize the Context (Problem), the Solution (Proposal), and the Impact (Value). Keep it under 200 words._ - -> **Example:** This RFC proposes replacing our current nightly batch billing system with an event-driven architecture using Kafka and AWS Lambda. Currently, billing delays cause a 5% increase in customer support tickets. The proposed solution will enable real-time invoicing, reducing billing latency from 24 hours to <5 minutes. - -## 2. Context and Motivation - -_Instruction: Why are we doing this? Why now? Link to the Product Requirement Document (PRD)._ - -### 2.1 Current State - -_Instruction: Describe the existing architecture. Use a "Context Diagram" if possible. Be honest about the flaws._ - -- **Architecture:** Currently, Service A communicates with Service B via a shared SQL database. -- **Limitations:** This creates a tight coupling; when Service A locks the table, Service B times out. - -### 2.2 The Problem - -_Instruction: What is the specific pain point?_ - -- **User Impact:** Customers cannot download receipts during the nightly batch window. -- **Business Impact:** We are losing $X/month in churn due to billing errors. -- **Technical Debt:** The current codebase is untestable and has 0% unit test coverage. - -## 3. Goals and Non-Goals - -_Instruction: This is the contract Definition of Success. Be precise._ - -### 3.1 Functional Goals - -- [ ] Users must be able to export data in CSV format. -- [ ] System must support multi-tenant data isolation. - -### 3.2 Non-Goals (Out of Scope) - -_Instruction: Explicitly state what you are NOT doing. This prevents scope creep._ - -- [ ] We will NOT support PDF export in this version (CSV only). -- [ ] We will NOT migrate data older than 3 years. -- [ ] We will NOT build a custom UI (API only). - -## 4. Proposed Solution (High-Level Design) - -_Instruction: The "Big Picture." Diagrams are mandatory here._ - -### 4.1 System Architecture Diagram - -_Instruction: Insert a C4 System Context or Container diagram. Show the "Black Boxes."_ - -- (Place Diagram Here - e.g., Mermaid diagram) - -For example, - -```mermaid -%%{init: {'theme':'base', 'themeVariables': { 'primaryColor':'#f8f9fa','primaryTextColor':'#2c3e50','primaryBorderColor':'#4a5568','lineColor':'#4a90e2','secondaryColor':'#ffffff','tertiaryColor':'#e9ecef','background':'#f5f7fa','mainBkg':'#f8f9fa','nodeBorder':'#4a5568','clusterBkg':'#ffffff','clusterBorder':'#cbd5e0','edgeLabelBackground':'#ffffff'}}}%% - -flowchart TB - %% --------------------------------------------------------- - %% CLEAN ENTERPRISE DESIGN - %% Professional • Trustworthy • Corporate Standards - %% --------------------------------------------------------- - - %% STYLE DEFINITIONS - classDef person fill:#5a67d8,stroke:#4c51bf,stroke-width:3px,color:#ffffff,font-weight:600,font-size:14px - - classDef systemCore fill:#4a90e2,stroke:#357abd,stroke-width:2.5px,color:#ffffff,font-weight:600,font-size:14px - - classDef systemSupport fill:#667eea,stroke:#5a67d8,stroke-width:2.5px,color:#ffffff,font-weight:600,font-size:13px - - classDef database fill:#48bb78,stroke:#38a169,stroke-width:2.5px,color:#ffffff,font-weight:600,font-size:13px - - classDef external fill:#718096,stroke:#4a5568,stroke-width:2.5px,color:#ffffff,font-weight:600,font-size:13px,stroke-dasharray:6 3 - - %% NODES - CLEAN ENTERPRISE HIERARCHY - - User(("◉
User
")):::person - - subgraph SystemBoundary["◆ Primary System Boundary"] - direction TB - - LoadBalancer{{"Load Balancer
NGINX
Layer 7 Proxy"}}:::systemCore - - API["API Application
Go • Gin Framework
REST Endpoints"]:::systemCore - - Worker(["Background Worker
Go Runtime
Async Processing"]):::systemSupport - - Cache[("◆
Cache Layer
Redis
In-Memory")]:::database - - PrimaryDB[("●
Primary Database
PostgreSQL
Persistent Storage")]:::database - end - - ExternalAPI{{"External API
Third Party
HTTP/REST"}}:::external - - %% RELATIONSHIPS - CLEAN FLOW - - User -->|"1. HTTPS Request
TLS 1.3"| LoadBalancer - LoadBalancer -->|"2. Proxy Pass
Round Robin"| API - - API <-->|"3. Cache
Read/Write"| Cache - API -->|"4. Persist Data
Transactional"| PrimaryDB - API -.->|"5. Enqueue Event
Async"| Worker - - Worker -->|"6. Process Job
Execution"| PrimaryDB - Worker -.->|"7. HTTP Call
Webhooks"| ExternalAPI - - %% STYLE BOUNDARY - style SystemBoundary fill:#ffffff,stroke:#cbd5e0,stroke-width:2px,color:#2d3748,stroke-dasharray:8 4,font-weight:600,font-size:12px -``` - -### 4.2 Architectural Pattern - -_Instruction: Name the pattern (e.g., "Event Sourcing", "BFF - Backend for Frontend")._ - -- We are adopting a Publisher-Subscriber pattern where the Order Service publishes `OrderCreated` events, and the Billing Service consumes them asynchronously. - -### 4.3 Key Components - -| Component | Responsibility | Technology Stack | Justification | -| ----------------- | --------------------------- | ----------------- | -------------------------------------------- | -| Ingestion Service | Validates incoming webhooks | Go, Gin Framework | High concurrency performance needed. | -| Event Bus | Decouples services | Kafka | Durable log, replay capability. | -| Projections DB | Read-optimized views | MongoDB | Flexible schema for diverse receipt formats. | - -## 5. Detailed Design - -_Instruction: The "Meat" of the document. Sufficient detail for an engineer to start coding._ - -### 5.1 API Interfaces - -_Instruction: Define the contract. Use OpenAPI/Swagger snippets or Protocol Buffer definitions._ - -**Endpoint:** `POST /api/v1/invoices` - -- **Auth:** Bearer Token (Scope: `invoice:write`) -- **Idempotency:** Required header `X-Idempotency-Key` -- **Request Body:** - -```json -{ "user_id": "uuid", "amount": 100.0, "currency": "USD" } -``` - -### 5.2 Data Model / Schema - -_Instruction: Provide ERDs (Entity Relationship Diagrams) or JSON schemas. Discuss normalization vs. denormalization._ - -**Table:** `invoices` (PostgreSQL) - -| Column | Type | Constraints | Description | -| --------- | ---- | ----------------- | --------------------- | -| `id` | UUID | PK | | -| `user_id` | UUID | FK -> Users | Partition Key | -| `status` | ENUM | 'PENDING', 'PAID' | Indexed for filtering | - -### 5.3 Algorithms and State Management - -_Instruction: Describe complex logic, state machines, or consistency models._ - -- **State Machine:** An invoice moves from `DRAFT` -> `LOCKED` -> `PROCESSING` -> `PAID`. -- **Concurrency:** We use Optimistic Locking on the `version` column to prevent double-payments. - -## 6. Alternatives Considered - -_Instruction: Prove you thought about trade-offs. Why is your solution better than the others?_ - -| Option | Pros | Cons | Reason for Rejection | -| -------------------------------- | ---------------------------------- | ----------------------------------------- | ----------------------------------------------------------------------------- | -| Option A: Synchronous HTTP Calls | Simple to implement, Easy to debug | Tight coupling, cascading failures | Latency requirements (200ms) make blocking calls risky. | -| Option B: RabbitMQ | Lightweight, Built-in routing | Less durable than Kafka, harder to replay | We need message replay for auditing (Compliance requirement). | -| Option C: Kafka (Selected) | High throughput, Replayability | Operational complexity | **Selected:** The need for auditability/replay outweighs the complexity cost. | - -## 7. Cross-Cutting Concerns - -### 7.1 Security and Privacy - -- **Authentication:** Services authenticate via mTLS. -- **Authorization:** Policy enforcement point at the API Gateway (OPA - Open Policy Agent). -- **Data Protection:** PII (Names, Emails) is encrypted at rest using AES-256. -- **Threat Model:** Primary threat is compromised API Key; remediation is rapid rotation and rate limiting. - -### 7.2 Observability Strategy - -- **Metrics:** We will track `invoice_creation_latency` (Histogram) and `payment_failure_count` (Counter). -- **Tracing:** All services propagate `X-Trace-ID` headers (OpenTelemetry). -- **Alerting:** PagerDuty triggers if `5xx` error rate > 1% for 5 minutes. - -### 7.3 Scalability and Capacity Planning - -- **Traffic Estimates:** 1M transactions/day = ~12 TPS avg / 100 TPS peak. -- **Storage Growth:** 1KB per record \* 1M = 1GB/day. -- **Bottleneck:** The PostgreSQL Write node is the bottleneck. We will implement Read Replicas to offload traffic. - -## 8. Migration, Rollout, and Testing - -### 8.1 Deployment Strategy - -- [ ] Phase 1: Deploy services in "Shadow Mode" (process traffic but do not email users). -- [ ] Phase 2: Enable Feature Flag `new-billing-engine` for 1% of internal users. -- [ ] Phase 3: Ramp to 100%. - -### 8.2 Data Migration Plan - -- **Backfill:** We will run a script to migrate the last 90 days of invoices from the legacy SQL server. -- **Verification:** A "Reconciliation Job" will run nightly to compare Legacy vs. New totals. - -### 8.3 Test Plan - -- **Unit Tests:** -- **Integration Tests:** -- **End-to-End Tests:** - -## 9. Open Questions / Unresolved Issues - -_Instruction: List known unknowns. These must be resolved before the doc is marked "Approved"._ - -- [ ] Will the Legal team approve the 3rd party library for PDF generation? -- [ ] Does the current VPC peering allow connection to the legacy mainframe? diff --git a/.agents/skills/docx/LICENSE.txt b/.agents/skills/docx/LICENSE.txt deleted file mode 100644 index c55ab4222..000000000 --- a/.agents/skills/docx/LICENSE.txt +++ /dev/null @@ -1,30 +0,0 @@ -© 2025 Anthropic, PBC. All rights reserved. - -LICENSE: Use of these materials (including all code, prompts, assets, files, -and other components of this Skill) is governed by your agreement with -Anthropic regarding use of Anthropic's services. If no separate agreement -exists, use is governed by Anthropic's Consumer Terms of Service or -Commercial Terms of Service, as applicable: -https://www.anthropic.com/legal/consumer-terms -https://www.anthropic.com/legal/commercial-terms -Your applicable agreement is referred to as the "Agreement." "Services" are -as defined in the Agreement. - -ADDITIONAL RESTRICTIONS: Notwithstanding anything in the Agreement to the -contrary, users may not: - -- Extract these materials from the Services or retain copies of these - materials outside the Services -- Reproduce or copy these materials, except for temporary copies created - automatically during authorized use of the Services -- Create derivative works based on these materials -- Distribute, sublicense, or transfer these materials to any third party -- Make, offer to sell, sell, or import any inventions embodied in these - materials -- Reverse engineer, decompile, or disassemble these materials - -The receipt, viewing, or possession of these materials does not convey or -imply any license or right beyond those expressly granted above. - -Anthropic retains all right, title, and interest in these materials, -including all copyrights, patents, and other intellectual property rights. diff --git a/.agents/skills/docx/SKILL.md b/.agents/skills/docx/SKILL.md deleted file mode 100644 index 1fe9417f3..000000000 --- a/.agents/skills/docx/SKILL.md +++ /dev/null @@ -1,592 +0,0 @@ ---- -name: docx -description: "Use this skill whenever the user wants to create, read, edit, or manipulate Word documents (.docx files). Triggers include: any mention of 'Word doc', 'word document', '.docx', or requests to produce professional documents with formatting like tables of contents, headings, page numbers, or letterheads. Also use when extracting or reorganizing content from .docx files, inserting or replacing images in documents, performing find-and-replace in Word files, working with tracked changes or comments, or converting content into a polished Word document. If the user asks for a 'report', 'memo', 'letter', 'template', or similar deliverable as a Word or .docx file, use this skill. Do NOT use for PDFs, spreadsheets, Google Docs, or general coding tasks unrelated to document generation." -license: Proprietary. LICENSE.txt has complete terms -metadata: - provider: atomic ---- - -# DOCX creation, editing, and analysis - -## Overview - -A .docx file is a ZIP archive containing XML files. - -## Quick Reference - -| Task | Approach | -|------|----------| -| Read/analyze content | `pandoc` or unpack for raw XML | -| Create new document | Use `docx-js` - see Creating New Documents below | -| Edit existing document | Unpack → edit XML → repack - see Editing Existing Documents below | - -### Converting .doc to .docx - -Legacy `.doc` files must be converted before editing: - -```bash -python scripts/office/soffice.py --headless --convert-to docx document.doc -``` - -### Reading Content - -```bash -# Text extraction with tracked changes -pandoc --track-changes=all document.docx -o output.md - -# Raw XML access -python scripts/office/unpack.py document.docx unpacked/ -``` - -### Converting to Images - -```bash -python scripts/office/soffice.py --headless --convert-to pdf document.docx -pdftoppm -jpeg -r 150 document.pdf page -``` - -### Accepting Tracked Changes - -To produce a clean document with all tracked changes accepted (requires LibreOffice): - -```bash -python scripts/accept_changes.py input.docx output.docx -``` - ---- - -## Creating New Documents - -Generate .docx files with JavaScript, then validate. Install: `npm install -g docx` - -### Setup -```javascript -const { Document, Packer, Paragraph, TextRun, Table, TableRow, TableCell, ImageRun, - Header, Footer, AlignmentType, PageOrientation, LevelFormat, ExternalHyperlink, - InternalHyperlink, Bookmark, FootnoteReferenceRun, PositionalTab, - PositionalTabAlignment, PositionalTabRelativeTo, PositionalTabLeader, - TabStopType, TabStopPosition, Column, SectionType, - TableOfContents, HeadingLevel, BorderStyle, WidthType, ShadingType, - VerticalAlign, PageNumber, PageBreak } = require('docx'); - -const doc = new Document({ sections: [{ children: [/* content */] }] }); -Packer.toBuffer(doc).then(buffer => fs.writeFileSync("doc.docx", buffer)); -``` - -### Validation -After creating the file, validate it. If validation fails, unpack, fix the XML, and repack. -```bash -python scripts/office/validate.py doc.docx -``` - -### Page Size - -```javascript -// CRITICAL: docx-js defaults to A4, not US Letter -// Always set page size explicitly for consistent results -sections: [{ - properties: { - page: { - size: { - width: 12240, // 8.5 inches in DXA - height: 15840 // 11 inches in DXA - }, - margin: { top: 1440, right: 1440, bottom: 1440, left: 1440 } // 1 inch margins - } - }, - children: [/* content */] -}] -``` - -**Common page sizes (DXA units, 1440 DXA = 1 inch):** - -| Paper | Width | Height | Content Width (1" margins) | -|-------|-------|--------|---------------------------| -| US Letter | 12,240 | 15,840 | 9,360 | -| A4 (default) | 11,906 | 16,838 | 9,026 | - -**Landscape orientation:** docx-js swaps width/height internally, so pass portrait dimensions and let it handle the swap: -```javascript -size: { - width: 12240, // Pass SHORT edge as width - height: 15840, // Pass LONG edge as height - orientation: PageOrientation.LANDSCAPE // docx-js swaps them in the XML -}, -// Content width = 15840 - left margin - right margin (uses the long edge) -``` - -### Styles (Override Built-in Headings) - -Use Arial as the default font (universally supported). Keep titles black for readability. - -```javascript -const doc = new Document({ - styles: { - default: { document: { run: { font: "Arial", size: 24 } } }, // 12pt default - paragraphStyles: [ - // IMPORTANT: Use exact IDs to override built-in styles - { id: "Heading1", name: "Heading 1", basedOn: "Normal", next: "Normal", quickFormat: true, - run: { size: 32, bold: true, font: "Arial" }, - paragraph: { spacing: { before: 240, after: 240 }, outlineLevel: 0 } }, // outlineLevel required for TOC - { id: "Heading2", name: "Heading 2", basedOn: "Normal", next: "Normal", quickFormat: true, - run: { size: 28, bold: true, font: "Arial" }, - paragraph: { spacing: { before: 180, after: 180 }, outlineLevel: 1 } }, - ] - }, - sections: [{ - children: [ - new Paragraph({ heading: HeadingLevel.HEADING_1, children: [new TextRun("Title")] }), - ] - }] -}); -``` - -### Lists (NEVER use unicode bullets) - -```javascript -// ❌ WRONG - never manually insert bullet characters -new Paragraph({ children: [new TextRun("• Item")] }) // BAD -new Paragraph({ children: [new TextRun("\u2022 Item")] }) // BAD - -// ✅ CORRECT - use numbering config with LevelFormat.BULLET -const doc = new Document({ - numbering: { - config: [ - { reference: "bullets", - levels: [{ level: 0, format: LevelFormat.BULLET, text: "•", alignment: AlignmentType.LEFT, - style: { paragraph: { indent: { left: 720, hanging: 360 } } } }] }, - { reference: "numbers", - levels: [{ level: 0, format: LevelFormat.DECIMAL, text: "%1.", alignment: AlignmentType.LEFT, - style: { paragraph: { indent: { left: 720, hanging: 360 } } } }] }, - ] - }, - sections: [{ - children: [ - new Paragraph({ numbering: { reference: "bullets", level: 0 }, - children: [new TextRun("Bullet item")] }), - new Paragraph({ numbering: { reference: "numbers", level: 0 }, - children: [new TextRun("Numbered item")] }), - ] - }] -}); - -// ⚠️ Each reference creates INDEPENDENT numbering -// Same reference = continues (1,2,3 then 4,5,6) -// Different reference = restarts (1,2,3 then 1,2,3) -``` - -### Tables - -**CRITICAL: Tables need dual widths** - set both `columnWidths` on the table AND `width` on each cell. Without both, tables render incorrectly on some platforms. - -```javascript -// CRITICAL: Always set table width for consistent rendering -// CRITICAL: Use ShadingType.CLEAR (not SOLID) to prevent black backgrounds -const border = { style: BorderStyle.SINGLE, size: 1, color: "CCCCCC" }; -const borders = { top: border, bottom: border, left: border, right: border }; - -new Table({ - width: { size: 9360, type: WidthType.DXA }, // Always use DXA (percentages break in Google Docs) - columnWidths: [4680, 4680], // Must sum to table width (DXA: 1440 = 1 inch) - rows: [ - new TableRow({ - children: [ - new TableCell({ - borders, - width: { size: 4680, type: WidthType.DXA }, // Also set on each cell - shading: { fill: "D5E8F0", type: ShadingType.CLEAR }, // CLEAR not SOLID - margins: { top: 80, bottom: 80, left: 120, right: 120 }, // Cell padding (internal, not added to width) - children: [new Paragraph({ children: [new TextRun("Cell")] })] - }) - ] - }) - ] -}) -``` - -**Table width calculation:** - -Always use `WidthType.DXA` — `WidthType.PERCENTAGE` breaks in Google Docs. - -```javascript -// Table width = sum of columnWidths = content width -// US Letter with 1" margins: 12240 - 2880 = 9360 DXA -width: { size: 9360, type: WidthType.DXA }, -columnWidths: [7000, 2360] // Must sum to table width -``` - -**Width rules:** -- **Always use `WidthType.DXA`** — never `WidthType.PERCENTAGE` (incompatible with Google Docs) -- Table width must equal the sum of `columnWidths` -- Cell `width` must match corresponding `columnWidth` -- Cell `margins` are internal padding - they reduce content area, not add to cell width -- For full-width tables: use content width (page width minus left and right margins) - -### Images - -```javascript -// CRITICAL: type parameter is REQUIRED -new Paragraph({ - children: [new ImageRun({ - type: "png", // Required: png, jpg, jpeg, gif, bmp, svg - data: fs.readFileSync("image.png"), - transformation: { width: 200, height: 150 }, - altText: { title: "Title", description: "Desc", name: "Name" } // All three required - })] -}) -``` - -### Page Breaks - -```javascript -// CRITICAL: PageBreak must be inside a Paragraph -new Paragraph({ children: [new PageBreak()] }) - -// Or use pageBreakBefore -new Paragraph({ pageBreakBefore: true, children: [new TextRun("New page")] }) -``` - -### Hyperlinks - -```javascript -// External link -new Paragraph({ - children: [new ExternalHyperlink({ - children: [new TextRun({ text: "Click here", style: "Hyperlink" })], - link: "https://example.com", - })] -}) - -// Internal link (bookmark + reference) -// 1. Create bookmark at destination -new Paragraph({ heading: HeadingLevel.HEADING_1, children: [ - new Bookmark({ id: "chapter1", children: [new TextRun("Chapter 1")] }), -]}) -// 2. Link to it -new Paragraph({ children: [new InternalHyperlink({ - children: [new TextRun({ text: "See Chapter 1", style: "Hyperlink" })], - anchor: "chapter1", -})]}) -``` - -### Footnotes - -```javascript -const doc = new Document({ - footnotes: { - 1: { children: [new Paragraph("Source: Annual Report 2024")] }, - 2: { children: [new Paragraph("See appendix for methodology")] }, - }, - sections: [{ - children: [new Paragraph({ - children: [ - new TextRun("Revenue grew 15%"), - new FootnoteReferenceRun(1), - new TextRun(" using adjusted metrics"), - new FootnoteReferenceRun(2), - ], - })] - }] -}); -``` - -### Tab Stops - -```javascript -// Right-align text on same line (e.g., date opposite a title) -new Paragraph({ - children: [ - new TextRun("Company Name"), - new TextRun("\tJanuary 2025"), - ], - tabStops: [{ type: TabStopType.RIGHT, position: TabStopPosition.MAX }], -}) - -// Dot leader (e.g., TOC-style) -new Paragraph({ - children: [ - new TextRun("Introduction"), - new TextRun({ children: [ - new PositionalTab({ - alignment: PositionalTabAlignment.RIGHT, - relativeTo: PositionalTabRelativeTo.MARGIN, - leader: PositionalTabLeader.DOT, - }), - "3", - ]}), - ], -}) -``` - -### Multi-Column Layouts - -```javascript -// Equal-width columns -sections: [{ - properties: { - column: { - count: 2, // number of columns - space: 720, // gap between columns in DXA (720 = 0.5 inch) - equalWidth: true, - separate: true, // vertical line between columns - }, - }, - children: [/* content flows naturally across columns */] -}] - -// Custom-width columns (equalWidth must be false) -sections: [{ - properties: { - column: { - equalWidth: false, - children: [ - new Column({ width: 5400, space: 720 }), - new Column({ width: 3240 }), - ], - }, - }, - children: [/* content */] -}] -``` - -Force a column break with a new section using `type: SectionType.NEXT_COLUMN`. - -### Table of Contents - -```javascript -// CRITICAL: Headings must use HeadingLevel ONLY - no custom styles -new TableOfContents("Table of Contents", { hyperlink: true, headingStyleRange: "1-3" }) -``` - -### Headers/Footers - -```javascript -sections: [{ - properties: { - page: { margin: { top: 1440, right: 1440, bottom: 1440, left: 1440 } } // 1440 = 1 inch - }, - headers: { - default: new Header({ children: [new Paragraph({ children: [new TextRun("Header")] })] }) - }, - footers: { - default: new Footer({ children: [new Paragraph({ - children: [new TextRun("Page "), new TextRun({ children: [PageNumber.CURRENT] })] - })] }) - }, - children: [/* content */] -}] -``` - -### Critical Rules for docx-js - -- **Set page size explicitly** - docx-js defaults to A4; use US Letter (12240 x 15840 DXA) for US documents -- **Landscape: pass portrait dimensions** - docx-js swaps width/height internally; pass short edge as `width`, long edge as `height`, and set `orientation: PageOrientation.LANDSCAPE` -- **Never use `\n`** - use separate Paragraph elements -- **Never use unicode bullets** - use `LevelFormat.BULLET` with numbering config -- **PageBreak must be in Paragraph** - standalone creates invalid XML -- **ImageRun requires `type`** - always specify png/jpg/etc -- **Always set table `width` with DXA** - never use `WidthType.PERCENTAGE` (breaks in Google Docs) -- **Tables need dual widths** - `columnWidths` array AND cell `width`, both must match -- **Table width = sum of columnWidths** - for DXA, ensure they add up exactly -- **Always add cell margins** - use `margins: { top: 80, bottom: 80, left: 120, right: 120 }` for readable padding -- **Use `ShadingType.CLEAR`** - never SOLID for table shading -- **Never use tables as dividers/rules** - cells have minimum height and render as empty boxes (including in headers/footers); use `border: { bottom: { style: BorderStyle.SINGLE, size: 6, color: "2E75B6", space: 1 } }` on a Paragraph instead. For two-column footers, use tab stops (see Tab Stops section), not tables -- **TOC requires HeadingLevel only** - no custom styles on heading paragraphs -- **Override built-in styles** - use exact IDs: "Heading1", "Heading2", etc. -- **Include `outlineLevel`** - required for TOC (0 for H1, 1 for H2, etc.) - ---- - -## Editing Existing Documents - -**Follow all 3 steps in order.** - -### Step 1: Unpack -```bash -python scripts/office/unpack.py document.docx unpacked/ -``` -Extracts XML, pretty-prints, merges adjacent runs, and converts smart quotes to XML entities (`“` etc.) so they survive editing. Use `--merge-runs false` to skip run merging. - -### Step 2: Edit XML - -Edit files in `unpacked/word/`. See XML Reference below for patterns. - -**Use "Claude" as the author** for tracked changes and comments, unless the user explicitly requests use of a different name. - -**Use the Edit tool directly for string replacement. Do not write Python scripts.** Scripts introduce unnecessary complexity. The Edit tool shows exactly what is being replaced. - -**CRITICAL: Use smart quotes for new content.** When adding text with apostrophes or quotes, use XML entities to produce smart quotes: -```xml - -Here’s a quote: “Hello” -``` -| Entity | Character | -|--------|-----------| -| `‘` | ‘ (left single) | -| `’` | ’ (right single / apostrophe) | -| `“` | “ (left double) | -| `”` | ” (right double) | - -**Adding comments:** Use `comment.py` to handle boilerplate across multiple XML files (text must be pre-escaped XML): -```bash -python scripts/comment.py unpacked/ 0 "Comment text with & and ’" -python scripts/comment.py unpacked/ 1 "Reply text" --parent 0 # reply to comment 0 -python scripts/comment.py unpacked/ 0 "Text" --author "Custom Author" # custom author name -``` -Then add markers to document.xml (see Comments in XML Reference). - -### Step 3: Pack -```bash -python scripts/office/pack.py unpacked/ output.docx --original document.docx -``` -Validates with auto-repair, condenses XML, and creates DOCX. Use `--validate false` to skip. - -**Auto-repair will fix:** -- `durableId` >= 0x7FFFFFFF (regenerates valid ID) -- Missing `xml:space="preserve"` on `` with whitespace - -**Auto-repair won't fix:** -- Malformed XML, invalid element nesting, missing relationships, schema violations - -### Common Pitfalls - -- **Replace entire `` elements**: When adding tracked changes, replace the whole `...` block with `......` as siblings. Don't inject tracked change tags inside a run. -- **Preserve `` formatting**: Copy the original run's `` block into your tracked change runs to maintain bold, font size, etc. - ---- - -## XML Reference - -### Schema Compliance - -- **Element order in ``**: ``, ``, ``, ``, ``, `` last -- **Whitespace**: Add `xml:space="preserve"` to `` with leading/trailing spaces -- **RSIDs**: Must be 8-digit hex (e.g., `00AB1234`) - -### Tracked Changes - -**Insertion:** -```xml - - inserted text - -``` - -**Deletion:** -```xml - - deleted text - -``` - -**Inside ``**: Use `` instead of ``, and `` instead of ``. - -**Minimal edits** - only mark what changes: -```xml - -The term is - - 30 - - - 60 - - days. -``` - -**Deleting entire paragraphs/list items** - when removing ALL content from a paragraph, also mark the paragraph mark as deleted so it merges with the next paragraph. Add `` inside ``: -```xml - - - ... - - - - - - Entire paragraph content being deleted... - - -``` -Without the `` in ``, accepting changes leaves an empty paragraph/list item. - -**Rejecting another author's insertion** - nest deletion inside their insertion: -```xml - - - their inserted text - - -``` - -**Restoring another author's deletion** - add insertion after (don't modify their deletion): -```xml - - deleted text - - - deleted text - -``` - -### Comments - -After running `comment.py` (see Step 2), add markers to document.xml. For replies, use `--parent` flag and nest markers inside the parent's. - -**CRITICAL: `` and `` are siblings of ``, never inside ``.** - -```xml - - - - deleted - - more text - - - - - - - text - - - - -``` - -### Images - -1. Add image file to `word/media/` -2. Add relationship to `word/_rels/document.xml.rels`: -```xml - -``` -3. Add content type to `[Content_Types].xml`: -```xml - -``` -4. Reference in document.xml: -```xml - - - - - - - - - - - - -``` - ---- - -## Dependencies - -- **pandoc**: Text extraction -- **docx**: `npm install -g docx` (new documents) -- **LibreOffice**: PDF conversion (auto-configured for sandboxed environments via `scripts/office/soffice.py`) -- **Poppler**: `pdftoppm` for images diff --git a/.agents/skills/docx/scripts/__init__.py b/.agents/skills/docx/scripts/__init__.py deleted file mode 100755 index 8b1378917..000000000 --- a/.agents/skills/docx/scripts/__init__.py +++ /dev/null @@ -1 +0,0 @@ - diff --git a/.agents/skills/docx/scripts/accept_changes.py b/.agents/skills/docx/scripts/accept_changes.py deleted file mode 100755 index 8e3631619..000000000 --- a/.agents/skills/docx/scripts/accept_changes.py +++ /dev/null @@ -1,135 +0,0 @@ -"""Accept all tracked changes in a DOCX file using LibreOffice. - -Requires LibreOffice (soffice) to be installed. -""" - -import argparse -import logging -import shutil -import subprocess -from pathlib import Path - -from office.soffice import get_soffice_env - -logger = logging.getLogger(__name__) - -LIBREOFFICE_PROFILE = "/tmp/libreoffice_docx_profile" -MACRO_DIR = f"{LIBREOFFICE_PROFILE}/user/basic/Standard" - -ACCEPT_CHANGES_MACRO = """ - - - Sub AcceptAllTrackedChanges() - Dim document As Object - Dim dispatcher As Object - - document = ThisComponent.CurrentController.Frame - dispatcher = createUnoService("com.sun.star.frame.DispatchHelper") - - dispatcher.executeDispatch(document, ".uno:AcceptAllTrackedChanges", "", 0, Array()) - ThisComponent.store() - ThisComponent.close(True) - End Sub -""" - - -def accept_changes( - input_file: str, - output_file: str, -) -> tuple[None, str]: - input_path = Path(input_file) - output_path = Path(output_file) - - if not input_path.exists(): - return None, f"Error: Input file not found: {input_file}" - - if not input_path.suffix.lower() == ".docx": - return None, f"Error: Input file is not a DOCX file: {input_file}" - - try: - output_path.parent.mkdir(parents=True, exist_ok=True) - shutil.copy2(input_path, output_path) - except Exception as e: - return None, f"Error: Failed to copy input file to output location: {e}" - - if not _setup_libreoffice_macro(): - return None, "Error: Failed to setup LibreOffice macro" - - cmd = [ - "soffice", - "--headless", - f"-env:UserInstallation=file://{LIBREOFFICE_PROFILE}", - "--norestore", - "vnd.sun.star.script:Standard.Module1.AcceptAllTrackedChanges?language=Basic&location=application", - str(output_path.absolute()), - ] - - try: - result = subprocess.run( - cmd, - capture_output=True, - text=True, - timeout=30, - check=False, - env=get_soffice_env(), - ) - except subprocess.TimeoutExpired: - return ( - None, - f"Successfully accepted all tracked changes: {input_file} -> {output_file}", - ) - - if result.returncode != 0: - return None, f"Error: LibreOffice failed: {result.stderr}" - - return ( - None, - f"Successfully accepted all tracked changes: {input_file} -> {output_file}", - ) - - -def _setup_libreoffice_macro() -> bool: - macro_dir = Path(MACRO_DIR) - macro_file = macro_dir / "Module1.xba" - - if macro_file.exists() and "AcceptAllTrackedChanges" in macro_file.read_text(): - return True - - if not macro_dir.exists(): - subprocess.run( - [ - "soffice", - "--headless", - f"-env:UserInstallation=file://{LIBREOFFICE_PROFILE}", - "--terminate_after_init", - ], - capture_output=True, - timeout=10, - check=False, - env=get_soffice_env(), - ) - macro_dir.mkdir(parents=True, exist_ok=True) - - try: - macro_file.write_text(ACCEPT_CHANGES_MACRO) - return True - except Exception as e: - logger.warning(f"Failed to setup LibreOffice macro: {e}") - return False - - -if __name__ == "__main__": - parser = argparse.ArgumentParser( - description="Accept all tracked changes in a DOCX file" - ) - parser.add_argument("input_file", help="Input DOCX file with tracked changes") - parser.add_argument( - "output_file", help="Output DOCX file (clean, no tracked changes)" - ) - args = parser.parse_args() - - _, message = accept_changes(args.input_file, args.output_file) - print(message) - - if "Error" in message: - raise SystemExit(1) diff --git a/.agents/skills/docx/scripts/comment.py b/.agents/skills/docx/scripts/comment.py deleted file mode 100755 index 36e1c935f..000000000 --- a/.agents/skills/docx/scripts/comment.py +++ /dev/null @@ -1,318 +0,0 @@ -"""Add comments to DOCX documents. - -Usage: - python comment.py unpacked/ 0 "Comment text" - python comment.py unpacked/ 1 "Reply text" --parent 0 - -Text should be pre-escaped XML (e.g., & for &, ’ for smart quotes). - -After running, add markers to document.xml: - - ... commented content ... - - -""" - -import argparse -import random -import shutil -import sys -from datetime import datetime, timezone -from pathlib import Path - -import defusedxml.minidom - -TEMPLATE_DIR = Path(__file__).parent / "templates" -NS = { - "w": "http://schemas.openxmlformats.org/wordprocessingml/2006/main", - "w14": "http://schemas.microsoft.com/office/word/2010/wordml", - "w15": "http://schemas.microsoft.com/office/word/2012/wordml", - "w16cid": "http://schemas.microsoft.com/office/word/2016/wordml/cid", - "w16cex": "http://schemas.microsoft.com/office/word/2018/wordml/cex", -} - -COMMENT_XML = """\ - - - - - - - - - - - - - {text} - - -""" - -COMMENT_MARKER_TEMPLATE = """ -Add to document.xml (markers must be direct children of w:p, never inside w:r): - - ... - - """ - -REPLY_MARKER_TEMPLATE = """ -Nest markers inside parent {pid}'s markers (markers must be direct children of w:p, never inside w:r): - - ... - - - """ - - -def _generate_hex_id() -> str: - return f"{random.randint(0, 0x7FFFFFFE):08X}" - - -SMART_QUOTE_ENTITIES = { - "\u201c": "“", - "\u201d": "”", - "\u2018": "‘", - "\u2019": "’", -} - - -def _encode_smart_quotes(text: str) -> str: - for char, entity in SMART_QUOTE_ENTITIES.items(): - text = text.replace(char, entity) - return text - - -def _append_xml(xml_path: Path, root_tag: str, content: str) -> None: - dom = defusedxml.minidom.parseString(xml_path.read_text(encoding="utf-8")) - root = dom.getElementsByTagName(root_tag)[0] - ns_attrs = " ".join(f'xmlns:{k}="{v}"' for k, v in NS.items()) - wrapper_dom = defusedxml.minidom.parseString(f"{content}") - for child in wrapper_dom.documentElement.childNodes: - if child.nodeType == child.ELEMENT_NODE: - root.appendChild(dom.importNode(child, True)) - output = _encode_smart_quotes(dom.toxml(encoding="UTF-8").decode("utf-8")) - xml_path.write_text(output, encoding="utf-8") - - -def _find_para_id(comments_path: Path, comment_id: int) -> str | None: - dom = defusedxml.minidom.parseString(comments_path.read_text(encoding="utf-8")) - for c in dom.getElementsByTagName("w:comment"): - if c.getAttribute("w:id") == str(comment_id): - for p in c.getElementsByTagName("w:p"): - if pid := p.getAttribute("w14:paraId"): - return pid - return None - - -def _get_next_rid(rels_path: Path) -> int: - dom = defusedxml.minidom.parseString(rels_path.read_text(encoding="utf-8")) - max_rid = 0 - for rel in dom.getElementsByTagName("Relationship"): - rid = rel.getAttribute("Id") - if rid and rid.startswith("rId"): - try: - max_rid = max(max_rid, int(rid[3:])) - except ValueError: - pass - return max_rid + 1 - - -def _has_relationship(rels_path: Path, target: str) -> bool: - dom = defusedxml.minidom.parseString(rels_path.read_text(encoding="utf-8")) - for rel in dom.getElementsByTagName("Relationship"): - if rel.getAttribute("Target") == target: - return True - return False - - -def _has_content_type(ct_path: Path, part_name: str) -> bool: - dom = defusedxml.minidom.parseString(ct_path.read_text(encoding="utf-8")) - for override in dom.getElementsByTagName("Override"): - if override.getAttribute("PartName") == part_name: - return True - return False - - -def _ensure_comment_relationships(unpacked_dir: Path) -> None: - rels_path = unpacked_dir / "word" / "_rels" / "document.xml.rels" - if not rels_path.exists(): - return - - if _has_relationship(rels_path, "comments.xml"): - return - - dom = defusedxml.minidom.parseString(rels_path.read_text(encoding="utf-8")) - root = dom.documentElement - next_rid = _get_next_rid(rels_path) - - rels = [ - ( - "http://schemas.openxmlformats.org/officeDocument/2006/relationships/comments", - "comments.xml", - ), - ( - "http://schemas.microsoft.com/office/2011/relationships/commentsExtended", - "commentsExtended.xml", - ), - ( - "http://schemas.microsoft.com/office/2016/09/relationships/commentsIds", - "commentsIds.xml", - ), - ( - "http://schemas.microsoft.com/office/2018/08/relationships/commentsExtensible", - "commentsExtensible.xml", - ), - ] - - for rel_type, target in rels: - rel = dom.createElement("Relationship") - rel.setAttribute("Id", f"rId{next_rid}") - rel.setAttribute("Type", rel_type) - rel.setAttribute("Target", target) - root.appendChild(rel) - next_rid += 1 - - rels_path.write_bytes(dom.toxml(encoding="UTF-8")) - - -def _ensure_comment_content_types(unpacked_dir: Path) -> None: - ct_path = unpacked_dir / "[Content_Types].xml" - if not ct_path.exists(): - return - - if _has_content_type(ct_path, "/word/comments.xml"): - return - - dom = defusedxml.minidom.parseString(ct_path.read_text(encoding="utf-8")) - root = dom.documentElement - - overrides = [ - ( - "/word/comments.xml", - "application/vnd.openxmlformats-officedocument.wordprocessingml.comments+xml", - ), - ( - "/word/commentsExtended.xml", - "application/vnd.openxmlformats-officedocument.wordprocessingml.commentsExtended+xml", - ), - ( - "/word/commentsIds.xml", - "application/vnd.openxmlformats-officedocument.wordprocessingml.commentsIds+xml", - ), - ( - "/word/commentsExtensible.xml", - "application/vnd.openxmlformats-officedocument.wordprocessingml.commentsExtensible+xml", - ), - ] - - for part_name, content_type in overrides: - override = dom.createElement("Override") - override.setAttribute("PartName", part_name) - override.setAttribute("ContentType", content_type) - root.appendChild(override) - - ct_path.write_bytes(dom.toxml(encoding="UTF-8")) - - -def add_comment( - unpacked_dir: str, - comment_id: int, - text: str, - author: str = "Claude", - initials: str = "C", - parent_id: int | None = None, -) -> tuple[str, str]: - word = Path(unpacked_dir) / "word" - if not word.exists(): - return "", f"Error: {word} not found" - - para_id, durable_id = _generate_hex_id(), _generate_hex_id() - ts = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") - - comments = word / "comments.xml" - first_comment = not comments.exists() - if first_comment: - shutil.copy(TEMPLATE_DIR / "comments.xml", comments) - _ensure_comment_relationships(Path(unpacked_dir)) - _ensure_comment_content_types(Path(unpacked_dir)) - _append_xml( - comments, - "w:comments", - COMMENT_XML.format( - id=comment_id, - author=author, - date=ts, - initials=initials, - para_id=para_id, - text=text, - ), - ) - - ext = word / "commentsExtended.xml" - if not ext.exists(): - shutil.copy(TEMPLATE_DIR / "commentsExtended.xml", ext) - if parent_id is not None: - parent_para = _find_para_id(comments, parent_id) - if not parent_para: - return "", f"Error: Parent comment {parent_id} not found" - _append_xml( - ext, - "w15:commentsEx", - f'', - ) - else: - _append_xml( - ext, - "w15:commentsEx", - f'', - ) - - ids = word / "commentsIds.xml" - if not ids.exists(): - shutil.copy(TEMPLATE_DIR / "commentsIds.xml", ids) - _append_xml( - ids, - "w16cid:commentsIds", - f'', - ) - - extensible = word / "commentsExtensible.xml" - if not extensible.exists(): - shutil.copy(TEMPLATE_DIR / "commentsExtensible.xml", extensible) - _append_xml( - extensible, - "w16cex:commentsExtensible", - f'', - ) - - action = "reply" if parent_id is not None else "comment" - return para_id, f"Added {action} {comment_id} (para_id={para_id})" - - -if __name__ == "__main__": - p = argparse.ArgumentParser(description="Add comments to DOCX documents") - p.add_argument("unpacked_dir", help="Unpacked DOCX directory") - p.add_argument("comment_id", type=int, help="Comment ID (must be unique)") - p.add_argument("text", help="Comment text") - p.add_argument("--author", default="Claude", help="Author name") - p.add_argument("--initials", default="C", help="Author initials") - p.add_argument("--parent", type=int, help="Parent comment ID (for replies)") - args = p.parse_args() - - para_id, msg = add_comment( - args.unpacked_dir, - args.comment_id, - args.text, - args.author, - args.initials, - args.parent, - ) - print(msg) - if "Error" in msg: - sys.exit(1) - cid = args.comment_id - if args.parent is not None: - print(REPLY_MARKER_TEMPLATE.format(pid=args.parent, cid=cid)) - else: - print(COMMENT_MARKER_TEMPLATE.format(cid=cid)) diff --git a/.agents/skills/docx/scripts/office/helpers/merge_runs.py b/.agents/skills/docx/scripts/office/helpers/merge_runs.py deleted file mode 100644 index ad7c25eec..000000000 --- a/.agents/skills/docx/scripts/office/helpers/merge_runs.py +++ /dev/null @@ -1,199 +0,0 @@ -"""Merge adjacent runs with identical formatting in DOCX. - -Merges adjacent elements that have identical properties. -Works on runs in paragraphs and inside tracked changes (, ). - -Also: -- Removes rsid attributes from runs (revision metadata that doesn't affect rendering) -- Removes proofErr elements (spell/grammar markers that block merging) -""" - -from pathlib import Path - -import defusedxml.minidom - - -def merge_runs(input_dir: str) -> tuple[int, str]: - doc_xml = Path(input_dir) / "word" / "document.xml" - - if not doc_xml.exists(): - return 0, f"Error: {doc_xml} not found" - - try: - dom = defusedxml.minidom.parseString(doc_xml.read_text(encoding="utf-8")) - root = dom.documentElement - - _remove_elements(root, "proofErr") - _strip_run_rsid_attrs(root) - - containers = {run.parentNode for run in _find_elements(root, "r")} - - merge_count = 0 - for container in containers: - merge_count += _merge_runs_in(container) - - doc_xml.write_bytes(dom.toxml(encoding="UTF-8")) - return merge_count, f"Merged {merge_count} runs" - - except Exception as e: - return 0, f"Error: {e}" - - - - -def _find_elements(root, tag: str) -> list: - results = [] - - def traverse(node): - if node.nodeType == node.ELEMENT_NODE: - name = node.localName or node.tagName - if name == tag or name.endswith(f":{tag}"): - results.append(node) - for child in node.childNodes: - traverse(child) - - traverse(root) - return results - - -def _get_child(parent, tag: str): - for child in parent.childNodes: - if child.nodeType == child.ELEMENT_NODE: - name = child.localName or child.tagName - if name == tag or name.endswith(f":{tag}"): - return child - return None - - -def _get_children(parent, tag: str) -> list: - results = [] - for child in parent.childNodes: - if child.nodeType == child.ELEMENT_NODE: - name = child.localName or child.tagName - if name == tag or name.endswith(f":{tag}"): - results.append(child) - return results - - -def _is_adjacent(elem1, elem2) -> bool: - node = elem1.nextSibling - while node: - if node == elem2: - return True - if node.nodeType == node.ELEMENT_NODE: - return False - if node.nodeType == node.TEXT_NODE and node.data.strip(): - return False - node = node.nextSibling - return False - - - - -def _remove_elements(root, tag: str): - for elem in _find_elements(root, tag): - if elem.parentNode: - elem.parentNode.removeChild(elem) - - -def _strip_run_rsid_attrs(root): - for run in _find_elements(root, "r"): - for attr in list(run.attributes.values()): - if "rsid" in attr.name.lower(): - run.removeAttribute(attr.name) - - - - -def _merge_runs_in(container) -> int: - merge_count = 0 - run = _first_child_run(container) - - while run: - while True: - next_elem = _next_element_sibling(run) - if next_elem and _is_run(next_elem) and _can_merge(run, next_elem): - _merge_run_content(run, next_elem) - container.removeChild(next_elem) - merge_count += 1 - else: - break - - _consolidate_text(run) - run = _next_sibling_run(run) - - return merge_count - - -def _first_child_run(container): - for child in container.childNodes: - if child.nodeType == child.ELEMENT_NODE and _is_run(child): - return child - return None - - -def _next_element_sibling(node): - sibling = node.nextSibling - while sibling: - if sibling.nodeType == sibling.ELEMENT_NODE: - return sibling - sibling = sibling.nextSibling - return None - - -def _next_sibling_run(node): - sibling = node.nextSibling - while sibling: - if sibling.nodeType == sibling.ELEMENT_NODE: - if _is_run(sibling): - return sibling - sibling = sibling.nextSibling - return None - - -def _is_run(node) -> bool: - name = node.localName or node.tagName - return name == "r" or name.endswith(":r") - - -def _can_merge(run1, run2) -> bool: - rpr1 = _get_child(run1, "rPr") - rpr2 = _get_child(run2, "rPr") - - if (rpr1 is None) != (rpr2 is None): - return False - if rpr1 is None: - return True - return rpr1.toxml() == rpr2.toxml() - - -def _merge_run_content(target, source): - for child in list(source.childNodes): - if child.nodeType == child.ELEMENT_NODE: - name = child.localName or child.tagName - if name != "rPr" and not name.endswith(":rPr"): - target.appendChild(child) - - -def _consolidate_text(run): - t_elements = _get_children(run, "t") - - for i in range(len(t_elements) - 1, 0, -1): - curr, prev = t_elements[i], t_elements[i - 1] - - if _is_adjacent(prev, curr): - prev_text = prev.firstChild.data if prev.firstChild else "" - curr_text = curr.firstChild.data if curr.firstChild else "" - merged = prev_text + curr_text - - if prev.firstChild: - prev.firstChild.data = merged - else: - prev.appendChild(run.ownerDocument.createTextNode(merged)) - - if merged.startswith(" ") or merged.endswith(" "): - prev.setAttribute("xml:space", "preserve") - elif prev.hasAttribute("xml:space"): - prev.removeAttribute("xml:space") - - run.removeChild(curr) diff --git a/.agents/skills/docx/scripts/office/helpers/simplify_redlines.py b/.agents/skills/docx/scripts/office/helpers/simplify_redlines.py deleted file mode 100644 index db963bb99..000000000 --- a/.agents/skills/docx/scripts/office/helpers/simplify_redlines.py +++ /dev/null @@ -1,197 +0,0 @@ -"""Simplify tracked changes by merging adjacent w:ins or w:del elements. - -Merges adjacent elements from the same author into a single element. -Same for elements. This makes heavily-redlined documents easier to -work with by reducing the number of tracked change wrappers. - -Rules: -- Only merges w:ins with w:ins, w:del with w:del (same element type) -- Only merges if same author (ignores timestamp differences) -- Only merges if truly adjacent (only whitespace between them) -""" - -import xml.etree.ElementTree as ET -import zipfile -from pathlib import Path - -import defusedxml.minidom - -WORD_NS = "http://schemas.openxmlformats.org/wordprocessingml/2006/main" - - -def simplify_redlines(input_dir: str) -> tuple[int, str]: - doc_xml = Path(input_dir) / "word" / "document.xml" - - if not doc_xml.exists(): - return 0, f"Error: {doc_xml} not found" - - try: - dom = defusedxml.minidom.parseString(doc_xml.read_text(encoding="utf-8")) - root = dom.documentElement - - merge_count = 0 - - containers = _find_elements(root, "p") + _find_elements(root, "tc") - - for container in containers: - merge_count += _merge_tracked_changes_in(container, "ins") - merge_count += _merge_tracked_changes_in(container, "del") - - doc_xml.write_bytes(dom.toxml(encoding="UTF-8")) - return merge_count, f"Simplified {merge_count} tracked changes" - - except Exception as e: - return 0, f"Error: {e}" - - -def _merge_tracked_changes_in(container, tag: str) -> int: - merge_count = 0 - - tracked = [ - child - for child in container.childNodes - if child.nodeType == child.ELEMENT_NODE and _is_element(child, tag) - ] - - if len(tracked) < 2: - return 0 - - i = 0 - while i < len(tracked) - 1: - curr = tracked[i] - next_elem = tracked[i + 1] - - if _can_merge_tracked(curr, next_elem): - _merge_tracked_content(curr, next_elem) - container.removeChild(next_elem) - tracked.pop(i + 1) - merge_count += 1 - else: - i += 1 - - return merge_count - - -def _is_element(node, tag: str) -> bool: - name = node.localName or node.tagName - return name == tag or name.endswith(f":{tag}") - - -def _get_author(elem) -> str: - author = elem.getAttribute("w:author") - if not author: - for attr in elem.attributes.values(): - if attr.localName == "author" or attr.name.endswith(":author"): - return attr.value - return author - - -def _can_merge_tracked(elem1, elem2) -> bool: - if _get_author(elem1) != _get_author(elem2): - return False - - node = elem1.nextSibling - while node and node != elem2: - if node.nodeType == node.ELEMENT_NODE: - return False - if node.nodeType == node.TEXT_NODE and node.data.strip(): - return False - node = node.nextSibling - - return True - - -def _merge_tracked_content(target, source): - while source.firstChild: - child = source.firstChild - source.removeChild(child) - target.appendChild(child) - - -def _find_elements(root, tag: str) -> list: - results = [] - - def traverse(node): - if node.nodeType == node.ELEMENT_NODE: - name = node.localName or node.tagName - if name == tag or name.endswith(f":{tag}"): - results.append(node) - for child in node.childNodes: - traverse(child) - - traverse(root) - return results - - -def get_tracked_change_authors(doc_xml_path: Path) -> dict[str, int]: - if not doc_xml_path.exists(): - return {} - - try: - tree = ET.parse(doc_xml_path) - root = tree.getroot() - except ET.ParseError: - return {} - - namespaces = {"w": WORD_NS} - author_attr = f"{{{WORD_NS}}}author" - - authors: dict[str, int] = {} - for tag in ["ins", "del"]: - for elem in root.findall(f".//w:{tag}", namespaces): - author = elem.get(author_attr) - if author: - authors[author] = authors.get(author, 0) + 1 - - return authors - - -def _get_authors_from_docx(docx_path: Path) -> dict[str, int]: - try: - with zipfile.ZipFile(docx_path, "r") as zf: - if "word/document.xml" not in zf.namelist(): - return {} - with zf.open("word/document.xml") as f: - tree = ET.parse(f) - root = tree.getroot() - - namespaces = {"w": WORD_NS} - author_attr = f"{{{WORD_NS}}}author" - - authors: dict[str, int] = {} - for tag in ["ins", "del"]: - for elem in root.findall(f".//w:{tag}", namespaces): - author = elem.get(author_attr) - if author: - authors[author] = authors.get(author, 0) + 1 - return authors - except (zipfile.BadZipFile, ET.ParseError): - return {} - - -def infer_author(modified_dir: Path, original_docx: Path, default: str = "Claude") -> str: - modified_xml = modified_dir / "word" / "document.xml" - modified_authors = get_tracked_change_authors(modified_xml) - - if not modified_authors: - return default - - original_authors = _get_authors_from_docx(original_docx) - - new_changes: dict[str, int] = {} - for author, count in modified_authors.items(): - original_count = original_authors.get(author, 0) - diff = count - original_count - if diff > 0: - new_changes[author] = diff - - if not new_changes: - return default - - if len(new_changes) == 1: - return next(iter(new_changes)) - - raise ValueError( - f"Multiple authors added new changes: {new_changes}. " - "Cannot infer which author to validate." - ) diff --git a/.agents/skills/docx/scripts/office/pack.py b/.agents/skills/docx/scripts/office/pack.py deleted file mode 100755 index db29ed8b1..000000000 --- a/.agents/skills/docx/scripts/office/pack.py +++ /dev/null @@ -1,159 +0,0 @@ -"""Pack a directory into a DOCX, PPTX, or XLSX file. - -Validates with auto-repair, condenses XML formatting, and creates the Office file. - -Usage: - python pack.py [--original ] [--validate true|false] - -Examples: - python pack.py unpacked/ output.docx --original input.docx - python pack.py unpacked/ output.pptx --validate false -""" - -import argparse -import sys -import shutil -import tempfile -import zipfile -from pathlib import Path - -import defusedxml.minidom - -from validators import DOCXSchemaValidator, PPTXSchemaValidator, RedliningValidator - -def pack( - input_directory: str, - output_file: str, - original_file: str | None = None, - validate: bool = True, - infer_author_func=None, -) -> tuple[None, str]: - input_dir = Path(input_directory) - output_path = Path(output_file) - suffix = output_path.suffix.lower() - - if not input_dir.is_dir(): - return None, f"Error: {input_dir} is not a directory" - - if suffix not in {".docx", ".pptx", ".xlsx"}: - return None, f"Error: {output_file} must be a .docx, .pptx, or .xlsx file" - - if validate and original_file: - original_path = Path(original_file) - if original_path.exists(): - success, output = _run_validation( - input_dir, original_path, suffix, infer_author_func - ) - if output: - print(output) - if not success: - return None, f"Error: Validation failed for {input_dir}" - - with tempfile.TemporaryDirectory() as temp_dir: - temp_content_dir = Path(temp_dir) / "content" - shutil.copytree(input_dir, temp_content_dir) - - for pattern in ["*.xml", "*.rels"]: - for xml_file in temp_content_dir.rglob(pattern): - _condense_xml(xml_file) - - output_path.parent.mkdir(parents=True, exist_ok=True) - with zipfile.ZipFile(output_path, "w", zipfile.ZIP_DEFLATED) as zf: - for f in temp_content_dir.rglob("*"): - if f.is_file(): - zf.write(f, f.relative_to(temp_content_dir)) - - return None, f"Successfully packed {input_dir} to {output_file}" - - -def _run_validation( - unpacked_dir: Path, - original_file: Path, - suffix: str, - infer_author_func=None, -) -> tuple[bool, str | None]: - output_lines = [] - validators = [] - - if suffix == ".docx": - author = "Claude" - if infer_author_func: - try: - author = infer_author_func(unpacked_dir, original_file) - except ValueError as e: - print(f"Warning: {e} Using default author 'Claude'.", file=sys.stderr) - - validators = [ - DOCXSchemaValidator(unpacked_dir, original_file), - RedliningValidator(unpacked_dir, original_file, author=author), - ] - elif suffix == ".pptx": - validators = [PPTXSchemaValidator(unpacked_dir, original_file)] - - if not validators: - return True, None - - total_repairs = sum(v.repair() for v in validators) - if total_repairs: - output_lines.append(f"Auto-repaired {total_repairs} issue(s)") - - success = all(v.validate() for v in validators) - - if success: - output_lines.append("All validations PASSED!") - - return success, "\n".join(output_lines) if output_lines else None - - -def _condense_xml(xml_file: Path) -> None: - try: - with open(xml_file, encoding="utf-8") as f: - dom = defusedxml.minidom.parse(f) - - for element in dom.getElementsByTagName("*"): - if element.tagName.endswith(":t"): - continue - - for child in list(element.childNodes): - if ( - child.nodeType == child.TEXT_NODE - and child.nodeValue - and child.nodeValue.strip() == "" - ) or child.nodeType == child.COMMENT_NODE: - element.removeChild(child) - - xml_file.write_bytes(dom.toxml(encoding="UTF-8")) - except Exception as e: - print(f"ERROR: Failed to parse {xml_file.name}: {e}", file=sys.stderr) - raise - - -if __name__ == "__main__": - parser = argparse.ArgumentParser( - description="Pack a directory into a DOCX, PPTX, or XLSX file" - ) - parser.add_argument("input_directory", help="Unpacked Office document directory") - parser.add_argument("output_file", help="Output Office file (.docx/.pptx/.xlsx)") - parser.add_argument( - "--original", - help="Original file for validation comparison", - ) - parser.add_argument( - "--validate", - type=lambda x: x.lower() == "true", - default=True, - metavar="true|false", - help="Run validation with auto-repair (default: true)", - ) - args = parser.parse_args() - - _, message = pack( - args.input_directory, - args.output_file, - original_file=args.original, - validate=args.validate, - ) - print(message) - - if "Error" in message: - sys.exit(1) diff --git a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-chart.xsd b/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-chart.xsd deleted file mode 100644 index 6454ef9a9..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-chart.xsd +++ /dev/null @@ -1,1499 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-chartDrawing.xsd b/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-chartDrawing.xsd deleted file mode 100644 index afa4f463e..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-chartDrawing.xsd +++ /dev/null @@ -1,146 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-diagram.xsd b/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-diagram.xsd deleted file mode 100644 index 64e66b8ab..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-diagram.xsd +++ /dev/null @@ -1,1085 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-lockedCanvas.xsd b/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-lockedCanvas.xsd deleted file mode 100644 index 687eea829..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-lockedCanvas.xsd +++ /dev/null @@ -1,11 +0,0 @@ - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-main.xsd b/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-main.xsd deleted file mode 100644 index 6ac81b06b..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-main.xsd +++ /dev/null @@ -1,3081 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-picture.xsd b/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-picture.xsd deleted file mode 100644 index 1dbf05140..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-picture.xsd +++ /dev/null @@ -1,23 +0,0 @@ - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-spreadsheetDrawing.xsd b/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-spreadsheetDrawing.xsd deleted file mode 100644 index f1af17db4..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-spreadsheetDrawing.xsd +++ /dev/null @@ -1,185 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-wordprocessingDrawing.xsd b/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-wordprocessingDrawing.xsd deleted file mode 100644 index 0a185ab6e..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-wordprocessingDrawing.xsd +++ /dev/null @@ -1,287 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/pml.xsd b/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/pml.xsd deleted file mode 100644 index 14ef48886..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/pml.xsd +++ /dev/null @@ -1,1676 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-additionalCharacteristics.xsd b/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-additionalCharacteristics.xsd deleted file mode 100644 index c20f3bf14..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-additionalCharacteristics.xsd +++ /dev/null @@ -1,28 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-bibliography.xsd b/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-bibliography.xsd deleted file mode 100644 index ac6025226..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-bibliography.xsd +++ /dev/null @@ -1,144 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-commonSimpleTypes.xsd b/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-commonSimpleTypes.xsd deleted file mode 100644 index 424b8ba8d..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-commonSimpleTypes.xsd +++ /dev/null @@ -1,174 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-customXmlDataProperties.xsd b/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-customXmlDataProperties.xsd deleted file mode 100644 index 2bddce292..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-customXmlDataProperties.xsd +++ /dev/null @@ -1,25 +0,0 @@ - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-customXmlSchemaProperties.xsd b/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-customXmlSchemaProperties.xsd deleted file mode 100644 index 8a8c18ba2..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-customXmlSchemaProperties.xsd +++ /dev/null @@ -1,18 +0,0 @@ - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesCustom.xsd b/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesCustom.xsd deleted file mode 100644 index 5c42706a0..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesCustom.xsd +++ /dev/null @@ -1,59 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesExtended.xsd b/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesExtended.xsd deleted file mode 100644 index 853c341c8..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesExtended.xsd +++ /dev/null @@ -1,56 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesVariantTypes.xsd b/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesVariantTypes.xsd deleted file mode 100644 index da835ee82..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesVariantTypes.xsd +++ /dev/null @@ -1,195 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-math.xsd b/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-math.xsd deleted file mode 100644 index 87ad2658f..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-math.xsd +++ /dev/null @@ -1,582 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-relationshipReference.xsd b/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-relationshipReference.xsd deleted file mode 100644 index 9e86f1b2b..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-relationshipReference.xsd +++ /dev/null @@ -1,25 +0,0 @@ - - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/sml.xsd b/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/sml.xsd deleted file mode 100644 index d0be42e75..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/sml.xsd +++ /dev/null @@ -1,4439 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-main.xsd b/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-main.xsd deleted file mode 100644 index 8821dd183..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-main.xsd +++ /dev/null @@ -1,570 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-officeDrawing.xsd b/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-officeDrawing.xsd deleted file mode 100644 index ca2575c75..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-officeDrawing.xsd +++ /dev/null @@ -1,509 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-presentationDrawing.xsd b/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-presentationDrawing.xsd deleted file mode 100644 index dd079e603..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-presentationDrawing.xsd +++ /dev/null @@ -1,12 +0,0 @@ - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-spreadsheetDrawing.xsd b/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-spreadsheetDrawing.xsd deleted file mode 100644 index 3dd6cf625..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-spreadsheetDrawing.xsd +++ /dev/null @@ -1,108 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-wordprocessingDrawing.xsd b/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-wordprocessingDrawing.xsd deleted file mode 100644 index f1041e34e..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-wordprocessingDrawing.xsd +++ /dev/null @@ -1,96 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/wml.xsd b/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/wml.xsd deleted file mode 100644 index 9c5b7a633..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/wml.xsd +++ /dev/null @@ -1,3646 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/xml.xsd b/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/xml.xsd deleted file mode 100644 index 0f13678d8..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ISO-IEC29500-4_2016/xml.xsd +++ /dev/null @@ -1,116 +0,0 @@ - - - - - - See http://www.w3.org/XML/1998/namespace.html and - http://www.w3.org/TR/REC-xml for information about this namespace. - - This schema document describes the XML namespace, in a form - suitable for import by other schema documents. - - Note that local names in this namespace are intended to be defined - only by the World Wide Web Consortium or its subgroups. The - following names are currently defined in this namespace and should - not be used with conflicting semantics by any Working Group, - specification, or document instance: - - base (as an attribute name): denotes an attribute whose value - provides a URI to be used as the base for interpreting any - relative URIs in the scope of the element on which it - appears; its value is inherited. This name is reserved - by virtue of its definition in the XML Base specification. - - lang (as an attribute name): denotes an attribute whose value - is a language code for the natural language of the content of - any element; its value is inherited. This name is reserved - by virtue of its definition in the XML specification. - - space (as an attribute name): denotes an attribute whose - value is a keyword indicating what whitespace processing - discipline is intended for the content of the element; its - value is inherited. This name is reserved by virtue of its - definition in the XML specification. - - Father (in any context at all): denotes Jon Bosak, the chair of - the original XML Working Group. This name is reserved by - the following decision of the W3C XML Plenary and - XML Coordination groups: - - In appreciation for his vision, leadership and dedication - the W3C XML Plenary on this 10th day of February, 2000 - reserves for Jon Bosak in perpetuity the XML name - xml:Father - - - - - This schema defines attributes and an attribute group - suitable for use by - schemas wishing to allow xml:base, xml:lang or xml:space attributes - on elements they define. - - To enable this, such a schema must import this schema - for the XML namespace, e.g. as follows: - <schema . . .> - . . . - <import namespace="http://www.w3.org/XML/1998/namespace" - schemaLocation="http://www.w3.org/2001/03/xml.xsd"/> - - Subsequently, qualified reference to any of the attributes - or the group defined below will have the desired effect, e.g. - - <type . . .> - . . . - <attributeGroup ref="xml:specialAttrs"/> - - will define a type which will schema-validate an instance - element with any of those attributes - - - - In keeping with the XML Schema WG's standard versioning - policy, this schema document will persist at - http://www.w3.org/2001/03/xml.xsd. - At the date of issue it can also be found at - http://www.w3.org/2001/xml.xsd. - The schema document at that URI may however change in the future, - in order to remain compatible with the latest version of XML Schema - itself. In other words, if the XML Schema namespace changes, the version - of this document at - http://www.w3.org/2001/xml.xsd will change - accordingly; the version at - http://www.w3.org/2001/03/xml.xsd will not change. - - - - - - In due course, we should install the relevant ISO 2- and 3-letter - codes as the enumerated possible values . . . - - - - - - - - - - - - - - - See http://www.w3.org/TR/xmlbase/ for - information about this attribute. - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ecma/fouth-edition/opc-contentTypes.xsd b/.agents/skills/docx/scripts/office/schemas/ecma/fouth-edition/opc-contentTypes.xsd deleted file mode 100644 index a6de9d273..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ecma/fouth-edition/opc-contentTypes.xsd +++ /dev/null @@ -1,42 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ecma/fouth-edition/opc-coreProperties.xsd b/.agents/skills/docx/scripts/office/schemas/ecma/fouth-edition/opc-coreProperties.xsd deleted file mode 100644 index 10e978b66..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ecma/fouth-edition/opc-coreProperties.xsd +++ /dev/null @@ -1,50 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ecma/fouth-edition/opc-digSig.xsd b/.agents/skills/docx/scripts/office/schemas/ecma/fouth-edition/opc-digSig.xsd deleted file mode 100644 index 4248bf7a3..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ecma/fouth-edition/opc-digSig.xsd +++ /dev/null @@ -1,49 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/ecma/fouth-edition/opc-relationships.xsd b/.agents/skills/docx/scripts/office/schemas/ecma/fouth-edition/opc-relationships.xsd deleted file mode 100644 index 564974671..000000000 --- a/.agents/skills/docx/scripts/office/schemas/ecma/fouth-edition/opc-relationships.xsd +++ /dev/null @@ -1,33 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/mce/mc.xsd b/.agents/skills/docx/scripts/office/schemas/mce/mc.xsd deleted file mode 100644 index ef725457c..000000000 --- a/.agents/skills/docx/scripts/office/schemas/mce/mc.xsd +++ /dev/null @@ -1,75 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/microsoft/wml-2010.xsd b/.agents/skills/docx/scripts/office/schemas/microsoft/wml-2010.xsd deleted file mode 100644 index f65f77773..000000000 --- a/.agents/skills/docx/scripts/office/schemas/microsoft/wml-2010.xsd +++ /dev/null @@ -1,560 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/microsoft/wml-2012.xsd b/.agents/skills/docx/scripts/office/schemas/microsoft/wml-2012.xsd deleted file mode 100644 index 6b00755a9..000000000 --- a/.agents/skills/docx/scripts/office/schemas/microsoft/wml-2012.xsd +++ /dev/null @@ -1,67 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/microsoft/wml-2018.xsd b/.agents/skills/docx/scripts/office/schemas/microsoft/wml-2018.xsd deleted file mode 100644 index f321d333a..000000000 --- a/.agents/skills/docx/scripts/office/schemas/microsoft/wml-2018.xsd +++ /dev/null @@ -1,14 +0,0 @@ - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/microsoft/wml-cex-2018.xsd b/.agents/skills/docx/scripts/office/schemas/microsoft/wml-cex-2018.xsd deleted file mode 100644 index 364c6a9b8..000000000 --- a/.agents/skills/docx/scripts/office/schemas/microsoft/wml-cex-2018.xsd +++ /dev/null @@ -1,20 +0,0 @@ - - - - - - - - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/microsoft/wml-cid-2016.xsd b/.agents/skills/docx/scripts/office/schemas/microsoft/wml-cid-2016.xsd deleted file mode 100644 index fed9d15b7..000000000 --- a/.agents/skills/docx/scripts/office/schemas/microsoft/wml-cid-2016.xsd +++ /dev/null @@ -1,13 +0,0 @@ - - - - - - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/microsoft/wml-sdtdatahash-2020.xsd b/.agents/skills/docx/scripts/office/schemas/microsoft/wml-sdtdatahash-2020.xsd deleted file mode 100644 index 680cf1540..000000000 --- a/.agents/skills/docx/scripts/office/schemas/microsoft/wml-sdtdatahash-2020.xsd +++ /dev/null @@ -1,4 +0,0 @@ - - - - diff --git a/.agents/skills/docx/scripts/office/schemas/microsoft/wml-symex-2015.xsd b/.agents/skills/docx/scripts/office/schemas/microsoft/wml-symex-2015.xsd deleted file mode 100644 index 89ada9083..000000000 --- a/.agents/skills/docx/scripts/office/schemas/microsoft/wml-symex-2015.xsd +++ /dev/null @@ -1,8 +0,0 @@ - - - - - - - - diff --git a/.agents/skills/docx/scripts/office/soffice.py b/.agents/skills/docx/scripts/office/soffice.py deleted file mode 100644 index c7f7e3289..000000000 --- a/.agents/skills/docx/scripts/office/soffice.py +++ /dev/null @@ -1,183 +0,0 @@ -""" -Helper for running LibreOffice (soffice) in environments where AF_UNIX -sockets may be blocked (e.g., sandboxed VMs). Detects the restriction -at runtime and applies an LD_PRELOAD shim if needed. - -Usage: - from office.soffice import run_soffice, get_soffice_env - - # Option 1 – run soffice directly - result = run_soffice(["--headless", "--convert-to", "pdf", "input.docx"]) - - # Option 2 – get env dict for your own subprocess calls - env = get_soffice_env() - subprocess.run(["soffice", ...], env=env) -""" - -import os -import socket -import subprocess -import tempfile -from pathlib import Path - - -def get_soffice_env() -> dict: - env = os.environ.copy() - env["SAL_USE_VCLPLUGIN"] = "svp" - - if _needs_shim(): - shim = _ensure_shim() - env["LD_PRELOAD"] = str(shim) - - return env - - -def run_soffice(args: list[str], **kwargs) -> subprocess.CompletedProcess: - env = get_soffice_env() - return subprocess.run(["soffice"] + args, env=env, **kwargs) - - - -_SHIM_SO = Path(tempfile.gettempdir()) / "lo_socket_shim.so" - - -def _needs_shim() -> bool: - try: - s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) - s.close() - return False - except OSError: - return True - - -def _ensure_shim() -> Path: - if _SHIM_SO.exists(): - return _SHIM_SO - - src = Path(tempfile.gettempdir()) / "lo_socket_shim.c" - src.write_text(_SHIM_SOURCE) - subprocess.run( - ["gcc", "-shared", "-fPIC", "-o", str(_SHIM_SO), str(src), "-ldl"], - check=True, - capture_output=True, - ) - src.unlink() - return _SHIM_SO - - - -_SHIM_SOURCE = r""" -#define _GNU_SOURCE -#include -#include -#include -#include -#include -#include -#include - -static int (*real_socket)(int, int, int); -static int (*real_socketpair)(int, int, int, int[2]); -static int (*real_listen)(int, int); -static int (*real_accept)(int, struct sockaddr *, socklen_t *); -static int (*real_close)(int); -static int (*real_read)(int, void *, size_t); - -/* Per-FD bookkeeping (FDs >= 1024 are passed through unshimmed). */ -static int is_shimmed[1024]; -static int peer_of[1024]; -static int wake_r[1024]; /* accept() blocks reading this */ -static int wake_w[1024]; /* close() writes to this */ -static int listener_fd = -1; /* FD that received listen() */ - -__attribute__((constructor)) -static void init(void) { - real_socket = dlsym(RTLD_NEXT, "socket"); - real_socketpair = dlsym(RTLD_NEXT, "socketpair"); - real_listen = dlsym(RTLD_NEXT, "listen"); - real_accept = dlsym(RTLD_NEXT, "accept"); - real_close = dlsym(RTLD_NEXT, "close"); - real_read = dlsym(RTLD_NEXT, "read"); - for (int i = 0; i < 1024; i++) { - peer_of[i] = -1; - wake_r[i] = -1; - wake_w[i] = -1; - } -} - -/* ---- socket ---------------------------------------------------------- */ -int socket(int domain, int type, int protocol) { - if (domain == AF_UNIX) { - int fd = real_socket(domain, type, protocol); - if (fd >= 0) return fd; - /* socket(AF_UNIX) blocked – fall back to socketpair(). */ - int sv[2]; - if (real_socketpair(domain, type, protocol, sv) == 0) { - if (sv[0] >= 0 && sv[0] < 1024) { - is_shimmed[sv[0]] = 1; - peer_of[sv[0]] = sv[1]; - int wp[2]; - if (pipe(wp) == 0) { - wake_r[sv[0]] = wp[0]; - wake_w[sv[0]] = wp[1]; - } - } - return sv[0]; - } - errno = EPERM; - return -1; - } - return real_socket(domain, type, protocol); -} - -/* ---- listen ---------------------------------------------------------- */ -int listen(int sockfd, int backlog) { - if (sockfd >= 0 && sockfd < 1024 && is_shimmed[sockfd]) { - listener_fd = sockfd; - return 0; - } - return real_listen(sockfd, backlog); -} - -/* ---- accept ---------------------------------------------------------- */ -int accept(int sockfd, struct sockaddr *addr, socklen_t *addrlen) { - if (sockfd >= 0 && sockfd < 1024 && is_shimmed[sockfd]) { - /* Block until close() writes to the wake pipe. */ - if (wake_r[sockfd] >= 0) { - char buf; - real_read(wake_r[sockfd], &buf, 1); - } - errno = ECONNABORTED; - return -1; - } - return real_accept(sockfd, addr, addrlen); -} - -/* ---- close ----------------------------------------------------------- */ -int close(int fd) { - if (fd >= 0 && fd < 1024 && is_shimmed[fd]) { - int was_listener = (fd == listener_fd); - is_shimmed[fd] = 0; - - if (wake_w[fd] >= 0) { /* unblock accept() */ - char c = 0; - write(wake_w[fd], &c, 1); - real_close(wake_w[fd]); - wake_w[fd] = -1; - } - if (wake_r[fd] >= 0) { real_close(wake_r[fd]); wake_r[fd] = -1; } - if (peer_of[fd] >= 0) { real_close(peer_of[fd]); peer_of[fd] = -1; } - - if (was_listener) - _exit(0); /* conversion done – exit */ - } - return real_close(fd); -} -""" - - - -if __name__ == "__main__": - import sys - result = run_soffice(sys.argv[1:]) - sys.exit(result.returncode) diff --git a/.agents/skills/docx/scripts/office/unpack.py b/.agents/skills/docx/scripts/office/unpack.py deleted file mode 100755 index 00152533a..000000000 --- a/.agents/skills/docx/scripts/office/unpack.py +++ /dev/null @@ -1,132 +0,0 @@ -"""Unpack Office files (DOCX, PPTX, XLSX) for editing. - -Extracts the ZIP archive, pretty-prints XML files, and optionally: -- Merges adjacent runs with identical formatting (DOCX only) -- Simplifies adjacent tracked changes from same author (DOCX only) - -Usage: - python unpack.py [options] - -Examples: - python unpack.py document.docx unpacked/ - python unpack.py presentation.pptx unpacked/ - python unpack.py document.docx unpacked/ --merge-runs false -""" - -import argparse -import sys -import zipfile -from pathlib import Path - -import defusedxml.minidom - -from helpers.merge_runs import merge_runs as do_merge_runs -from helpers.simplify_redlines import simplify_redlines as do_simplify_redlines - -SMART_QUOTE_REPLACEMENTS = { - "\u201c": "“", - "\u201d": "”", - "\u2018": "‘", - "\u2019": "’", -} - - -def unpack( - input_file: str, - output_directory: str, - merge_runs: bool = True, - simplify_redlines: bool = True, -) -> tuple[None, str]: - input_path = Path(input_file) - output_path = Path(output_directory) - suffix = input_path.suffix.lower() - - if not input_path.exists(): - return None, f"Error: {input_file} does not exist" - - if suffix not in {".docx", ".pptx", ".xlsx"}: - return None, f"Error: {input_file} must be a .docx, .pptx, or .xlsx file" - - try: - output_path.mkdir(parents=True, exist_ok=True) - - with zipfile.ZipFile(input_path, "r") as zf: - zf.extractall(output_path) - - xml_files = list(output_path.rglob("*.xml")) + list(output_path.rglob("*.rels")) - for xml_file in xml_files: - _pretty_print_xml(xml_file) - - message = f"Unpacked {input_file} ({len(xml_files)} XML files)" - - if suffix == ".docx": - if simplify_redlines: - simplify_count, _ = do_simplify_redlines(str(output_path)) - message += f", simplified {simplify_count} tracked changes" - - if merge_runs: - merge_count, _ = do_merge_runs(str(output_path)) - message += f", merged {merge_count} runs" - - for xml_file in xml_files: - _escape_smart_quotes(xml_file) - - return None, message - - except zipfile.BadZipFile: - return None, f"Error: {input_file} is not a valid Office file" - except Exception as e: - return None, f"Error unpacking: {e}" - - -def _pretty_print_xml(xml_file: Path) -> None: - try: - content = xml_file.read_text(encoding="utf-8") - dom = defusedxml.minidom.parseString(content) - xml_file.write_bytes(dom.toprettyxml(indent=" ", encoding="utf-8")) - except Exception: - pass - - -def _escape_smart_quotes(xml_file: Path) -> None: - try: - content = xml_file.read_text(encoding="utf-8") - for char, entity in SMART_QUOTE_REPLACEMENTS.items(): - content = content.replace(char, entity) - xml_file.write_text(content, encoding="utf-8") - except Exception: - pass - - -if __name__ == "__main__": - parser = argparse.ArgumentParser( - description="Unpack an Office file (DOCX, PPTX, XLSX) for editing" - ) - parser.add_argument("input_file", help="Office file to unpack") - parser.add_argument("output_directory", help="Output directory") - parser.add_argument( - "--merge-runs", - type=lambda x: x.lower() == "true", - default=True, - metavar="true|false", - help="Merge adjacent runs with identical formatting (DOCX only, default: true)", - ) - parser.add_argument( - "--simplify-redlines", - type=lambda x: x.lower() == "true", - default=True, - metavar="true|false", - help="Merge adjacent tracked changes from same author (DOCX only, default: true)", - ) - args = parser.parse_args() - - _, message = unpack( - args.input_file, - args.output_directory, - merge_runs=args.merge_runs, - simplify_redlines=args.simplify_redlines, - ) - print(message) - - if "Error" in message: - sys.exit(1) diff --git a/.agents/skills/docx/scripts/office/validate.py b/.agents/skills/docx/scripts/office/validate.py deleted file mode 100755 index 03b01f6e3..000000000 --- a/.agents/skills/docx/scripts/office/validate.py +++ /dev/null @@ -1,111 +0,0 @@ -""" -Command line tool to validate Office document XML files against XSD schemas and tracked changes. - -Usage: - python validate.py [--original ] [--auto-repair] [--author NAME] - -The first argument can be either: -- An unpacked directory containing the Office document XML files -- A packed Office file (.docx/.pptx/.xlsx) which will be unpacked to a temp directory - -Auto-repair fixes: -- paraId/durableId values that exceed OOXML limits -- Missing xml:space="preserve" on w:t elements with whitespace -""" - -import argparse -import sys -import tempfile -import zipfile -from pathlib import Path - -from validators import DOCXSchemaValidator, PPTXSchemaValidator, RedliningValidator - - -def main(): - parser = argparse.ArgumentParser(description="Validate Office document XML files") - parser.add_argument( - "path", - help="Path to unpacked directory or packed Office file (.docx/.pptx/.xlsx)", - ) - parser.add_argument( - "--original", - required=False, - default=None, - help="Path to original file (.docx/.pptx/.xlsx). If omitted, all XSD errors are reported and redlining validation is skipped.", - ) - parser.add_argument( - "-v", - "--verbose", - action="store_true", - help="Enable verbose output", - ) - parser.add_argument( - "--auto-repair", - action="store_true", - help="Automatically repair common issues (hex IDs, whitespace preservation)", - ) - parser.add_argument( - "--author", - default="Claude", - help="Author name for redlining validation (default: Claude)", - ) - args = parser.parse_args() - - path = Path(args.path) - assert path.exists(), f"Error: {path} does not exist" - - original_file = None - if args.original: - original_file = Path(args.original) - assert original_file.is_file(), f"Error: {original_file} is not a file" - assert original_file.suffix.lower() in [".docx", ".pptx", ".xlsx"], ( - f"Error: {original_file} must be a .docx, .pptx, or .xlsx file" - ) - - file_extension = (original_file or path).suffix.lower() - assert file_extension in [".docx", ".pptx", ".xlsx"], ( - f"Error: Cannot determine file type from {path}. Use --original or provide a .docx/.pptx/.xlsx file." - ) - - if path.is_file() and path.suffix.lower() in [".docx", ".pptx", ".xlsx"]: - temp_dir = tempfile.mkdtemp() - with zipfile.ZipFile(path, "r") as zf: - zf.extractall(temp_dir) - unpacked_dir = Path(temp_dir) - else: - assert path.is_dir(), f"Error: {path} is not a directory or Office file" - unpacked_dir = path - - match file_extension: - case ".docx": - validators = [ - DOCXSchemaValidator(unpacked_dir, original_file, verbose=args.verbose), - ] - if original_file: - validators.append( - RedliningValidator(unpacked_dir, original_file, verbose=args.verbose, author=args.author) - ) - case ".pptx": - validators = [ - PPTXSchemaValidator(unpacked_dir, original_file, verbose=args.verbose), - ] - case _: - print(f"Error: Validation not supported for file type {file_extension}") - sys.exit(1) - - if args.auto_repair: - total_repairs = sum(v.repair() for v in validators) - if total_repairs: - print(f"Auto-repaired {total_repairs} issue(s)") - - success = all(v.validate() for v in validators) - - if success: - print("All validations PASSED!") - - sys.exit(0 if success else 1) - - -if __name__ == "__main__": - main() diff --git a/.agents/skills/docx/scripts/office/validators/__init__.py b/.agents/skills/docx/scripts/office/validators/__init__.py deleted file mode 100644 index db092ece7..000000000 --- a/.agents/skills/docx/scripts/office/validators/__init__.py +++ /dev/null @@ -1,15 +0,0 @@ -""" -Validation modules for Word document processing. -""" - -from .base import BaseSchemaValidator -from .docx import DOCXSchemaValidator -from .pptx import PPTXSchemaValidator -from .redlining import RedliningValidator - -__all__ = [ - "BaseSchemaValidator", - "DOCXSchemaValidator", - "PPTXSchemaValidator", - "RedliningValidator", -] diff --git a/.agents/skills/docx/scripts/office/validators/base.py b/.agents/skills/docx/scripts/office/validators/base.py deleted file mode 100644 index db4a06a22..000000000 --- a/.agents/skills/docx/scripts/office/validators/base.py +++ /dev/null @@ -1,847 +0,0 @@ -""" -Base validator with common validation logic for document files. -""" - -import re -from pathlib import Path - -import defusedxml.minidom -import lxml.etree - - -class BaseSchemaValidator: - - IGNORED_VALIDATION_ERRORS = [ - "hyphenationZone", - "purl.org/dc/terms", - ] - - UNIQUE_ID_REQUIREMENTS = { - "comment": ("id", "file"), - "commentrangestart": ("id", "file"), - "commentrangeend": ("id", "file"), - "bookmarkstart": ("id", "file"), - "bookmarkend": ("id", "file"), - "sldid": ("id", "file"), - "sldmasterid": ("id", "global"), - "sldlayoutid": ("id", "global"), - "cm": ("authorid", "file"), - "sheet": ("sheetid", "file"), - "definedname": ("id", "file"), - "cxnsp": ("id", "file"), - "sp": ("id", "file"), - "pic": ("id", "file"), - "grpsp": ("id", "file"), - } - - EXCLUDED_ID_CONTAINERS = { - "sectionlst", - } - - ELEMENT_RELATIONSHIP_TYPES = {} - - SCHEMA_MAPPINGS = { - "word": "ISO-IEC29500-4_2016/wml.xsd", - "ppt": "ISO-IEC29500-4_2016/pml.xsd", - "xl": "ISO-IEC29500-4_2016/sml.xsd", - "[Content_Types].xml": "ecma/fouth-edition/opc-contentTypes.xsd", - "app.xml": "ISO-IEC29500-4_2016/shared-documentPropertiesExtended.xsd", - "core.xml": "ecma/fouth-edition/opc-coreProperties.xsd", - "custom.xml": "ISO-IEC29500-4_2016/shared-documentPropertiesCustom.xsd", - ".rels": "ecma/fouth-edition/opc-relationships.xsd", - "people.xml": "microsoft/wml-2012.xsd", - "commentsIds.xml": "microsoft/wml-cid-2016.xsd", - "commentsExtensible.xml": "microsoft/wml-cex-2018.xsd", - "commentsExtended.xml": "microsoft/wml-2012.xsd", - "chart": "ISO-IEC29500-4_2016/dml-chart.xsd", - "theme": "ISO-IEC29500-4_2016/dml-main.xsd", - "drawing": "ISO-IEC29500-4_2016/dml-main.xsd", - } - - MC_NAMESPACE = "http://schemas.openxmlformats.org/markup-compatibility/2006" - XML_NAMESPACE = "http://www.w3.org/XML/1998/namespace" - - PACKAGE_RELATIONSHIPS_NAMESPACE = ( - "http://schemas.openxmlformats.org/package/2006/relationships" - ) - OFFICE_RELATIONSHIPS_NAMESPACE = ( - "http://schemas.openxmlformats.org/officeDocument/2006/relationships" - ) - CONTENT_TYPES_NAMESPACE = ( - "http://schemas.openxmlformats.org/package/2006/content-types" - ) - - MAIN_CONTENT_FOLDERS = {"word", "ppt", "xl"} - - OOXML_NAMESPACES = { - "http://schemas.openxmlformats.org/officeDocument/2006/math", - "http://schemas.openxmlformats.org/officeDocument/2006/relationships", - "http://schemas.openxmlformats.org/schemaLibrary/2006/main", - "http://schemas.openxmlformats.org/drawingml/2006/main", - "http://schemas.openxmlformats.org/drawingml/2006/chart", - "http://schemas.openxmlformats.org/drawingml/2006/chartDrawing", - "http://schemas.openxmlformats.org/drawingml/2006/diagram", - "http://schemas.openxmlformats.org/drawingml/2006/picture", - "http://schemas.openxmlformats.org/drawingml/2006/spreadsheetDrawing", - "http://schemas.openxmlformats.org/drawingml/2006/wordprocessingDrawing", - "http://schemas.openxmlformats.org/wordprocessingml/2006/main", - "http://schemas.openxmlformats.org/presentationml/2006/main", - "http://schemas.openxmlformats.org/spreadsheetml/2006/main", - "http://schemas.openxmlformats.org/officeDocument/2006/sharedTypes", - "http://www.w3.org/XML/1998/namespace", - } - - def __init__(self, unpacked_dir, original_file=None, verbose=False): - self.unpacked_dir = Path(unpacked_dir).resolve() - self.original_file = Path(original_file) if original_file else None - self.verbose = verbose - - self.schemas_dir = Path(__file__).parent.parent / "schemas" - - patterns = ["*.xml", "*.rels"] - self.xml_files = [ - f for pattern in patterns for f in self.unpacked_dir.rglob(pattern) - ] - - if not self.xml_files: - print(f"Warning: No XML files found in {self.unpacked_dir}") - - def validate(self): - raise NotImplementedError("Subclasses must implement the validate method") - - def repair(self) -> int: - return self.repair_whitespace_preservation() - - def repair_whitespace_preservation(self) -> int: - repairs = 0 - - for xml_file in self.xml_files: - try: - content = xml_file.read_text(encoding="utf-8") - dom = defusedxml.minidom.parseString(content) - modified = False - - for elem in dom.getElementsByTagName("*"): - if elem.tagName.endswith(":t") and elem.firstChild: - text = elem.firstChild.nodeValue - if text and (text.startswith((' ', '\t')) or text.endswith((' ', '\t'))): - if elem.getAttribute("xml:space") != "preserve": - elem.setAttribute("xml:space", "preserve") - text_preview = repr(text[:30]) + "..." if len(text) > 30 else repr(text) - print(f" Repaired: {xml_file.name}: Added xml:space='preserve' to {elem.tagName}: {text_preview}") - repairs += 1 - modified = True - - if modified: - xml_file.write_bytes(dom.toxml(encoding="UTF-8")) - - except Exception: - pass - - return repairs - - def validate_xml(self): - errors = [] - - for xml_file in self.xml_files: - try: - lxml.etree.parse(str(xml_file)) - except lxml.etree.XMLSyntaxError as e: - errors.append( - f" {xml_file.relative_to(self.unpacked_dir)}: " - f"Line {e.lineno}: {e.msg}" - ) - except Exception as e: - errors.append( - f" {xml_file.relative_to(self.unpacked_dir)}: " - f"Unexpected error: {str(e)}" - ) - - if errors: - print(f"FAILED - Found {len(errors)} XML violations:") - for error in errors: - print(error) - return False - else: - if self.verbose: - print("PASSED - All XML files are well-formed") - return True - - def validate_namespaces(self): - errors = [] - - for xml_file in self.xml_files: - try: - root = lxml.etree.parse(str(xml_file)).getroot() - declared = set(root.nsmap.keys()) - {None} - - for attr_val in [ - v for k, v in root.attrib.items() if k.endswith("Ignorable") - ]: - undeclared = set(attr_val.split()) - declared - errors.extend( - f" {xml_file.relative_to(self.unpacked_dir)}: " - f"Namespace '{ns}' in Ignorable but not declared" - for ns in undeclared - ) - except lxml.etree.XMLSyntaxError: - continue - - if errors: - print(f"FAILED - {len(errors)} namespace issues:") - for error in errors: - print(error) - return False - if self.verbose: - print("PASSED - All namespace prefixes properly declared") - return True - - def validate_unique_ids(self): - errors = [] - global_ids = {} - - for xml_file in self.xml_files: - try: - root = lxml.etree.parse(str(xml_file)).getroot() - file_ids = {} - - mc_elements = root.xpath( - ".//mc:AlternateContent", namespaces={"mc": self.MC_NAMESPACE} - ) - for elem in mc_elements: - elem.getparent().remove(elem) - - for elem in root.iter(): - tag = ( - elem.tag.split("}")[-1].lower() - if "}" in elem.tag - else elem.tag.lower() - ) - - if tag in self.UNIQUE_ID_REQUIREMENTS: - in_excluded_container = any( - ancestor.tag.split("}")[-1].lower() in self.EXCLUDED_ID_CONTAINERS - for ancestor in elem.iterancestors() - ) - if in_excluded_container: - continue - - attr_name, scope = self.UNIQUE_ID_REQUIREMENTS[tag] - - id_value = None - for attr, value in elem.attrib.items(): - attr_local = ( - attr.split("}")[-1].lower() - if "}" in attr - else attr.lower() - ) - if attr_local == attr_name: - id_value = value - break - - if id_value is not None: - if scope == "global": - if id_value in global_ids: - prev_file, prev_line, prev_tag = global_ids[ - id_value - ] - errors.append( - f" {xml_file.relative_to(self.unpacked_dir)}: " - f"Line {elem.sourceline}: Global ID '{id_value}' in <{tag}> " - f"already used in {prev_file} at line {prev_line} in <{prev_tag}>" - ) - else: - global_ids[id_value] = ( - xml_file.relative_to(self.unpacked_dir), - elem.sourceline, - tag, - ) - elif scope == "file": - key = (tag, attr_name) - if key not in file_ids: - file_ids[key] = {} - - if id_value in file_ids[key]: - prev_line = file_ids[key][id_value] - errors.append( - f" {xml_file.relative_to(self.unpacked_dir)}: " - f"Line {elem.sourceline}: Duplicate {attr_name}='{id_value}' in <{tag}> " - f"(first occurrence at line {prev_line})" - ) - else: - file_ids[key][id_value] = elem.sourceline - - except (lxml.etree.XMLSyntaxError, Exception) as e: - errors.append( - f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}" - ) - - if errors: - print(f"FAILED - Found {len(errors)} ID uniqueness violations:") - for error in errors: - print(error) - return False - else: - if self.verbose: - print("PASSED - All required IDs are unique") - return True - - def validate_file_references(self): - errors = [] - - rels_files = list(self.unpacked_dir.rglob("*.rels")) - - if not rels_files: - if self.verbose: - print("PASSED - No .rels files found") - return True - - all_files = [] - for file_path in self.unpacked_dir.rglob("*"): - if ( - file_path.is_file() - and file_path.name != "[Content_Types].xml" - and not file_path.name.endswith(".rels") - ): - all_files.append(file_path.resolve()) - - all_referenced_files = set() - - if self.verbose: - print( - f"Found {len(rels_files)} .rels files and {len(all_files)} target files" - ) - - for rels_file in rels_files: - try: - rels_root = lxml.etree.parse(str(rels_file)).getroot() - - rels_dir = rels_file.parent - - referenced_files = set() - broken_refs = [] - - for rel in rels_root.findall( - ".//ns:Relationship", - namespaces={"ns": self.PACKAGE_RELATIONSHIPS_NAMESPACE}, - ): - target = rel.get("Target") - if target and not target.startswith( - ("http", "mailto:") - ): - if target.startswith("/"): - target_path = self.unpacked_dir / target.lstrip("/") - elif rels_file.name == ".rels": - target_path = self.unpacked_dir / target - else: - base_dir = rels_dir.parent - target_path = base_dir / target - - try: - target_path = target_path.resolve() - if target_path.exists() and target_path.is_file(): - referenced_files.add(target_path) - all_referenced_files.add(target_path) - else: - broken_refs.append((target, rel.sourceline)) - except (OSError, ValueError): - broken_refs.append((target, rel.sourceline)) - - if broken_refs: - rel_path = rels_file.relative_to(self.unpacked_dir) - for broken_ref, line_num in broken_refs: - errors.append( - f" {rel_path}: Line {line_num}: Broken reference to {broken_ref}" - ) - - except Exception as e: - rel_path = rels_file.relative_to(self.unpacked_dir) - errors.append(f" Error parsing {rel_path}: {e}") - - unreferenced_files = set(all_files) - all_referenced_files - - if unreferenced_files: - for unref_file in sorted(unreferenced_files): - unref_rel_path = unref_file.relative_to(self.unpacked_dir) - errors.append(f" Unreferenced file: {unref_rel_path}") - - if errors: - print(f"FAILED - Found {len(errors)} relationship validation errors:") - for error in errors: - print(error) - print( - "CRITICAL: These errors will cause the document to appear corrupt. " - + "Broken references MUST be fixed, " - + "and unreferenced files MUST be referenced or removed." - ) - return False - else: - if self.verbose: - print( - "PASSED - All references are valid and all files are properly referenced" - ) - return True - - def validate_all_relationship_ids(self): - import lxml.etree - - errors = [] - - for xml_file in self.xml_files: - if xml_file.suffix == ".rels": - continue - - rels_dir = xml_file.parent / "_rels" - rels_file = rels_dir / f"{xml_file.name}.rels" - - if not rels_file.exists(): - continue - - try: - rels_root = lxml.etree.parse(str(rels_file)).getroot() - rid_to_type = {} - - for rel in rels_root.findall( - f".//{{{self.PACKAGE_RELATIONSHIPS_NAMESPACE}}}Relationship" - ): - rid = rel.get("Id") - rel_type = rel.get("Type", "") - if rid: - if rid in rid_to_type: - rels_rel_path = rels_file.relative_to(self.unpacked_dir) - errors.append( - f" {rels_rel_path}: Line {rel.sourceline}: " - f"Duplicate relationship ID '{rid}' (IDs must be unique)" - ) - type_name = ( - rel_type.split("/")[-1] if "/" in rel_type else rel_type - ) - rid_to_type[rid] = type_name - - xml_root = lxml.etree.parse(str(xml_file)).getroot() - - r_ns = self.OFFICE_RELATIONSHIPS_NAMESPACE - rid_attrs_to_check = ["id", "embed", "link"] - for elem in xml_root.iter(): - for attr_name in rid_attrs_to_check: - rid_attr = elem.get(f"{{{r_ns}}}{attr_name}") - if not rid_attr: - continue - xml_rel_path = xml_file.relative_to(self.unpacked_dir) - elem_name = ( - elem.tag.split("}")[-1] if "}" in elem.tag else elem.tag - ) - - if rid_attr not in rid_to_type: - errors.append( - f" {xml_rel_path}: Line {elem.sourceline}: " - f"<{elem_name}> r:{attr_name} references non-existent relationship '{rid_attr}' " - f"(valid IDs: {', '.join(sorted(rid_to_type.keys())[:5])}{'...' if len(rid_to_type) > 5 else ''})" - ) - elif attr_name == "id" and self.ELEMENT_RELATIONSHIP_TYPES: - expected_type = self._get_expected_relationship_type( - elem_name - ) - if expected_type: - actual_type = rid_to_type[rid_attr] - if expected_type not in actual_type.lower(): - errors.append( - f" {xml_rel_path}: Line {elem.sourceline}: " - f"<{elem_name}> references '{rid_attr}' which points to '{actual_type}' " - f"but should point to a '{expected_type}' relationship" - ) - - except Exception as e: - xml_rel_path = xml_file.relative_to(self.unpacked_dir) - errors.append(f" Error processing {xml_rel_path}: {e}") - - if errors: - print(f"FAILED - Found {len(errors)} relationship ID reference errors:") - for error in errors: - print(error) - print("\nThese ID mismatches will cause the document to appear corrupt!") - return False - else: - if self.verbose: - print("PASSED - All relationship ID references are valid") - return True - - def _get_expected_relationship_type(self, element_name): - elem_lower = element_name.lower() - - if elem_lower in self.ELEMENT_RELATIONSHIP_TYPES: - return self.ELEMENT_RELATIONSHIP_TYPES[elem_lower] - - if elem_lower.endswith("id") and len(elem_lower) > 2: - prefix = elem_lower[:-2] - if prefix.endswith("master"): - return prefix.lower() - elif prefix.endswith("layout"): - return prefix.lower() - else: - if prefix == "sld": - return "slide" - return prefix.lower() - - if elem_lower.endswith("reference") and len(elem_lower) > 9: - prefix = elem_lower[:-9] - return prefix.lower() - - return None - - def validate_content_types(self): - errors = [] - - content_types_file = self.unpacked_dir / "[Content_Types].xml" - if not content_types_file.exists(): - print("FAILED - [Content_Types].xml file not found") - return False - - try: - root = lxml.etree.parse(str(content_types_file)).getroot() - declared_parts = set() - declared_extensions = set() - - for override in root.findall( - f".//{{{self.CONTENT_TYPES_NAMESPACE}}}Override" - ): - part_name = override.get("PartName") - if part_name is not None: - declared_parts.add(part_name.lstrip("/")) - - for default in root.findall( - f".//{{{self.CONTENT_TYPES_NAMESPACE}}}Default" - ): - extension = default.get("Extension") - if extension is not None: - declared_extensions.add(extension.lower()) - - declarable_roots = { - "sld", - "sldLayout", - "sldMaster", - "presentation", - "document", - "workbook", - "worksheet", - "theme", - } - - media_extensions = { - "png": "image/png", - "jpg": "image/jpeg", - "jpeg": "image/jpeg", - "gif": "image/gif", - "bmp": "image/bmp", - "tiff": "image/tiff", - "wmf": "image/x-wmf", - "emf": "image/x-emf", - } - - all_files = list(self.unpacked_dir.rglob("*")) - all_files = [f for f in all_files if f.is_file()] - - for xml_file in self.xml_files: - path_str = str(xml_file.relative_to(self.unpacked_dir)).replace( - "\\", "/" - ) - - if any( - skip in path_str - for skip in [".rels", "[Content_Types]", "docProps/", "_rels/"] - ): - continue - - try: - root_tag = lxml.etree.parse(str(xml_file)).getroot().tag - root_name = root_tag.split("}")[-1] if "}" in root_tag else root_tag - - if root_name in declarable_roots and path_str not in declared_parts: - errors.append( - f" {path_str}: File with <{root_name}> root not declared in [Content_Types].xml" - ) - - except Exception: - continue - - for file_path in all_files: - if file_path.suffix.lower() in {".xml", ".rels"}: - continue - if file_path.name == "[Content_Types].xml": - continue - if "_rels" in file_path.parts or "docProps" in file_path.parts: - continue - - extension = file_path.suffix.lstrip(".").lower() - if extension and extension not in declared_extensions: - if extension in media_extensions: - relative_path = file_path.relative_to(self.unpacked_dir) - errors.append( - f' {relative_path}: File with extension \'{extension}\' not declared in [Content_Types].xml - should add: ' - ) - - except Exception as e: - errors.append(f" Error parsing [Content_Types].xml: {e}") - - if errors: - print(f"FAILED - Found {len(errors)} content type declaration errors:") - for error in errors: - print(error) - return False - else: - if self.verbose: - print( - "PASSED - All content files are properly declared in [Content_Types].xml" - ) - return True - - def validate_file_against_xsd(self, xml_file, verbose=False): - xml_file = Path(xml_file).resolve() - unpacked_dir = self.unpacked_dir.resolve() - - is_valid, current_errors = self._validate_single_file_xsd( - xml_file, unpacked_dir - ) - - if is_valid is None: - return None, set() - elif is_valid: - return True, set() - - original_errors = self._get_original_file_errors(xml_file) - - assert current_errors is not None - new_errors = current_errors - original_errors - - new_errors = { - e for e in new_errors - if not any(pattern in e for pattern in self.IGNORED_VALIDATION_ERRORS) - } - - if new_errors: - if verbose: - relative_path = xml_file.relative_to(unpacked_dir) - print(f"FAILED - {relative_path}: {len(new_errors)} new error(s)") - for error in list(new_errors)[:3]: - truncated = error[:250] + "..." if len(error) > 250 else error - print(f" - {truncated}") - return False, new_errors - else: - if verbose: - print( - f"PASSED - No new errors (original had {len(current_errors)} errors)" - ) - return True, set() - - def validate_against_xsd(self): - new_errors = [] - original_error_count = 0 - valid_count = 0 - skipped_count = 0 - - for xml_file in self.xml_files: - relative_path = str(xml_file.relative_to(self.unpacked_dir)) - is_valid, new_file_errors = self.validate_file_against_xsd( - xml_file, verbose=False - ) - - if is_valid is None: - skipped_count += 1 - continue - elif is_valid and not new_file_errors: - valid_count += 1 - continue - elif is_valid: - original_error_count += 1 - valid_count += 1 - continue - - new_errors.append(f" {relative_path}: {len(new_file_errors)} new error(s)") - for error in list(new_file_errors)[:3]: - new_errors.append( - f" - {error[:250]}..." if len(error) > 250 else f" - {error}" - ) - - if self.verbose: - print(f"Validated {len(self.xml_files)} files:") - print(f" - Valid: {valid_count}") - print(f" - Skipped (no schema): {skipped_count}") - if original_error_count: - print(f" - With original errors (ignored): {original_error_count}") - print( - f" - With NEW errors: {len(new_errors) > 0 and len([e for e in new_errors if not e.startswith(' ')]) or 0}" - ) - - if new_errors: - print("\nFAILED - Found NEW validation errors:") - for error in new_errors: - print(error) - return False - else: - if self.verbose: - print("\nPASSED - No new XSD validation errors introduced") - return True - - def _get_schema_path(self, xml_file): - if xml_file.name in self.SCHEMA_MAPPINGS: - return self.schemas_dir / self.SCHEMA_MAPPINGS[xml_file.name] - - if xml_file.suffix == ".rels": - return self.schemas_dir / self.SCHEMA_MAPPINGS[".rels"] - - if "charts/" in str(xml_file) and xml_file.name.startswith("chart"): - return self.schemas_dir / self.SCHEMA_MAPPINGS["chart"] - - if "theme/" in str(xml_file) and xml_file.name.startswith("theme"): - return self.schemas_dir / self.SCHEMA_MAPPINGS["theme"] - - if xml_file.parent.name in self.MAIN_CONTENT_FOLDERS: - return self.schemas_dir / self.SCHEMA_MAPPINGS[xml_file.parent.name] - - return None - - def _clean_ignorable_namespaces(self, xml_doc): - xml_string = lxml.etree.tostring(xml_doc, encoding="unicode") - xml_copy = lxml.etree.fromstring(xml_string) - - for elem in xml_copy.iter(): - attrs_to_remove = [] - - for attr in elem.attrib: - if "{" in attr: - ns = attr.split("}")[0][1:] - if ns not in self.OOXML_NAMESPACES: - attrs_to_remove.append(attr) - - for attr in attrs_to_remove: - del elem.attrib[attr] - - self._remove_ignorable_elements(xml_copy) - - return lxml.etree.ElementTree(xml_copy) - - def _remove_ignorable_elements(self, root): - elements_to_remove = [] - - for elem in list(root): - if not hasattr(elem, "tag") or callable(elem.tag): - continue - - tag_str = str(elem.tag) - if tag_str.startswith("{"): - ns = tag_str.split("}")[0][1:] - if ns not in self.OOXML_NAMESPACES: - elements_to_remove.append(elem) - continue - - self._remove_ignorable_elements(elem) - - for elem in elements_to_remove: - root.remove(elem) - - def _preprocess_for_mc_ignorable(self, xml_doc): - root = xml_doc.getroot() - - if f"{{{self.MC_NAMESPACE}}}Ignorable" in root.attrib: - del root.attrib[f"{{{self.MC_NAMESPACE}}}Ignorable"] - - return xml_doc - - def _validate_single_file_xsd(self, xml_file, base_path): - schema_path = self._get_schema_path(xml_file) - if not schema_path: - return None, None - - try: - with open(schema_path, "rb") as xsd_file: - parser = lxml.etree.XMLParser() - xsd_doc = lxml.etree.parse( - xsd_file, parser=parser, base_url=str(schema_path) - ) - schema = lxml.etree.XMLSchema(xsd_doc) - - with open(xml_file, "r") as f: - xml_doc = lxml.etree.parse(f) - - xml_doc, _ = self._remove_template_tags_from_text_nodes(xml_doc) - xml_doc = self._preprocess_for_mc_ignorable(xml_doc) - - relative_path = xml_file.relative_to(base_path) - if ( - relative_path.parts - and relative_path.parts[0] in self.MAIN_CONTENT_FOLDERS - ): - xml_doc = self._clean_ignorable_namespaces(xml_doc) - - if schema.validate(xml_doc): - return True, set() - else: - errors = set() - for error in schema.error_log: - errors.add(error.message) - return False, errors - - except Exception as e: - return False, {str(e)} - - def _get_original_file_errors(self, xml_file): - if self.original_file is None: - return set() - - import tempfile - import zipfile - - xml_file = Path(xml_file).resolve() - unpacked_dir = self.unpacked_dir.resolve() - relative_path = xml_file.relative_to(unpacked_dir) - - with tempfile.TemporaryDirectory() as temp_dir: - temp_path = Path(temp_dir) - - with zipfile.ZipFile(self.original_file, "r") as zip_ref: - zip_ref.extractall(temp_path) - - original_xml_file = temp_path / relative_path - - if not original_xml_file.exists(): - return set() - - is_valid, errors = self._validate_single_file_xsd( - original_xml_file, temp_path - ) - return errors if errors else set() - - def _remove_template_tags_from_text_nodes(self, xml_doc): - warnings = [] - template_pattern = re.compile(r"\{\{[^}]*\}\}") - - xml_string = lxml.etree.tostring(xml_doc, encoding="unicode") - xml_copy = lxml.etree.fromstring(xml_string) - - def process_text_content(text, content_type): - if not text: - return text - matches = list(template_pattern.finditer(text)) - if matches: - for match in matches: - warnings.append( - f"Found template tag in {content_type}: {match.group()}" - ) - return template_pattern.sub("", text) - return text - - for elem in xml_copy.iter(): - if not hasattr(elem, "tag") or callable(elem.tag): - continue - tag_str = str(elem.tag) - if tag_str.endswith("}t") or tag_str == "t": - continue - - elem.text = process_text_content(elem.text, "text content") - elem.tail = process_text_content(elem.tail, "tail content") - - return lxml.etree.ElementTree(xml_copy), warnings - - -if __name__ == "__main__": - raise RuntimeError("This module should not be run directly.") diff --git a/.agents/skills/docx/scripts/office/validators/docx.py b/.agents/skills/docx/scripts/office/validators/docx.py deleted file mode 100644 index fec405e69..000000000 --- a/.agents/skills/docx/scripts/office/validators/docx.py +++ /dev/null @@ -1,446 +0,0 @@ -""" -Validator for Word document XML files against XSD schemas. -""" - -import random -import re -import tempfile -import zipfile - -import defusedxml.minidom -import lxml.etree - -from .base import BaseSchemaValidator - - -class DOCXSchemaValidator(BaseSchemaValidator): - - WORD_2006_NAMESPACE = "http://schemas.openxmlformats.org/wordprocessingml/2006/main" - W14_NAMESPACE = "http://schemas.microsoft.com/office/word/2010/wordml" - W16CID_NAMESPACE = "http://schemas.microsoft.com/office/word/2016/wordml/cid" - - ELEMENT_RELATIONSHIP_TYPES = {} - - def validate(self): - if not self.validate_xml(): - return False - - all_valid = True - if not self.validate_namespaces(): - all_valid = False - - if not self.validate_unique_ids(): - all_valid = False - - if not self.validate_file_references(): - all_valid = False - - if not self.validate_content_types(): - all_valid = False - - if not self.validate_against_xsd(): - all_valid = False - - if not self.validate_whitespace_preservation(): - all_valid = False - - if not self.validate_deletions(): - all_valid = False - - if not self.validate_insertions(): - all_valid = False - - if not self.validate_all_relationship_ids(): - all_valid = False - - if not self.validate_id_constraints(): - all_valid = False - - if not self.validate_comment_markers(): - all_valid = False - - self.compare_paragraph_counts() - - return all_valid - - def validate_whitespace_preservation(self): - errors = [] - - for xml_file in self.xml_files: - if xml_file.name != "document.xml": - continue - - try: - root = lxml.etree.parse(str(xml_file)).getroot() - - for elem in root.iter(f"{{{self.WORD_2006_NAMESPACE}}}t"): - if elem.text: - text = elem.text - if re.search(r"^[ \t\n\r]", text) or re.search( - r"[ \t\n\r]$", text - ): - xml_space_attr = f"{{{self.XML_NAMESPACE}}}space" - if ( - xml_space_attr not in elem.attrib - or elem.attrib[xml_space_attr] != "preserve" - ): - text_preview = ( - repr(text)[:50] + "..." - if len(repr(text)) > 50 - else repr(text) - ) - errors.append( - f" {xml_file.relative_to(self.unpacked_dir)}: " - f"Line {elem.sourceline}: w:t element with whitespace missing xml:space='preserve': {text_preview}" - ) - - except (lxml.etree.XMLSyntaxError, Exception) as e: - errors.append( - f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}" - ) - - if errors: - print(f"FAILED - Found {len(errors)} whitespace preservation violations:") - for error in errors: - print(error) - return False - else: - if self.verbose: - print("PASSED - All whitespace is properly preserved") - return True - - def validate_deletions(self): - errors = [] - - for xml_file in self.xml_files: - if xml_file.name != "document.xml": - continue - - try: - root = lxml.etree.parse(str(xml_file)).getroot() - namespaces = {"w": self.WORD_2006_NAMESPACE} - - for t_elem in root.xpath(".//w:del//w:t", namespaces=namespaces): - if t_elem.text: - text_preview = ( - repr(t_elem.text)[:50] + "..." - if len(repr(t_elem.text)) > 50 - else repr(t_elem.text) - ) - errors.append( - f" {xml_file.relative_to(self.unpacked_dir)}: " - f"Line {t_elem.sourceline}: found within : {text_preview}" - ) - - for instr_elem in root.xpath( - ".//w:del//w:instrText", namespaces=namespaces - ): - text_preview = ( - repr(instr_elem.text or "")[:50] + "..." - if len(repr(instr_elem.text or "")) > 50 - else repr(instr_elem.text or "") - ) - errors.append( - f" {xml_file.relative_to(self.unpacked_dir)}: " - f"Line {instr_elem.sourceline}: found within (use ): {text_preview}" - ) - - except (lxml.etree.XMLSyntaxError, Exception) as e: - errors.append( - f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}" - ) - - if errors: - print(f"FAILED - Found {len(errors)} deletion validation violations:") - for error in errors: - print(error) - return False - else: - if self.verbose: - print("PASSED - No w:t elements found within w:del elements") - return True - - def count_paragraphs_in_unpacked(self): - count = 0 - - for xml_file in self.xml_files: - if xml_file.name != "document.xml": - continue - - try: - root = lxml.etree.parse(str(xml_file)).getroot() - paragraphs = root.findall(f".//{{{self.WORD_2006_NAMESPACE}}}p") - count = len(paragraphs) - except Exception as e: - print(f"Error counting paragraphs in unpacked document: {e}") - - return count - - def count_paragraphs_in_original(self): - original = self.original_file - if original is None: - return 0 - - count = 0 - - try: - with tempfile.TemporaryDirectory() as temp_dir: - with zipfile.ZipFile(original, "r") as zip_ref: - zip_ref.extractall(temp_dir) - - doc_xml_path = temp_dir + "/word/document.xml" - root = lxml.etree.parse(doc_xml_path).getroot() - - paragraphs = root.findall(f".//{{{self.WORD_2006_NAMESPACE}}}p") - count = len(paragraphs) - - except Exception as e: - print(f"Error counting paragraphs in original document: {e}") - - return count - - def validate_insertions(self): - errors = [] - - for xml_file in self.xml_files: - if xml_file.name != "document.xml": - continue - - try: - root = lxml.etree.parse(str(xml_file)).getroot() - namespaces = {"w": self.WORD_2006_NAMESPACE} - - invalid_elements = root.xpath( - ".//w:ins//w:delText[not(ancestor::w:del)]", namespaces=namespaces - ) - - for elem in invalid_elements: - text_preview = ( - repr(elem.text or "")[:50] + "..." - if len(repr(elem.text or "")) > 50 - else repr(elem.text or "") - ) - errors.append( - f" {xml_file.relative_to(self.unpacked_dir)}: " - f"Line {elem.sourceline}: within : {text_preview}" - ) - - except (lxml.etree.XMLSyntaxError, Exception) as e: - errors.append( - f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}" - ) - - if errors: - print(f"FAILED - Found {len(errors)} insertion validation violations:") - for error in errors: - print(error) - return False - else: - if self.verbose: - print("PASSED - No w:delText elements within w:ins elements") - return True - - def compare_paragraph_counts(self): - original_count = self.count_paragraphs_in_original() - new_count = self.count_paragraphs_in_unpacked() - - diff = new_count - original_count - diff_str = f"+{diff}" if diff > 0 else str(diff) - print(f"\nParagraphs: {original_count} → {new_count} ({diff_str})") - - def _parse_id_value(self, val: str, base: int = 16) -> int: - return int(val, base) - - def validate_id_constraints(self): - errors = [] - para_id_attr = f"{{{self.W14_NAMESPACE}}}paraId" - durable_id_attr = f"{{{self.W16CID_NAMESPACE}}}durableId" - - for xml_file in self.xml_files: - try: - for elem in lxml.etree.parse(str(xml_file)).iter(): - if val := elem.get(para_id_attr): - if self._parse_id_value(val, base=16) >= 0x80000000: - errors.append( - f" {xml_file.name}:{elem.sourceline}: paraId={val} >= 0x80000000" - ) - - if val := elem.get(durable_id_attr): - if xml_file.name == "numbering.xml": - try: - if self._parse_id_value(val, base=10) >= 0x7FFFFFFF: - errors.append( - f" {xml_file.name}:{elem.sourceline}: " - f"durableId={val} >= 0x7FFFFFFF" - ) - except ValueError: - errors.append( - f" {xml_file.name}:{elem.sourceline}: " - f"durableId={val} must be decimal in numbering.xml" - ) - else: - if self._parse_id_value(val, base=16) >= 0x7FFFFFFF: - errors.append( - f" {xml_file.name}:{elem.sourceline}: " - f"durableId={val} >= 0x7FFFFFFF" - ) - except Exception: - pass - - if errors: - print(f"FAILED - {len(errors)} ID constraint violations:") - for e in errors: - print(e) - elif self.verbose: - print("PASSED - All paraId/durableId values within constraints") - return not errors - - def validate_comment_markers(self): - errors = [] - - document_xml = None - comments_xml = None - for xml_file in self.xml_files: - if xml_file.name == "document.xml" and "word" in str(xml_file): - document_xml = xml_file - elif xml_file.name == "comments.xml": - comments_xml = xml_file - - if not document_xml: - if self.verbose: - print("PASSED - No document.xml found (skipping comment validation)") - return True - - try: - doc_root = lxml.etree.parse(str(document_xml)).getroot() - namespaces = {"w": self.WORD_2006_NAMESPACE} - - range_starts = { - elem.get(f"{{{self.WORD_2006_NAMESPACE}}}id") - for elem in doc_root.xpath( - ".//w:commentRangeStart", namespaces=namespaces - ) - } - range_ends = { - elem.get(f"{{{self.WORD_2006_NAMESPACE}}}id") - for elem in doc_root.xpath( - ".//w:commentRangeEnd", namespaces=namespaces - ) - } - references = { - elem.get(f"{{{self.WORD_2006_NAMESPACE}}}id") - for elem in doc_root.xpath( - ".//w:commentReference", namespaces=namespaces - ) - } - - orphaned_ends = range_ends - range_starts - for comment_id in sorted( - orphaned_ends, key=lambda x: int(x) if x and x.isdigit() else 0 - ): - errors.append( - f' document.xml: commentRangeEnd id="{comment_id}" has no matching commentRangeStart' - ) - - orphaned_starts = range_starts - range_ends - for comment_id in sorted( - orphaned_starts, key=lambda x: int(x) if x and x.isdigit() else 0 - ): - errors.append( - f' document.xml: commentRangeStart id="{comment_id}" has no matching commentRangeEnd' - ) - - comment_ids = set() - if comments_xml and comments_xml.exists(): - comments_root = lxml.etree.parse(str(comments_xml)).getroot() - comment_ids = { - elem.get(f"{{{self.WORD_2006_NAMESPACE}}}id") - for elem in comments_root.xpath( - ".//w:comment", namespaces=namespaces - ) - } - - marker_ids = range_starts | range_ends | references - invalid_refs = marker_ids - comment_ids - for comment_id in sorted( - invalid_refs, key=lambda x: int(x) if x and x.isdigit() else 0 - ): - if comment_id: - errors.append( - f' document.xml: marker id="{comment_id}" references non-existent comment' - ) - - except (lxml.etree.XMLSyntaxError, Exception) as e: - errors.append(f" Error parsing XML: {e}") - - if errors: - print(f"FAILED - {len(errors)} comment marker violations:") - for error in errors: - print(error) - return False - else: - if self.verbose: - print("PASSED - All comment markers properly paired") - return True - - def repair(self) -> int: - repairs = super().repair() - repairs += self.repair_durableId() - return repairs - - def repair_durableId(self) -> int: - repairs = 0 - - for xml_file in self.xml_files: - try: - content = xml_file.read_text(encoding="utf-8") - dom = defusedxml.minidom.parseString(content) - modified = False - - for elem in dom.getElementsByTagName("*"): - if not elem.hasAttribute("w16cid:durableId"): - continue - - durable_id = elem.getAttribute("w16cid:durableId") - needs_repair = False - - if xml_file.name == "numbering.xml": - try: - needs_repair = ( - self._parse_id_value(durable_id, base=10) >= 0x7FFFFFFF - ) - except ValueError: - needs_repair = True - else: - try: - needs_repair = ( - self._parse_id_value(durable_id, base=16) >= 0x7FFFFFFF - ) - except ValueError: - needs_repair = True - - if needs_repair: - value = random.randint(1, 0x7FFFFFFE) - if xml_file.name == "numbering.xml": - new_id = str(value) - else: - new_id = f"{value:08X}" - - elem.setAttribute("w16cid:durableId", new_id) - print( - f" Repaired: {xml_file.name}: durableId {durable_id} → {new_id}" - ) - repairs += 1 - modified = True - - if modified: - xml_file.write_bytes(dom.toxml(encoding="UTF-8")) - - except Exception: - pass - - return repairs - - -if __name__ == "__main__": - raise RuntimeError("This module should not be run directly.") diff --git a/.agents/skills/docx/scripts/office/validators/pptx.py b/.agents/skills/docx/scripts/office/validators/pptx.py deleted file mode 100644 index 09842aa99..000000000 --- a/.agents/skills/docx/scripts/office/validators/pptx.py +++ /dev/null @@ -1,275 +0,0 @@ -""" -Validator for PowerPoint presentation XML files against XSD schemas. -""" - -import re - -from .base import BaseSchemaValidator - - -class PPTXSchemaValidator(BaseSchemaValidator): - - PRESENTATIONML_NAMESPACE = ( - "http://schemas.openxmlformats.org/presentationml/2006/main" - ) - - ELEMENT_RELATIONSHIP_TYPES = { - "sldid": "slide", - "sldmasterid": "slidemaster", - "notesmasterid": "notesmaster", - "sldlayoutid": "slidelayout", - "themeid": "theme", - "tablestyleid": "tablestyles", - } - - def validate(self): - if not self.validate_xml(): - return False - - all_valid = True - if not self.validate_namespaces(): - all_valid = False - - if not self.validate_unique_ids(): - all_valid = False - - if not self.validate_uuid_ids(): - all_valid = False - - if not self.validate_file_references(): - all_valid = False - - if not self.validate_slide_layout_ids(): - all_valid = False - - if not self.validate_content_types(): - all_valid = False - - if not self.validate_against_xsd(): - all_valid = False - - if not self.validate_notes_slide_references(): - all_valid = False - - if not self.validate_all_relationship_ids(): - all_valid = False - - if not self.validate_no_duplicate_slide_layouts(): - all_valid = False - - return all_valid - - def validate_uuid_ids(self): - import lxml.etree - - errors = [] - uuid_pattern = re.compile( - r"^[\{\(]?[0-9A-Fa-f]{8}-?[0-9A-Fa-f]{4}-?[0-9A-Fa-f]{4}-?[0-9A-Fa-f]{4}-?[0-9A-Fa-f]{12}[\}\)]?$" - ) - - for xml_file in self.xml_files: - try: - root = lxml.etree.parse(str(xml_file)).getroot() - - for elem in root.iter(): - for attr, value in elem.attrib.items(): - attr_name = attr.split("}")[-1].lower() - if attr_name == "id" or attr_name.endswith("id"): - if self._looks_like_uuid(value): - if not uuid_pattern.match(value): - errors.append( - f" {xml_file.relative_to(self.unpacked_dir)}: " - f"Line {elem.sourceline}: ID '{value}' appears to be a UUID but contains invalid hex characters" - ) - - except (lxml.etree.XMLSyntaxError, Exception) as e: - errors.append( - f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}" - ) - - if errors: - print(f"FAILED - Found {len(errors)} UUID ID validation errors:") - for error in errors: - print(error) - return False - else: - if self.verbose: - print("PASSED - All UUID-like IDs contain valid hex values") - return True - - def _looks_like_uuid(self, value): - clean_value = value.strip("{}()").replace("-", "") - return len(clean_value) == 32 and all(c.isalnum() for c in clean_value) - - def validate_slide_layout_ids(self): - import lxml.etree - - errors = [] - - slide_masters = list(self.unpacked_dir.glob("ppt/slideMasters/*.xml")) - - if not slide_masters: - if self.verbose: - print("PASSED - No slide masters found") - return True - - for slide_master in slide_masters: - try: - root = lxml.etree.parse(str(slide_master)).getroot() - - rels_file = slide_master.parent / "_rels" / f"{slide_master.name}.rels" - - if not rels_file.exists(): - errors.append( - f" {slide_master.relative_to(self.unpacked_dir)}: " - f"Missing relationships file: {rels_file.relative_to(self.unpacked_dir)}" - ) - continue - - rels_root = lxml.etree.parse(str(rels_file)).getroot() - - valid_layout_rids = set() - for rel in rels_root.findall( - f".//{{{self.PACKAGE_RELATIONSHIPS_NAMESPACE}}}Relationship" - ): - rel_type = rel.get("Type", "") - if "slideLayout" in rel_type: - valid_layout_rids.add(rel.get("Id")) - - for sld_layout_id in root.findall( - f".//{{{self.PRESENTATIONML_NAMESPACE}}}sldLayoutId" - ): - r_id = sld_layout_id.get( - f"{{{self.OFFICE_RELATIONSHIPS_NAMESPACE}}}id" - ) - layout_id = sld_layout_id.get("id") - - if r_id and r_id not in valid_layout_rids: - errors.append( - f" {slide_master.relative_to(self.unpacked_dir)}: " - f"Line {sld_layout_id.sourceline}: sldLayoutId with id='{layout_id}' " - f"references r:id='{r_id}' which is not found in slide layout relationships" - ) - - except (lxml.etree.XMLSyntaxError, Exception) as e: - errors.append( - f" {slide_master.relative_to(self.unpacked_dir)}: Error: {e}" - ) - - if errors: - print(f"FAILED - Found {len(errors)} slide layout ID validation errors:") - for error in errors: - print(error) - print( - "Remove invalid references or add missing slide layouts to the relationships file." - ) - return False - else: - if self.verbose: - print("PASSED - All slide layout IDs reference valid slide layouts") - return True - - def validate_no_duplicate_slide_layouts(self): - import lxml.etree - - errors = [] - slide_rels_files = list(self.unpacked_dir.glob("ppt/slides/_rels/*.xml.rels")) - - for rels_file in slide_rels_files: - try: - root = lxml.etree.parse(str(rels_file)).getroot() - - layout_rels = [ - rel - for rel in root.findall( - f".//{{{self.PACKAGE_RELATIONSHIPS_NAMESPACE}}}Relationship" - ) - if "slideLayout" in rel.get("Type", "") - ] - - if len(layout_rels) > 1: - errors.append( - f" {rels_file.relative_to(self.unpacked_dir)}: has {len(layout_rels)} slideLayout references" - ) - - except Exception as e: - errors.append( - f" {rels_file.relative_to(self.unpacked_dir)}: Error: {e}" - ) - - if errors: - print("FAILED - Found slides with duplicate slideLayout references:") - for error in errors: - print(error) - return False - else: - if self.verbose: - print("PASSED - All slides have exactly one slideLayout reference") - return True - - def validate_notes_slide_references(self): - import lxml.etree - - errors = [] - notes_slide_references = {} - - slide_rels_files = list(self.unpacked_dir.glob("ppt/slides/_rels/*.xml.rels")) - - if not slide_rels_files: - if self.verbose: - print("PASSED - No slide relationship files found") - return True - - for rels_file in slide_rels_files: - try: - root = lxml.etree.parse(str(rels_file)).getroot() - - for rel in root.findall( - f".//{{{self.PACKAGE_RELATIONSHIPS_NAMESPACE}}}Relationship" - ): - rel_type = rel.get("Type", "") - if "notesSlide" in rel_type: - target = rel.get("Target", "") - if target: - normalized_target = target.replace("../", "") - - slide_name = rels_file.stem.replace( - ".xml", "" - ) - - if normalized_target not in notes_slide_references: - notes_slide_references[normalized_target] = [] - notes_slide_references[normalized_target].append( - (slide_name, rels_file) - ) - - except (lxml.etree.XMLSyntaxError, Exception) as e: - errors.append( - f" {rels_file.relative_to(self.unpacked_dir)}: Error: {e}" - ) - - for target, references in notes_slide_references.items(): - if len(references) > 1: - slide_names = [ref[0] for ref in references] - errors.append( - f" Notes slide '{target}' is referenced by multiple slides: {', '.join(slide_names)}" - ) - for slide_name, rels_file in references: - errors.append(f" - {rels_file.relative_to(self.unpacked_dir)}") - - if errors: - print( - f"FAILED - Found {len([e for e in errors if not e.startswith(' ')])} notes slide reference validation errors:" - ) - for error in errors: - print(error) - print("Each slide may optionally have its own slide file.") - return False - else: - if self.verbose: - print("PASSED - All notes slide references are unique") - return True - - -if __name__ == "__main__": - raise RuntimeError("This module should not be run directly.") diff --git a/.agents/skills/docx/scripts/office/validators/redlining.py b/.agents/skills/docx/scripts/office/validators/redlining.py deleted file mode 100644 index 71c81b6bf..000000000 --- a/.agents/skills/docx/scripts/office/validators/redlining.py +++ /dev/null @@ -1,247 +0,0 @@ -""" -Validator for tracked changes in Word documents. -""" - -import subprocess -import tempfile -import zipfile -from pathlib import Path - - -class RedliningValidator: - - def __init__(self, unpacked_dir, original_docx, verbose=False, author="Claude"): - self.unpacked_dir = Path(unpacked_dir) - self.original_docx = Path(original_docx) - self.verbose = verbose - self.author = author - self.namespaces = { - "w": "http://schemas.openxmlformats.org/wordprocessingml/2006/main" - } - - def repair(self) -> int: - return 0 - - def validate(self): - modified_file = self.unpacked_dir / "word" / "document.xml" - if not modified_file.exists(): - print(f"FAILED - Modified document.xml not found at {modified_file}") - return False - - try: - import xml.etree.ElementTree as ET - - tree = ET.parse(modified_file) - root = tree.getroot() - - del_elements = root.findall(".//w:del", self.namespaces) - ins_elements = root.findall(".//w:ins", self.namespaces) - - author_del_elements = [ - elem - for elem in del_elements - if elem.get(f"{{{self.namespaces['w']}}}author") == self.author - ] - author_ins_elements = [ - elem - for elem in ins_elements - if elem.get(f"{{{self.namespaces['w']}}}author") == self.author - ] - - if not author_del_elements and not author_ins_elements: - if self.verbose: - print(f"PASSED - No tracked changes by {self.author} found.") - return True - - except Exception: - pass - - with tempfile.TemporaryDirectory() as temp_dir: - temp_path = Path(temp_dir) - - try: - with zipfile.ZipFile(self.original_docx, "r") as zip_ref: - zip_ref.extractall(temp_path) - except Exception as e: - print(f"FAILED - Error unpacking original docx: {e}") - return False - - original_file = temp_path / "word" / "document.xml" - if not original_file.exists(): - print( - f"FAILED - Original document.xml not found in {self.original_docx}" - ) - return False - - try: - import xml.etree.ElementTree as ET - - modified_tree = ET.parse(modified_file) - modified_root = modified_tree.getroot() - original_tree = ET.parse(original_file) - original_root = original_tree.getroot() - except ET.ParseError as e: - print(f"FAILED - Error parsing XML files: {e}") - return False - - self._remove_author_tracked_changes(original_root) - self._remove_author_tracked_changes(modified_root) - - modified_text = self._extract_text_content(modified_root) - original_text = self._extract_text_content(original_root) - - if modified_text != original_text: - error_message = self._generate_detailed_diff( - original_text, modified_text - ) - print(error_message) - return False - - if self.verbose: - print(f"PASSED - All changes by {self.author} are properly tracked") - return True - - def _generate_detailed_diff(self, original_text, modified_text): - error_parts = [ - f"FAILED - Document text doesn't match after removing {self.author}'s tracked changes", - "", - "Likely causes:", - " 1. Modified text inside another author's or tags", - " 2. Made edits without proper tracked changes", - " 3. Didn't nest inside when deleting another's insertion", - "", - "For pre-redlined documents, use correct patterns:", - " - To reject another's INSERTION: Nest inside their ", - " - To restore another's DELETION: Add new AFTER their ", - "", - ] - - git_diff = self._get_git_word_diff(original_text, modified_text) - if git_diff: - error_parts.extend(["Differences:", "============", git_diff]) - else: - error_parts.append("Unable to generate word diff (git not available)") - - return "\n".join(error_parts) - - def _get_git_word_diff(self, original_text, modified_text): - try: - with tempfile.TemporaryDirectory() as temp_dir: - temp_path = Path(temp_dir) - - original_file = temp_path / "original.txt" - modified_file = temp_path / "modified.txt" - - original_file.write_text(original_text, encoding="utf-8") - modified_file.write_text(modified_text, encoding="utf-8") - - result = subprocess.run( - [ - "git", - "diff", - "--word-diff=plain", - "--word-diff-regex=.", - "-U0", - "--no-index", - str(original_file), - str(modified_file), - ], - capture_output=True, - text=True, - ) - - if result.stdout.strip(): - lines = result.stdout.split("\n") - content_lines = [] - in_content = False - for line in lines: - if line.startswith("@@"): - in_content = True - continue - if in_content and line.strip(): - content_lines.append(line) - - if content_lines: - return "\n".join(content_lines) - - result = subprocess.run( - [ - "git", - "diff", - "--word-diff=plain", - "-U0", - "--no-index", - str(original_file), - str(modified_file), - ], - capture_output=True, - text=True, - ) - - if result.stdout.strip(): - lines = result.stdout.split("\n") - content_lines = [] - in_content = False - for line in lines: - if line.startswith("@@"): - in_content = True - continue - if in_content and line.strip(): - content_lines.append(line) - return "\n".join(content_lines) - - except (subprocess.CalledProcessError, FileNotFoundError, Exception): - pass - - return None - - def _remove_author_tracked_changes(self, root): - ins_tag = f"{{{self.namespaces['w']}}}ins" - del_tag = f"{{{self.namespaces['w']}}}del" - author_attr = f"{{{self.namespaces['w']}}}author" - - for parent in root.iter(): - to_remove = [] - for child in parent: - if child.tag == ins_tag and child.get(author_attr) == self.author: - to_remove.append(child) - for elem in to_remove: - parent.remove(elem) - - deltext_tag = f"{{{self.namespaces['w']}}}delText" - t_tag = f"{{{self.namespaces['w']}}}t" - - for parent in root.iter(): - to_process = [] - for child in parent: - if child.tag == del_tag and child.get(author_attr) == self.author: - to_process.append((child, list(parent).index(child))) - - for del_elem, del_index in reversed(to_process): - for elem in del_elem.iter(): - if elem.tag == deltext_tag: - elem.tag = t_tag - - for child in reversed(list(del_elem)): - parent.insert(del_index, child) - parent.remove(del_elem) - - def _extract_text_content(self, root): - p_tag = f"{{{self.namespaces['w']}}}p" - t_tag = f"{{{self.namespaces['w']}}}t" - - paragraphs = [] - for p_elem in root.findall(f".//{p_tag}"): - text_parts = [] - for t_elem in p_elem.findall(f".//{t_tag}"): - if t_elem.text: - text_parts.append(t_elem.text) - paragraph_text = "".join(text_parts) - if paragraph_text: - paragraphs.append(paragraph_text) - - return "\n".join(paragraphs) - - -if __name__ == "__main__": - raise RuntimeError("This module should not be run directly.") diff --git a/.agents/skills/docx/scripts/templates/comments.xml b/.agents/skills/docx/scripts/templates/comments.xml deleted file mode 100644 index cd01a7d71..000000000 --- a/.agents/skills/docx/scripts/templates/comments.xml +++ /dev/null @@ -1,3 +0,0 @@ - - - diff --git a/.agents/skills/docx/scripts/templates/commentsExtended.xml b/.agents/skills/docx/scripts/templates/commentsExtended.xml deleted file mode 100644 index 411003cc4..000000000 --- a/.agents/skills/docx/scripts/templates/commentsExtended.xml +++ /dev/null @@ -1,3 +0,0 @@ - - - diff --git a/.agents/skills/docx/scripts/templates/commentsExtensible.xml b/.agents/skills/docx/scripts/templates/commentsExtensible.xml deleted file mode 100644 index f5572d710..000000000 --- a/.agents/skills/docx/scripts/templates/commentsExtensible.xml +++ /dev/null @@ -1,3 +0,0 @@ - - - diff --git a/.agents/skills/docx/scripts/templates/commentsIds.xml b/.agents/skills/docx/scripts/templates/commentsIds.xml deleted file mode 100644 index 32f1629f2..000000000 --- a/.agents/skills/docx/scripts/templates/commentsIds.xml +++ /dev/null @@ -1,3 +0,0 @@ - - - diff --git a/.agents/skills/docx/scripts/templates/people.xml b/.agents/skills/docx/scripts/templates/people.xml deleted file mode 100644 index 3803d2de0..000000000 --- a/.agents/skills/docx/scripts/templates/people.xml +++ /dev/null @@ -1,3 +0,0 @@ - - - diff --git a/.agents/skills/evaluation/references/metrics.md b/.agents/skills/evaluation/references/metrics.md deleted file mode 100644 index b9d5f11db..000000000 --- a/.agents/skills/evaluation/references/metrics.md +++ /dev/null @@ -1,339 +0,0 @@ -# Evaluation Reference: Metrics and Implementation - -This document provides implementation details for evaluation metrics and evaluation systems. - -## Core Metric Definitions - -### Factual Accuracy - -Factual accuracy measures whether claims in agent output match ground truth. - -``` -Excellent (1.0): All claims verified against ground truth, no errors -Good (0.8): Minor errors that do not affect main conclusions -Acceptable (0.6): Major claims correct, minor inaccuracies present -Poor (0.3): Significant factual errors in key claims -Failed (0.0): Fundamental factual errors that invalidate output -``` - -Calculation approach: -- Extract claims from output -- Verify each claim against ground truth -- Weight claims by importance (major claims more weight) -- Calculate weighted average of claim accuracy - -### Completeness - -Completeness measures whether output covers all requested aspects. - -``` -Excellent (1.0): All requested aspects thoroughly covered -Good (0.8): Most aspects covered with minor gaps -Acceptable (0.6): Key aspects covered, some gaps -Poor (0.3): Major aspects missing from output -Failed (0.0): Fundamental aspects not addressed -``` - -### Citation Accuracy - -Citation accuracy measures whether cited sources match claimed sources. - -``` -Excellent (1.0): All citations accurate and complete -Good (0.8): Minor citation formatting issues -Acceptable (0.6): Major citations accurate -Poor (0.3): Significant citation problems -Failed (0.0): Citations missing or completely incorrect -``` - -### Source Quality - -Source quality measures whether appropriate primary sources were used. - -``` -Excellent (1.0): Primary authoritative sources -Good (0.8): Mostly primary sources with some secondary -Acceptable (0.6): Mix of primary and secondary sources -Poor (0.3): Mostly secondary or unreliable sources -Failed (0.0): No credible sources cited -``` - -### Tool Efficiency - -Tool efficiency measures whether the agent used appropriate tools a reasonable number of times. - -``` -Excellent (1.0): Optimal tool selection and call count -Good (0.8): Good tool selection with minor inefficiencies -Acceptable (0.6): Appropriate tools with some redundancy -Poor (0.3): Wrong tools or excessive call counts -Failed (0.0): Severe tool misuse or extremely excessive calls -``` - -## Rubric Implementation - -```python -EVALUATION_DIMENSIONS = { - "factual_accuracy": { - "weight": 0.30, - "description": "Claims match ground truth", - "levels": { - "excellent": 1.0, - "good": 0.8, - "acceptable": 0.6, - "poor": 0.3, - "failed": 0.0 - } - }, - "completeness": { - "weight": 0.25, - "description": "All requested aspects covered", - "levels": { - "excellent": 1.0, - "good": 0.8, - "acceptable": 0.6, - "poor": 0.3, - "failed": 0.0 - } - }, - "citation_accuracy": { - "weight": 0.15, - "description": "Citations match sources", - "levels": { - "excellent": 1.0, - "good": 0.8, - "acceptable": 0.6, - "poor": 0.3, - "failed": 0.0 - } - }, - "source_quality": { - "weight": 0.10, - "description": "Appropriate primary sources used", - "levels": { - "excellent": 1.0, - "good": 0.8, - "acceptable": 0.6, - "poor": 0.3, - "failed": 0.0 - } - }, - "tool_efficiency": { - "weight": 0.20, - "description": "Right tools used reasonably", - "levels": { - "excellent": 1.0, - "good": 0.8, - "acceptable": 0.6, - "poor": 0.3, - "failed": 0.0 - } - } -} - -def calculate_overall_score(dimension_scores, rubric): - """Calculate weighted overall score from dimension scores.""" - total_weight = 0 - weighted_sum = 0 - - for dimension, score in dimension_scores.items(): - if dimension in rubric: - weight = rubric[dimension]["weight"] - weighted_sum += score * weight - total_weight += weight - - return weighted_sum / total_weight if total_weight > 0 else 0 -``` - -## Test Set Management - -```python -class TestSet: - def __init__(self, name): - self.name = name - self.tests = [] - self.tags = {} - - def add_test(self, test_case): - """Add test case to test set.""" - self.tests.append(test_case) - - # Index by tags - for tag in test_case.get("tags", []): - if tag not in self.tags: - self.tags[tag] = [] - self.tags[tag].append(len(self.tests) - 1) - - def filter(self, **criteria): - """Filter tests by criteria.""" - filtered = [] - for test in self.tests: - match = True - for key, value in criteria.items(): - if test.get(key) != value: - match = False - break - if match: - filtered.append(test) - return filtered - - def get_complexity_distribution(self): - """Get distribution of tests by complexity.""" - distribution = {} - for test in self.tests: - complexity = test.get("complexity", "medium") - distribution[complexity] = distribution.get(complexity, 0) + 1 - return distribution -``` - -## Evaluation Runner - -```python -class EvaluationRunner: - def __init__(self, test_set, rubric, agent): - self.test_set = test_set - self.rubric = rubric - self.agent = agent - self.results = [] - - def run_all(self, verbose=False): - """Run evaluation on all tests.""" - self.results = [] - - for i, test in enumerate(self.test_set.tests): - if verbose: - print(f"Running test {i+1}/{len(self.test_set.tests)}") - - result = self.run_test(test) - self.results.append(result) - - return self.summarize() - - def run_test(self, test): - """Run single evaluation test.""" - # Get agent output - output = self.agent.run(test["input"]) - - # Evaluate - evaluation = self.evaluate_output(output, test) - - return { - "test": test, - "output": output, - "evaluation": evaluation - } - - def evaluate_output(self, output, test): - """Evaluate agent output against test.""" - ground_truth = test.get("expected", {}) - - dimension_scores = {} - for dimension, config in self.rubric.items(): - score = self.evaluate_dimension( - output, ground_truth, dimension, config - ) - dimension_scores[dimension] = score - - overall = calculate_overall_score(dimension_scores, self.rubric) - - return { - "overall_score": overall, - "dimension_scores": dimension_scores, - "passed": overall >= 0.7 - } - - def summarize(self): - """Summarize evaluation results.""" - if not self.results: - return {"error": "No results"} - - passed = sum(1 for r in self.results if r["evaluation"]["passed"]) - - dimension_totals = {} - for dimension in self.rubric.keys(): - dimension_totals[dimension] = { - "total": 0, - "count": 0 - } - - for result in self.results: - for dimension, score in result["evaluation"]["dimension_scores"].items(): - if dimension in dimension_totals: - dimension_totals[dimension]["total"] += score - dimension_totals[dimension]["count"] += 1 - - dimension_averages = {} - for dimension, data in dimension_totals.items(): - if data["count"] > 0: - dimension_averages[dimension] = data["total"] / data["count"] - - return { - "total_tests": len(self.results), - "passed": passed, - "failed": len(self.results) - passed, - "pass_rate": passed / len(self.results) if self.results else 0, - "dimension_averages": dimension_averages, - "failures": [ - r for r in self.results - if not r["evaluation"]["passed"] - ] - } -``` - -## Production Monitoring - -```python -class ProductionMonitor: - def __init__(self, sample_rate=0.01): - self.sample_rate = sample_rate - self.samples = [] - self.alert_thresholds = { - "pass_rate_warning": 0.85, - "pass_rate_critical": 0.70 - } - - def sample_and_evaluate(self, query, output): - """Sample production interaction for evaluation.""" - if random.random() > self.sample_rate: - return None - - evaluation = evaluate_output(output, {}, EVALUATION_RUBRIC) - - sample = { - "query": query[:200], - "output_preview": output[:200], - "score": evaluation["overall_score"], - "passed": evaluation["passed"], - "timestamp": current_timestamp() - } - - self.samples.append(sample) - return sample - - def get_metrics(self): - """Calculate current metrics from samples.""" - if not self.samples: - return {"status": "insufficient_data"} - - passed = sum(1 for s in self.samples if s["passed"]) - pass_rate = passed / len(self.samples) - - avg_score = sum(s["score"] for s in self.samples) / len(self.samples) - - return { - "sample_count": len(self.samples), - "pass_rate": pass_rate, - "average_score": avg_score, - "status": self._get_status(pass_rate) - } - - def _get_status(self, pass_rate): - """Get status based on pass rate.""" - if pass_rate < self.alert_thresholds["pass_rate_critical"]: - return "critical" - elif pass_rate < self.alert_thresholds["pass_rate_warning"]: - return "warning" - else: - return "healthy" -``` - diff --git a/.agents/skills/evaluation/scripts/evaluator.py b/.agents/skills/evaluation/scripts/evaluator.py deleted file mode 100644 index f5dab5664..000000000 --- a/.agents/skills/evaluation/scripts/evaluator.py +++ /dev/null @@ -1,627 +0,0 @@ -"""Agent Evaluation Framework for context-engineered agent systems. - -Use when: building evaluation pipelines, scoring agent outputs against -multi-dimensional rubrics, managing test sets, or monitoring production -agent quality. Provides composable classes that can be used independently -or wired together into a full evaluation pipeline. - -Typical usage:: - - evaluator = AgentEvaluator() - test_set = TestSet("my_tests").create_standard_tests() - runner = EvaluationRunner(evaluator, test_set) - summary = runner.run_all(verbose=True) - print(summary) -""" - -from typing import Dict, List, Any, Optional -from dataclasses import dataclass -from enum import Enum -import time - -__all__ = [ - "ScoreLevel", - "RubricDimension", - "DEFAULT_RUBRIC", - "AgentEvaluator", - "TestSet", - "EvaluationRunner", - "ProductionMonitor", -] - - -class ScoreLevel(Enum): - """Use when: mapping qualitative judgments to numeric scores.""" - - EXCELLENT = 1.0 - GOOD = 0.8 - ACCEPTABLE = 0.6 - POOR = 0.3 - FAILED = 0.0 - - -@dataclass -class RubricDimension: - """Definition of a single evaluation dimension. - - Use when: defining custom rubric dimensions beyond the defaults. - """ - - name: str - weight: float - description: str - levels: Dict[str, str] # level_name -> description - - -DEFAULT_RUBRIC: Dict[str, RubricDimension] = { - "factual_accuracy": RubricDimension( - name="factual_accuracy", - weight=0.30, - description="Claims in output match ground truth", - levels={ - "excellent": "All claims verified, no errors", - "good": "Minor errors not affecting main conclusions", - "acceptable": "Major claims correct, minor inaccuracies", - "poor": "Significant factual errors", - "failed": "Fundamental factual errors", - }, - ), - "completeness": RubricDimension( - name="completeness", - weight=0.25, - description="Output covers all requested aspects", - levels={ - "excellent": "All aspects thoroughly covered", - "good": "Most aspects covered, minor gaps", - "acceptable": "Key aspects covered, some gaps", - "poor": "Major aspects missing", - "failed": "Fundamental aspects missing", - }, - ), - "citation_accuracy": RubricDimension( - name="citation_accuracy", - weight=0.15, - description="Citations match claimed sources", - levels={ - "excellent": "All citations accurate and complete", - "good": "Minor citation issues", - "acceptable": "Major citations accurate", - "poor": "Significant citation problems", - "failed": "Citations missing or incorrect", - }, - ), - "source_quality": RubricDimension( - name="source_quality", - weight=0.10, - description="Uses appropriate primary sources", - levels={ - "excellent": "Primary sources, authoritative", - "good": "Mostly primary, some secondary", - "acceptable": "Mix of primary and secondary", - "poor": "Mostly secondary or unreliable", - "failed": "No credible sources", - }, - ), - "tool_efficiency": RubricDimension( - name="tool_efficiency", - weight=0.20, - description="Uses right tools reasonable number of times", - levels={ - "excellent": "Optimal tool selection and count", - "good": "Good tool selection, minor inefficiencies", - "acceptable": "Appropriate tools, some redundancy", - "poor": "Wrong tools or excessive calls", - "failed": "Severe tool misuse", - }, - ), -} - - -# --------------------------------------------------------------------------- -# Evaluation Engine -# --------------------------------------------------------------------------- - - -class AgentEvaluator: - """Main evaluation engine for agent outputs. - - Use when: scoring a single agent output against a multi-dimensional rubric. - Instantiate with a custom rubric or rely on ``DEFAULT_RUBRIC``. - """ - - def __init__(self, rubric: Optional[Dict[str, RubricDimension]] = None) -> None: - self.rubric: Dict[str, RubricDimension] = rubric or DEFAULT_RUBRIC - self.evaluation_history: List[Dict[str, Any]] = [] - - def evaluate( - self, - task: Dict[str, Any], - output: str, - ground_truth: Optional[Dict[str, Any]] = None, - tool_calls: Optional[List[Dict[str, Any]]] = None, - ) -> Dict[str, Any]: - """Evaluate agent output against task requirements. - - Use when: you have a single (task, output) pair and need per-dimension - scores plus an overall pass/fail verdict. - - Returns evaluation results with per-dimension scores. - """ - scores: Dict[str, Dict[str, Any]] = {} - - for dimension_name, dimension in self.rubric.items(): - score = self._evaluate_dimension( - dimension=dimension, - task=task, - output=output, - ground_truth=ground_truth, - tool_calls=tool_calls, - ) - - scores[dimension_name] = { - "score": score, - "weight": dimension.weight, - "level": self._score_to_level(score), - } - - # Calculate weighted overall - overall: float = sum( - s["score"] * self.rubric[k].weight for k, s in scores.items() - ) - - result: Dict[str, Any] = { - "overall_score": overall, - "dimension_scores": scores, - "passed": overall >= 0.7, - "timestamp": time.time(), - } - - self.evaluation_history.append(result) - return result - - def _evaluate_dimension( - self, - dimension: RubricDimension, - task: Dict[str, Any], - output: str, - ground_truth: Optional[Dict[str, Any]] = None, - tool_calls: Optional[List[Dict[str, Any]]] = None, - ) -> float: - """Evaluate a single dimension. - - Use when: extending the evaluator with custom dimension logic. - In production, replace heuristics with LLM judgment or human evaluation. - """ - output_lower: str = output.lower() - task_type: str = task.get("type", "") - - if dimension.name == "factual_accuracy": - if ground_truth: - return self._check_factual_accuracy(output, ground_truth) - return 0.7 # Default assumption - - elif dimension.name == "completeness": - required: List[str] = task.get("requirements", []) - if required: - covered = sum(1 for r in required if r.lower() in output_lower) - return covered / len(required) - return 0.8 - - elif dimension.name == "citation_accuracy": - if task.get("requires_citations"): - # Look for citation patterns like [1], [Author 2024], [source] - # Avoid false positives from code brackets or JSON - citation_pattern = r'\[\d+\]|\[[A-Z][a-z]+(?:\s+(?:et al\.?|&)\s+[A-Z][a-z]+)?\s*[\d,]+\]|\[(?:source|ref|cite)[^\]]*\]' - import re as _re - citations_found = _re.findall(citation_pattern, output) - if len(citations_found) >= 1: - return 1.0 - elif any(marker in output_lower for marker in ["according to", "cited in", "reported by"]): - return 0.7 - return 0.4 - return 0.8 # Citations not required - - elif dimension.name == "source_quality": - quality_markers = ["according to", "reported by", "data from", "study"] - quality_count = sum(1 for m in quality_markers if m in output_lower) - return min(1.0, 0.5 + quality_count * 0.1) - - elif dimension.name == "tool_efficiency": - if tool_calls: - expected_count = self._estimate_expected_tools(task_type) - actual_count = len(tool_calls) - if actual_count <= expected_count: - return 1.0 - elif actual_count <= expected_count * 1.5: - return 0.7 - else: - return 0.4 - return 0.8 # No tool calls needed or recorded - - return 0.5 # Default - - def _check_factual_accuracy( - self, output: str, ground_truth: Dict[str, Any] - ) -> float: - """Check output against ground truth. - - Use when: ground truth key_claims are available for comparison. - """ - if not ground_truth: - return 0.7 - - key_claims: List[str] = ground_truth.get("key_claims", []) - if not key_claims: - return 0.7 - - output_lower: str = output.lower() - matched: int = sum(1 for claim in key_claims if claim.lower() in output_lower) - - if matched == len(key_claims): - return 1.0 - elif matched >= len(key_claims) * 0.7: - return 0.8 - elif matched >= len(key_claims) * 0.5: - return 0.6 - else: - return 0.3 - - def _estimate_expected_tools(self, task_type: str) -> int: - """Estimate expected tool count for task type.""" - estimates: Dict[str, int] = { - "research": 3, - "create": 2, - "analyze": 2, - "general": 1, - } - return estimates.get(task_type, 1) - - def _score_to_level(self, score: float) -> str: - """Convert numeric score to level name.""" - if score >= 0.9: - return "excellent" - elif score >= 0.7: - return "good" - elif score >= 0.5: - return "acceptable" - elif score >= 0.25: - return "poor" - else: - return "failed" - - -# --------------------------------------------------------------------------- -# Test Set Management -# --------------------------------------------------------------------------- - - -class TestSet: - """Manage evaluation test sets with tagging and complexity stratification. - - Use when: building, filtering, or analyzing collections of evaluation - test cases. Supports tag-based indexing and complexity distribution - analysis. - """ - - def __init__(self, name: str) -> None: - self.name: str = name - self.tests: List[Dict[str, Any]] = [] - self.tags: Dict[str, List[int]] = {} - - def add_test(self, test: Dict[str, Any]) -> None: - """Add a test case to the test set. - - Use when: incrementally building a test set from individual cases. - """ - self.tests.append(test) - idx: int = len(self.tests) - 1 - - for tag in test.get("tags", []): - if tag not in self.tags: - self.tags[tag] = [] - self.tags[tag].append(idx) - - def filter(self, **criteria: Any) -> List[Dict[str, Any]]: - """Filter tests by criteria. - - Use when: selecting a subset of tests matching specific field values. - """ - results: List[Dict[str, Any]] = [] - for test in self.tests: - match = True - for key, value in criteria.items(): - if test.get(key) != value: - match = False - break - if match: - results.append(test) - return results - - def get_complexity_distribution(self) -> Dict[str, int]: - """Get distribution of tests by complexity. - - Use when: verifying test set balance across difficulty levels. - """ - distribution: Dict[str, int] = {} - for test in self.tests: - complexity: str = test.get("complexity", "medium") - distribution[complexity] = distribution.get(complexity, 0) + 1 - return distribution - - def create_standard_tests(self) -> "TestSet": - """Populate with standard test cases for context engineering evaluation. - - Use when: bootstrapping a test set quickly for initial development. - """ - tests: List[Dict[str, Any]] = [ - { - "name": "simple_lookup", - "input": "What is the capital of France?", - "expected": {"type": "fact", "answer": "Paris"}, - "complexity": "simple", - "tags": ["knowledge", "simple"], - }, - { - "name": "context_retrieval", - "input": "Based on the user preferences, recommend a restaurant", - "context": { - "user_preferences": { - "cuisine": "Italian", - "price_range": "moderate", - } - }, - "complexity": "medium", - "tags": ["retrieval", "reasoning"], - }, - { - "name": "multi_step_reasoning", - "input": "Analyze the sales data and create a summary report", - "complexity": "complex", - "tags": ["analysis", "multi-step"], - }, - ] - - for test in tests: - self.add_test(test) - - return self - - -# --------------------------------------------------------------------------- -# Evaluation Runner -# --------------------------------------------------------------------------- - - -class EvaluationRunner: - """Run evaluations across an entire test set and produce summaries. - - Use when: executing a full evaluation pass over a test set, comparing - agent versions, or generating evaluation reports. - """ - - def __init__(self, evaluator: AgentEvaluator, test_set: TestSet) -> None: - self.evaluator: AgentEvaluator = evaluator - self.test_set: TestSet = test_set - self.results: List[Dict[str, Any]] = [] - - def run_all(self, verbose: bool = False) -> Dict[str, Any]: - """Run evaluation on all tests in the test set. - - Use when: performing a complete evaluation pass. - """ - self.results = [] - - for i, test in enumerate(self.test_set.tests): - if verbose: - print( - f"Running test {i + 1}/{len(self.test_set.tests)}: {test['name']}" - ) - - result = self.run_test(test) - self.results.append(result) - - return self.summarize() - - def run_test(self, test: Dict[str, Any]) -> Dict[str, Any]: - """Run a single evaluation test. - - Use when: evaluating an individual test case outside of a full run. - In production, replace the simulated output with actual agent execution. - """ - # In production, run actual agent - # Here we simulate - output: str = f"Simulated output for: {test.get('input', '')}" - - evaluation: Dict[str, Any] = self.evaluator.evaluate( - task=test, - output=output, - ground_truth=test.get("expected"), - tool_calls=[], - ) - - return { - "test": test, - "output": output, - "evaluation": evaluation, - "passed": evaluation["passed"], - } - - def summarize(self) -> Dict[str, Any]: - """Summarize evaluation results with per-dimension averages. - - Use when: generating a report after a full evaluation run. - """ - if not self.results: - return {"error": "No results"} - - passed: int = sum(1 for r in self.results if r["passed"]) - - # Dimension averages - dimension_totals: Dict[str, Dict[str, float]] = {} - for dim_name in self.evaluator.rubric.keys(): - dimension_totals[dim_name] = {"total": 0.0, "count": 0.0} - - for result in self.results: - for dim_name, score in result["evaluation"]["dimension_scores"].items(): - dimension_totals[dim_name]["total"] += score["score"] - dimension_totals[dim_name]["count"] += 1 - - dimension_averages: Dict[str, float] = {} - for dim_name, data in dimension_totals.items(): - if data["count"] > 0: - dimension_averages[dim_name] = data["total"] / data["count"] - - return { - "total_tests": len(self.results), - "passed": passed, - "failed": len(self.results) - passed, - "pass_rate": passed / len(self.results) if self.results else 0, - "dimension_averages": dimension_averages, - "failures": [ - { - "test": r["test"]["name"], - "score": r["evaluation"]["overall_score"], - } - for r in self.results - if not r["passed"] - ], - } - - -# --------------------------------------------------------------------------- -# Production Monitoring -# --------------------------------------------------------------------------- - - -class ProductionMonitor: - """Monitor agent performance in production via sampling. - - Use when: setting up continuous quality monitoring for a deployed agent. - Samples interactions at a configurable rate and tracks pass rate, average - score, and alert status. - """ - - def __init__(self, sample_rate: float = 0.01) -> None: - import random - - self.sample_rate: float = sample_rate - self._rng: random.Random = random.Random() - self.samples: List[Dict[str, Any]] = [] - self.alert_thresholds: Dict[str, float] = { - "pass_rate_warning": 0.85, - "pass_rate_critical": 0.70, - } - - def should_sample(self) -> bool: - """Determine if current interaction should be sampled. - - Use when: deciding at request time whether to evaluate this interaction. - """ - return self._rng.random() < self.sample_rate - - def record_sample( - self, query: str, output: str, evaluation: Dict[str, Any] - ) -> None: - """Record a production sample for evaluation. - - Use when: storing evaluated production interactions for trend analysis. - """ - sample: Dict[str, Any] = { - "query": query[:200], - "output_preview": output[:200], - "score": evaluation.get("overall_score", 0), - "passed": evaluation.get("passed", False), - "timestamp": time.time(), - } - self.samples.append(sample) - - def get_metrics(self) -> Dict[str, Any]: - """Calculate current metrics from collected samples. - - Use when: checking production health or generating monitoring reports. - """ - if not self.samples: - return {"status": "insufficient_data"} - - passed: int = sum(1 for s in self.samples if s["passed"]) - pass_rate: float = passed / len(self.samples) - avg_score: float = sum(s["score"] for s in self.samples) / len(self.samples) - - status: str = "healthy" - if pass_rate < self.alert_thresholds["pass_rate_critical"]: - status = "critical" - elif pass_rate < self.alert_thresholds["pass_rate_warning"]: - status = "warning" - - return { - "sample_count": len(self.samples), - "pass_rate": pass_rate, - "average_score": avg_score, - "status": status, - "alerts": self._generate_alerts(pass_rate, avg_score), - } - - def _generate_alerts( - self, pass_rate: float, avg_score: float - ) -> List[Dict[str, str]]: - """Generate alerts based on metrics.""" - alerts: List[Dict[str, str]] = [] - - if pass_rate < self.alert_thresholds["pass_rate_critical"]: - alerts.append( - { - "type": "critical", - "message": f"Pass rate ({pass_rate:.2f}) below critical threshold", - } - ) - elif pass_rate < self.alert_thresholds["pass_rate_warning"]: - alerts.append( - { - "type": "warning", - "message": f"Pass rate ({pass_rate:.2f}) below warning threshold", - } - ) - - if avg_score < 0.6: - alerts.append( - { - "type": "quality", - "message": f"Average score ({avg_score:.2f}) indicates quality issues", - } - ) - - return alerts - - -# --------------------------------------------------------------------------- -# CLI entry point -# --------------------------------------------------------------------------- - -if __name__ == "__main__": - print("=== Agent Evaluation Framework Demo ===\n") - - # 1. Create evaluator with default rubric - evaluator = AgentEvaluator() - print(f"Rubric dimensions: {list(evaluator.rubric.keys())}\n") - - # 2. Build a standard test set - test_set = TestSet("demo").create_standard_tests() - print(f"Test set: {test_set.name}") - print(f"Test count: {len(test_set.tests)}") - print(f"Complexity distribution: {test_set.get_complexity_distribution()}\n") - - # 3. Run evaluation - runner = EvaluationRunner(evaluator, test_set) - summary = runner.run_all(verbose=True) - - print(f"\n--- Summary ---") - print(f"Total: {summary['total_tests']}") - print(f"Passed: {summary['passed']}") - print(f"Failed: {summary['failed']}") - print(f"Pass rate: {summary['pass_rate']:.1%}") - print(f"Dimension averages: {summary['dimension_averages']}") - - if summary["failures"]: - print(f"\nFailures:") - for f in summary["failures"]: - print(f" - {f['test']}: {f['score']:.2f}") diff --git a/.agents/skills/explain-code/SKILL.md b/.agents/skills/explain-code/SKILL.md deleted file mode 100644 index 15b4ce6a2..000000000 --- a/.agents/skills/explain-code/SKILL.md +++ /dev/null @@ -1,232 +0,0 @@ ---- -name: explain-code -description: Explain code functionality in detail. -metadata: - provider: atomic ---- - -# Analyze and Explain Code Functionality - -## Available Tools - -- **Playwright CLI** (`playwright-cli`): Use to retrieve external documentation, browse web content, and extract data from documentation sites, forums, and GitHub repositories — especially useful for understanding third-party dependencies and their APIs. Invoke via the `/playwright-cli` skill or run `npx playwright-cli` commands directly. - - -- PREFER to use the playwright-cli (refer to playwright-cli skill) OVER web fetch/search tools - - ALWAYS load the playwright-cli skill before usage with the Skill tool. - - ALWAYS ASSUME you have the playwright-cli tool installed (if the `playwright-cli` command fails, fallback to `npx playwright-cli`). - - -### Web Fetch Strategy (token-efficient order) - -When you need external documentation about a library, framework, or API, use the **playwright-cli** skill (or `curl`) and apply these techniques in order — stop as soon as you have what you need: - -1. **Check `/llms.txt` first** — Many modern docs sites publish an AI-friendly index at `/llms.txt` (spec: [llmstxt.org](https://llmstxt.org/llms.txt)). Try `curl https:///llms.txt` before anything else; it often links directly to the most relevant pages in plain text. -2. **Request Markdown via `Accept: text/markdown`** — For any HTML page, try `curl -H "Accept: text/markdown"` first. Sites behind Cloudflare with [Markdown for Agents](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/) will return pre-converted Markdown (look for `content-type: text/markdown` and the `x-markdown-tokens` header), which is far cheaper than raw HTML. -3. **Fall back to HTML parsing** — If neither above yields usable content, navigate the page with `playwright-cli` to extract the rendered DOM, or `curl` the raw HTML and parse locally. - -**Persist useful findings to `research/web/`:** When you fetch a document worth keeping for future sessions (API references, SDK guides, troubleshooting docs), save it to `research/web/-.md` with a short header noting the source URL and fetch date. Check this directory first before re-fetching. - -## Instructions - -Follow this systematic approach to explain code: **$ARGUMENTS** - -1. **Code Context Analysis** - - Identify the programming language and framework - - Understand the broader context and purpose of the code - - Identify the file location and its role in the project - - Review related imports, dependencies, and configurations - -2. **High-Level Overview** - - Provide a summary of what the code does - - Explain the main purpose and functionality - - Identify the problem the code is solving - - Describe how it fits into the larger system - -3. **Code Structure Breakdown** - - Break down the code into logical sections - - Identify classes, functions, and methods - - Explain the overall architecture and design patterns - - Map out data flow and control flow - -4. **Line-by-Line Analysis** - - Explain complex or non-obvious lines of code - - Describe variable declarations and their purposes - - Explain function calls and their parameters - - Clarify conditional logic and loops - -5. **Algorithm and Logic Explanation** - - Describe the algorithm or approach being used - - Explain the logic behind complex calculations - - Break down nested conditions and loops - - Clarify recursive or asynchronous operations - -6. **Data Structures and Types** - - Explain data types and structures being used - - Describe how data is transformed or processed - - Explain object relationships and hierarchies - - Clarify input and output formats - -7. **Framework and Library Usage** - - Explain framework-specific patterns and conventions - - Describe library functions and their purposes - - Explain API calls and their expected responses - - Clarify configuration and setup code - - Use the **playwright-cli** skill (following the Web Fetch Strategy above) to look up external library documentation when needed - -8. **Error Handling and Edge Cases** - - Explain error handling mechanisms - - Describe exception handling and recovery - - Identify edge cases being handled - - Explain validation and defensive programming - -9. **Performance Considerations** - - Identify performance-critical sections - - Explain optimization techniques being used - - Describe complexity and scalability implications - - Point out potential bottlenecks or inefficiencies - -10. **Security Implications** - - Identify security-related code sections - - Explain authentication and authorization logic - - Describe input validation and sanitization - - Point out potential security vulnerabilities - -11. **Testing and Debugging** - - Explain how the code can be tested - - Identify debugging points and logging - - Describe mock data or test scenarios - - Explain test helpers and utilities - - Use `playwright-cli` to reproduce browser interactions and validate frontend behavior when applicable - -12. **Dependencies and Integrations** - - Explain external service integrations - - Describe database operations and queries - - Explain API interactions and protocols - - Clarify third-party library usage - -**Explanation Format Examples:** - -**For Complex Algorithms:** - -``` -This function implements a depth-first search algorithm: - -1. Line 1-3: Initialize a stack with the starting node and a visited set -2. Line 4-8: Main loop - continue until stack is empty -3. Line 9-11: Pop a node and check if it's the target -4. Line 12-15: Add unvisited neighbors to the stack -5. Line 16: Return null if target not found - -Time Complexity: O(V + E) where V is vertices and E is edges -Space Complexity: O(V) for the visited set and stack -``` - -**For API Integration Code:** - -``` -This code handles user authentication with a third-party service: - -1. Extract credentials from request headers -2. Validate credential format and required fields -3. Make API call to authentication service -4. Handle response and extract user data -5. Create session token and set cookies -6. Return user profile or error response - -Error Handling: Catches network errors, invalid credentials, and service unavailability -Security: Uses HTTPS, validates inputs, and sanitizes responses -``` - -**For Database Operations:** - -``` -This function performs a complex database query with joins: - -1. Build base query with primary table -2. Add LEFT JOIN for related user data -3. Apply WHERE conditions for filtering -4. Add ORDER BY for consistent sorting -5. Implement pagination with LIMIT/OFFSET -6. Execute query and handle potential errors -7. Transform raw results into domain objects - -Performance Notes: Uses indexes on filtered columns, implements connection pooling -``` - -13. **Common Patterns and Idioms** - - Identify language-specific patterns and idioms - - Explain design patterns being implemented - - Describe architectural patterns in use - - Clarify naming conventions and code style - -14. **Potential Improvements** - - Suggest code improvements and optimizations - - Identify possible refactoring opportunities - - Point out maintainability concerns - - Recommend best practices and standards - -15. **Related Code and Context** - - Reference related functions and classes - - Explain how this code interacts with other components - - Describe the calling context and usage patterns - - Point to relevant documentation and resources - -16. **Debugging and Troubleshooting** - - Explain how to debug issues in this code - - Identify common failure points - - Describe logging and monitoring approaches - - Suggest testing strategies - -**Language-Specific Considerations:** - -**JavaScript/TypeScript:** - -- Explain async/await and Promise handling -- Describe closure and scope behavior -- Clarify this binding and arrow functions -- Explain event handling and callbacks - -**Python:** - -- Explain list comprehensions and generators -- Describe decorator usage and purpose -- Clarify context managers and with statements -- Explain class inheritance and method resolution - -**Java:** - -- Explain generics and type parameters -- Describe annotation usage and processing -- Clarify stream operations and lambda expressions -- Explain exception hierarchy and handling - -**C#:** - -- Explain LINQ queries and expressions -- Describe async/await and Task handling -- Clarify delegate and event usage -- Explain nullable reference types - -**Go:** - -- Explain goroutines and channel usage -- Describe interface implementation -- Clarify error handling patterns -- Explain package structure and imports - -**Rust:** - -- Explain ownership and borrowing -- Describe lifetime annotations -- Clarify pattern matching and Option/Result types -- Explain trait implementations - -Remember to: - -- Use clear, non-technical language when possible -- Provide examples and analogies for complex concepts -- Structure explanations logically from high-level to detailed -- Include visual diagrams or flowcharts when helpful -- Tailor the explanation level to the intended audience -- Use the **playwright-cli** skill (following the Web Fetch Strategy above) to look up external library documentation when encountering unfamiliar dependencies, and for live browser inspection when static code analysis is not enough diff --git a/.agents/skills/filesystem-context/references/implementation-patterns.md b/.agents/skills/filesystem-context/references/implementation-patterns.md deleted file mode 100644 index 6487d650d..000000000 --- a/.agents/skills/filesystem-context/references/implementation-patterns.md +++ /dev/null @@ -1,549 +0,0 @@ -# Filesystem Context Implementation Patterns - -This reference provides detailed implementation patterns for filesystem-based context engineering. - -## Pattern Catalog - -### 1. Scratch Pad Manager - -A centralized manager for handling large tool outputs and intermediate results. - -```python -import os -import json -from datetime import datetime -from pathlib import Path - -class ScratchPadManager: - """Manages temporary file storage for agent context offloading.""" - - def __init__(self, base_path: str = "scratch", token_threshold: int = 2000): - self.base_path = Path(base_path) - self.base_path.mkdir(parents=True, exist_ok=True) - self.token_threshold = token_threshold - self.manifest = {} - - def should_offload(self, content: str) -> bool: - """Determine if content exceeds threshold for offloading.""" - # Rough token estimate: 1 token ≈ 4 characters - estimated_tokens = len(content) // 4 - return estimated_tokens > self.token_threshold - - def offload(self, content: str, source: str, summary: str = None) -> dict: - """Write content to file, return reference.""" - timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") - filename = f"{source}_{timestamp}.txt" - file_path = self.base_path / filename - - file_path.write_text(content) - - reference = { - "type": "file_reference", - "path": str(file_path), - "source": source, - "timestamp": timestamp, - "size_chars": len(content), - "summary": summary or self._extract_summary(content) - } - - self.manifest[filename] = reference - return reference - - def _extract_summary(self, content: str, max_chars: int = 500) -> str: - """Extract first meaningful content as summary.""" - lines = content.strip().split('\n') - summary_lines = [] - char_count = 0 - - for line in lines: - if char_count + len(line) > max_chars: - break - summary_lines.append(line) - char_count += len(line) - - return '\n'.join(summary_lines) - - def cleanup(self, max_age_hours: int = 24): - """Remove scratch files older than threshold.""" - cutoff = datetime.now().timestamp() - (max_age_hours * 3600) - - for file_path in self.base_path.glob("*.txt"): - if file_path.stat().st_mtime < cutoff: - file_path.unlink() - if file_path.name in self.manifest: - del self.manifest[file_path.name] -``` - -### 2. Plan Persistence - -Structured plan storage with progress tracking. - -```python -import yaml -from dataclasses import dataclass, field, asdict -from enum import Enum -from typing import List, Optional - -class StepStatus(Enum): - PENDING = "pending" - IN_PROGRESS = "in_progress" - COMPLETED = "completed" - BLOCKED = "blocked" - CANCELLED = "cancelled" - -@dataclass -class PlanStep: - id: int - description: str - status: StepStatus = StepStatus.PENDING - notes: Optional[str] = None - -@dataclass -class AgentPlan: - objective: str - steps: List[PlanStep] = field(default_factory=list) - status: str = "in_progress" - - def save(self, path: str = "scratch/current_plan.yaml"): - """Persist plan to filesystem.""" - data = { - "objective": self.objective, - "status": self.status, - "steps": [ - { - "id": s.id, - "description": s.description, - "status": s.status.value, - "notes": s.notes - } - for s in self.steps - ] - } - with open(path, 'w') as f: - yaml.dump(data, f, default_flow_style=False) - - @classmethod - def load(cls, path: str = "scratch/current_plan.yaml") -> "AgentPlan": - """Load plan from filesystem.""" - with open(path, 'r') as f: - data = yaml.safe_load(f) - - plan = cls(objective=data["objective"], status=data.get("status", "in_progress")) - for step_data in data.get("steps", []): - plan.steps.append(PlanStep( - id=step_data["id"], - description=step_data["description"], - status=StepStatus(step_data["status"]), - notes=step_data.get("notes") - )) - return plan - - def current_step(self) -> Optional[PlanStep]: - """Get the first non-completed step.""" - for step in self.steps: - if step.status != StepStatus.COMPLETED: - return step - return None - - def complete_step(self, step_id: int, notes: str = None): - """Mark step as completed.""" - for step in self.steps: - if step.id == step_id: - step.status = StepStatus.COMPLETED - if notes: - step.notes = notes - break -``` - -### 3. Sub-Agent Workspace - -File-based communication between agents. - -```python -from pathlib import Path -from datetime import datetime -import json - -class AgentWorkspace: - """Manages file-based workspace for an agent.""" - - def __init__(self, agent_id: str, base_path: str = "workspace/agents"): - self.agent_id = agent_id - self.path = Path(base_path) / agent_id - self.path.mkdir(parents=True, exist_ok=True) - - # Standard files - self.findings_file = self.path / "findings.md" - self.status_file = self.path / "status.json" - self.log_file = self.path / "activity.log" - - def write_finding(self, content: str, append: bool = True): - """Write or append a finding.""" - mode = 'a' if append else 'w' - with open(self.findings_file, mode) as f: - if append: - f.write(f"\n---\n## {datetime.now().isoformat()}\n\n") - f.write(content) - - def update_status(self, status: str, progress: float = None, details: dict = None): - """Update agent status for coordinator visibility.""" - status_data = { - "agent_id": self.agent_id, - "status": status, - "updated_at": datetime.now().isoformat(), - "progress": progress, - "details": details or {} - } - self.status_file.write_text(json.dumps(status_data, indent=2)) - - def log(self, message: str): - """Append to activity log.""" - with open(self.log_file, 'a') as f: - f.write(f"[{datetime.now().isoformat()}] {message}\n") - - def read_peer_findings(self, peer_id: str) -> str: - """Read findings from another agent's workspace.""" - peer_path = self.path.parent / peer_id / "findings.md" - if peer_path.exists(): - return peer_path.read_text() - return "" - - -class CoordinatorWorkspace: - """Coordinator that reads from sub-agent workspaces.""" - - def __init__(self, base_path: str = "workspace/agents"): - self.base_path = Path(base_path) - - def get_all_statuses(self) -> dict: - """Collect status from all sub-agents.""" - statuses = {} - for agent_dir in self.base_path.iterdir(): - if agent_dir.is_dir(): - status_file = agent_dir / "status.json" - if status_file.exists(): - statuses[agent_dir.name] = json.loads(status_file.read_text()) - return statuses - - def aggregate_findings(self) -> str: - """Combine all agent findings into synthesis.""" - findings = [] - for agent_dir in self.base_path.iterdir(): - if agent_dir.is_dir(): - findings_file = agent_dir / "findings.md" - if findings_file.exists(): - findings.append(f"# {agent_dir.name}\n\n{findings_file.read_text()}") - return "\n\n".join(findings) -``` - -### 4. Dynamic Skill Loader - -Load skill content on demand. - -```python -from pathlib import Path -from typing import List, Optional -import yaml - -@dataclass -class SkillMetadata: - name: str - description: str - path: str - triggers: List[str] = field(default_factory=list) - -class SkillLoader: - """Manages dynamic loading of agent skills.""" - - def __init__(self, skills_path: str = "skills"): - self.skills_path = Path(skills_path) - self.skill_index = self._build_index() - - def _build_index(self) -> dict: - """Build index of available skills from SKILL.md frontmatter.""" - index = {} - for skill_dir in self.skills_path.iterdir(): - if skill_dir.is_dir(): - skill_file = skill_dir / "SKILL.md" - if skill_file.exists(): - metadata = self._parse_frontmatter(skill_file) - if metadata: - index[metadata.name] = metadata - return index - - def _parse_frontmatter(self, path: Path) -> Optional[SkillMetadata]: - """Extract YAML frontmatter from skill file.""" - content = path.read_text() - if content.startswith('---'): - end = content.find('---', 3) - if end > 0: - frontmatter = yaml.safe_load(content[3:end]) - return SkillMetadata( - name=frontmatter.get('name', path.parent.name), - description=frontmatter.get('description', ''), - path=str(path), - triggers=frontmatter.get('triggers', []) - ) - return None - - def get_static_context(self) -> str: - """Generate minimal static context listing available skills.""" - lines = ["Available skills (load with read_file when relevant):"] - for name, meta in self.skill_index.items(): - lines.append(f"- {name}: {meta.description[:100]}") - return "\n".join(lines) - - def load_skill(self, name: str) -> str: - """Load full skill content.""" - if name in self.skill_index: - return Path(self.skill_index[name].path).read_text() - raise ValueError(f"Unknown skill: {name}") - - def find_relevant_skills(self, query: str) -> List[str]: - """Find skills that might be relevant to a query.""" - query_lower = query.lower() - relevant = [] - for name, meta in self.skill_index.items(): - if any(trigger in query_lower for trigger in meta.triggers): - relevant.append(name) - elif name.replace('-', ' ') in query_lower: - relevant.append(name) - return relevant -``` - -### 5. Terminal Output Persistence - -Capture and persist terminal sessions. - -```python -import subprocess -from pathlib import Path -from datetime import datetime -import re - -class TerminalCapture: - """Captures and persists terminal output for agent access.""" - - def __init__(self, terminals_path: str = "terminals"): - self.terminals_path = Path(terminals_path) - self.terminals_path.mkdir(parents=True, exist_ok=True) - self.session_counter = 0 - - def run_command(self, command: str, capture: bool = True) -> dict: - """Run command and optionally capture output to file.""" - self.session_counter += 1 - - result = subprocess.run( - command, - shell=True, - capture_output=True, - text=True - ) - - output = { - "command": command, - "exit_code": result.returncode, - "stdout": result.stdout, - "stderr": result.stderr, - "timestamp": datetime.now().isoformat() - } - - if capture: - output["file"] = self._persist_output(output) - - return output - - def _persist_output(self, output: dict) -> str: - """Write output to terminal file.""" - filename = f"{self.session_counter}.txt" - file_path = self.terminals_path / filename - - content = f"""--- -command: {output['command']} -exit_code: {output['exit_code']} -timestamp: {output['timestamp']} ---- - -=== STDOUT === -{output['stdout']} - -=== STDERR === -{output['stderr']} -""" - file_path.write_text(content) - return str(file_path) - - def grep_terminals(self, pattern: str, context_lines: int = 3) -> List[dict]: - """Search all terminal outputs for pattern.""" - matches = [] - regex = re.compile(pattern, re.IGNORECASE) - - for term_file in self.terminals_path.glob("*.txt"): - content = term_file.read_text() - lines = content.split('\n') - - for i, line in enumerate(lines): - if regex.search(line): - start = max(0, i - context_lines) - end = min(len(lines), i + context_lines + 1) - matches.append({ - "file": str(term_file), - "line_number": i + 1, - "context": '\n'.join(lines[start:end]) - }) - - return matches -``` - -### 6. Self-Modification Guard - -Safe pattern for agent self-learning. - -```python -import yaml -from pathlib import Path -from datetime import datetime -from typing import Any - -class PreferenceStore: - """Guarded storage for agent-learned preferences.""" - - MAX_ENTRIES = 100 - MAX_VALUE_LENGTH = 1000 - - def __init__(self, path: str = "agent/preferences.yaml"): - self.path = Path(path) - self.path.parent.mkdir(parents=True, exist_ok=True) - self.preferences = self._load() - - def _load(self) -> dict: - """Load preferences from file.""" - if self.path.exists(): - return yaml.safe_load(self.path.read_text()) or {} - return {} - - def _save(self): - """Persist preferences to file.""" - self.path.write_text(yaml.dump(self.preferences, default_flow_style=False)) - - def remember(self, key: str, value: Any, source: str = "user"): - """Store a preference with validation.""" - # Validate key - if not key or len(key) > 100: - raise ValueError("Invalid key length") - - # Validate value - value_str = str(value) - if len(value_str) > self.MAX_VALUE_LENGTH: - raise ValueError(f"Value exceeds max length of {self.MAX_VALUE_LENGTH}") - - # Check entry limit - if len(self.preferences) >= self.MAX_ENTRIES and key not in self.preferences: - raise ValueError(f"Max entries ({self.MAX_ENTRIES}) reached") - - # Store with metadata - self.preferences[key] = { - "value": value, - "source": source, - "updated_at": datetime.now().isoformat() - } - self._save() - - def recall(self, key: str, default: Any = None) -> Any: - """Retrieve a preference.""" - entry = self.preferences.get(key) - if entry: - return entry["value"] - return default - - def list_all(self) -> dict: - """Get all preferences for context injection.""" - return {k: v["value"] for k, v in self.preferences.items()} - - def forget(self, key: str): - """Remove a preference.""" - if key in self.preferences: - del self.preferences[key] - self._save() -``` - -## Integration Example - -Combining patterns in an agent harness: - -```python -class FilesystemContextAgent: - """Agent with filesystem-based context management.""" - - def __init__(self): - self.scratch = ScratchPadManager() - self.skills = SkillLoader() - self.preferences = PreferenceStore() - self.workspace = AgentWorkspace("main_agent") - - def handle_tool_output(self, tool_name: str, output: str) -> str: - """Process tool output, offloading if necessary.""" - if self.scratch.should_offload(output): - ref = self.scratch.offload(output, source=tool_name) - return f"[{tool_name} output saved to {ref['path']}. Summary: {ref['summary'][:200]}]" - return output - - def get_system_prompt(self) -> str: - """Build system prompt with dynamic skill references.""" - base_prompt = "You are a helpful assistant." - skill_context = self.skills.get_static_context() - user_prefs = self.preferences.list_all() - - pref_section = "" - if user_prefs: - pref_section = "\n\nUser preferences:\n" + "\n".join( - f"- {k}: {v}" for k, v in user_prefs.items() - ) - - return f"{base_prompt}\n\n{skill_context}{pref_section}" -``` - -## File Organization Best Practices - -``` -project/ -├── scratch/ # Ephemeral working files -│ ├── tool_outputs/ # Large tool results -│ │ └── search_20260107.txt -│ └── plans/ # Active task plans -│ └── current_plan.yaml -├── workspace/ # Agent workspaces -│ └── agents/ -│ ├── research_agent/ -│ │ ├── findings.md -│ │ └── status.json -│ └── code_agent/ -│ ├── findings.md -│ └── status.json -├── agent/ # Agent configuration -│ ├── preferences.yaml # Learned preferences -│ └── patterns.md # Discovered patterns -├── skills/ # Loadable skills -│ └── {skill-name}/ -│ └── SKILL.md -├── terminals/ # Terminal output -│ ├── 1.txt -│ └── 2.txt -└── history/ # Chat history archives - └── session_001.txt -``` - -## Token Accounting Metrics - -Track these metrics to validate filesystem patterns: - -1. **Static context ratio**: tokens in static context / total tokens -2. **Dynamic load rate**: how often skills/files are loaded per task -3. **Offload savings**: tokens saved by writing to files vs keeping in context -4. **Retrieval precision**: percentage of loaded content actually used - -Target benchmarks: -- Static context ratio < 20% -- Offload savings > 50% for tool-heavy workflows -- Retrieval precision > 70% (loaded content is relevant) - diff --git a/.agents/skills/filesystem-context/scripts/filesystem_context.py b/.agents/skills/filesystem-context/scripts/filesystem_context.py deleted file mode 100644 index 574ef4f68..000000000 --- a/.agents/skills/filesystem-context/scripts/filesystem_context.py +++ /dev/null @@ -1,425 +0,0 @@ -""" -Filesystem Context Manager -- composable utilities for filesystem-based context engineering. - -Provides three core patterns for managing agent context through the filesystem: -1. ScratchPadManager -- offload large tool outputs to files, return compact references -2. AgentPlan / PlanStep -- persist plans to disk so agents survive context window refreshes -3. ToolOutputHandler -- automatic offload-or-inline decision for tool outputs - -Use when: - - Tool outputs exceed ~2000 tokens and would bloat the context window - - Agents need plan persistence across long-horizon, multi-turn tasks - - Building agent systems that offload intermediate results to files - -Example (library usage):: - - from filesystem_context import ScratchPadManager, ToolOutputHandler - - handler = ToolOutputHandler(ScratchPadManager(base_path="scratch")) - result = handler.process_output("web_search", large_output_string) - -Example (CLI demo):: - - python filesystem_context.py -""" - -from __future__ import annotations - -import json -import os -import shutil -from dataclasses import dataclass, field -from datetime import datetime -from pathlib import Path -from typing import Any, Dict, List, Optional - -__all__: list[str] = [ - "ScratchPadManager", - "PlanStep", - "AgentPlan", - "ToolOutputHandler", -] - - -# ============================================================================= -# Pattern 1: Scratch Pad Manager -# ============================================================================= - - -class ScratchPadManager: - """Manage temporary file storage for offloading large tool outputs. - - Use when: tool outputs exceed a token threshold and would bloat the - context window. Writes content to a scratch directory and returns a - compact reference the agent can include in context instead. - """ - - def __init__(self, base_path: str = "scratch", token_threshold: int = 2000) -> None: - self.base_path: Path = Path(base_path) - self.base_path.mkdir(parents=True, exist_ok=True) - self.token_threshold: int = token_threshold - - def estimate_tokens(self, content: str) -> int: - """Return a rough token estimate (~4 characters per token). - - Use when: deciding whether content should be offloaded before - writing it to disk. - """ - return len(content) // 4 - - def should_offload(self, content: str) -> bool: - """Return True if *content* exceeds the configured token threshold. - - Use when: making an inline-vs-offload decision for a tool output. - """ - return self.estimate_tokens(content) > self.token_threshold - - def offload(self, content: str, source: str) -> Dict[str, Any]: - """Write *content* to a timestamped scratch file and return a reference dict. - - Use when: a tool output has been determined to exceed the threshold - and should be persisted to disk. - - Returns a dict with keys: path, source, tokens_saved, summary. - """ - timestamp: str = datetime.now().strftime("%Y%m%d_%H%M%S_%f") - filename: str = f"{source}_{timestamp}.txt" - file_path: Path = self.base_path / filename - - file_path.write_text(content) - - # Extract summary from first meaningful lines - lines: list[str] = content.strip().split("\n")[:5] - summary: str = "\n".join(lines) - if len(summary) > 300: - summary = summary[:300] + "..." - - return { - "path": str(file_path), - "source": source, - "tokens_saved": self.estimate_tokens(content), - "summary": summary, - } - - def format_reference(self, ref: Dict[str, Any]) -> str: - """Format a reference dict as a compact string for context inclusion. - - Use when: constructing the replacement message that goes into context - in place of the full tool output. - """ - return ( - f"[Output from {ref['source']} saved to {ref['path']}. " - f"~{ref['tokens_saved']} tokens. " - f"Summary: {ref['summary'][:200]}]" - ) - - def cleanup(self, max_age_seconds: int = 3600) -> int: - """Remove scratch files older than *max_age_seconds*. - - Use when: ending a session or when the scratch directory has grown - large enough to slow directory listings. - - Returns the number of files removed. - """ - removed: int = 0 - now: float = datetime.now().timestamp() - for f in self.base_path.iterdir(): - if f.is_file() and (now - f.stat().st_mtime) > max_age_seconds: - f.unlink() - removed += 1 - return removed - - -# ============================================================================= -# Pattern 2: Plan Persistence -# ============================================================================= - - -@dataclass -class PlanStep: - """Individual step in an agent plan. - - Use when: building a plan that will be persisted to disk for later - re-reading across context window boundaries. - """ - - id: int - description: str - status: str = "pending" # pending | in_progress | completed | blocked - notes: Optional[str] = None - - -@dataclass -class AgentPlan: - """Persistent plan that survives context window limitations. - - Use when: an agent needs to track a multi-step objective across turns - or context refreshes. Write the plan to disk so the agent can re-read - it at any point, even after summarization or context window refresh. - """ - - objective: str - steps: List[PlanStep] = field(default_factory=list) - created_at: str = field(default_factory=lambda: datetime.now().isoformat()) - - def to_dict(self) -> Dict[str, Any]: - """Serialize the plan to a plain dict suitable for JSON output.""" - return { - "objective": self.objective, - "created_at": self.created_at, - "steps": [ - { - "id": s.id, - "description": s.description, - "status": s.status, - "notes": s.notes, - } - for s in self.steps - ], - } - - def save(self, path: str = "scratch/current_plan.json") -> None: - """Persist plan to *path* as JSON. - - Use when: a plan has been created or updated and must survive a - potential context refresh. - """ - Path(path).parent.mkdir(parents=True, exist_ok=True) - with open(path, "w") as f: - json.dump(self.to_dict(), f, indent=2) - print(f"Plan saved to {path}") - - @classmethod - def load(cls, path: str = "scratch/current_plan.json") -> AgentPlan: - """Load a plan from *path*. - - Use when: resuming work in a new context window or after - summarization -- re-read the plan to restore task awareness. - """ - with open(path, "r") as f: - data: Dict[str, Any] = json.load(f) - - plan = cls(objective=data["objective"]) - plan.created_at = data.get("created_at", "") - - for step_data in data.get("steps", []): - plan.steps.append( - PlanStep( - id=step_data["id"], - description=step_data["description"], - status=step_data["status"], - notes=step_data.get("notes"), - ) - ) - return plan - - def current_step(self) -> Optional[PlanStep]: - """Return the first non-completed step, or None if all are done. - - Use when: determining what to work on next after re-reading a plan. - """ - for step in self.steps: - if step.status not in ("completed", "cancelled"): - return step - return None - - def complete_step(self, step_id: int, notes: Optional[str] = None) -> None: - """Mark step *step_id* as completed, optionally attaching *notes*. - - Use when: an agent finishes a plan step and needs to record - progress before persisting the updated plan. - """ - for step in self.steps: - if step.id == step_id: - step.status = "completed" - if notes: - step.notes = notes - return - raise ValueError(f"Step {step_id} not found") - - def progress_summary(self) -> str: - """Generate a compact progress string for context injection. - - Use when: the agent needs a one-line status to include in context - without re-reading the full plan. - """ - completed: int = sum(1 for s in self.steps if s.status == "completed") - total: int = len(self.steps) - current: Optional[PlanStep] = self.current_step() - - summary: str = f"Objective: {self.objective}\n" - summary += f"Progress: {completed}/{total} steps completed\n" - if current: - summary += f"Current step: [{current.id}] {current.description}" - else: - summary += "All steps completed." - - return summary - - -# ============================================================================= -# Pattern 3: Tool Output Handler -# ============================================================================= - - -class ToolOutputHandler: - """Automatically decide whether to inline or offload tool outputs. - - Use when: building an agent loop that processes heterogeneous tool - outputs -- some small enough to inline, others requiring offload. - """ - - def __init__(self, scratch_pad: Optional[ScratchPadManager] = None) -> None: - self.scratch_pad: ScratchPadManager = scratch_pad or ScratchPadManager() - - def process_output(self, tool_name: str, output: str) -> str: - """Return *output* directly if small, or a file reference if large. - - Use when: handling a tool's return value in an agent loop. Pass - the result into context; offloading happens transparently. - """ - if self.scratch_pad.should_offload(output): - ref: Dict[str, Any] = self.scratch_pad.offload(output, source=tool_name) - return self.scratch_pad.format_reference(ref) - return output - - -# ============================================================================= -# Demonstration -# ============================================================================= - - -def _demo_scratch_pad() -> None: - """Demonstrate the scratch pad offloading pattern.""" - print("=" * 60) - print("DEMO: Scratch Pad for Tool Output Offloading") - print("=" * 60) - - scratch = ScratchPadManager(base_path="demo_scratch", token_threshold=100) - - # Small output stays in context - small_output: str = "API returned: {'status': 'ok', 'data': [1, 2, 3]}" - print(f"\nSmall output ({scratch.estimate_tokens(small_output)} tokens):") - print(f" Should offload: {scratch.should_offload(small_output)}") - - # Large output gets offloaded - large_output: str = """ -Search Results for "context engineering": - -1. Context Engineering: The Art of Curating LLM Context - URL: https://example.com/article1 - Snippet: Context engineering is the discipline of managing what information - enters the language model's context window. Unlike prompt engineering which - focuses on instruction crafting, context engineering addresses the holistic - curation of all information... - -2. Building Production Agents with Effective Context Management - URL: https://example.com/article2 - Snippet: Production agent systems require sophisticated context management - strategies. This includes compression, caching, and strategic partitioning - of work across sub-agents with isolated contexts... - -3. The Lost-in-Middle Problem and How to Avoid It - URL: https://example.com/article3 - Snippet: Research shows that language models exhibit U-shaped attention - patterns, with information in the middle of long contexts receiving less - attention than content at the beginning or end... - -... (imagine 50 more results) ... -""" - - print(f"\nLarge output ({scratch.estimate_tokens(large_output)} tokens):") - print(f" Should offload: {scratch.should_offload(large_output)}") - - if scratch.should_offload(large_output): - ref = scratch.offload(large_output, source="web_search") - print(f"\nOffloaded to: {ref['path']}") - print(f"Tokens saved: {ref['tokens_saved']}") - print(f"\nReference for context:\n{scratch.format_reference(ref)}") - - -def _demo_plan_persistence() -> None: - """Demonstrate the plan persistence pattern.""" - print("\n" + "=" * 60) - print("DEMO: Plan Persistence for Long-Horizon Tasks") - print("=" * 60) - - plan = AgentPlan(objective="Refactor authentication module") - plan.steps = [ - PlanStep(id=1, description="Audit current auth endpoints"), - PlanStep(id=2, description="Design new token validation flow"), - PlanStep(id=3, description="Implement changes"), - PlanStep(id=4, description="Write tests"), - PlanStep(id=5, description="Deploy and monitor"), - ] - - print("\nInitial plan:") - print(plan.progress_summary()) - - plan.save("demo_scratch/current_plan.json") - - # Simulate completing first step - plan.complete_step(1, notes="Found 12 endpoints, 3 need updates") - plan.steps[1].status = "in_progress" - - print("\nAfter completing step 1:") - print(plan.progress_summary()) - - plan.save("demo_scratch/current_plan.json") - - # Simulate loading from file (as if in new context) - print("\n--- Simulating context refresh ---") - loaded_plan = AgentPlan.load("demo_scratch/current_plan.json") - print("\nPlan loaded from file:") - print(loaded_plan.progress_summary()) - - -def _demo_tool_handler() -> None: - """Demonstrate the integrated tool output handler.""" - print("\n" + "=" * 60) - print("DEMO: Integrated Tool Output Handler") - print("=" * 60) - - handler = ToolOutputHandler( - scratch_pad=ScratchPadManager(base_path="demo_scratch", token_threshold=50) - ) - - outputs: list[tuple[str, str]] = [ - ("calculator", "42"), - ("file_read", "Error: File not found"), - ( - "database_query", - """ - Results (250 rows): - | id | name | email | created_at | status | - |----|------|-------|------------|--------| - | 1 | John | j@e.c | 2024-01-01 | active | - | 2 | Jane | j@e.c | 2024-01-02 | active | - ... (248 more rows) ... - """, - ), - ] - - for tool_name, output in outputs: - processed: str = handler.process_output(tool_name, output) - print(f"\n{tool_name}:") - print(f" Original length: {len(output)} chars") - print(f" Processed: {processed[:100]}...") - - -def _cleanup_demo() -> None: - """Remove demo files created during the demonstration.""" - demo_path = Path("demo_scratch") - if demo_path.exists(): - shutil.rmtree(demo_path) - print("\nDemo files cleaned up.") - - -if __name__ == "__main__": - _demo_scratch_pad() - _demo_plan_persistence() - _demo_tool_handler() - - print("\n" + "=" * 60) - print("Cleaning up demo files...") - _cleanup_demo() diff --git a/.agents/skills/find-skills/SKILL.md b/.agents/skills/find-skills/SKILL.md deleted file mode 100644 index 0e86636a6..000000000 --- a/.agents/skills/find-skills/SKILL.md +++ /dev/null @@ -1,144 +0,0 @@ ---- -name: find-skills -description: Helps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skill should be used when the user is looking for functionality that might exist as an installable skill. -metadata: - provider: atomic ---- - -# Find Skills - -This skill helps you discover and install skills from the open agent skills ecosystem. - -## When to Use This Skill - -Use this skill when the user: - -- Asks "how do I do X" where X might be a common task with an existing skill -- Says "find a skill for X" or "is there a skill for X" -- Asks "can you do X" where X is a specialized capability -- Expresses interest in extending agent capabilities -- Wants to search for tools, templates, or workflows -- Mentions they wish they had help with a specific domain (design, testing, deployment, etc.) - -## What is the Skills CLI? - -The Skills CLI (`npx skills`) is the package manager for the open agent skills ecosystem. Skills are modular packages that extend agent capabilities with specialized knowledge, workflows, and tools. - -**Key commands:** - -- `npx skills find [query]` - Search for skills interactively or by keyword -- `npx skills add ` - Install a skill from GitHub or other sources -- `npx skills check` - Check for skill updates -- `npx skills update` - Update all installed skills - -**Browse skills at:** https://skills.sh/ - -## How to Help Users Find Skills - -### Step 1: Understand What They Need - -When a user asks for help with something, identify: - -1. The domain (e.g., React, testing, design, deployment) -2. The specific task (e.g., writing tests, creating animations, reviewing PRs) -3. Whether this is a common enough task that a skill likely exists - -### Step 2: Check the Leaderboard First - -Before running a CLI search, check the [skills.sh leaderboard](https://skills.sh/) to see if a well-known skill already exists for the domain. The leaderboard ranks skills by total installs, surfacing the most popular and battle-tested options. - -For example, top skills for web development include: -- `vercel-labs/agent-skills` — React, Next.js, web design (100K+ installs each) -- `anthropics/skills` — Frontend design, document processing (100K+ installs) - -### Step 3: Search for Skills - -If the leaderboard doesn't cover the user's need, run the find command: - -```bash -npx skills find [query] -``` - -For example: - -- User asks "how do I make my React app faster?" → `npx skills find react performance` -- User asks "can you help me with PR reviews?" → `npx skills find pr review` -- User asks "I need to create a changelog" → `npx skills find changelog` - -### Step 4: Verify Quality Before Recommending - -**Do not recommend a skill based solely on search results.** Always verify: - -1. **Install count** — Prefer skills with 1K+ installs. Be cautious with anything under 100. -2. **Source reputation** — Official sources (`vercel-labs`, `anthropics`, `microsoft`) are more trustworthy than unknown authors. -3. **GitHub stars** — Check the source repository. A skill from a repo with <100 stars should be treated with skepticism. - -### Step 5: Present Options to the User - -When you find relevant skills, present them to the user with: - -1. The skill name and what it does -2. The install count and source -3. The install command they can run -4. A link to learn more at skills.sh - -Example response: - -``` -I found a skill that might help! The "react-best-practices" skill provides -React and Next.js performance optimization guidelines from Vercel Engineering. -(185K installs) - -To install it: -npx skills add vercel-labs/agent-skills@react-best-practices - -Learn more: https://skills.sh/vercel-labs/agent-skills/react-best-practices -``` - -### Step 6: Offer to Install - -If the user wants to proceed, you can install the skill for them: - -```bash -npx skills add -g -y -``` - -The `-g` flag installs globally (user-level) and `-y` skips confirmation prompts. - -## Common Skill Categories - -When searching, consider these common categories: - -| Category | Example Queries | -| --------------- | ---------------------------------------- | -| Web Development | react, nextjs, typescript, css, tailwind | -| Testing | testing, jest, playwright, e2e | -| DevOps | deploy, docker, kubernetes, ci-cd | -| Documentation | docs, readme, changelog, api-docs | -| Code Quality | review, lint, refactor, best-practices | -| Design | ui, ux, design-system, accessibility | -| Productivity | workflow, automation, git | - -## Tips for Effective Searches - -1. **Use specific keywords**: "react testing" is better than just "testing" -2. **Try alternative terms**: If "deploy" doesn't work, try "deployment" or "ci-cd" -3. **Check popular sources**: Many skills come from `vercel-labs/agent-skills` or `ComposioHQ/awesome-claude-skills` - -## When No Skills Are Found - -If no relevant skills exist: - -1. Acknowledge that no existing skill was found -2. Offer to help with the task directly using your general capabilities -3. Suggest the user could create their own skill with `npx skills init` - -Example: - -``` -I searched for skills related to "xyz" but didn't find any matches. -I can still help you with this task directly! Would you like me to proceed? - -If this is something you do often, you could create your own skill: -npx skills init my-xyz-skill -``` diff --git a/.agents/skills/gh-commit/SKILL.md b/.agents/skills/gh-commit/SKILL.md index 3e1e4a459..e56be2b06 100644 --- a/.agents/skills/gh-commit/SKILL.md +++ b/.agents/skills/gh-commit/SKILL.md @@ -2,7 +2,7 @@ name: gh-commit description: Create well-formatted commits with conventional commit format. metadata: - provider: atomic + internal: true --- # Smart Git Commit @@ -234,7 +234,7 @@ dde0159 Claude Code [] Test work item (#7) (origin/main, origin/HEAD) ## Important Notes -- By default, pre-commit checks (defined in `.pre-commit-config.yaml`) will run to ensure code quality +- By default, pre-commit checks (defined in `prek.toml`) will run to ensure code quality - IMPORTANT: DO NOT SKIP pre-commit checks - ALWAYS attribute AI-Assisted Code Authorship - If specific files are already staged, the command will only commit those files diff --git a/.agents/skills/gh-create-pr/SKILL.md b/.agents/skills/gh-create-pr/SKILL.md index 73ee33453..5ce4fbcbf 100644 --- a/.agents/skills/gh-create-pr/SKILL.md +++ b/.agents/skills/gh-create-pr/SKILL.md @@ -2,7 +2,7 @@ name: gh-create-pr description: Commit unstaged changes, push changes, submit a pull request. metadata: - provider: atomic + internal: true --- # Create Pull Request @@ -88,7 +88,7 @@ Use this structure for the PR body. Omit sections that are not applicable. ## Important Notes -- By default, pre-commit checks (defined in `.pre-commit-config.yaml`) will run to ensure code quality +- By default, pre-commit checks (defined in `prek.toml`) will run to ensure code quality - IMPORTANT: DO NOT SKIP pre-commit checks - ALWAYS attribute AI-Assisted Code Authorship in commit messages - Always review the diff before generating the title and description to ensure accuracy diff --git a/.agents/skills/hosted-agents/references/infrastructure-patterns.md b/.agents/skills/hosted-agents/references/infrastructure-patterns.md deleted file mode 100644 index 2de33fd58..000000000 --- a/.agents/skills/hosted-agents/references/infrastructure-patterns.md +++ /dev/null @@ -1,700 +0,0 @@ -# Infrastructure Patterns for Hosted Agents - -This reference provides detailed implementation patterns for building hosted agent infrastructure. These patterns are derived from production systems at scale. - -## Sandbox Architecture - -### Modal Integration Pattern - -Modal provides the sandbox infrastructure with near-instant startup and filesystem snapshots. - -```python -import modal - -# Define the base image with all dependencies -image = modal.Image.debian_slim().pip_install([ - "opencode", - "gitpython", - "psycopg2-binary", -]) - -# Create the app -app = modal.App("coding-agent") - -# Sandbox class with snapshot support -@app.cls(image=image, timeout=3600) -class AgentSandbox: - def __init__(self, repo_url: str, snapshot_id: str = None): - self.repo_url = repo_url - self.snapshot_id = snapshot_id - - @modal.enter() - def setup(self): - if self.snapshot_id: - # Restore from snapshot - modal.Sandbox.restore(self.snapshot_id) - else: - # Fresh setup from image - self._clone_and_setup() - - def _clone_and_setup(self): - """Clone repo and run initial setup.""" - token = self._get_github_app_token() - os.system(f"git clone https://x-access-token:{token}@github.com/{self.repo_url}") - os.system("npm install") - os.system("npm run build") - - @modal.method() - def execute_prompt(self, prompt: str, user_identity: dict) -> dict: - """Execute a prompt in the sandbox.""" - # Update git config for this user - os.system(f'git config user.name "{user_identity["name"]}"') - os.system(f'git config user.email "{user_identity["email"]}"') - - # Run the agent - result = self.agent.run(prompt) - - return { - "result": result, - "snapshot_id": modal.Sandbox.snapshot() - } -``` - -### Image Build Pipeline - -Build images on a schedule to keep them fresh: - -```python -import schedule -import time -from datetime import datetime - -class ImageBuilder: - def __init__(self, repositories: list[str]): - self.repositories = repositories - self.images = {} - - def build_all_images(self): - """Build images for all repositories.""" - for repo in self.repositories: - try: - image = self._build_image(repo) - self.images[repo] = { - "image": image, - "built_at": datetime.utcnow(), - "commit": self._get_latest_commit(repo) - } - except Exception as e: - # Log but continue with other repos - log.error(f"Failed to build image for {repo}: {e}") - - def _build_image(self, repo: str) -> str: - """Build a single repository image.""" - sandbox = modal.Sandbox.create() - - # Clone with app token - token = get_app_installation_token(repo) - sandbox.exec(f"git clone https://x-access-token:{token}@github.com/{repo} /workspace") - - # Install dependencies - sandbox.exec("cd /workspace && npm install") - - # Run build - sandbox.exec("cd /workspace && npm run build") - - # Warm caches - sandbox.exec("cd /workspace && npm run dev &") - time.sleep(5) # Let dev server start - sandbox.exec("cd /workspace && npm test -- --run") - - # Create snapshot - return sandbox.snapshot() - - def get_latest_image(self, repo: str) -> str: - """Get the most recent image for a repository.""" - if repo not in self.images: - raise ValueError(f"No image available for {repo}") - return self.images[repo]["image"] - -# Schedule builds every 30 minutes -builder = ImageBuilder(["org/frontend", "org/backend", "org/shared"]) -schedule.every(30).minutes.do(builder.build_all_images) -``` - -### Warm Pool Management - -Maintain pre-warmed sandboxes for instant session starts: - -```python -from collections import defaultdict -from dataclasses import dataclass -from datetime import datetime, timedelta - -@dataclass -class WarmSandbox: - sandbox_id: str - repo: str - created_at: datetime - image_version: str - is_claimed: bool = False - -class WarmPoolManager: - def __init__(self, target_pool_size: int = 3): - self.target_size = target_pool_size - self.pools = defaultdict(list) # repo -> [WarmSandbox] - self.max_age = timedelta(minutes=25) # Expire before next image build - - def get_warm_sandbox(self, repo: str) -> WarmSandbox | None: - """Get a pre-warmed sandbox if available.""" - pool = self.pools[repo] - - for sandbox in pool: - if not sandbox.is_claimed and self._is_valid(sandbox): - sandbox.is_claimed = True - return sandbox - - return None - - def _is_valid(self, sandbox: WarmSandbox) -> bool: - """Check if sandbox is still valid.""" - age = datetime.utcnow() - sandbox.created_at - current_image = self.image_builder.get_latest_image(sandbox.repo) - - return ( - age < self.max_age and - sandbox.image_version == current_image - ) - - def maintain_pool(self, repo: str): - """Ensure pool has target number of warm sandboxes.""" - # Remove expired sandboxes - self.pools[repo] = [s for s in self.pools[repo] if self._is_valid(s)] - - # Add new sandboxes to reach target - current_count = len([s for s in self.pools[repo] if not s.is_claimed]) - needed = self.target_size - current_count - - for _ in range(needed): - sandbox = self._create_warm_sandbox(repo) - self.pools[repo].append(sandbox) - - def _create_warm_sandbox(self, repo: str) -> WarmSandbox: - """Create a new warm sandbox from latest image.""" - image = self.image_builder.get_latest_image(repo) - sandbox_id = modal.Sandbox.create(image=image) - - # Sync to latest (runs in background) - self._sync_to_latest(sandbox_id, repo) - - return WarmSandbox( - sandbox_id=sandbox_id, - repo=repo, - created_at=datetime.utcnow(), - image_version=image - ) -``` - -## API Layer Patterns - -### Cloudflare Durable Objects for Session State - -Each session gets its own Durable Object with isolated SQLite: - -```typescript -// Session Durable Object -export class SessionDO implements DurableObject { - private storage: DurableObjectStorage; - private sql: SqlStorage; - private connections: Map = new Map(); - - constructor(ctx: DurableObjectState) { - this.storage = ctx.storage; - this.sql = ctx.storage.sql; - this.initializeSchema(); - } - - private initializeSchema() { - this.sql.exec(` - CREATE TABLE IF NOT EXISTS messages ( - id INTEGER PRIMARY KEY, - role TEXT NOT NULL, - content TEXT NOT NULL, - author_id TEXT, - author_name TEXT, - created_at TEXT DEFAULT CURRENT_TIMESTAMP - ); - - CREATE TABLE IF NOT EXISTS artifacts ( - id INTEGER PRIMARY KEY, - type TEXT NOT NULL, - path TEXT, - content TEXT, - created_at TEXT DEFAULT CURRENT_TIMESTAMP - ); - - CREATE TABLE IF NOT EXISTS events ( - id INTEGER PRIMARY KEY, - type TEXT NOT NULL, - data TEXT, - created_at TEXT DEFAULT CURRENT_TIMESTAMP - ); - `); - } - - async fetch(request: Request): Promise { - const url = new URL(request.url); - - if (request.headers.get("Upgrade") === "websocket") { - return this.handleWebSocket(request); - } - - switch (url.pathname) { - case "/message": - return this.handleMessage(request); - case "/status": - return this.getStatus(); - default: - return new Response("Not found", { status: 404 }); - } - } - - private handleWebSocket(request: Request): Response { - const pair = new WebSocketPair(); - const [client, server] = Object.values(pair); - - const connectionId = crypto.randomUUID(); - this.connections.set(connectionId, server); - - server.accept(); - server.addEventListener("close", () => { - this.connections.delete(connectionId); - }); - - return new Response(null, { status: 101, webSocket: client }); - } - - private broadcast(message: object) { - const data = JSON.stringify(message); - for (const ws of this.connections.values()) { - ws.send(data); - } - } - - async handleMessage(request: Request): Promise { - const { content, author } = await request.json(); - - // Store message - this.sql.exec( - `INSERT INTO messages (role, content, author_id, author_name) VALUES (?, ?, ?, ?)`, - ["user", content, author.id, author.name] - ); - - // Broadcast to all connected clients - this.broadcast({ - type: "message", - role: "user", - content, - author, - }); - - // Forward to sandbox for processing - const result = await this.forwardToSandbox(content, author); - - return Response.json(result); - } -} -``` - -### Real-Time Event Streaming - -Stream events from sandbox to all connected clients: - -```typescript -class EventStream { - private sessionDO: DurableObjectStub; - - async streamFromSandbox(sandboxId: string, sessionId: string) { - const sandbox = await modal.Sandbox.get(sandboxId); - - // Subscribe to sandbox events - for await (const event of sandbox.events()) { - // Forward to Durable Object for broadcast - await this.sessionDO.fetch( - new Request(`https://internal/event`, { - method: "POST", - body: JSON.stringify({ - type: event.type, - data: event.data, - }), - }) - ); - } - } -} -``` - -## Client Integration Patterns - -### Slack Bot with Repository Classification - -```python -from slack_bolt import App -from slack_bolt.adapter.socket_mode import SocketModeHandler - -app = App(token=os.environ["SLACK_BOT_TOKEN"]) - -# Repository descriptions for classification -REPO_DESCRIPTIONS = [ - { - "name": "frontend-monorepo", - "description": "React frontend application with dashboard, user portal, and admin interfaces", - "hints": ["dashboard", "UI", "component", "page", "frontend"] - }, - { - "name": "backend-services", - "description": "Node.js API services including auth, payments, and core business logic", - "hints": ["API", "endpoint", "service", "backend", "database"] - }, - { - "name": "mobile-app", - "description": "React Native mobile application for iOS and Android", - "hints": ["mobile", "app", "iOS", "Android", "native"] - } -] - -async def classify_repository(message: str, channel: str, thread: list[str]) -> str: - """Use fast model to classify which repo the message refers to.""" - prompt = f"""Classify which repository this message is about. - -Message: {message} -Channel: #{channel} -Thread context: {' | '.join(thread[-3:])} - -Repositories: -{json.dumps(REPO_DESCRIPTIONS, indent=2)} - -Return ONLY the repository name, or "unknown" if unclear.""" - - response = await openai.chat.completions.create( - model="gpt-4o-mini", - messages=[{"role": "user", "content": prompt}], - max_tokens=50 - ) - - return response.choices[0].message.content.strip() - -@app.event("app_mention") -async def handle_mention(event, say, client): - """Handle @mentions of the bot.""" - channel = event["channel"] - message = event["text"] - thread_ts = event.get("thread_ts", event["ts"]) - - # Get thread context if in a thread - thread_messages = [] - if "thread_ts" in event: - result = await client.conversations_replies( - channel=channel, - ts=thread_ts - ) - thread_messages = [m["text"] for m in result["messages"]] - - # Get channel info for context - channel_info = await client.conversations_info(channel=channel) - channel_name = channel_info["channel"]["name"] - - # Classify repository - repo = await classify_repository(message, channel_name, thread_messages) - - if repo == "unknown": - await say( - text="I'm not sure which repository you're referring to. Could you specify?", - thread_ts=thread_ts - ) - return - - # Start session and process - session = await start_session(repo, event["user"]) - - await say( - text=f":robot_face: Starting work in `{repo}`...", - thread_ts=thread_ts - ) - - result = await session.process(message) - - # Post result with Block Kit formatting - await say( - blocks=format_result_blocks(result), - thread_ts=thread_ts - ) -``` - -### Chrome Extension DOM Extraction - -Extract DOM structure instead of sending screenshots: - -```typescript -// content-script.ts -interface ElementInfo { - tag: string; - classes: string[]; - id?: string; - text?: string; - rect: DOMRect; - reactComponent?: string; -} - -function extractDOMInfo(element: Element): ElementInfo { - // Get React component name if available - let reactComponent: string | undefined; - const fiberKey = Object.keys(element).find((key) => - key.startsWith("__reactFiber") - ); - if (fiberKey) { - const fiber = (element as any)[fiberKey]; - reactComponent = fiber?.type?.name || fiber?.type?.displayName; - } - - return { - tag: element.tagName.toLowerCase(), - classes: Array.from(element.classList), - id: element.id || undefined, - text: element.textContent?.slice(0, 100), - rect: element.getBoundingClientRect(), - reactComponent, - }; -} - -function extractSelectedArea(selection: DOMRect): ElementInfo[] { - const elements: ElementInfo[] = []; - - // Find all elements within selection bounds - document.querySelectorAll("*").forEach((el) => { - const rect = el.getBoundingClientRect(); - if ( - rect.top >= selection.top && - rect.left >= selection.left && - rect.bottom <= selection.bottom && - rect.right <= selection.right - ) { - elements.push(extractDOMInfo(el)); - } - }); - - return elements; -} - -// Message handler for sidebar -chrome.runtime.onMessage.addListener((request, sender, sendResponse) => { - if (request.type === "EXTRACT_SELECTION") { - const elements = extractSelectedArea(request.selection); - sendResponse({ elements }); - } -}); -``` - -## Multiplayer Implementation - -### Authorship Tracking - -Track which user made each change: - -```python -@dataclass -class PromptContext: - content: str - author: Author - session_id: str - timestamp: datetime - -@dataclass -class Author: - id: str - name: str - email: str - github_token: str # For PR creation - -class MultiplayerSession: - def __init__(self, session_id: str): - self.session_id = session_id - self.participants: dict[str, Author] = {} - self.prompt_queue: list[PromptContext] = [] - - def add_participant(self, author: Author): - """Add a participant to the session.""" - self.participants[author.id] = author - self.broadcast_event("participant_joined", author) - - async def process_prompt(self, prompt: PromptContext): - """Process prompt with author attribution.""" - # Update git config for this author - await self.sandbox.exec( - f'git config user.name "{prompt.author.name}"' - ) - await self.sandbox.exec( - f'git config user.email "{prompt.author.email}"' - ) - - # Run agent - result = await self.agent.run(prompt.content) - - # If changes were made, create PR with author's token - if result.has_changes: - await self.create_pr( - branch=result.branch, - author=prompt.author - ) - - return result - - async def create_pr(self, branch: str, author: Author): - """Create PR using the author's GitHub token.""" - async with aiohttp.ClientSession() as session: - headers = { - "Authorization": f"Bearer {author.github_token}", - "Accept": "application/vnd.github.v3+json" - } - - await session.post( - f"https://api.github.com/repos/{self.repo}/pulls", - headers=headers, - json={ - "title": self.generate_pr_title(), - "body": self.generate_pr_body(), - "head": branch, - "base": "main" - } - ) -``` - -## Metrics and Monitoring - -### Key Metrics to Track - -```python -from dataclasses import dataclass -from datetime import datetime, timedelta - -@dataclass -class SessionMetrics: - session_id: str - started_at: datetime - first_token_at: datetime | None - completed_at: datetime | None - pr_created: bool - pr_merged: bool - prompts_count: int - participants_count: int - - @property - def time_to_first_token(self) -> timedelta | None: - if self.first_token_at: - return self.first_token_at - self.started_at - return None - -class MetricsAggregator: - def get_adoption_metrics(self, period: timedelta) -> dict: - """Get adoption metrics for a time period.""" - sessions = self.get_sessions_in_period(period) - - total_prs = sum(1 for s in sessions if s.pr_created) - merged_prs = sum(1 for s in sessions if s.pr_merged) - - return { - "total_sessions": len(sessions), - "prs_created": total_prs, - "prs_merged": merged_prs, - "merge_rate": merged_prs / total_prs if total_prs > 0 else 0, - "avg_time_to_first_token": self._avg_ttft(sessions), - "unique_users": len(set(s.author_id for s in sessions)), - "multiplayer_sessions": sum( - 1 for s in sessions if s.participants_count > 1 - ) - } - - def get_repository_metrics(self) -> dict[str, dict]: - """Get metrics broken down by repository.""" - metrics = {} - - for repo in self.repositories: - repo_sessions = self.get_sessions_for_repo(repo) - total_prs = self.get_total_prs(repo) - agent_prs = sum(1 for s in repo_sessions if s.pr_merged) - - metrics[repo] = { - "agent_pr_percentage": agent_prs / total_prs * 100, - "session_count": len(repo_sessions), - "avg_prompts_per_session": sum( - s.prompts_count for s in repo_sessions - ) / len(repo_sessions) - } - - return metrics -``` - -## Security Considerations - -### Sandbox Isolation - -```python -class SandboxSecurityConfig: - """Security configuration for sandboxes.""" - - # Network restrictions - allowed_hosts = [ - "github.com", - "api.github.com", - "registry.npmjs.org", - "pypi.org", - ] - - # Resource limits - max_memory_mb = 4096 - max_cpu_cores = 2 - max_disk_gb = 10 - max_runtime_hours = 4 - - # Secrets handling - secrets_to_inject = [ - "GITHUB_APP_TOKEN", - "NPM_TOKEN", - ] - - # Blocked operations - blocked_commands = [ - "curl", # Use fetch tools instead - "wget", - "ssh", - ] -``` - -### Token Handling - -```python -class TokenManager: - """Manage tokens for GitHub operations.""" - - def get_app_installation_token(self, repo: str) -> str: - """Get short-lived token for repo access.""" - # Token expires in 1 hour - return github_app.create_installation_token( - installation_id=self.get_installation_id(repo), - permissions={"contents": "write", "pull_requests": "write"} - ) - - def get_user_token(self, user_id: str) -> str: - """Get user's OAuth token for PR creation.""" - # Stored encrypted, decrypted at runtime - encrypted = self.storage.get(f"user_token:{user_id}") - return self.decrypt(encrypted) -``` - -## References - -- [Modal Documentation](https://modal.com/docs) -- [Cloudflare Durable Objects](https://developers.cloudflare.com/durable-objects/) -- [Cloudflare Agents SDK](https://developers.cloudflare.com/agents/) -- [GitHub Apps Authentication](https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app) -- [Slack Bolt for Python](https://slack.dev/bolt-python/) -- [Chrome Extension APIs](https://developer.chrome.com/docs/extensions/) diff --git a/.agents/skills/hosted-agents/scripts/sandbox_manager.py b/.agents/skills/hosted-agents/scripts/sandbox_manager.py deleted file mode 100644 index af0a50595..000000000 --- a/.agents/skills/hosted-agents/scripts/sandbox_manager.py +++ /dev/null @@ -1,590 +0,0 @@ -""" -Sandbox Manager for Hosted Agent Infrastructure. - -Use when: building background coding agents that need sandboxed execution -environments with pre-built images, warm pools, and session snapshots. - -This module provides composable building blocks for sandbox lifecycle -management. Each class handles one concern (image building, warm pools, -session coordination) and can be used independently or combined via -SandboxManager. - -Note: This is pseudocode demonstrating architectural patterns. -Adapt for your specific infrastructure (Modal, Fly.io, etc.). -""" - -from dataclasses import dataclass, field -from datetime import datetime, timedelta -from typing import Optional, Callable, Any -from enum import Enum -import asyncio - -__all__ = [ - "SandboxState", - "UserIdentity", - "SandboxConfig", - "Sandbox", - "RepositoryImage", - "ImageBuilder", - "WarmSandbox", - "WarmPoolManager", - "SandboxManager", - "AgentSession", -] - - -class SandboxState(Enum): - """Sandbox lifecycle states.""" - CREATING = "creating" - SYNCING = "syncing" - READY = "ready" - EXECUTING = "executing" - SNAPSHOTTING = "snapshotting" - TERMINATED = "terminated" - - -@dataclass -class UserIdentity: - """User identity for commit attribution. - - Use when: configuring sandbox git identity so commits are - attributed to the prompting user, not the app. - """ - id: str - name: str - email: str - github_token: str - - -@dataclass -class SandboxConfig: - """Configuration for sandbox creation. - - Use when: defining resource limits and timeouts for a new sandbox - to prevent cost runaway and resource exhaustion. - """ - repo_url: str - base_image: str - memory_mb: int = 4096 - cpu_cores: int = 2 - disk_gb: int = 10 - timeout_hours: int = 4 - - -@dataclass -class Sandbox: - """Represents a sandboxed execution environment. - - Use when: interacting with a running sandbox to execute commands, - read/write files, or take snapshots for session continuity. - """ - id: str - config: SandboxConfig - state: SandboxState - created_at: datetime - snapshot_id: Optional[str] = None - current_user: Optional[UserIdentity] = None - - # Event handlers - on_state_change: Optional[Callable[[SandboxState], None]] = None - - async def execute_command(self, command: str) -> dict[str, Any]: - """Execute a command in the sandbox. - - Use when: running shell commands (git, build tools, tests) - inside the isolated environment. - - Returns: - dict with keys "stdout", "stderr", "exit_code". - """ - # Implementation depends on infrastructure - pass - - async def read_file(self, path: str) -> str: - """Read a file from the sandbox filesystem. - - Use when: agent needs to inspect source code or config files. - Safe to call before git sync completes. - """ - pass - - async def write_file(self, path: str, content: str) -> None: - """Write a file to the sandbox filesystem. - - Use when: agent needs to modify source code. Block this - until git sync completes to avoid write conflicts. - """ - pass - - async def snapshot(self) -> str: - """Create a snapshot of current filesystem state. - - Use when: preserving session state before sandbox termination - so follow-up prompts can restore instantly. - """ - self.state = SandboxState.SNAPSHOTTING - snapshot_id = await self._create_snapshot() - self.snapshot_id = snapshot_id - self.state = SandboxState.READY - return snapshot_id - - async def _create_snapshot(self) -> str: - """Create snapshot (infrastructure-specific).""" - pass - - async def restore(self, snapshot_id: str) -> None: - """Restore sandbox to a previous snapshot.""" - pass - - async def terminate(self) -> None: - """Terminate the sandbox.""" - self.state = SandboxState.TERMINATED - - -@dataclass -class RepositoryImage: - """Pre-built image for a repository. - - Use when: checking whether a cached environment image exists - and whether it is recent enough to use. - """ - repo_url: str - image_id: str - commit_sha: str - built_at: datetime - - def is_stale(self, max_age: timedelta = timedelta(minutes=30)) -> bool: - """Check if image is older than max age.""" - return datetime.utcnow() - self.built_at > max_age - - -class ImageBuilder: - """Builds and manages repository images. - - Use when: setting up the periodic image build loop that - pre-bakes development environments for fast sandbox spin-up. - """ - - def __init__(self, github_app_token_provider: Callable[[], str]) -> None: - self.token_provider = github_app_token_provider - self.images: dict[str, RepositoryImage] = {} - - async def build_image(self, repo_url: str) -> RepositoryImage: - """Build a new image for a repository. - - Use when: the current image is stale or no image exists yet. - Runs clone, dependency install, build, and cache warming. - """ - print(f"Building image for {repo_url}...") - - # Get fresh token for clone - token = self.token_provider() - - # These operations run in build environment - build_steps: list[str] = [ - # Clone repository - f"git clone https://x-access-token:{token}@github.com/{repo_url} /workspace", - - # Install dependencies - "cd /workspace && npm install", - - # Run build - "cd /workspace && npm run build", - - # Warm caches by running once - "cd /workspace && npm run dev &", - "sleep 5", # Let dev server start - "cd /workspace && npm test -- --run || true", # Run tests to warm cache - ] - - # Execute build steps (infrastructure-specific) - for step in build_steps: - await self._execute_build_step(step) - - # Get current commit - commit_sha: str = await self._get_commit_sha() - - # Create and store image - image = RepositoryImage( - repo_url=repo_url, - image_id=await self._finalize_image(), - commit_sha=commit_sha, - built_at=datetime.utcnow() - ) - - self.images[repo_url] = image - return image - - def get_latest_image(self, repo_url: str) -> Optional[RepositoryImage]: - """Get the most recent image for a repository.""" - return self.images.get(repo_url) - - async def _execute_build_step(self, command: str) -> None: - """Execute a build step (infrastructure-specific).""" - pass - - async def _get_commit_sha(self) -> str: - """Get current HEAD commit SHA.""" - pass - - async def _finalize_image(self) -> str: - """Finalize and store the image, return image ID.""" - pass - - -@dataclass -class WarmSandbox: - """A pre-warmed sandbox ready for use. - - Use when: tracking warm pool inventory and claiming a sandbox - for an incoming user session. - """ - sandbox: Sandbox - repo_url: str - created_at: datetime - image_version: str - is_claimed: bool = False - sync_complete: bool = False - - -class WarmPoolManager: - """Manages pools of pre-warmed sandboxes. - - Use when: reducing cold start latency by maintaining ready-to-use - sandboxes that are pre-synced to the latest code. - """ - - def __init__( - self, - image_builder: ImageBuilder, - target_pool_size: int = 3, - max_age: timedelta = timedelta(minutes=25) - ) -> None: - self.image_builder = image_builder - self.target_size = target_pool_size - self.max_age = max_age - self.pools: dict[str, list[WarmSandbox]] = {} - - async def get_warm_sandbox(self, repo_url: str) -> Optional[WarmSandbox]: - """Get a pre-warmed sandbox if available. - - Use when: a user submits a prompt and needs a sandbox immediately. - Returns None if no valid warm sandbox is available. - """ - if repo_url not in self.pools: - return None - - for warm in self.pools[repo_url]: - if not warm.is_claimed and self._is_valid(warm): - warm.is_claimed = True - return warm - - return None - - def _is_valid(self, warm: WarmSandbox) -> bool: - """Check if a warm sandbox is still valid.""" - age: timedelta = datetime.utcnow() - warm.created_at - if age > self.max_age: - return False - - # Check if image is still current - current = self.image_builder.get_latest_image(warm.repo_url) - if not current or current.image_id != warm.image_version: - return False - - return True - - async def maintain_pool(self, repo_url: str) -> None: - """Ensure pool has target number of warm sandboxes. - - Use when: called periodically or after an image rebuild to - keep the warm pool populated. - """ - if repo_url not in self.pools: - self.pools[repo_url] = [] - - # Remove invalid sandboxes - valid: list[WarmSandbox] = [w for w in self.pools[repo_url] if self._is_valid(w)] - self.pools[repo_url] = valid - - # Count available (unclaimed) sandboxes - available: int = len([w for w in valid if not w.is_claimed]) - needed: int = self.target_size - available - - # Create new warm sandboxes - for _ in range(max(0, needed)): - warm = await self._create_warm_sandbox(repo_url) - self.pools[repo_url].append(warm) - - async def _create_warm_sandbox(self, repo_url: str) -> WarmSandbox: - """Create a new warm sandbox.""" - image: Optional[RepositoryImage] = self.image_builder.get_latest_image(repo_url) - if not image: - raise ValueError(f"No image available for {repo_url}") - - # Create sandbox from image - sandbox: Sandbox = await self._create_sandbox_from_image(image) - - warm = WarmSandbox( - sandbox=sandbox, - repo_url=repo_url, - created_at=datetime.utcnow(), - image_version=image.image_id, - sync_complete=False - ) - - # Start syncing to latest in background - asyncio.create_task(self._sync_to_latest(warm)) - - return warm - - async def _sync_to_latest(self, warm: WarmSandbox) -> None: - """Sync sandbox to latest commit on base branch.""" - await warm.sandbox.execute_command("git fetch origin main") - await warm.sandbox.execute_command("git reset --hard origin/main") - warm.sync_complete = True - - async def _create_sandbox_from_image(self, image: RepositoryImage) -> Sandbox: - """Create a sandbox from an image (infrastructure-specific).""" - pass - - -class SandboxManager: - """Main manager for sandbox lifecycle. - - Use when: orchestrating the full sandbox lifecycle including - image building, warm pools, and session management. This is the - top-level entry point that composes ImageBuilder and WarmPoolManager. - """ - - def __init__( - self, - repositories: list[str], - github_app_token_provider: Callable[[], str], - build_interval: timedelta = timedelta(minutes=30) - ) -> None: - self.repositories = repositories - self.image_builder = ImageBuilder(github_app_token_provider) - self.warm_pool = WarmPoolManager(self.image_builder) - self.build_interval = build_interval - self.active_sessions: dict[str, Sandbox] = {} - - async def start_build_loop(self) -> None: - """Start the background image build loop. - - Use when: initializing the system. Runs indefinitely, rebuilding - images every build_interval to keep environments fresh. - """ - while True: - for repo in self.repositories: - try: - await self.image_builder.build_image(repo) - await self.warm_pool.maintain_pool(repo) - except Exception as e: - print(f"Failed to build {repo}: {e}") - - await asyncio.sleep(self.build_interval.total_seconds()) - - async def start_session( - self, - repo_url: str, - user: UserIdentity, - snapshot_id: Optional[str] = None - ) -> Sandbox: - """Start a new session for a user. - - Use when: a user submits a prompt. Tries warm pool first, - then snapshot restore, then cold start as fallback. - """ - # Try to get from warm pool first - warm: Optional[WarmSandbox] = await self.warm_pool.get_warm_sandbox(repo_url) - - if warm: - sandbox = warm.sandbox - # Wait for sync if not complete - if not warm.sync_complete: - await self._wait_for_sync(warm) - elif snapshot_id: - # Restore from previous session snapshot - sandbox = await self._restore_from_snapshot(snapshot_id) - else: - # Cold start from latest image - sandbox = await self._cold_start(repo_url) - - # Configure for user - await self._configure_for_user(sandbox, user) - - # Track session - session_id: str = f"{user.id}_{datetime.utcnow().isoformat()}" - self.active_sessions[session_id] = sandbox - - return sandbox - - async def on_user_typing(self, user: UserIdentity, repo_url: str) -> None: - """Called when user starts typing a prompt. - - Use when: implementing predictive warm-up. Starts preparing a - sandbox so it is ready by the time the user submits. - """ - warm: Optional[WarmSandbox] = await self.warm_pool.get_warm_sandbox(repo_url) - - if not warm: - # Start warming one now - asyncio.create_task(self.warm_pool.maintain_pool(repo_url)) - - async def end_session(self, session_id: str) -> Optional[str]: - """End a session and return snapshot ID for potential follow-up. - - Use when: a session completes. Always snapshots before termination - to prevent state loss. - """ - if session_id not in self.active_sessions: - return None - - sandbox: Sandbox = self.active_sessions[session_id] - - # Create snapshot before terminating - snapshot_id: str = await sandbox.snapshot() - - # Terminate sandbox - await sandbox.terminate() - - del self.active_sessions[session_id] - - return snapshot_id - - async def _configure_for_user( - self, - sandbox: Sandbox, - user: UserIdentity - ) -> None: - """Configure sandbox for a specific user.""" - sandbox.current_user = user - - # Set git identity - await sandbox.execute_command( - f'git config user.name "{user.name}"' - ) - await sandbox.execute_command( - f'git config user.email "{user.email}"' - ) - - async def _wait_for_sync(self, warm: WarmSandbox) -> None: - """Wait for sync to complete.""" - while not warm.sync_complete: - await asyncio.sleep(0.1) - - async def _restore_from_snapshot(self, snapshot_id: str) -> Sandbox: - """Restore a sandbox from a snapshot.""" - pass - - async def _cold_start(self, repo_url: str) -> Sandbox: - """Start a sandbox from cold (no warm pool available).""" - pass - - -class AgentSession: - """Agent session with file read/write coordination. - - Use when: wrapping a Sandbox to enforce the pattern where reads - are allowed before sync completes but writes are blocked until - sync finishes, preventing write conflicts. - """ - - def __init__(self, sandbox: Sandbox) -> None: - self.sandbox = sandbox - self.sync_complete: bool = False - self.pending_writes: list[tuple[str, str]] = [] - - async def read_file(self, path: str) -> str: - """Read a file -- allowed even before sync completes. - - Use when: agent needs to research code immediately. Safe because - in large repos, files being worked on are unlikely to have - changed in the last 30 minutes since image build. - """ - return await self.sandbox.read_file(path) - - async def write_file(self, path: str, content: str) -> None: - """Write a file -- blocks until sync is complete. - - Use when: agent needs to modify source code. Queues the write - and waits for git sync to finish to prevent conflicts. - """ - if not self.sync_complete: - # Queue the write - self.pending_writes.append((path, content)) - await self._wait_for_sync() - - await self.sandbox.write_file(path, content) - - def mark_sync_complete(self) -> None: - """Called when git sync is complete.""" - self.sync_complete = True - - async def _wait_for_sync(self) -> None: - """Wait for sync to complete, then flush pending writes.""" - while not self.sync_complete: - await asyncio.sleep(0.1) - - # Flush pending writes - for path, content in self.pending_writes: - await self.sandbox.write_file(path, content) - self.pending_writes.clear() - - -if __name__ == "__main__": - async def _demo() -> None: - """Demonstrate sandbox manager usage end-to-end.""" - - def get_github_token() -> str: - """Get GitHub App installation token.""" - # Implementation: call GitHub API to get installation token - return "ghs_xxxx" - - # Initialize manager with target repositories - manager = SandboxManager( - repositories=[ - "myorg/frontend", - "myorg/backend", - "myorg/shared-libs" - ], - github_app_token_provider=get_github_token - ) - - # Start background build loop - asyncio.create_task(manager.start_build_loop()) - - # Simulate user session - user = UserIdentity( - id="user123", - name="Alice Developer", - email="alice@example.com", - github_token="gho_user_token" - ) - - # User starts typing -- predictively warm a sandbox - await manager.on_user_typing(user, "myorg/frontend") - - # User submits prompt -- get sandbox - sandbox: Sandbox = await manager.start_session("myorg/frontend", user) - - # Create session wrapper for read/write coordination - session = AgentSession(sandbox) - - # Agent can read immediately (before sync completes) - readme: str = await session.read_file("/workspace/README.md") - - # Agent work happens here... - - # End session and get snapshot for follow-up - # Find the session_id that was generated during start_session - active_ids = list(manager.active_sessions.keys()) - if active_ids: - session_id = active_ids[0] - snapshot_id: Optional[str] = await manager.end_session(session_id) - print(f"Session ended, snapshot: {snapshot_id}") - else: - print("No active session found") - - asyncio.run(_demo()) diff --git a/.agents/skills/impeccable/SKILL.md b/.agents/skills/impeccable/SKILL.md deleted file mode 100644 index 25044b37a..000000000 --- a/.agents/skills/impeccable/SKILL.md +++ /dev/null @@ -1,176 +0,0 @@ ---- -name: impeccable -description: Use when the user wants to design, redesign, shape, critique, audit, polish, clarify, distill, harden, optimize, adapt, animate, colorize, extract, or otherwise improve a frontend interface. Covers websites, landing pages, dashboards, product UI, app shells, components, forms, settings, onboarding, and empty states. Handles UX review, visual hierarchy, information architecture, cognitive load, accessibility, performance, responsive behavior, theming, anti-patterns, typography, fonts, spacing, layout, alignment, color, motion, micro-interactions, UX copy, error states, edge cases, i18n, and reusable design systems or tokens. Also use for bland designs that need to become bolder or more delightful, loud designs that should become quieter, live browser iteration on UI elements, or ambitious visual effects that should feel technically extraordinary. Not for backend-only or non-UI tasks. ---- - -Designs and iterates production-grade frontend interfaces. Real working code, committed design choices, exceptional craft. - -## Setup (non-optional) - -Before any design work or file edits, pass these gates. Skipping them produces generic output that ignores the project. - -| Gate | Required check | If fail | -|---|---|---| -| Context | The PRODUCT.md / DESIGN.md loader result is known from `node .agents/skills/impeccable/scripts/load-context.mjs`. | Run the loader before continuing. | -| Product | PRODUCT.md exists and is not empty or placeholder (`[TODO]` markers, <200 chars). | Run `$impeccable teach`, refresh context, then resume. Never synthesize PRODUCT.md from the user's original prompt alone. | -| Command | The matching command reference is loaded when a sub-command is used. | Load the reference before continuing. | -| Craft | `$impeccable craft` has a user-confirmed shape brief for this task. `teach` / PRODUCT.md never counts as shape. | Run `$impeccable shape` and wait for explicit brief confirmation. | -| Image | Required visual probes / mocks are generated or skipped with a reason. | Resolve the image-generation gate in `shape.md` or `craft.md` before code. | -| Mutation | All active gates above pass. | Do not edit project files yet. | - -Codex-style agents must state this before editing files: - -```text -IMPECCABLE_PREFLIGHT: context=pass product=pass command_reference=pass shape=pass|not_required image_gate=pass|skipped: mutation=open -``` - -For `$impeccable craft`, `shape=pass` is only valid after a separate user response approving the shape design brief, or when the user provided an already-confirmed brief in the request. Do not mark `shape=pass` after writing PRODUCT.md, summarizing assumptions, or drafting an unconfirmed brief yourself. - -Other harnesses should follow the same checklist when they can expose this state. - -### 1. Context gathering - -Two files, case-insensitive. The loader looks at the project root by default and falls back to `.agents/context/` and `docs/` if the root is clean. Override with `IMPECCABLE_CONTEXT_DIR=path/to/dir` (absolute or relative to cwd). - -- **PRODUCT.md**: required. Users, brand, tone, anti-references, strategic principles. -- **DESIGN.md**: optional, strongly recommended. Colors, typography, elevation, components. - -Load both in one call: - -```bash -node .agents/skills/impeccable/scripts/load-context.mjs -``` - -Consume the full JSON output. Never pipe through `head`, `tail`, `grep`, or `jq`. The output's `contextDir` field tells you where the files were resolved from. - -If the output is already in this session's conversation history, don't re-run. Exceptions requiring a fresh load: you just ran `$impeccable teach` or `$impeccable document` (they rewrite the files), or the user manually edited one. - -`$impeccable live` already warms context via `live.mjs`. If you've run `live.mjs`, don't also run `load-context.mjs` this session. - -If PRODUCT.md is missing, empty, or placeholder (`[TODO]` markers, <200 chars): run `$impeccable teach`, then resume the user's original task with the fresh context. If the original task was `$impeccable craft`, resume into `$impeccable shape` before any implementation work. - -If DESIGN.md is missing: nudge once per session (*"Run `$impeccable document` for more on-brand output"*), then proceed. - -### 2. Register - -Every design task is **brand** (marketing, landing, campaign, long-form content, portfolio: design IS the product) or **product** (app UI, admin, dashboard, tool: design SERVES the product). - -Identify before designing. Priority: (1) cue in the task itself ("landing page" vs "dashboard"); (2) the surface in focus (the page, file, or route being worked on); (3) `register` field in PRODUCT.md. First match wins. - -If PRODUCT.md lacks the `register` field (legacy), infer it once from its "Users" and "Product Purpose" sections, then cache the inferred value for the session. Suggest the user run `$impeccable teach` to add the field explicitly. - -Load the matching reference: [reference/brand.md](reference/brand.md) or [reference/product.md](reference/product.md). The shared design laws below apply to both. - -## Shared design laws - -Apply to every design, both registers. Match implementation complexity to the aesthetic vision: maximalism needs elaborate code, minimalism needs precision. Interpret creatively. Vary across projects; never converge on the same choices. GPT is capable of extraordinary work. Don't hold back. - -### Color - -- Use OKLCH. Reduce chroma as lightness approaches 0 or 100; high chroma at extremes looks garish. -- Never use `#000` or `#fff`. Tint every neutral toward the brand hue (chroma 0.005–0.01 is enough). -- Pick a **color strategy** before picking colors. Four steps on the commitment axis: - - **Restrained**: tinted neutrals + one accent ≤10%. Product default; brand minimalism. - - **Committed**: one saturated color carries 30–60% of the surface. Brand default for identity-driven pages. - - **Full palette**: 3–4 named roles, each used deliberately. Brand campaigns; product data viz. - - **Drenched**: the surface IS the color. Brand heroes, campaign pages. -- The "one accent ≤10%" rule is Restrained only. Committed / Full palette / Drenched exceed it on purpose. Don't collapse every design to Restrained by reflex. - -### Theme - -Dark vs. light is never a default. Not dark "because tools look cool dark." Not light "to be safe." - -Before choosing, write one sentence of physical scene: who uses this, where, under what ambient light, in what mood. If the sentence doesn't force the answer, it's not concrete enough. Add detail until it does. - -"Observability dashboard" does not force an answer. "SRE glancing at incident severity on a 27-inch monitor at 2am in a dim room" does. Run the sentence, not the category. - -### Typography - -- Cap body line length at 65–75ch. -- Hierarchy through scale + weight contrast (≥1.25 ratio between steps). Avoid flat scales. - -### Layout - -- Vary spacing for rhythm. Same padding everywhere is monotony. -- Cards are the lazy answer. Use them only when they're truly the best affordance. Nested cards are always wrong. -- Don't wrap everything in a container. Most things don't need one. - -### Motion - -- Don't animate CSS layout properties. -- Ease out with exponential curves (ease-out-quart / quint / expo). No bounce, no elastic. - -### Absolute bans - -Match-and-refuse. If you're about to write any of these, rewrite the element with different structure. - -- **Side-stripe borders.** `border-left` or `border-right` greater than 1px as a colored accent on cards, list items, callouts, or alerts. Never intentional. Rewrite with full borders, background tints, leading numbers/icons, or nothing. -- **Gradient text.** `background-clip: text` combined with a gradient background. Decorative, never meaningful. Use a single solid color. Emphasis via weight or size. -- **Glassmorphism as default.** Blurs and glass cards used decoratively. Rare and purposeful, or nothing. -- **The hero-metric template.** Big number, small label, supporting stats, gradient accent. SaaS cliché. -- **Identical card grids.** Same-sized cards with icon + heading + text, repeated endlessly. -- **Modal as first thought.** Modals are usually laziness. Exhaust inline / progressive alternatives first. - -### Copy - -- Every word earns its place. No restated headings, no intros that repeat the title. -- **No em dashes.** Use commas, colons, semicolons, periods, or parentheses. Also not `--`. - -### The AI slop test - -If someone could look at this interface and say "AI made that" without doubt, it's failed. Cross-register failures are the absolute bans above. Register-specific failures live in each reference. - -**Category-reflex check.** Run at two altitudes; the second one catches what the first one misses. - -- **First-order:** if someone could guess the theme + palette from the category alone ("observability → dark blue", "healthcare → white + teal", "finance → navy + gold", "crypto → neon on black"), it's the first training-data reflex. Rework the scene sentence and color strategy until the answer isn't obvious from the domain. -- **Second-order:** if someone could guess the aesthetic family from category-plus-anti-references ("AI workflow tool that's not SaaS-cream → editorial-typographic", "fintech that's not navy-and-gold → terminal-native dark mode"), it's the trap one tier deeper. The first reflex was avoided; the second wasn't. Rework until both answers are not obvious. The brand register's [reflex-reject aesthetic lanes](reference/brand.md) list catches the currently-saturated families. - -## Commands - -| Command | Category | Description | Reference | -|---|---|---|---| -| `craft [feature]` | Build | Shape, then build a feature end-to-end | [reference/craft.md](reference/craft.md) | -| `shape [feature]` | Build | Plan UX/UI before writing code | [reference/shape.md](reference/shape.md) | -| `teach` | Build | Set up PRODUCT.md and DESIGN.md context | [reference/teach.md](reference/teach.md) | -| `document` | Build | Generate DESIGN.md from existing project code | [reference/document.md](reference/document.md) | -| `extract [target]` | Build | Pull reusable tokens and components into design system | [reference/extract.md](reference/extract.md) | -| `critique [target]` | Evaluate | UX design review with heuristic scoring | [reference/critique.md](reference/critique.md) | -| `audit [target]` | Evaluate | Technical quality checks (a11y, perf, responsive) | [reference/audit.md](reference/audit.md) | -| `polish [target]` | Refine | Final quality pass before shipping | [reference/polish.md](reference/polish.md) | -| `bolder [target]` | Refine | Amplify safe or bland designs | [reference/bolder.md](reference/bolder.md) | -| `quieter [target]` | Refine | Tone down aggressive or overstimulating designs | [reference/quieter.md](reference/quieter.md) | -| `distill [target]` | Refine | Strip to essence, remove complexity | [reference/distill.md](reference/distill.md) | -| `harden [target]` | Refine | Production-ready: errors, i18n, edge cases | [reference/harden.md](reference/harden.md) | -| `onboard [target]` | Refine | Design first-run flows, empty states, activation | [reference/onboard.md](reference/onboard.md) | -| `animate [target]` | Enhance | Add purposeful animations and motion | [reference/animate.md](reference/animate.md) | -| `colorize [target]` | Enhance | Add strategic color to monochromatic UIs | [reference/colorize.md](reference/colorize.md) | -| `typeset [target]` | Enhance | Improve typography hierarchy and fonts | [reference/typeset.md](reference/typeset.md) | -| `layout [target]` | Enhance | Fix spacing, rhythm, and visual hierarchy | [reference/layout.md](reference/layout.md) | -| `delight [target]` | Enhance | Add personality and memorable touches | [reference/delight.md](reference/delight.md) | -| `overdrive [target]` | Enhance | Push past conventional limits | [reference/overdrive.md](reference/overdrive.md) | -| `clarify [target]` | Fix | Improve UX copy, labels, and error messages | [reference/clarify.md](reference/clarify.md) | -| `adapt [target]` | Fix | Adapt for different devices and screen sizes | [reference/adapt.md](reference/adapt.md) | -| `optimize [target]` | Fix | Diagnose and fix UI performance | [reference/optimize.md](reference/optimize.md) | -| `live` | Iterate | Visual variant mode: pick elements in the browser, generate alternatives | [reference/live.md](reference/live.md) | - -Plus two management commands: `pin ` and `unpin `, detailed below. - -### Routing rules - -1. **No argument**: render the table above as the user-facing command menu, grouped by category. Ask what they'd like to do. -2. **First word matches a command**: load its reference file and follow its instructions. Everything after the command name is the target. -3. **First word doesn't match**: general design invocation. Apply the setup steps, shared design laws, and the loaded register reference, using the full argument as context. - -Setup (context gathering, register) is already loaded by then; sub-commands don't re-invoke `$impeccable`. - -If the first word is `craft`, setup still runs first, but [reference/craft.md](reference/craft.md) owns the rest of the flow. If setup invokes `teach` as a blocker, finish teach, refresh context, then resume the original command and target. - -## Pin / Unpin - -**Pin** creates a standalone shortcut so `$` invokes `$impeccable ` directly. **Unpin** removes it. The script writes to every harness directory present in the project. - -```bash -node .agents/skills/impeccable/scripts/pin.mjs -``` - -Valid `` is any command from the table above. Report the script's result concisely. Confirm the new shortcut on success, relay stderr verbatim on error. \ No newline at end of file diff --git a/.agents/skills/impeccable/agents/openai.yaml b/.agents/skills/impeccable/agents/openai.yaml deleted file mode 100644 index ee6cae772..000000000 --- a/.agents/skills/impeccable/agents/openai.yaml +++ /dev/null @@ -1,4 +0,0 @@ -interface: - display_name: Impeccable - short_description: Use when the user wants to design, redesign, shape, critique, audit, polish, clarify,... - default_prompt: Use Impeccable to redesign, critique, audit, or polish this frontend. \ No newline at end of file diff --git a/.agents/skills/impeccable/reference/craft.md b/.agents/skills/impeccable/reference/craft.md deleted file mode 100644 index 337b5c9b7..000000000 --- a/.agents/skills/impeccable/reference/craft.md +++ /dev/null @@ -1,193 +0,0 @@ -# Craft Flow - -Build a feature with impeccable UX and UI quality through a structured process: shape the design, land the visual direction, build real production code, then inspect and improve in-browser until the result meets a high-end studio bar. - -## Build Gate - -Craft cannot build until all of these are true: - -1. PRODUCT context is valid and current. -2. The shape design brief is explicitly confirmed by the user for this task, unless the user already provided a confirmed brief. -3. Implementation references from the brief are loaded. -4. The shape visual probe decision is recorded: generated, skipped with reason, or already resolved. -5. The north-star mock decision is recorded: generated, skipped with reason, or not applicable. - -PRODUCT.md and `teach` answers do **not** satisfy the shape gate. They are project context only. A compact self-authored brief does not satisfy the shape gate either. `shape=pass` requires a separate user response approving the shape brief or an already-confirmed brief supplied by the user. - -Invalid image-skip reasons include: "the final implementation will be semantic HTML/CSS/SVG", "the diagram should stay editable", "a raster mock would not be used directly", or "the product is fictional." Generated probes and mocks are direction artifacts; they are not implementation assets. - -## Craft Contract - -Craft is not a first pass. It is a loop with these required artifacts: - -1. Confirmed design brief from `shape`. -2. Approved visual direction, from generated probes / mocks when image generation is available. -3. Mock fidelity inventory: the visible ingredients from the approved direction that must survive into code. -4. Semantic, functional implementation using the project's real stack and conventions. -5. Browser evidence across relevant viewports. -6. At least one critique-and-fix pass after the first browser inspection, unless the first pass has no material defects. - -Do not let generated mockups replace interface structure, copy, accessibility, responsive behavior, or state design. But do treat the approved mock as a concrete visual contract for composition, hierarchy, density, atmosphere, signature motifs, image needs, and distinctive visual moves. "North star" means "preserve the important visible ingredients in semantic code," not "use it as loose mood." - -## Step 1: Shape the Design - -Run $impeccable shape, passing along whatever feature description the user provided. - -Wait for the design brief to be fully confirmed by the user before proceeding. The brief is your blueprint, and every implementation decision should trace back to it. - -If this craft run resumed after `teach` created PRODUCT.md, run shape now. Do not treat the teach interview, PRODUCT.md, or a summary of project context as a substitute for shape. Shape is task-specific and must cover scope, content/states, visual direction, constraints, anti-goals, probes when applicable, and explicit brief confirmation. - -If the user has already run $impeccable shape and has a confirmed design brief, skip this step and use the existing brief. - -## Step 2: Load References - -Based on the design brief's "Recommended References" section, consult the relevant impeccable reference files. At minimum, always consult: - -- [spatial-design.md](spatial-design.md) for layout and spacing -- [typography.md](typography.md) for type hierarchy - -Then add references based on the brief's needs: -- Complex interactions or forms? Consult [interaction-design.md](interaction-design.md) -- Animation or transitions? Consult [motion-design.md](motion-design.md) -- Color-heavy or themed? Consult [color-and-contrast.md](color-and-contrast.md) -- Responsive requirements? Consult [responsive-design.md](responsive-design.md) -- Heavy on copy, labels, or errors? Consult [ux-writing.md](ux-writing.md) - -## Step 3: Land the Visual Direction (Capability-Gated) - -Before implementation, generate high-fidelity visual comps when all of these are true: - -- The work is **net-new** or visually open-ended enough that composition exploration will improve the build. -- The brief's scope is **mid-fi, high-fi, or production-ready**. -- The current harness has **built-in image generation capability** (for example, Codex with a native image tool). Do **not** ask the user to set up external APIs, shell scripts, or one-off tooling just to do this. - -When those conditions are met, this step is mandatory for **both brand and product work** in Codex and any harness with built-in image generation. Use native image generation; in Codex, use the built-in `image_gen` tool via the imagegen skill. If image generation is unavailable, do not ask the user to install APIs or tooling. State in one line that the image step is skipped because the harness lacks native image generation, then proceed. - -Do not skip this step because the eventual UI should be semantic, editable, code-native, responsive, or accessible. Those are implementation requirements, not reasons to avoid visual exploration. - -### Purpose - -Use the mock step to find a stronger visual lane than code-first generation would reliably discover on its own. The brief remains authoritative on user, purpose, content, constraints, states, and anti-goals. The mock clarifies composition, hierarchy, density, typography, and visual tone. - -### What to generate - -Generate **1 to 3** high-fidelity north-star comps based on the confirmed brief. If shape already produced direction probes, use those results as input and generate a more resolved mock from the winning lane, not another unrelated exploration. - -- For brand work, push visual identity, composition, and mood aggressively. -- For product work, still push hierarchy, topology, density, and tone, but keep the comps grounded in realistic product structure and states. -- For landing pages and long-form brand surfaces, show enough of the next section or second fold to establish the system beyond the hero. - -The comps must be genuinely different in primary visual direction, not just color variants. - -### Approval loop - -Show the comps and ask what should carry forward. If the user asks for changes or the best direction is still weak, generate a focused revision before implementation. Continue until one direction is approved, or until the user explicitly delegates the choice. - -If the user delegates, pick the strongest direction and explain the decision using the brief, not personal taste. - -Before moving to implementation, summarize: - -- What to carry into code -- What **not** to literalize from the mock - -This summary is required before Step 4. It is the handoff between visual exploration and semantic implementation. - -### Mock fidelity inventory - -Before building, inventory the approved mock's major visible ingredients: - -- Hero silhouette and dominant composition. -- Signature motifs: planets, devices, portraits, charts, route lines, insets, badges, or other memorable objects. -- Nav and primary CTA treatment. -- Section sequence visible in the mock, especially the second fold. -- Image-native content the concept depends on. -- Typography, density, color/material treatment, and motion cues. - -For each ingredient, decide how it will be implemented: semantic HTML/CSS/SVG, generated asset, sourced project asset, icon library, canvas/WebGL, or an explicitly accepted omission. Do not substitute a different hero composition or new visual driver after approval unless the user approves the change. - -Treat the mock as a **north star**, not a screenshot to trace. Do **not** rasterize core UI text or let the mock override the confirmed brief. But if the live result lacks the mock's major visible ingredients, the implementation is wrong. - -## Step 4: Asset Extraction (Need-Gated) - -If the chosen direction includes image-native visual ingredients that would materially improve the implementation, generate them as separate assets before building. - -Good candidates: - -- stickers -- badges -- seals -- tickets -- graphic labels -- textures -- abstract objects -- decorative marks -- non-semantic scene elements - -For travel, editorial, portfolio, venue, product showcase, entertainment, education, or any other image-led brand surface, visual assets are usually core content, not decoration. Do not ship abstract CSS panels where the approved mock or subject matter calls for real imagery, generated plates, illustrations, maps, product/object renders, or destination scenes. - -Do **not** export assets for core UI text, navigation, body copy, or any structure that should stay semantic and editable in code. - -Usually **1 to 5** extracted assets is enough. If the design can be built cleanly in HTML/CSS/SVG, prefer that over raster assets. If the mock contains major visual content that cannot be built credibly in code, asset extraction is not optional. - -## Step 5: Build to Production Quality - -Implement the feature following the design brief. Build in passes so structure, visual system, states, motion/media, and responsive behavior each get deliberate attention. The list below is the definition of done, not inspiration. - -### Production bar - -- Use real or realistic content. Remove placeholder copy, placeholder images, dead links, fake controls, and unused scaffold before presenting. -- Preserve the approved mock's major ingredients. Missing hero objects, missing world/product imagery, different section structure, downgraded CTA/nav treatment, or generic replacements for distinctive motifs are blocking defects unless the user accepted the change. -- Build semantically first: real headings, landmarks, labels, form associations, button/link semantics, accessible names, and state announcements where needed. -- Calibrate spacing, alignment, grid placement, and vertical rhythm deliberately. Do not accept default gaps, arbitrary margins, unbalanced whitespace, or accidental optical misalignment. -- Make typography intentional: chosen font loading strategy, clear hierarchy, readable measure, stable line breaks, tuned wrapping, and no overflow at mobile or large desktop sizes. -- Design realistic state coverage: default, hover where supported, focus-visible, active, disabled, loading, error, success, empty, overflow, long text, short text, and first-run states where relevant. -- Make interaction quality feel finished: keyboard paths, touch targets, feedback timing, scroll behavior, transitions between states, and no hover-only functionality. -- Use icons from the project's established icon set when available. If no set exists, choose a coherent library or use accessible text controls; do not mix unrelated icon styles. -- Optimize imagery and media: correct dimensions, useful alt text, lazy loading below the fold, modern formats when practical, responsive `srcset` / `picture` for raster assets, and no project-referenced asset left outside the workspace. -- Make motion feel premium: use atmospheric blur, filter, mask, shadow, or reveal effects when they improve the experience; avoid casual layout-property animation, bound expensive effects, verify smoothness in-browser, respect reduced motion, and avoid choreography that blocks task completion. -- Preserve maintainability: reusable local patterns, clear component boundaries, project conventions, no rasterized UI text, and no hard-coded one-off hacks when a better local pattern exists. -- Fit the technical context: production build passes, no obvious console errors, no avoidable layout shift, no needless dependency, and no broken asset path. -- If you discover a design question that materially changes the brief or approved direction, stop and ask rather than guessing. - -## Step 6: Browser-Based Iteration - -**This step is critical.** Do not stop after the first implementation pass. - -Open the result in a browser. In Codex, use browser-use or equivalent browser automation when available; otherwise use Playwright or ask the user for screenshots. Inspect screenshots, not just DOM or terminal output. - -### Required viewport pass - -Check the experience at the viewports that matter for the brief. Default minimum: - -- Mobile narrow -- Tablet or small laptop -- Desktop wide - -For each viewport, capture or inspect the rendered state and look for visual defects: overlap, clipping, weak hierarchy, off-grid alignment, awkward whitespace, cramped controls, unreadable type, broken imagery, hover-only functionality, layout shift, and text overflow. - -### Critique and fix loop - -After the first browser pass, write a short critique for yourself and patch the implementation. Repeat browser inspection after fixes. Continue until no material issues remain against this checklist: - -1. **Does it match the brief?** Compare the live result against every section of the design brief. Fix discrepancies. -2. **Does it match the approved mock?** Compare screenshots against the mock fidelity inventory: hero silhouette, major motifs, imagery, nav/CTA, section sequence, density, color/materials, and second-fold substance. Missing major ingredients are P0 defects. -3. **Does it pass the AI slop test?** If someone saw this and said "AI made this," would they believe it immediately? If yes, it needs more design intention. -4. **Check against impeccable's DON'T guidelines.** Fix any anti-pattern violations. -5. **Check every state.** Navigate through empty, error, loading, and edge case states. Each one should feel intentional, not like an afterthought. -6. **Check responsive behavior.** The design should adapt compositionally, not merely shrink. -7. **Check craft details.** Spacing consistency, optical alignment, type hierarchy, color contrast, image quality, icon coherence, interactive feedback, motion timing, and focus treatment. -8. **Check performance basics.** No obviously oversized images, avoidable layout thrash, blocking animations, or heavy assets without a reason. - -The exit bar is not "it works." It is: the rendered result looks intentional at all checked viewports, all expected states are handled, no placeholders remain unless explicitly accepted, and the implementation quality would be defensible in a high-end studio review. - -## Step 7: Present - -Present the result to the user: -- Show the feature in its primary state -- Summarize the browser/viewports checked and the most important fixes made after inspection -- Walk through the key states (empty, error, responsive) -- Explain design decisions that connect back to the design brief and, when used, the chosen north-star mock. Include any accepted deviations from the mock; do not hide unimplemented mock ingredients. -- Note any remaining limitations or follow-up risks honestly -- Ask: "What's working? What isn't?" - -Iterate based on feedback. Good design is rarely right on the first pass. diff --git a/.agents/skills/impeccable/reference/critique.md b/.agents/skills/impeccable/reference/critique.md deleted file mode 100644 index 4e9b73db0..000000000 --- a/.agents/skills/impeccable/reference/critique.md +++ /dev/null @@ -1,213 +0,0 @@ -> **Additional context needed**: what the interface is trying to accomplish. - -### Gather Assessments - -Launch two independent assessments. **Neither may see the other's output.** This isolation is what makes the combined score honest. Running both in one head silently anchors them to each other; do not shortcut it for cost, speed, or context-size reasons. - -Delegate each assessment to a separate sub-agent (Claude Code's `Agent` tool, Codex's subagent spawning, etc.). Each returns structured findings as text. Do NOT output findings to the user yet. - -Fall back to sequential in-head work only if the environment genuinely cannot spawn sub-agents. - -**Tab isolation**: When browser automation is available, each assessment MUST create its own new tab. Never reuse an existing tab, even if one is already open at the correct URL. This prevents the two assessments from interfering with each other's page state. - -#### Assessment A: LLM Design Review - -Read the relevant source files (HTML, CSS, JS/TS) and, if browser automation is available, visually inspect the live page. **Create a new tab** for this; do not reuse existing tabs. After navigation, label the tab by setting the document title: -```javascript -document.title = '[LLM] ' + document.title; -``` -Think like a design director. Evaluate: - -**AI Slop Detection (CRITICAL)**: Does this look like every other AI-generated interface? Review against ALL **DON'T** guidelines from the parent impeccable skill (already loaded in this context). Check for AI color palette, gradient text, dark glows, glassmorphism, hero metric layouts, identical card grids, generic fonts, and all other tells. **The test**: If someone said "AI made this," would you believe them immediately? - -**Holistic Design Review**: visual hierarchy (eye flow, primary action clarity), information architecture (structure, grouping, cognitive load), emotional resonance (does it match brand and audience?), discoverability (are interactive elements obvious?), composition (balance, whitespace, rhythm), typography (hierarchy, readability, font choices), color (purposeful use, cohesion, accessibility), states & edge cases (empty, loading, error, success), microcopy (clarity, tone, helpfulness). - -**Cognitive Load** (consult [cognitive-load](cognitive-load.md)): -- Run the 8-item cognitive load checklist. Report failure count: 0-1 = low (good), 2-3 = moderate, 4+ = critical. -- Count visible options at each decision point. If >4, flag it. -- Check for progressive disclosure: is complexity revealed only when needed? - -**Emotional Journey**: -- What emotion does this interface evoke? Is that intentional? -- **Peak-end rule**: Is the most intense moment positive? Does the experience end well? -- **Emotional valleys**: Check for anxiety spikes at high-stakes moments (payment, delete, commit). Are there design interventions (progress indicators, reassurance copy, undo options)? - -**Nielsen's Heuristics** (consult [heuristics-scoring](heuristics-scoring.md)): -Score each of the 10 heuristics 0-4. This scoring will be presented in the report. - -Return structured findings covering: AI slop verdict, heuristic scores, cognitive load assessment, what's working (2-3 items), priority issues (3-5 with what/why/fix), minor observations, and provocative questions. - -#### Assessment B: Automated Detection - -Run the bundled deterministic detector, which flags 27 specific patterns (AI slop tells + general design quality). - -**CLI scan**: -```bash -npx impeccable --json [--fast] [target] -``` - -- Pass HTML/JSX/TSX/Vue/Svelte files or directories as `[target]` (anything with markup). Do not pass CSS-only files. -- For URLs, skip the CLI scan (it requires Puppeteer). Use browser visualization instead. -- For large directories (200+ scannable files), use `--fast` (regex-only, skips jsdom) -- For 500+ files, narrow scope or ask the user -- Exit code 0 = clean, 2 = findings - -**Browser visualization**: **required** when browser automation tools are available AND the target is a viewable page. The `[Human]` overlay tab is the user-facing deliverable; the critique is incomplete without it. Skip only if the target is not a viewable page (CSS-only file, non-browser target). - -The overlay is a **visual aid for the user**. It highlights issues directly in their browser. Do NOT scroll through the page to screenshot overlays. Instead, read the console output to get the results programmatically. - -1. **Start the live detection server**: - ```bash - npx impeccable live & - ``` - Note the port printed to stdout (auto-assigned). Use `--port=PORT` to fix it. -2. **Create a new tab** and navigate to the page (use dev server URL for local files, or direct URL). Do not reuse existing tabs. -3. **Label the tab** via `javascript_tool` so the user can distinguish it: - ```javascript - document.title = '[Human] ' + document.title; - ``` -4. **Scroll to top** to ensure the page is scrolled to the very top before injection -5. **Inject** via `javascript_tool` (replace PORT with the port from step 1): - ```javascript - const s = document.createElement('script'); s.src = 'http://localhost:PORT/detect.js'; document.head.appendChild(s); - ``` -6. Wait 2-3 seconds for the detector to render overlays -7. **Read results from console** using `read_console_messages` with pattern `impeccable`. The detector logs all findings with the `[impeccable]` prefix. Do NOT scroll through the page to take screenshots of the overlays. -8. **Cleanup**: Stop the live server when done: - ```bash - npx impeccable live stop - ``` - -For multi-view targets, inject on 3-5 representative pages. If injection fails, continue with CLI results only. - -Return: CLI findings (JSON), browser console findings (if applicable), and any false positives noted. - -### Generate Combined Critique Report - -Synthesize both assessments into a single report. Do NOT simply concatenate. Weave the findings together, noting where the LLM review and detector agree, where the detector caught issues the LLM missed, and where detector findings are false positives. - -Structure your feedback as a design director would: - -#### Design Health Score -> *Consult [heuristics-scoring](heuristics-scoring.md)* - -Present the Nielsen's 10 heuristics scores as a table: - -| # | Heuristic | Score | Key Issue | -|---|-----------|-------|-----------| -| 1 | Visibility of System Status | ? | [specific finding or "n/a" if solid] | -| 2 | Match System / Real World | ? | | -| 3 | User Control and Freedom | ? | | -| 4 | Consistency and Standards | ? | | -| 5 | Error Prevention | ? | | -| 6 | Recognition Rather Than Recall | ? | | -| 7 | Flexibility and Efficiency | ? | | -| 8 | Aesthetic and Minimalist Design | ? | | -| 9 | Error Recovery | ? | | -| 10 | Help and Documentation | ? | | -| **Total** | | **??/40** | **[Rating band]** | - -Be honest with scores. A 4 means genuinely excellent. Most real interfaces score 20-32. - -#### Anti-Patterns Verdict - -**Start here.** Does this look AI-generated? - -**LLM assessment**: Your own evaluation of AI slop tells. Cover overall aesthetic feel, layout sameness, generic composition, missed opportunities for personality. - -**Deterministic scan**: Summarize what the automated detector found, with counts and file locations. Note any additional issues the detector caught that you missed, and flag any false positives. - -**Visual overlays** (if browser was used): Tell the user that overlays are now visible in the **[Human]** tab in their browser, highlighting the detected issues. Summarize what the console output reported. - -#### Overall Impression -A brief gut reaction: what works, what doesn't, and the single biggest opportunity. - -#### What's Working -Highlight 2-3 things done well. Be specific about why they work. - -#### Priority Issues -The 3-5 most impactful design problems, ordered by importance. - -For each issue, tag with **P0-P3 severity** (consult [heuristics-scoring](heuristics-scoring.md) for severity definitions): -- **[P?] What**: Name the problem clearly -- **Why it matters**: How this hurts users or undermines goals -- **Fix**: What to do about it (be concrete) -- **Suggested command**: Which command could address this (from: $impeccable adapt, $impeccable animate, $impeccable audit, $impeccable bolder, $impeccable clarify, $impeccable colorize, $impeccable critique, $impeccable delight, $impeccable distill, $impeccable document, $impeccable harden, $impeccable layout, $impeccable onboard, $impeccable optimize, $impeccable overdrive, $impeccable polish, $impeccable quieter, $impeccable shape, $impeccable typeset) - -#### Persona Red Flags -> *Consult [personas](personas.md)* - -Auto-select 2-3 personas most relevant to this interface type (use the selection table in the reference). If `AGENTS.md` contains a `## Design Context` section from `impeccable teach`, also generate 1-2 project-specific personas from the audience/brand info. - -For each selected persona, walk through the primary user action and list specific red flags found: - -**Alex (Power User)**: No keyboard shortcuts detected. Form requires 8 clicks for primary action. Forced modal onboarding. High abandonment risk. - -**Jordan (First-Timer)**: Icon-only nav in sidebar. Technical jargon in error messages ("404 Not Found"). No visible help. Will abandon at step 2. - -Be specific. Name the exact elements and interactions that fail each persona. Don't write generic persona descriptions; write what broke for them. - -#### Minor Observations -Quick notes on smaller issues worth addressing. - -#### Questions to Consider -Provocative questions that might unlock better solutions: -- "What if the primary action were more prominent?" -- "Does this need to feel this complex?" -- "What would a confident version of this look like?" - -**Remember**: -- Be direct. Vague feedback wastes everyone's time. -- Be specific. "The submit button," not "some elements." -- Say what's wrong AND why it matters to users. -- Give concrete suggestions. Cut "consider exploring..." entirely. -- Prioritize ruthlessly. If everything is important, nothing is. -- Don't soften criticism. Developers need honest feedback to ship great design. - -### Ask the User - -**After presenting findings**, use targeted questions based on what was actually found. STOP and use Codex's structured user-input/question tool when available; if unavailable, ask directly in chat to clarify what you cannot infer. These answers will shape the action plan. - -Ask questions along these lines (adapt to the specific findings; do NOT ask generic questions): - -1. **Priority direction**: Based on the issues found, ask which category matters most to the user right now. For example: "I found problems with visual hierarchy, color usage, and information overload. Which area should we tackle first?" Offer the top 2-3 issue categories as options. - -2. **Design intent**: If the critique found a tonal mismatch, ask whether it was intentional. For example: "The interface feels clinical and corporate. Is that the intended tone, or should it feel warmer/bolder/more playful?" Offer 2-3 tonal directions as options based on what would fix the issues found. - -3. **Scope**: Ask how much the user wants to take on. For example: "I found N issues. Want to address everything, or focus on the top 3?" Offer scope options like "Top 3 only", "All issues", "Critical issues only". - -4. **Constraints** (optional; only ask if relevant): If the findings touch many areas, ask if anything is off-limits. For example: "Should any sections stay as-is?" This prevents the plan from touching things the user considers done. - -**Rules for questions**: -- Every question must reference specific findings from the report. Never ask generic "who is your audience?" questions. -- Keep it to 2-4 questions maximum. Respect the user's time. -- Offer concrete options, not open-ended prompts. -- If findings are straightforward (e.g., only 1-2 clear issues), skip questions and go directly to Recommended Actions. - -### Recommended Actions - -**After receiving the user's answers**, present a prioritized action summary reflecting the user's priorities and scope from Ask the User. - -#### Action Summary - -List recommended commands in priority order, based on the user's answers: - -1. **`$command-name`**: Brief description of what to fix (specific context from critique findings) -2. **`$command-name`**: Brief description (specific context) -... - -**Rules for recommendations**: -- Only recommend commands from: $impeccable adapt, $impeccable animate, $impeccable audit, $impeccable bolder, $impeccable clarify, $impeccable colorize, $impeccable critique, $impeccable delight, $impeccable distill, $impeccable document, $impeccable harden, $impeccable layout, $impeccable onboard, $impeccable optimize, $impeccable overdrive, $impeccable polish, $impeccable quieter, $impeccable shape, $impeccable typeset -- Order by the user's stated priorities first, then by impact -- Each item's description should carry enough context that the command knows what to focus on -- Map each Priority Issue to the appropriate command -- Skip commands that would address zero issues -- If the user chose a limited scope, only include items within that scope -- If the user marked areas as off-limits, exclude commands that would touch those areas -- End with `$impeccable polish` as the final step if any fixes were recommended - -After presenting the summary, tell the user: - -> You can ask me to run these one at a time, all at once, or in any order you prefer. -> -> Re-run `$impeccable critique` after fixes to see your score improve. diff --git a/.agents/skills/impeccable/reference/shape.md b/.agents/skills/impeccable/reference/shape.md deleted file mode 100644 index 4eaa99a30..000000000 --- a/.agents/skills/impeccable/reference/shape.md +++ /dev/null @@ -1,151 +0,0 @@ -Shape the UX and UI for a feature before any code is written. This command produces a **design brief**: a structured artifact that guides implementation through discovery, not guesswork. - -**Scope**: Design planning only. This command does NOT write code. It produces the thinking that makes code good. - -**Output**: A design brief that can be handed off to $impeccable craft, or directly to $impeccable for freeform implementation. When visual direction probes are used, the images are supporting artifacts, not the primary output. - -## Philosophy - -Most AI-generated UIs fail not because of bad code, but because of skipped thinking. They jump to "here's a card grid" without asking "what is the user trying to accomplish?" This command inverts that: understand deeply first, so implementation is precise. - -## Phase 1: Discovery Interview - -**Do NOT write any code or make any design decisions during this phase.** Your only job is to understand the feature deeply enough to make excellent design decisions later. - -This is a required interaction, not optional guidance. Ask these questions in conversation, adapting based on answers. Don't dump them all at once; have a natural dialogue. STOP and use Codex's structured user-input/question tool when available; if unavailable, ask directly in chat to clarify what you cannot infer. - -### Interview cadence - -Discovery must include at least one user-answer round unless PRODUCT.md, DESIGN.md, or an already-confirmed brief directly answers the needed design inputs. With a sparse prompt, do **not** synthesize a complete brief for confirmation on the first response. - -- Use the harness's structured question tool when one exists. Otherwise, ask directly in chat and stop. -- Ask **2-3 questions per round**, then wait for answers. -- Treat PRODUCT.md and DESIGN.md as anchors; they reduce repeated questions but do **not** replace shape for craft. Shape is task-specific. -- Round 1 should clarify purpose, audience/context, and success or emotional outcome. -- Round 2 should clarify content/data/states and scope/fidelity. -- Round 3 should clarify visual direction, constraints, and anti-goals when still unresolved. - -### Purpose & Context -- What is this feature for? What problem does it solve? -- Who specifically will use it? (Not "users"; be specific: role, context, frequency) -- What does success look like? How will you know this feature is working? -- What's the user's state of mind when they reach this feature? (Rushed? Exploring? Anxious? Focused?) - -### Content & Data -- What content or data does this feature display or collect? -- What are the realistic ranges? (Minimum, typical, maximum, e.g., 0 items, 5 items, 500 items) -- What are the edge cases? (Empty state, error state, first-time use, power user) -- Is any content dynamic? What changes and how often? - -### Design Direction - -Force a visual decision on three fronts. Skip anything PRODUCT.md or DESIGN.md already answers; ask only what's missing. - -- **Color strategy for this surface.** Pick one: Restrained / Committed / Full palette / Drenched. Can override the project default if the surface earns it (e.g. a drenched hero inside an otherwise Restrained product). -- **Theme via scene sentence.** Write one sentence of physical context for this surface: who uses it, where, under what ambient light, in what mood. The sentence forces dark vs light. If it doesn't, add detail until it does. -- **Two or three named anchor references.** Specific products, brands, objects. Not adjectives like "modern" or "clean." - -### Scope - -Always ask. Sketch quality and shipped quality are different outputs; don't guess between them. - -- **Fidelity.** Sketch / mid-fi / high-fi / production-ready? -- **Breadth.** One screen / a flow / a whole surface? -- **Interactivity.** Static visual / interactive prototype / shipped-quality component? -- **Time intent.** Quick exploration, or polish until it ships? - -Scope answers are task-scoped. Don't write them to PRODUCT.md or DESIGN.md; carry them through the design brief only. - -### Constraints -- Are there technical constraints? (Framework, performance budget, browser support) -- Are there content constraints? (Localization, dynamic text length, user-generated content) -- Mobile/responsive requirements? -- Accessibility requirements beyond WCAG AA? - -### Anti-Goals -- What should this NOT be? What would be a wrong direction? -- What's the biggest risk of getting this wrong? - -## Phase 1.5: Visual Direction Probe (Capability-Gated) - -After the discovery interview, generate a small set of visual direction probes **before** writing the final brief when all of these are true: - -- The work is **net-new** or directionally ambiguous enough that visual exploration will clarify the brief. -- The requested fidelity is **mid-fi, high-fi, or production-ready**. Skip for sketch-only planning. -- The current harness has **built-in image generation capability** (for example, Codex with a native image tool). Do **not** ask the user to set up external APIs, shell scripts, or one-off tooling just to do this. - -When those conditions are met, this step is mandatory for Codex and any harness with built-in image generation. Use native image generation; in Codex, use the built-in `image_gen` tool via the imagegen skill. If image generation is unavailable, do not ask the user to install APIs or tooling. State in one line that the image step is skipped because the harness lacks native image generation, then proceed. - -Use probes to explore visual lanes, not to replace the brief. - -Do not skip probes because the final UI will be semantic, editable, code-native, responsive, or accessible. Those are implementation requirements, not reasons to avoid visual exploration. - -### What to generate - -Generate **2 to 4** distinct direction probes based on the discovery answers, especially: - -- Color strategy -- Theme scene sentence -- Named anchor references -- Scope and fidelity - -The probes should differ in primary visual direction (hierarchy, topology, density, typographic voice, or color strategy), not just palette tweaks. - -### How to use the probes - -- Treat them as **direction tests**, not final designs. -- Use them to pressure-test whether the brief is pointing at the right lane. -- Ask the user which direction feels closest, what feels off, and what should carry forward. -- If the probes reveal a mismatch, revise the brief inputs before finalizing the brief. - -### Important limits - -- Do **not** skip discovery because image generation is available. -- Do **not** treat generated imagery as final UX specification, final copy, or final accessibility behavior. -- Do **not** use this step for minor refinements of existing work. It's for shaping a new surface or clarifying a big directional choice. - -If image generation is unavailable, or the task doesn't benefit from it, skip this phase only with a one-line reason and proceed directly to the design brief. - -## Phase 2: Design Brief - -After the interview and any required probes, synthesize everything into a structured design brief. Present it to the user for explicit confirmation before considering this command complete. Stop after asking for confirmation; do not proceed to craft or implementation in the same response unless the user has already approved the brief. - -### Brief Structure - -**1. Feature Summary** (2-3 sentences) -What this is, who it's for, what it needs to accomplish. - -**2. Primary User Action** -The single most important thing a user should do or understand here. - -**3. Design Direction** -Color strategy (Restrained / Committed / Full palette / Drenched) + the theme scene sentence + 2–3 named anchor references. Reference PRODUCT.md and DESIGN.md where they already answer, and note any per-surface overrides. - -If you ran the Visual Direction Probe step, name which probe direction won and what changed in the brief because of it. - -**4. Scope** -Fidelity, breadth, interactivity, and time intent from the Scope section of the interview. Task-scoped; these don't persist beyond the brief. - -**5. Layout Strategy** -High-level spatial approach: what gets emphasis, what's secondary, how information flows. Describe the visual hierarchy and rhythm, not specific CSS. - -**6. Key States** -List every state the feature needs: default, empty, loading, error, success, edge cases. For each, note what the user needs to see and feel. - -**7. Interaction Model** -How users interact with this feature. What happens on click, hover, scroll? What feedback do they get? What's the flow from entry to completion? - -**8. Content Requirements** -What copy, labels, empty state messages, error messages, and microcopy are needed. Note any dynamic content and its realistic ranges. - -**9. Recommended References** -Based on the brief, list which impeccable reference files would be most valuable during implementation (e.g., spatial-design.md for complex layouts, motion-design.md for animated features, interaction-design.md for form-heavy features). - -**10. Open Questions** -Anything unresolved that the implementer should resolve during build. - ---- - -STOP and use Codex's structured user-input/question tool when available; if unavailable, ask directly in chat to clarify what you cannot infer. Ask for explicit confirmation of the brief before finishing. If the user disagrees with any part, revisit the relevant discovery questions. A shape run is incomplete until the brief is confirmed. - -Once confirmed, the brief is complete. The user can now hand it to $impeccable, or use it to guide any other implementation approach. (If the user wants the full discovery-then-build flow in one step, they should use $impeccable craft instead, which runs this command internally.) diff --git a/.agents/skills/init/SKILL.md b/.agents/skills/init/SKILL.md deleted file mode 100644 index f179683ea..000000000 --- a/.agents/skills/init/SKILL.md +++ /dev/null @@ -1,140 +0,0 @@ ---- -name: init -description: Generate CLAUDE.md and AGENTS.md by exploring the codebase -metadata: - provider: atomic ---- - -# Generate CLAUDE.md and AGENTS.md - -You are tasked with exploring the current codebase with the codebase-analyzer, codebase-locator, codebase-pattern-finder sub-agents, detecting the primary project languages, checking whether the corresponding language servers already exist on the user's machine, optionally installing any missing language servers after explicit user confirmation, and then generating populated `CLAUDE.md` and `AGENTS.md` files at the project root. These files provide coding agents with the context they need to work effectively in this repository. - -## Steps - -1. **Explore the codebase to discover project metadata:** - - Read `package.json`, `Cargo.toml`, `go.mod`, `pyproject.toml`, `Gemfile`, `pom.xml`, or similar manifest files - - Scan the top-level directory structure (`src/`, `lib/`, `app/`, `tests/`, `docs/`, etc.) - - Check for existing config files: `.eslintrc`, `tsconfig.json`, `biome.json`, `oxlint.json`, `.prettierrc`, CI configs (`.github/workflows/`, `.gitlab-ci.yml`), etc. - - Read `README.md` if it exists for project description and setup instructions - - Check for `.env.example`, `.env.local`, or similar environment files - - Identify the package manager (bun, npm, yarn, pnpm, cargo, go, pip, etc.) - - Identify the primary project languages from manifests, lockfiles, and source file extensions - - Inspect editor/tooling config such as `.vscode`, language-specific config files, or existing LSP settings when present - -2. **Identify key project attributes:** - - **Project name**: From manifest file or directory name - - **Project purpose**: 1-2 sentence description from README or manifest - - **Project structure**: Key directories and their purposes - - **Tech stack**: Language, framework, runtime - - **Detected languages**: The main implementation languages in the repo - - **Recommended LSPs**: The language servers that best match those languages - - **Commands**: dev, build, test, lint, typecheck, format (from scripts in manifest) - - **Environment setup**: Required env vars, env example files - - **Verification command**: The command to run before commits (usually lint + typecheck + test) - - **Existing documentation**: Links to docs within the repo - -3. **Detect installed language servers and prepare an installation plan:** - - For each detected language, choose the most standard LSP for that ecosystem and prefer already configured tooling when the repo clearly indicates a preference - - Check whether each LSP is already available by using non-destructive discovery commands such as `command -v`, `--version`, or equivalent read-only checks - - Use this default mapping unless the repo clearly points to a different choice: - - TypeScript / JavaScript -> `typescript-language-server` (and ensure `typescript` is available when required) - - Python -> `pyright` - - Go -> `gopls` - - Rust -> `rust-analyzer` - - Ruby -> `ruby-lsp` - - PHP -> `intelephense` - - Lua -> `lua-language-server` - - Bash / shell -> `bash-language-server` - - YAML -> `yaml-language-server` - - Docker -> `dockerfile-language-server-nodejs` - - Terraform -> `terraform-ls` - - Java -> `jdtls` - - Kotlin -> `kotlin-language-server` - - C / C++ -> `clangd` - - C# -> `csharp-ls` - - If an LSP is missing, prepare the safest install command that fits the user's available tooling and platform; do not guess a package manager that is not installed - - If the required runtime or package manager is missing, stop short of installation and report what is needed instead - -4. **Ask for confirmation before installing anything:** - - Summarize the detected languages, the LSPs already present, the LSPs that are missing, and the exact install commands you plan to run - - Ask the user for confirmation before running any install command - - If the user declines, skip installation and continue with documentation generation - - After installation, verify each newly installed LSP with a version check or binary lookup and mention any failures clearly - -5. **Populate the template below** with discovered values. Replace every `{{placeholder}}` with actual values from the repo. Delete sections that don't apply (e.g., Environment if there are no env files). Remove the "How to Fill This Template" meta-section entirely. - -6. **Write the populated content** to both `CLAUDE.md` and `AGENTS.md` at the project root with identical content. - -## Template - -```markdown -# {{PROJECT_NAME}} - -## Overview - -{{1-2 sentences describing the project purpose}} - -## Project Structure - -| Path | Type | Purpose | -| ------------ | -------- | ----------- | -| \`{{path}}\` | {{type}} | {{purpose}} | - -## Quick Reference - -### Languages and Tooling - -- Languages: {{comma-separated detected languages}} -- LSPs: {{comma-separated installed or recommended language servers}} - -### Commands - -\`\`\`bash -{{dev_command}} # Start dev server / all services -{{build_command}} # Build the project -{{test_command}} # Run tests -{{lint_command}} # Lint & format check -{{typecheck_command}} # Type-check (if applicable) -\`\`\` - -### Environment - -- Copy \`{{env_example_file}}\` → \`{{env_local_file}}\` for local development -- Required vars: {{comma-separated list of required env vars}} - -## Progressive Disclosure - -Read relevant docs before starting: -| Topic | Location | -| ----- | -------- | -| {{topic}} | \`{{path_to_doc}}\` | - -## Universal Rules - -1. Run \`{{verify_command}}\` before commits -2. Keep PRs focused on a single concern -3. {{Add any project-specific universal rules}} - -## Code Quality - -Formatting and linting are handled by automated tools: - -- \`{{lint_command}}\` — {{linter/formatter names}} -- \`{{format_command}}\` — Auto-fix formatting (if separate from lint) - -Run before committing. Don't manually check style—let tools do it. -``` - -## Important Notes - -- **Keep it under 100 lines** (ideally under 60) after populating -- **Every instruction must be universally applicable** to all tasks in the repo -- **No code style rules** — delegate to linters/formatters -- **No task-specific instructions** — use the progressive disclosure table -- **No code snippets** — use `file:line` pointers instead -- **Include verification commands** the agent can run to validate work -- **Never install tooling without an explicit user confirmation first** -- **Prefer read-only discovery before installation** and verify any installed LSP afterward -- Delete any section from the template that doesn't apply to this project -- Do NOT include the "How to Fill This Template" section in the output -- Write identical content to both `CLAUDE.md` and `AGENTS.md` at the project root diff --git a/.agents/skills/liteparse/SKILL.md b/.agents/skills/liteparse/SKILL.md deleted file mode 100644 index 9b3096b71..000000000 --- a/.agents/skills/liteparse/SKILL.md +++ /dev/null @@ -1,223 +0,0 @@ ---- -name: liteparse -description: Use this skill when the user asks to parse, perform multi-format document conversion or spatially extract text from an unstructured file (PDF, DOCX, PPTX, XLSX, images, etc.) locally without cloud dependencies. -compatibility: Requires Node 18+ and `@llamaindex/liteparse` installed globally via npm (`npm i -g @llamaindex/liteparse`) -license: MIT -metadata: - provider: atomic - author: LlamaIndex - version: "0.1.0" ---- - -# LiteParse Skill - -Parse unstructured documents (PDF, DOCX, PPTX, XLSX, images, and more) locally with LiteParse: fast, lightweight, no cloud dependencies or LLM required. - -## Initial Setup - -When this skill is invoked, respond with: - -``` -I'm ready to use LiteParse to parse files locally. Before we begin, please confirm that: - -- `@llamaindex/liteparse` is installed globally (`npm i -g @llamaindex/liteparse`) -- The `lit` CLI command is available in your terminal - -If both are set, please provide: - -1. One or more files to parse (PDF, DOCX, PPTX, XLSX, images, etc.) -2. Any specific options: output format (json/text), page ranges, OCR preferences, DPI, etc. -3. What you'd like to do with the parsed content. - -I will produce the appropriate `lit` CLI command or TypeScript script, and once approved, report the results. -``` - -Then wait for the user's input. - ---- - -## Step 0 — Install LiteParse (if needed) - -If `liteparse` is not yet installed, install it globally: - -```bash -npm i -g @llamaindex/liteparse -``` - -Verify installation: - -```bash -lit --version -``` - -For Office document support (DOCX, PPTX, XLSX), LibreOffice is required: - -```bash -# macOS -brew install --cask libreoffice - -# Ubuntu/Debian -apt-get install libreoffice -``` - -For image parsing, ImageMagick is required: -```bash -# macOS -brew install imagemagick - -# Ubuntu/Debian -apt-get install imagemagick -``` - ---- - -## Step 1 — Produce the CLI Command or Script - -### Parse a Single File - -```bash -# Basic text extraction -lit parse document.pdf - -# JSON output saved to a file -lit parse document.pdf --format json -o output.json - -# Specific page range -lit parse document.pdf --target-pages "1-5,10,15-20" - -# Disable OCR (faster, text-only PDFs) -lit parse document.pdf --no-ocr - -# Use an external HTTP OCR server for higher accuracy -lit parse document.pdf --ocr-server-url http://localhost:8828/ocr - -# Higher DPI for better quality -lit parse document.pdf --dpi 300 -``` - -### Batch Parse a Directory - -```bash -lit batch-parse ./input-directory ./output-directory - -# Only process PDFs, recursively -lit batch-parse ./input ./output --extension .pdf --recursive -``` - -### Generate Page Screenshots - -Screenshots are useful for LLM agents that need to see visual layout. - -```bash -# All pages -lit screenshot document.pdf -o ./screenshots - -# Specific pages -lit screenshot document.pdf --pages "1,3,5" -o ./screenshots - -# High-DPI PNG -lit screenshot document.pdf --dpi 300 --format png -o ./screenshots - -# Page range -lit screenshot document.pdf --pages "1-10" -o ./screenshots -``` - ---- - -## Step 3 — Key Options Reference - -### OCR Options - -| Option | Description | -|--------|-------------| -| (default) | Tesseract.js — zero setup, built-in | -| `--ocr-language fra` | Set OCR language (ISO code) | -| `--ocr-server-url ` | Use external HTTP OCR server (EasyOCR, PaddleOCR, custom) | -| `--no-ocr` | Disable OCR entirely | - -### Output Options - -| Option | Description | -|--------|-------------| -| `--format json` | Structured JSON with bounding boxes | -| `--format text` | Plain text (default) | -| `-o ` | Save output to file | - -### Performance / Quality Options - -| Option | Description | -|--------|-------------| -| `--dpi ` | Rendering DPI (default: 150; use 300 for high quality) | -| `--max-pages ` | Limit pages parsed | -| `--target-pages ` | Parse specific pages (e.g. `"1-5,10"`) | -| `--no-precise-bbox` | Disable precise bounding boxes (faster) | -| `--skip-diagonal-text` | Ignore rotated/diagonal text | -| `--preserve-small-text` | Keep very small text that would otherwise be dropped | - ---- - -## Step 4 — Using a Config File - -For repeated use with consistent options, generate a `liteparse.config.json`: - -```json -{ - "ocrLanguage": "en", - "ocrEnabled": true, - "maxPages": 1000, - "dpi": 150, - "outputFormat": "json", - "preciseBoundingBox": true, - "skipDiagonalText": false, - "preserveVerySmallText": false -} -``` - -For an HTTP OCR server: - -```json -{ - "ocrServerUrl": "http://localhost:8828/ocr", - "ocrLanguage": "en", - "outputFormat": "json" -} -``` - -Use with: - -```bash -lit parse document.pdf --config liteparse.config.json -``` - ---- - -## Step 5 — HTTP OCR Server API (Advanced) - -If the user wants to plug in a custom OCR backend, the server must implement: - -- **Endpoint**: `POST /ocr` -- **Accepts**: `file` (multipart) and `language` (string) parameters -- **Returns**: -```json -{ - "results": [ - { "text": "Hello", "bbox": [x1, y1, x2, y2], "confidence": 0.98 } - ] -} -``` - -Ready-to-use wrappers exist for EasyOCR and PaddleOCR in the LiteParse repo. - ---- - -## Supported Input Formats - -| Category | Formats | -|----------|---------| -| PDF | `.pdf` | -| Word | `.doc`, `.docx`, `.docm`, `.odt`, `.rtf` | -| PowerPoint | `.ppt`, `.pptx`, `.pptm`, `.odp` | -| Spreadsheets | `.xls`, `.xlsx`, `.xlsm`, `.ods`, `.csv`, `.tsv` | -| Images | `.jpg`, `.jpeg`, `.png`, `.gif`, `.bmp`, `.tiff`, `.webp`, `.svg` | - -Office documents require LibreOffice; images require ImageMagick. LiteParse auto-converts these formats to PDF before parsing. diff --git a/.agents/skills/memory-systems/references/implementation.md b/.agents/skills/memory-systems/references/implementation.md deleted file mode 100644 index 8c248456f..000000000 --- a/.agents/skills/memory-systems/references/implementation.md +++ /dev/null @@ -1,551 +0,0 @@ -# Memory Systems: Technical Reference - -This document provides implementation details for memory system components. - -## Vector Store Implementation - -### Basic Vector Store - -```python -import numpy as np -from typing import List, Dict, Any -import json - - -def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float: - """Compute cosine similarity between two vectors.""" - norm_a = np.linalg.norm(a) - norm_b = np.linalg.norm(b) - if norm_a == 0 or norm_b == 0: - return 0.0 - return float(np.dot(a, b) / (norm_a * norm_b)) - - -class VectorStore: - def __init__(self, dimension=768): - self.dimension = dimension - self.vectors = [] - self.metadata = [] - self.texts = [] - - def add(self, text: str, metadata: Dict[str, Any] = None): - """Add document to store.""" - embedding = self._embed(text) - self.vectors.append(embedding) - self.metadata.append(metadata or {}) - self.texts.append(text) - return len(self.vectors) - 1 - - def search(self, query: str, limit: int = 5, - filters: Dict[str, Any] = None) -> List[Dict]: - """Search for similar documents.""" - query_embedding = self._embed(query) - - scores = [] - for i, vec in enumerate(self.vectors): - score = cosine_similarity(query_embedding, vec) - - # Apply filters - if filters and not self._matches_filters(self.metadata[i], filters): - score = -1 # Exclude - - scores.append((i, score)) - - # Sort by score - scores.sort(key=lambda x: x[1], reverse=True) - - # Return top k - results = [] - for idx, score in scores[:limit]: - if score > 0: # Only include positive matches - results.append({ - "index": idx, - "score": score, - "text": self._get_text(idx), - "metadata": self.metadata[idx] - }) - - return results - - def _embed(self, text: str) -> np.ndarray: - """Generate deterministic pseudo-embedding for demonstration. - In production, replace with actual embedding model.""" - np.random.seed(hash(text) % (2**32)) - vec = np.random.randn(self.dimension) - return vec / (np.linalg.norm(vec) + 1e-8) - - def _matches_filters(self, metadata: Dict, filters: Dict) -> bool: - """Check if metadata matches filters.""" - for key, value in filters.items(): - if key not in metadata: - return False - if isinstance(value, list): - if metadata[key] not in value: - return False - elif metadata[key] != value: - return False - return True - - def _get_text(self, index: int) -> str: - """Retrieve original text for index.""" - return self.texts[index] if index < len(self.texts) else "" -``` - -### Metadata-Enhanced Vector Store - -```python -class MetadataVectorStore(VectorStore): - def __init__(self, dimension=768): - super().__init__(dimension) - self.entity_index = {} # entity -> [indices] - self.time_index = {} # time_range -> [indices] - - def add(self, text: str, metadata: Dict[str, Any] = None): - """Add with enhanced indexing.""" - metadata = metadata or {} - index = super().add(text, metadata) - - # Index by entity - if "entity" in metadata: - entity = metadata["entity"] - if entity not in self.entity_index: - self.entity_index[entity] = [] - self.entity_index[entity].append(index) - - # Index by time - if "valid_from" in metadata: - time_key = self._time_range_key( - metadata.get("valid_from"), - metadata.get("valid_until") - ) - if time_key not in self.time_index: - self.time_index[time_key] = [] - self.time_index[time_key].append(index) - - return index - - def search_by_entity(self, query: str, entity: str, limit: int = 5) -> List[Dict]: - """Search within specific entity.""" - indices = self.entity_index.get(entity, []) - filtered = [self.metadata[i] for i in indices] - - # Score and rank - query_embedding = self._embed(query) - scored = [] - for i, meta in zip(indices, filtered): - vec = self.vectors[i] - score = cosine_similarity(query_embedding, vec) - scored.append((i, score, meta)) - - scored.sort(key=lambda x: x[1], reverse=True) - - return [{ - "index": idx, - "score": score, - "metadata": meta - } for idx, score, meta in scored[:limit]] -``` - -## Knowledge Graph Implementation - -### Property Graph Storage - -```python -from typing import Dict, List, Optional -import uuid - -class PropertyGraph: - def __init__(self): - self.nodes = {} # id -> properties - self.edges = [] # list of edge dicts - self.entity_registry = {} # name -> node_id (maintains identity) - self.indexes = { - "node_label": {}, # label -> [node_ids] - "edge_type": {} # type -> [edge_ids] - } - - def get_or_create_node(self, name: str, label: str, properties: Dict = None) -> str: - """Get existing node by name, or create a new one. - Uses entity_registry to ensure identity across interactions.""" - if name in self.entity_registry: - return self.entity_registry[name] - node_id = self.create_node(label, {**(properties or {}), "name": name}) - self.entity_registry[name] = node_id - return node_id - - def create_node(self, label: str, properties: Dict = None) -> str: - """Create node with label and properties.""" - node_id = str(uuid.uuid4()) - self.nodes[node_id] = { - "label": label, - "properties": properties or {} - } - - # Index by label - if label not in self.indexes["node_label"]: - self.indexes["node_label"][label] = [] - self.indexes["node_label"][label].append(node_id) - - return node_id - - def create_relationship(self, source_id: str, rel_type: str, - target_id: str, properties: Dict = None) -> str: - """Create directed relationship between nodes.""" - edge_id = str(uuid.uuid4()) - self.edges.append({ - "id": edge_id, - "source": source_id, - "target": target_id, - "type": rel_type, - "properties": properties or {} - }) - - # Index by type - if rel_type not in self.indexes["edge_type"]: - self.indexes["edge_type"][rel_type] = [] - self.indexes["edge_type"][rel_type].append(edge_id) - - return edge_id - - def query(self, cypher_like: str, params: Dict = None) -> List[Dict]: - """ - Simple query matching. - - Supports patterns like: - MATCH (e)-[r]->(o) WHERE e.id = $id RETURN r - """ - # In production, use actual graph database - # This is a simplified pattern matcher - results = [] - - if cypher_like.startswith("MATCH"): - # Parse basic pattern - pattern = self._parse_pattern(cypher_like) - results = self._match_pattern(pattern, params or {}) - - return results - - def _parse_pattern(self, query: str) -> Dict: - """Parse simplified MATCH pattern.""" - # Simplified parser for demonstration - return { - "source_label": self._extract_label(query, "source"), - "rel_type": self._extract_type(query), - "target_label": self._extract_label(query, "target"), - "where": self._extract_where(query) - } - - def _match_pattern(self, pattern: Dict, params: Dict) -> List[Dict]: - """Match pattern against graph.""" - results = [] - - for edge in self.edges: - # Match relationship type - if pattern["rel_type"] and edge["type"] != pattern["rel_type"]: - continue - - source = self.nodes.get(edge["source"], {}) - target = self.nodes.get(edge["target"], {}) - - # Match labels - if pattern["source_label"] and source.get("label") != pattern["source_label"]: - continue - if pattern["target_label"] and target.get("label") != pattern["target_label"]: - continue - - # Match where clause - if pattern["where"] and not self._match_where(edge, source, target, params): - continue - - results.append({ - "source": source, - "relationship": edge, - "target": target - }) - - return results -``` - -## Temporal Knowledge Graph - -```python -from datetime import datetime -from typing import Optional - -class TemporalKnowledgeGraph(PropertyGraph): - def __init__(self): - super().__init__() - self.temporal_index = {} # time_range -> [edge_ids] - - def create_temporal_relationship( - self, - source_id: str, - rel_type: str, - target_id: str, - valid_from: datetime, - valid_until: Optional[datetime] = None, - properties: Dict = None - ) -> str: - """Create relationship with temporal validity.""" - edge_id = super().create_relationship( - source_id, rel_type, target_id, properties - ) - - # Index temporally - time_key = self._time_range_key(valid_from, valid_until) - if time_key not in self.temporal_index: - self.temporal_index[time_key] = [] - self.temporal_index[time_key].append(edge_id) - - # Store validity on edge - edge = self._get_edge(edge_id) - edge["valid_from"] = valid_from.isoformat() - edge["valid_until"] = valid_until.isoformat() if valid_until else None - - return edge_id - - def query_at_time(self, query: str, query_time: datetime) -> List[Dict]: - """Query graph state at specific time.""" - # Find edges valid at query time - valid_edges = [] - for edge in self.edges: - valid_from = datetime.fromisoformat(edge.get("valid_from", "1970-01-01")) - valid_until = edge.get("valid_until") - - if valid_from <= query_time: - if valid_until is None or datetime.fromisoformat(valid_until) > query_time: - valid_edges.append(edge) - - # Match against pattern - pattern = self._parse_pattern(query) - results = [] - - for edge in valid_edges: - if pattern["rel_type"] and edge["type"] != pattern["rel_type"]: - continue - - source = self.nodes.get(edge["source"], {}) - target = self.nodes.get(edge["target"], {}) - - results.append({ - "source": source, - "relationship": edge, - "target": target - }) - - return results - - def _time_range_key(self, start: datetime, end: Optional[datetime]) -> str: - """Create time range key for indexing.""" - start_str = start.isoformat() - end_str = end.isoformat() if end else "infinity" - return f"{start_str}::{end_str}" -``` - -## Memory Consolidation - -```python -class MemoryConsolidator: - def __init__(self, graph: PropertyGraph, vector_store: VectorStore): - self.graph = graph - self.vector_store = vector_store - self.consolidation_threshold = 1000 # memories before consolidation - - def should_consolidate(self) -> bool: - """Check if consolidation should trigger.""" - total_memories = len(self.graph.nodes) + len(self.graph.edges) - return total_memories > self.consolidation_threshold - - def consolidate(self): - """Run consolidation process.""" - # Step 1: Identify duplicate or merged facts - duplicates = self.find_duplicates() - - # Step 2: Merge related facts - for group in duplicates: - self.merge_fact_group(group) - - # Step 3: Update validity periods - self.update_validity_periods() - - # Step 4: Rebuild indexes - self.rebuild_indexes() - - def find_duplicates(self) -> List[List]: - """Find groups of potentially duplicate facts.""" - # Group by subject and predicate - groups = {} - - for edge in self.graph.edges: - key = (edge["source"], edge["type"]) - if key not in groups: - groups[key] = [] - groups[key].append(edge) - - # Return groups with multiple edges - return [edges for edges in groups.values() if len(edges) > 1] - - def merge_fact_group(self, edges: List[Dict]): - """Merge group of duplicate edges.""" - if len(edges) == 1: - return - - # Keep most recent/relevant - keeper = max(edges, key=lambda e: e.get("properties", {}).get("confidence", 0)) - - # Merge metadata - for edge in edges: - if edge["id"] != keeper["id"]: - self.merge_properties(keeper, edge) - self.graph.edges.remove(edge) - - def merge_properties(self, target: Dict, source: Dict): - """Merge properties from source into target.""" - for key, value in source.get("properties", {}).items(): - if key not in target["properties"]: - target["properties"][key] = value - elif isinstance(value, list): - target["properties"][key].extend(value) -``` - -## Memory-Context Integration - -```python -class MemoryContextIntegrator: - def __init__(self, memory_system, context_limit=100000): - self.memory_system = memory_system - self.context_limit = context_limit - - def build_context(self, task: str, current_context: str = "") -> str: - """Build context including relevant memories.""" - # Extract entities from task - entities = self._extract_entities(task) - - # Retrieve memories for each entity - memories = [] - for entity in entities: - entity_memories = self.memory_system.retrieve_entity(entity) - memories.extend(entity_memories) - - # Format memories for context - memory_section = self._format_memories(memories) - - # Combine with current context - combined = current_context + "\n\n" + memory_section - - # Check limit and truncate if needed - if self._token_count(combined) > self.context_limit: - combined = self._truncate_context(combined, self.context_limit) - - return combined - - def _extract_entities(self, task: str) -> List[str]: - """Extract entity mentions from task.""" - # In production, use NER or entity extraction - import re - pattern = r"\[([^\]]+)\]" # [[entity_name]] convention - return re.findall(pattern, task) - - def _format_memories(self, memories: List[Dict]) -> str: - """Format memories for context injection.""" - sections = ["## Relevant Memories"] - - for memory in memories: - formatted = f"- {memory.get('content', '')}" - if "source" in memory: - formatted += f" (Source: {memory['source']})" - if "timestamp" in memory: - formatted += f" [Time: {memory['timestamp']}]" - sections.append(formatted) - - return "\n".join(sections) - - def _token_count(self, text: str) -> int: - """Estimate token count.""" - return len(text) // 4 # Rough approximation - - def _truncate_context(self, context: str, limit: int) -> str: - """Truncate context to fit limit.""" - tokens = context.split() - truncated = [] - count = 0 - - for token in tokens: - if count + 1 > limit: - break - truncated.append(token) - count += 1 - - return " ".join(truncated) -``` - -## Framework Integration Examples - -### Mem0 Quick Start - -```python -from mem0 import Memory - -# Initialize with default config (uses local storage) -m = Memory() - -# Store memories with user scoping -m.add("Prefers Python 3.12 with type hints", user_id="dev-alice") -m.add("Working on microservices migration", user_id="dev-alice") - -# Search with natural language -results = m.search("What language does the user prefer?", user_id="dev-alice") - -# Batch operations -m.add([ - "Sprint goal: complete auth service", - "Blocked on database schema review" -], user_id="dev-alice") -``` - -### Graphiti (Zep's Open-Source Temporal KG Engine) - -```python -from graphiti_core import Graphiti -from graphiti_core.nodes import EpisodeType - -# Initialize with Neo4j backend -graphiti = Graphiti("bolt://localhost:7687", "neo4j", "password") - -# Add episodes (conversations, events) -await graphiti.add_episode( - name="user_conversation_42", - episode_body="Alice mentioned she moved to Berlin in January.", - source=EpisodeType.message, - source_description="Chat with Alice" -) - -# Search combines semantic, keyword, and graph traversal -results = await graphiti.search("Where does Alice live?") -``` - -### Cognee (Open-Source Knowledge Engine for AI Memory) - -```python -import cognee -from cognee.modules.search.types import SearchType - -# ECL pipeline: add → cognify → memify → search -await cognee.add("./docs/") -await cognee.add("any-data") -await cognee.cognify() -await cognee.memify() - -# Graph-aware retrieval (default: GRAPH_COMPLETION) -results = await cognee.search( - query_text="any query to search in memory", - query_type=SearchType.GRAPH_COMPLETION, -) - -# Raw chunks when agent reasons over text itself -chunks = await cognee.search( - query_text="any query to search in memory", - query_type=SearchType.CHUNKS, -) -``` - diff --git a/.agents/skills/memory-systems/scripts/memory_store.py b/.agents/skills/memory-systems/scripts/memory_store.py deleted file mode 100644 index 6bb63fae9..000000000 --- a/.agents/skills/memory-systems/scripts/memory_store.py +++ /dev/null @@ -1,616 +0,0 @@ -"""Memory System Implementation. - -Provides composable building blocks for agent memory: vector stores with -metadata indexing, property graphs for entity relationships, and temporal -knowledge graphs for facts that change over time. - -Use when: - - Building a memory persistence layer for an agent that must retain - knowledge across sessions. - - Prototyping memory architectures before committing to a production - framework (Mem0, Zep/Graphiti, Letta, Cognee). - - Combining semantic search with graph-based entity retrieval in a - single integrated system. - -Typical usage:: - - from memory_store import IntegratedMemorySystem - mem = IntegratedMemorySystem() - mem.start_session("session-001") - mem.store_fact("Alice prefers dark mode", entity="Alice") - results = mem.retrieve_memories("theme preference") -""" - -import hashlib -import json -from datetime import datetime -from typing import Any, Dict, List, Optional - -import numpy as np - -__all__ = [ - "VectorStore", - "PropertyGraph", - "TemporalKnowledgeGraph", - "IntegratedMemorySystem", -] - - -class VectorStore: - """Simple vector store with metadata indexing. - - Use when: the agent needs semantic similarity search over stored facts - with optional entity and temporal filtering. - """ - - def __init__(self, dimension: int = 768) -> None: - self.dimension: int = dimension - self.vectors: List[np.ndarray] = [] - self.metadata: List[Dict[str, Any]] = [] - self.entity_index: Dict[str, List[int]] = {} - self.time_index: Dict[str, List[int]] = {} - - def add(self, text: str, metadata: Optional[Dict[str, Any]] = None) -> int: - """Add document to store. - - Use when: persisting a new fact or observation that the agent should - be able to retrieve later via semantic search. - """ - metadata = metadata or {} - embedding: np.ndarray = self._embed(text) - index: int = len(self.vectors) - - self.vectors.append(embedding) - self.metadata.append(metadata) - - # Index by entity - if "entity" in metadata: - entity: str = metadata["entity"] - if entity not in self.entity_index: - self.entity_index[entity] = [] - self.entity_index[entity].append(index) - - # Index by time - if "valid_from" in metadata: - time_key: str = self._time_key(metadata["valid_from"]) - if time_key not in self.time_index: - self.time_index[time_key] = [] - self.time_index[time_key].append(index) - - return index - - def search( - self, - query: str, - limit: int = 5, - filters: Optional[Dict[str, Any]] = None, - ) -> List[Dict[str, Any]]: - """Search for similar documents. - - Use when: retrieving memories relevant to a query, optionally - narrowed by metadata filters (entity, session, time range). - """ - query_embedding: np.ndarray = self._embed(query) - - scores: List[tuple[int, float]] = [] - for i, vec in enumerate(self.vectors): - score: float = float( - np.dot(query_embedding, vec) - / (np.linalg.norm(query_embedding) * np.linalg.norm(vec) + 1e-8) - ) - - # Apply filters - if filters and not self._matches_filters(self.metadata[i], filters): - score = -1.0 - - scores.append((i, score)) - - scores.sort(key=lambda x: x[1], reverse=True) - - results: List[Dict[str, Any]] = [] - for idx, score in scores[:limit]: - if score > 0: - results.append( - { - "index": idx, - "score": score, - "text": self.metadata[idx].get("text", ""), - "metadata": self.metadata[idx], - } - ) - - return results - - def search_by_entity( - self, entity: str, query: str = "", limit: int = 5 - ) -> List[Dict[str, Any]]: - """Search within specific entity. - - Use when: the agent needs all memories associated with a known - entity, optionally ranked by relevance to a query. - """ - indices: List[int] = self.entity_index.get(entity, []) - - if not indices: - return [] - - if query: - query_embedding: np.ndarray = self._embed(query) - scored: List[tuple[int, float, Dict[str, Any]]] = [] - for i in indices: - vec: np.ndarray = self.vectors[i] - score: float = float( - np.dot(query_embedding, vec) - / (np.linalg.norm(query_embedding) * np.linalg.norm(vec) + 1e-8) - ) - scored.append((i, score, self.metadata[i])) - - scored.sort(key=lambda x: x[1], reverse=True) - return [ - {"index": i, "score": s, "metadata": m} - for i, s, m in scored[:limit] - ] - else: - return [ - {"index": i, "score": 1.0, "metadata": self.metadata[i]} - for i in indices[:limit] - ] - - def _embed(self, text: str) -> np.ndarray: - """Generate embedding for text. - - In production, replace with an actual embedding model. This - deterministic stub uses the text hash as a random seed so that - identical texts always produce identical vectors. Uses a local - RNG to avoid corrupting global numpy random state. - """ - rng = np.random.default_rng(hash(text) % (2**32)) - return rng.standard_normal(self.dimension) - - def _time_key(self, timestamp: Any) -> str: - """Create time key for indexing.""" - if isinstance(timestamp, datetime): - return timestamp.strftime("%Y-%m") - return str(timestamp) - - def _matches_filters(self, metadata: Dict[str, Any], filters: Dict[str, Any]) -> bool: - """Check if metadata matches filters.""" - for key, value in filters.items(): - if key not in metadata: - return False - if isinstance(value, list): - if metadata[key] not in value: - return False - elif metadata[key] != value: - return False - return True - - -class PropertyGraph: - """Simple property graph storage. - - Use when: the agent needs to maintain entity relationships and - traverse connections between nodes (e.g., "find all projects - associated with this user"). - """ - - def __init__(self) -> None: - self.nodes: Dict[str, Dict[str, Any]] = {} - self.edges: Dict[str, Dict[str, Any]] = {} - self.entity_registry: Dict[str, str] = {} # name -> node_id - self.node_index: Dict[str, List[str]] = {} # label -> node_ids - self.edge_index: Dict[str, List[str]] = {} # type -> edge_ids - - def get_or_create_node( - self, name: str, label: str = "Entity", properties: Optional[Dict[str, Any]] = None - ) -> str: - """Get existing node by name, or create a new one. - - Use when: storing an entity that may already exist. The entity - registry ensures identity is maintained across interactions - ("John Doe" always maps to the same node). - """ - if name in self.entity_registry: - node_id: str = self.entity_registry[name] - if properties: - self.nodes[node_id]["properties"].update(properties) - return node_id - node_id = self.create_node(label, {**(properties or {}), "name": name}) - self.entity_registry[name] = node_id - return node_id - - def create_node(self, label: str, properties: Optional[Dict[str, Any]] = None) -> str: - """Create node with label and properties. - - Use when: adding a new entity to the graph that does not need - identity deduplication (prefer get_or_create_node otherwise). - """ - node_id: str = hashlib.md5(f"{label}{datetime.now().isoformat()}".encode()).hexdigest()[:16] - - self.nodes[node_id] = { - "id": node_id, - "label": label, - "properties": properties or {}, - "created_at": datetime.now().isoformat(), - } - - if label not in self.node_index: - self.node_index[label] = [] - self.node_index[label].append(node_id) - - return node_id - - def create_relationship( - self, - source_id: str, - rel_type: str, - target_id: str, - properties: Optional[Dict[str, Any]] = None, - ) -> str: - """Create directed relationship between nodes. - - Use when: recording a connection between two entities (e.g., - WORKS_AT, LIVES_IN, DEPENDS_ON). - """ - if source_id not in self.nodes: - raise ValueError(f"Unknown source node: {source_id}") - if target_id not in self.nodes: - raise ValueError(f"Unknown target node: {target_id}") - - edge_id: str = hashlib.md5( - f"{source_id}{rel_type}{target_id}{datetime.now().isoformat()}".encode() - ).hexdigest()[:16] - - self.edges[edge_id] = { - "id": edge_id, - "source": source_id, - "target": target_id, - "type": rel_type, - "properties": properties or {}, - "created_at": datetime.now().isoformat(), - } - - if rel_type not in self.edge_index: - self.edge_index[rel_type] = [] - self.edge_index[rel_type].append(edge_id) - - return edge_id - - def query(self, pattern: Dict[str, Any]) -> List[Dict[str, Any]]: - """Query graph with simple pattern matching. - - Use when: finding relationships that match a structural pattern - (e.g., all WORKS_AT edges from Person nodes). - """ - results: List[Dict[str, Any]] = [] - - # Match by edge type - if "type" in pattern: - edge_ids: List[str] = self.edge_index.get(pattern["type"], []) - for eid in edge_ids: - edge: Dict[str, Any] = self.edges[eid] - source: Dict[str, Any] = self.nodes.get(edge["source"], {}) - target: Dict[str, Any] = self.nodes.get(edge["target"], {}) - - # Match source label - if "source_label" in pattern: - if source.get("label") != pattern["source_label"]: - continue - - # Match target label - if "target_label" in pattern: - if target.get("label") != pattern["target_label"]: - continue - - results.append({"source": source, "edge": edge, "target": target}) - - return results - - def get_node(self, node_id: str) -> Optional[Dict[str, Any]]: - """Get node by ID.""" - return self.nodes.get(node_id) - - def get_relationships( - self, node_id: str, direction: str = "both" - ) -> List[Dict[str, Any]]: - """Get relationships for a node. - - Use when: retrieving all connections for a given entity to build - a complete entity context. - """ - relationships: List[Dict[str, Any]] = [] - - for edge in self.edges.values(): - if direction in ["outgoing", "both"] and edge["source"] == node_id: - relationships.append( - { - "edge": edge, - "target": self.nodes.get(edge["target"]), - "direction": "outgoing", - } - ) - if direction in ["incoming", "both"] and edge["target"] == node_id: - relationships.append( - { - "edge": edge, - "source": self.nodes.get(edge["source"]), - "direction": "incoming", - } - ) - - return relationships - - -class TemporalKnowledgeGraph(PropertyGraph): - """Property graph with temporal validity for facts. - - Use when: the agent must track facts that change over time and - answer time-scoped queries (e.g., "where did the user live in - March 2024?"). - """ - - def create_temporal_relationship( - self, - source_id: str, - rel_type: str, - target_id: str, - valid_from: datetime, - valid_until: Optional[datetime] = None, - properties: Optional[Dict[str, Any]] = None, - ) -> str: - """Create relationship with temporal validity. - - Use when: recording a fact that has a known start time and - may expire (e.g., employment, address, subscription status). - """ - edge_id: str = super().create_relationship( - source_id, rel_type, target_id, properties - ) - - # Add temporal properties - self.edges[edge_id]["valid_from"] = valid_from.isoformat() - self.edges[edge_id]["valid_until"] = ( - valid_until.isoformat() if valid_until else None - ) - - return edge_id - - def query_at_time( - self, query: Dict[str, Any], query_time: datetime - ) -> List[Dict[str, Any]]: - """Query graph state at specific time. - - Use when: answering point-in-time questions about entities - (e.g., "what was true on date X?"). - """ - results: List[Dict[str, Any]] = [] - - # Get base query results - base_results: List[Dict[str, Any]] = self.query(query) - - for result in base_results: - edge: Dict[str, Any] = result["edge"] - valid_from: datetime = datetime.fromisoformat( - edge.get("valid_from", "1970-01-01") - ) - valid_until: Optional[str] = edge.get("valid_until") - - # Check temporal validity - if valid_from <= query_time: - if valid_until is None or datetime.fromisoformat(valid_until) > query_time: - results.append( - { - **result, - "valid_from": valid_from, - "valid_until": valid_until, - } - ) - - return results - - def query_time_range( - self, - query: Dict[str, Any], - start_time: datetime, - end_time: datetime, - ) -> List[Dict[str, Any]]: - """Query facts valid during time range. - - Use when: retrieving all facts that overlap with a given time - window (e.g., "what changed between January and June?"). - """ - results: List[Dict[str, Any]] = [] - - base_results: List[Dict[str, Any]] = self.query(query) - - for result in base_results: - edge: Dict[str, Any] = result["edge"] - valid_from: datetime = datetime.fromisoformat( - edge.get("valid_from", "1970-01-01") - ) - valid_until: Optional[str] = edge.get("valid_until") - - # Check if overlaps with query range - until_dt: datetime = ( - datetime.fromisoformat(valid_until) if valid_until else datetime.max - ) - - if until_dt >= start_time and valid_from <= end_time: - results.append( - { - **result, - "valid_from": valid_from, - "valid_until": valid_until, - } - ) - - return results - - -# --------------------------------------------------------------------------- -# Memory System Integration -# --------------------------------------------------------------------------- - - -class IntegratedMemorySystem: - """Integrated memory system combining vector store and graph. - - Use when: the agent needs both semantic search over facts and - graph-based entity relationship traversal in a single unified - interface. This class composes VectorStore and TemporalKnowledgeGraph, - enriching vector search results with graph context. - """ - - def __init__(self) -> None: - self.vector_store: VectorStore = VectorStore() - self.graph: TemporalKnowledgeGraph = TemporalKnowledgeGraph() - self.session_id: str = "" - - def start_session(self, session_id: str) -> None: - """Start a new memory session. - - Use when: beginning a new conversation or task that should - scope its memories to a distinct session identifier. - """ - self.session_id = session_id - - def store_fact( - self, - fact: str, - entity: str, - timestamp: Optional[datetime] = None, - relationships: Optional[List[Dict[str, Any]]] = None, - ) -> None: - """Store a fact with entity and relationships. - - Use when: the agent observes a new piece of information that - should be persisted for future retrieval. Stores in both the - vector store (for semantic search) and the graph (for entity - traversal). - """ - # Store in vector store - self.vector_store.add( - fact, - { - "text": fact, - "entity": entity, - "valid_from": (timestamp or datetime.now()).isoformat(), - "session_id": self.session_id, - }, - ) - - # Get or create entity node (uses registry for identity) - entity_node_id: str = self.graph.get_or_create_node(entity) - - # Create relationships - if relationships: - for rel in relationships: - target_node_id: str = self.graph.get_or_create_node(rel["target"]) - self.graph.create_relationship( - entity_node_id, - rel["type"], - target_node_id, - properties=rel.get("properties", {}), - ) - - def retrieve_memories( - self, - query: str, - entity_filter: Optional[str] = None, - time_filter: Optional[Dict[str, Any]] = None, - limit: int = 5, - ) -> List[Dict[str, Any]]: - """Retrieve memories matching query. - - Use when: the agent needs to recall previously stored facts, - optionally filtered by entity or time. Results are enriched - with graph relationships for each matched entity. - """ - # Vector search - filters: Dict[str, Any] = {"session_id": self.session_id} - if entity_filter: - filters["entity"] = entity_filter - - results: List[Dict[str, Any]] = self.vector_store.search( - query, limit=limit, filters=filters - ) - - # Enrich with graph relationships - for result in results: - entity: Optional[str] = result["metadata"].get("entity") - if entity: - node_id: Optional[str] = self.graph.entity_registry.get(entity) - if node_id: - result["relationships"] = self.graph.get_relationships(node_id) - - return results - - def retrieve_entity_context(self, entity: str) -> Dict[str, Any]: - """Retrieve complete context for an entity. - - Use when: the agent needs a full picture of a single entity - including its properties, all relationships, and associated - vector memories. - """ - node_id: Optional[str] = self.graph.entity_registry.get(entity) - - # Get entity node - entity_node: Optional[Dict[str, Any]] = ( - self.graph.get_node(node_id) if node_id else None - ) - - # Get relationships - relationships: List[Dict[str, Any]] = ( - self.graph.get_relationships(node_id) if node_id else [] - ) - - # Get vector memories - memories: List[Dict[str, Any]] = self.vector_store.search_by_entity( - entity, limit=10 - ) - - return { - "entity": entity_node, - "relationships": relationships, - "memories": memories, - } - - def consolidate(self) -> None: - """Consolidate memories and remove outdated information. - - Use when: memory count exceeds a threshold, retrieval quality - degrades, or on a scheduled interval. In production, implement: - - Merge related facts into summaries - - Update validity periods on stale entries - - Archive obsolete facts (invalidate, do not discard) - """ - pass - - -if __name__ == "__main__": - # Quick smoke test demonstrating the integrated memory system. - mem = IntegratedMemorySystem() - mem.start_session("demo-session") - - # Store facts with entity relationships - mem.store_fact( - "Alice prefers dark mode", - entity="Alice", - relationships=[{"target": "dark mode", "type": "PREFERS"}], - ) - mem.store_fact( - "Alice works at Acme Corp", - entity="Alice", - relationships=[{"target": "Acme Corp", "type": "WORKS_AT"}], - ) - - # Semantic retrieval - results = mem.retrieve_memories("theme preference") - print(f"Search results: {len(results)} memories found") - for r in results: - print(f" score={r['score']:.3f} text={r['text']}") - - # Entity context - context = mem.retrieve_entity_context("Alice") - print(f"\nAlice context: {len(context['relationships'])} relationships, " - f"{len(context['memories'])} memories") diff --git a/.agents/skills/multi-agent-patterns/references/frameworks.md b/.agents/skills/multi-agent-patterns/references/frameworks.md deleted file mode 100644 index 6b49aa5f2..000000000 --- a/.agents/skills/multi-agent-patterns/references/frameworks.md +++ /dev/null @@ -1,433 +0,0 @@ -# Multi-Agent Patterns: Technical Reference - -This document provides implementation details for multi-agent architectures across different frameworks. - -## Supervisor Pattern - -### LangGraph Supervisor Implementation - -Implement a supervisor that routes to worker nodes: - -```python -from typing import TypedDict, Union -from langgraph.graph import StateGraph, END - -class AgentState(TypedDict): - task: str - current_agent: str - task_output: dict - messages: list - -def supervisor_node(state: AgentState) -> AgentState: - """ - Supervisor decides which worker to invoke next. - - Returns routing decision and updates state. - """ - task = state["task"] - messages = state.get("messages", []) - - # Determine next agent based on task and history - if "research" in task.lower(): - next_agent = "researcher" - elif "write" in task.lower() or "create" in task.lower(): - next_agent = "writer" - elif "review" in task.lower() or "analyze" in task.lower(): - next_agent = "reviewer" - else: - next_agent = "coordinator" - - return { - "task": task, - "current_agent": next_agent, - "task_output": {}, - "messages": messages + [{"supervisor": f"Routing to {next_agent}"}] - } - -def researcher_node(state: AgentState) -> AgentState: - """Research worker that gathers information.""" - # Perform research task - output = perform_research(state["task"]) - - return { - "task": state["task"], - "current_agent": "researcher", - "task_output": output, - "messages": state["messages"] + [{"researcher": "Research complete"}] - } - -def writer_node(state: AgentState) -> AgentState: - """Writer worker that creates content based on research.""" - output = create_content(state["task"], state["task_output"]) - - return { - "task": state["task"], - "current_agent": "writer", - "task_output": output, - "messages": state["messages"] + [{"writer": "Content created"}] - } - -def build_supervisor_graph(): - """Build the supervisor workflow graph.""" - workflow = StateGraph(AgentState) - - # Add nodes - workflow.add_node("supervisor", supervisor_node) - workflow.add_node("researcher", researcher_node) - workflow.add_node("writer", writer_node) - - # Add edges - workflow.add_edge("supervisor", "researcher") - workflow.add_edge("researcher", "supervisor") - workflow.add_edge("supervisor", "writer") - workflow.add_edge("writer", "supervisor") - - # Set entry point - workflow.set_entry_point("supervisor") - - return workflow.compile() -``` - -### AutoGen Supervisor - -Implement supervisor using GroupChat pattern: - -```python -from autogen import AssistantAgent, UserProxyAgent, GroupChat - -# Define specialized agents -researcher = AssistantAgent( - name="researcher", - system_message="""You are a research specialist. - Your goal is to gather accurate, comprehensive information - on topics assigned by the supervisor. Always cite sources - and note confidence levels.""", - llm_config=llm_config -) - -writer = AssistantAgent( - name="writer", - system_message="""You are a content creation specialist. - Your goal is to create well-structured content based on - research provided by the supervisor. Follow style guidelines - and ensure factual accuracy.""", - llm_config=llm_config -) - -# Define supervisor -supervisor = AssistantAgent( - name="supervisor", - system_message="""You are the project supervisor. - Your goal is to coordinate researchers and writers to - complete tasks efficiently. - - Process: - 1. Break down the task into research and writing phases - 2. Route to appropriate specialists - 3. Synthesize results into final output - 4. Ensure quality before completing""", - llm_config=llm_config -) - -# Configure group chat -group_chat = GroupChat( - agents=[supervisor, researcher, writer], - messages=[], - max_round=20 -) - -manager = GroupChatManager( - groupchat=group_chat, - llm_config=llm_config -) -``` - -## Swarm Pattern Implementation - -### LangGraph Swarms - -Implement peer-to-peer handoffs: - -```python -def create_agent(name, system_prompt, tools): - """Create an agent node for the swarm.""" - - def agent_node(state): - # Process current state with agent - response = invoke_agent(name, system_prompt, state["input"], tools) - - # Check for handoff - if "handoff" in response: - return {"next_agent": response["handoff"], "output": response["output"]} - else: - return {"next_agent": END, "output": response["output"]} - - return agent_node - -def build_swarm(): - """Build a peer-to-peer agent swarm.""" - workflow = StateGraph(State) - - # Create agents - triage = create_agent("triage", TRIAGE_PROMPT, [search, read]) - research = create_agent("research", RESEARCH_PROMPT, [search, browse, read]) - analysis = create_agent("analysis", ANALYSIS_PROMPT, [calculate, compare]) - writing = create_agent("writing", WRITING_PROMPT, [write, edit]) - - # Add to graph - workflow.add_node("triage", triage) - workflow.add_node("research", research) - workflow.add_node("analysis", analysis) - workflow.add_node("writing", writing) - - # Define handoff edges - workflow.add_edge("triage", "research") - workflow.add_edge("triage", "analysis") - workflow.add_edge("research", "writing") - workflow.add_edge("analysis", "writing") - - workflow.set_entry_point("triage") - - return workflow.compile() -``` - -## Hierarchical Pattern Implementation - -### CrewAI-Style Hierarchy - -```python -class ManagerAgent: - def __init__(self, name, system_prompt, llm): - self.name = name - self.system_prompt = system_prompt - self.llm = llm - self.workers = [] - - def add_worker(self, worker): - """Add a worker agent to the team.""" - self.workers.append(worker) - - def delegate(self, task): - """ - Analyze task and delegate to appropriate worker. - - Returns work assignment and expected output format. - """ - # Analyze task requirements - requirements = analyze_task_requirements(task) - - # Select best worker - best_worker = select_worker(self.workers, requirements) - - # Create assignment - assignment = { - "worker": best_worker.name, - "task": task, - "context": self.get_relevant_context(task), - "output_format": requirements.output_format, - "deadline": requirements.deadline - } - - return assignment - - def review_output(self, worker_output, requirements): - """ - Review worker output against requirements. - - Returns approval or revision request. - """ - quality_score = assess_quality(worker_output, requirements) - - if quality_score >= requirements.threshold: - return {"status": "approved", "output": worker_output} - else: - return { - "status": "revision_requested", - "feedback": generate_feedback(worker_output, requirements), - "revise_worker": requirements.revise_worker - } -``` - -## Context Isolation Patterns - -### Full Context Delegation - -```python -def delegate_with_full_context(planner_state, subagent): - """ - Pass entire planner context to subagent. - - Use for complex tasks requiring complete understanding. - """ - return { - "context": planner_state, - "subagent": subagent, - "isolation_mode": "full" - } -``` - -### Instruction Passing - -```python -def delegate_with_instructions(task_spec, subagent): - """ - Pass only instructions to subagent. - - Use for simple, well-defined subtasks. - """ - return { - "instructions": { - "objective": task_spec.objective, - "constraints": task_spec.constraints, - "inputs": task_spec.inputs, - "outputs": task_spec.output_schema - }, - "subagent": subagent, - "isolation_mode": "minimal" - } -``` - -### File System Coordination - -```python -class FileSystemCoordination: - def __init__(self, workspace_path): - self.workspace = workspace_path - - def write_shared_state(self, key, value): - """Write state accessible to all agents.""" - path = f"{self.workspace}/{key}.json" - with open(path, 'w') as f: - json.dump(value, f) - return path - - def read_shared_state(self, key): - """Read state written by any agent.""" - path = f"{self.workspace}/{key}.json" - with open(path, 'r') as f: - return json.load(f) - - def acquire_lock(self, resource, agent_id): - """Prevent concurrent access to shared resources.""" - lock_path = f"{self.workspace}/locks/{resource}.lock" - if os.path.exists(lock_path): - return False - with open(lock_path, 'w') as f: - f.write(agent_id) - return True -``` - -## Consensus Mechanisms - -### Weighted Voting - -```python -def weighted_consensus(agent_outputs, weights): - """ - Calculate weighted consensus from agent outputs. - - Weight = verbalized_confidence * domain_expertise - """ - weighted_sum = sum( - output.vote * weights[output.agent_id] - for output in agent_outputs - ) - total_weight = sum(weights[output.agent_id] for output in agent_outputs) - - return weighted_sum / total_weight -``` - -### Debate Protocol - -```python -class DebateProtocol: - def __init__(self, agents, max_rounds=5): - self.agents = agents - self.max_rounds = max_rounds - self.history = [] - - def run_debate(self, topic): - """Execute structured debate on topic.""" - # Initial statements - statements = {agent.name: agent.initial_statement(topic) - for agent in self.agents} - - for round_num in range(self.max_rounds): - # Generate critiques - critiques = {} - for agent in self.agents: - critiques[agent.name] = agent.critique( - topic, - statements, - exclude=[agent.name] - ) - - # Update statements with critique integration - for agent in self.agents: - statements[agent.name] = agent.integrate_critique( - statements[agent.name], - critiques - ) - - # Check for convergence - if self.check_convergence(statements): - break - - # Final evaluation - return self.evaluate_final(statements) -``` - -## Failure Recovery - -### Circuit Breaker - -```python -class AgentCircuitBreaker: - def __init__(self, failure_threshold=3, timeout_seconds=60): - self.failure_count = {} - self.failure_threshold = failure_threshold - self.timeout_seconds = timeout_seconds - - def call(self, agent, task): - """Execute agent task with circuit breaker protection.""" - if self.is_open(agent.name): - raise CircuitBreakerOpen(f"Agent {agent.name} temporarily unavailable") - - try: - result = agent.execute(task) - self.record_success(agent.name) - return result - except Exception as e: - self.record_failure(agent.name) - if self.failure_count[agent.name] >= self.failure_threshold: - self.open_circuit(agent.name) - raise -``` - -### Checkpoint and Resume - -```python -class CheckpointManager: - def __init__(self, checkpoint_dir): - self.checkpoint_dir = checkpoint_dir - os.makedirs(checkpoint_dir, exist_ok=True) - - def save_checkpoint(self, workflow_id, step, state): - """Save workflow state for potential resume.""" - checkpoint = { - "workflow_id": workflow_id, - "step": step, - "state": state, - "timestamp": time.time() - } - path = f"{self.checkpoint_dir}/{workflow_id}.json" - with open(path, 'w') as f: - json.dump(checkpoint, f) - - def load_checkpoint(self, workflow_id): - """Load last saved checkpoint for workflow.""" - path = f"{self.checkpoint_dir}/{workflow_id}.json" - with open(path, 'r') as f: - return json.load(f) -``` - diff --git a/.agents/skills/multi-agent-patterns/scripts/coordination.py b/.agents/skills/multi-agent-patterns/scripts/coordination.py deleted file mode 100644 index df0d3bc8e..000000000 --- a/.agents/skills/multi-agent-patterns/scripts/coordination.py +++ /dev/null @@ -1,613 +0,0 @@ -""" -Multi-Agent Coordination Utilities - -Provides reusable building blocks for multi-agent coordination patterns: -supervisor/orchestrator, peer-to-peer handoffs, consensus mechanisms, -and failure handling with circuit breakers. - -Use when: building multi-agent systems that need structured communication, -task delegation, consensus voting, or fault-tolerant agent coordination. - -Designed for composability — import individual classes or use the -``if __name__ == "__main__"`` demo to see all patterns in action. -""" - -from typing import Dict, List, Any, Optional -from dataclasses import dataclass, field -from enum import Enum -import time -import uuid - -__all__ = [ - "MessageType", - "AgentMessage", - "AgentCommunication", - "SupervisorAgent", - "HandoffProtocol", - "ConsensusManager", - "AgentFailureHandler", -] - - -class MessageType(Enum): - """Types of messages exchanged between agents.""" - - REQUEST = "request" - RESPONSE = "response" - HANDOVER = "handover" - FEEDBACK = "feedback" - ALERT = "alert" - - -@dataclass -class AgentMessage: - """Message exchanged between agents. - - Use when: agents need a structured envelope for inter-agent communication - that carries sender/receiver identity, type, priority, and payload. - """ - - sender: str - receiver: str - message_type: MessageType - content: Dict[str, Any] - timestamp: float = field(default_factory=time.time) - message_id: str = field(default_factory=lambda: str(uuid.uuid4())) - requires_response: bool = False - priority: int = 0 # 0 = normal, higher = more urgent - - -class AgentCommunication: - """Communication channel for multi-agent systems. - - Use when: multiple agents need an in-process message bus for sending, - receiving, and broadcasting messages with history tracking. - """ - - def __init__(self) -> None: - self.inbox: Dict[str, List[AgentMessage]] = {} - self.outbox: List[AgentMessage] = [] - self.message_history: List[AgentMessage] = [] - - def send(self, message: AgentMessage) -> None: - """Send a message to an agent.""" - if message.receiver not in self.inbox: - self.inbox[message.receiver] = [] - self.inbox[message.receiver].append(message) - self.outbox.append(message) - self.message_history.append(message) - - def receive(self, agent_id: str) -> List[AgentMessage]: - """Receive all messages for an agent, clearing its inbox.""" - messages = self.inbox.get(agent_id, []) - self.inbox[agent_id] = [] - return messages - - def broadcast( - self, - sender: str, - message_type: MessageType, - content: Dict[str, Any], - receivers: List[str], - ) -> None: - """Broadcast a message to multiple agents.""" - for receiver in receivers: - self.send( - AgentMessage( - sender=sender, - receiver=receiver, - message_type=message_type, - content=content, - ) - ) - - -# --------------------------------------------------------------------------- -# Supervisor Pattern -# --------------------------------------------------------------------------- - - -class SupervisorAgent: - """Central supervisor agent that coordinates worker agents. - - Use when: tasks have clear decomposition and a single coordinator should - delegate subtasks, track worker status, and aggregate results. - """ - - def __init__(self, name: str, communication: AgentCommunication) -> None: - self.name = name - self.communication = communication - self.workers: Dict[str, Dict[str, Any]] = {} - self.task_queue: List[Dict[str, Any]] = [] - self.completed_tasks: List[Dict[str, Any]] = [] - self.current_state: Dict[str, Any] = {} - - def register_worker(self, worker_id: str, capabilities: List[str]) -> None: - """Register a worker agent with the supervisor.""" - self.workers[worker_id] = { - "capabilities": capabilities, - "status": "available", - "current_task": None, - "metrics": {"tasks_completed": 0, "avg_response_time": 0.0}, - } - - def decompose_task(self, task: Dict[str, Any]) -> List[Dict[str, Any]]: - """Decompose a task into subtasks. - - Use when: a high-level task needs to be broken into assignable units. - In production, replace the rule-based logic with LLM-driven planning. - """ - subtasks: List[Dict[str, Any]] = [] - task_type = task.get("type", "general") - - if task_type == "research": - subtasks = [ - {"type": "search", "description": "Gather information"}, - {"type": "analyze", "description": "Analyze findings"}, - {"type": "synthesize", "description": "Synthesize results"}, - ] - elif task_type == "create": - subtasks = [ - {"type": "plan", "description": "Create plan"}, - {"type": "draft", "description": "Draft content"}, - {"type": "review", "description": "Review and refine"}, - ] - else: - subtasks = [ - { - "type": "execute", - "description": task.get("description", "Execute task"), - } - ] - - for subtask in subtasks: - subtask["parent_task"] = task.get("id") - subtask["priority"] = task.get("priority", 0) - - return subtasks - - def assign_task(self, subtask: Dict[str, Any], worker_id: str) -> None: - """Assign a subtask to a worker agent.""" - if worker_id not in self.workers: - raise ValueError(f"Unknown worker: {worker_id}") - - self.workers[worker_id]["status"] = "busy" - self.workers[worker_id]["current_task"] = subtask.get("id") - - self._send( - AgentMessage( - sender=self.name, - receiver=worker_id, - message_type=MessageType.REQUEST, - content={"action": "execute_task", "task": subtask}, - requires_response=True, - priority=subtask.get("priority", 0), - ) - ) - - def select_worker(self, subtask: Dict[str, Any]) -> str: - """Select the best available worker for a subtask. - - Use when: the supervisor needs capability-aware routing with - load-balancing (fewest completed tasks chosen first). - """ - required_capability = subtask.get("type", "general") - - candidates = [ - wid - for wid, info in self.workers.items() - if info["status"] == "available" - and required_capability in info["capabilities"] - ] - - if not candidates: - candidates = [ - wid - for wid, info in self.workers.items() - if info["status"] == "available" - ] - - if not candidates: - raise ValueError("No available workers") - - return min( - candidates, - key=lambda w: self.workers[w]["metrics"]["tasks_completed"], - ) - - def aggregate_results( - self, subtask_results: List[Dict[str, Any]] - ) -> Dict[str, Any]: - """Aggregate results from completed subtasks.""" - summaries = [ - r.get("summary", "") - for r in subtask_results - if r.get("success") - ] - successful = sum( - 1 for r in subtask_results if r.get("success", False) - ) - quality = successful / len(subtask_results) if subtask_results else 0.0 - - return { - "results": subtask_results, - "summary": " | ".join(summaries), - "quality_score": quality, - } - - def run_workflow(self, task: Dict[str, Any]) -> Dict[str, Any]: - """Execute a complete workflow with supervision. - - Use when: running an end-to-end supervised pipeline that decomposes - a task, assigns subtasks, collects results, and aggregates them. - - Note: This is a synchronous simulation. Workers do not execute - asynchronously — each subtask is simulated inline. In production, - replace ``_simulate_worker_response`` with actual async worker - execution and message passing. - """ - subtasks = self.decompose_task(task) - - results: List[Dict[str, Any]] = [] - for subtask in subtasks: - worker = self.select_worker(subtask) - self.assign_task(subtask, worker) - - # Simulate worker executing and responding - response = self._simulate_worker_response(worker, subtask) - self.communication.send( - AgentMessage( - sender=worker, - receiver=self.name, - message_type=MessageType.RESPONSE, - content=response, - ) - ) - self.workers[worker]["status"] = "available" - self.workers[worker]["metrics"]["tasks_completed"] += 1 - - messages = self.communication.receive(self.name) - for msg in messages: - if msg.message_type == MessageType.RESPONSE: - results.append(msg.content) - - final_result = self.aggregate_results(results) - - return { - "task": task, - "subtask_results": results, - "final_result": final_result, - "success": final_result["quality_score"] >= 0.8, - } - - def _simulate_worker_response( - self, worker_id: str, subtask: Dict[str, Any] - ) -> Dict[str, Any]: - """Simulate a worker completing a subtask. - - In production, replace with actual agent execution that sends - the subtask to a worker process and awaits a real response. - """ - return { - "success": True, - "summary": f"{worker_id} completed: {subtask.get('description', subtask.get('type', 'task'))}", - "worker": worker_id, - "subtask_type": subtask.get("type"), - } - - def _send(self, message: AgentMessage) -> None: - """Send message through the communication channel.""" - self.communication.send(message) - - -# --------------------------------------------------------------------------- -# Handoff Protocol -# --------------------------------------------------------------------------- - - -class HandoffProtocol: - """Protocol for agent-to-agent handoffs. - - Use when: implementing peer-to-peer or swarm patterns where agents - transfer control and task state to one another. - """ - - def __init__(self, communication: AgentCommunication) -> None: - self.communication = communication - - def create_handoff( - self, - from_agent: str, - to_agent: str, - context: Dict[str, Any], - reason: str, - ) -> AgentMessage: - """Create a handoff message with transferred context.""" - return AgentMessage( - sender=from_agent, - receiver=to_agent, - message_type=MessageType.HANDOVER, - content={ - "handoff_reason": reason, - "transferred_context": context, - "handoff_timestamp": time.time(), - }, - priority=1, - ) - - def accept_handoff(self, agent_id: str) -> Optional[AgentMessage]: - """Accept the first pending handoff for an agent, if any.""" - messages = self.communication.receive(agent_id) - - for msg in messages: - if msg.message_type == MessageType.HANDOVER: - return msg - - return None - - def transfer_with_state( - self, - from_agent: str, - to_agent: str, - state: Dict[str, Any], - task: Dict[str, Any], - ) -> bool: - """Transfer task state from one agent to another. - - Use when: a handoff must carry full task state and progress so the - receiving agent can resume without re-deriving context. - - Returns True if the receiving agent acknowledged the handoff. - """ - handoff = self.create_handoff( - from_agent=from_agent, - to_agent=to_agent, - context={ - "task_state": state, - "task_details": task, - "progress": state.get("progress", 0), - }, - reason="task_transfer", - ) - - self.communication.send(handoff) - - # In production, replace sleep with async await + timeout - time.sleep(0.1) - ack = self.communication.receive(from_agent) - - return any( - m.message_type == MessageType.RESPONSE - and m.content.get("status") == "handoff_received" - for m in ack - ) - - -# --------------------------------------------------------------------------- -# Consensus Mechanism -# --------------------------------------------------------------------------- - - -class ConsensusManager: - """Manager for multi-agent consensus building. - - Use when: multiple agents must vote on a decision and the system needs - weighted consensus that accounts for confidence and expertise rather - than naive majority voting. - """ - - def __init__(self) -> None: - self.votes: Dict[str, List[Dict[str, Any]]] = {} - self.debates: Dict[str, List[Dict[str, Any]]] = {} - - def initiate_vote( - self, topic_id: str, agents: List[str], options: List[str] - ) -> None: - """Initiate a voting round on a topic.""" - self.votes[topic_id] = [ - { - "agent": agent, - "topic": topic_id, - "options": options, - "status": "pending", - } - for agent in agents - ] - - def submit_vote( - self, - topic_id: str, - agent_id: str, - selection: str, - confidence: float, - ) -> None: - """Submit a vote for a topic with a confidence weight.""" - if topic_id not in self.votes: - raise ValueError(f"Unknown topic: {topic_id}") - - for vote in self.votes[topic_id]: - if vote["agent"] == agent_id: - vote["status"] = "cast" - vote["selection"] = selection - vote["confidence"] = confidence - break - - def calculate_weighted_consensus(self, topic_id: str) -> Dict[str, Any]: - """Calculate weighted consensus from cast votes. - - Use when: votes are in and the system needs to determine a winner - weighted by each agent's confidence rather than simple majority. - Weight = confidence * expertise_factor. - """ - if topic_id not in self.votes: - raise ValueError(f"Unknown topic: {topic_id}") - - votes = [ - v for v in self.votes[topic_id] if v.get("status") == "cast" - ] - - if not votes: - return {"status": "no_votes", "result": None} - - # Group by selection - selections: Dict[str, List[Dict[str, Any]]] = {} - for vote in votes: - selection = vote["selection"] - if selection not in selections: - selections[selection] = [] - selections[selection].append(vote) - - # Calculate weighted score for each selection - results: Dict[str, Dict[str, Any]] = {} - for selection, selection_votes in selections.items(): - weighted_sum = sum(v["confidence"] for v in selection_votes) - avg_confidence = ( - weighted_sum / len(selection_votes) if selection_votes else 0.0 - ) - results[selection] = { - "weighted_score": weighted_sum, - "avg_confidence": avg_confidence, - "vote_count": len(selection_votes), - } - - winner = max(results.keys(), key=lambda s: results[s]["weighted_score"]) - - return { - "status": "complete", - "result": winner, - "details": results, - "consensus_strength": ( - results[winner]["weighted_score"] / len(votes) if votes else 0.0 - ), - } - - -# --------------------------------------------------------------------------- -# Failure Handling -# --------------------------------------------------------------------------- - - -class AgentFailureHandler: - """Handler for agent failures in multi-agent systems. - - Use when: agents may fail and the system needs retry logic with - exponential backoff, circuit breakers, and automatic rerouting to - backup agents. - """ - - def __init__( - self, - communication: AgentCommunication, - max_retries: int = 3, - ) -> None: - self.communication = communication - self.max_retries = max_retries - self.failure_counts: Dict[str, int] = {} - self.circuit_breakers: Dict[str, float] = {} # agent -> unlock time - - def handle_failure( - self, agent_id: str, task_id: str, error: str - ) -> Dict[str, Any]: - """Handle a failure from an agent. - - Use when: an agent reports an error and the system must decide - whether to retry (with backoff) or reroute to a backup agent. - """ - self.failure_counts[agent_id] = ( - self.failure_counts.get(agent_id, 0) + 1 - ) - - if self.failure_counts[agent_id] >= self.max_retries: - self._activate_circuit_breaker(agent_id) - return { - "action": "reroute", - "reason": "circuit_breaker_activated", - "alternative": self._find_alternative_agent(agent_id), - } - - return { - "action": "retry", - "reason": error, - "retry_count": self.failure_counts[agent_id], - "delay": min(2 ** self.failure_counts[agent_id], 60), - } - - def _activate_circuit_breaker(self, agent_id: str) -> None: - """Temporarily disable an agent (1-minute cooldown).""" - self.circuit_breakers[agent_id] = time.time() + 60 - - def _find_alternative_agent(self, failed_agent: str) -> str: - """Find an alternative agent to handle the task. - - In production, check agent capabilities and availability. - """ - return "default_backup_agent" - - def is_available(self, agent_id: str) -> bool: - """Check if an agent is available (circuit breaker not active).""" - if agent_id in self.circuit_breakers: - if time.time() < self.circuit_breakers[agent_id]: - return False - del self.circuit_breakers[agent_id] - self.failure_counts[agent_id] = 0 - return True - - def record_success(self, agent_id: str) -> None: - """Record a successful task completion, resetting failure count.""" - self.failure_counts[agent_id] = 0 - - -# --------------------------------------------------------------------------- -# Demo / CLI entry point -# --------------------------------------------------------------------------- - - -if __name__ == "__main__": - print("=== Multi-Agent Coordination Demo ===\n") - - # 1. Communication channel - comm = AgentCommunication() - print("1. Created communication channel") - - # 2. Supervisor pattern - supervisor = SupervisorAgent("supervisor", comm) - supervisor.register_worker("researcher", ["search", "analyze"]) - supervisor.register_worker("writer", ["synthesize", "draft"]) - print("2. Registered supervisor with 2 workers: researcher, writer") - - # 3. Handoff protocol - protocol = HandoffProtocol(comm) - handoff_msg = protocol.create_handoff( - from_agent="researcher", - to_agent="writer", - context={"findings": ["item1", "item2"]}, - reason="research_complete", - ) - comm.send(handoff_msg) - received = protocol.accept_handoff("writer") - print( - f"3. Handoff from researcher -> writer: " - f"{'accepted' if received else 'none pending'}" - ) - - # 4. Consensus mechanism - consensus = ConsensusManager() - consensus.initiate_vote("best_approach", ["agent_a", "agent_b", "agent_c"], ["A", "B"]) - consensus.submit_vote("best_approach", "agent_a", "A", confidence=0.9) - consensus.submit_vote("best_approach", "agent_b", "B", confidence=0.6) - consensus.submit_vote("best_approach", "agent_c", "A", confidence=0.8) - result = consensus.calculate_weighted_consensus("best_approach") - print( - f"4. Consensus result: {result['result']} " - f"(strength: {result['consensus_strength']:.2f})" - ) - - # 5. Failure handling - handler = AgentFailureHandler(comm, max_retries=3) - action1 = handler.handle_failure("flaky_agent", "task_1", "timeout") - action2 = handler.handle_failure("flaky_agent", "task_1", "timeout") - action3 = handler.handle_failure("flaky_agent", "task_1", "timeout") - print(f"5. After 3 failures: action={action3['action']}") - print(f" Agent available? {handler.is_available('flaky_agent')}") - - print("\n=== Demo Complete ===") diff --git a/.agents/skills/opentui/SKILL.md b/.agents/skills/opentui/SKILL.md deleted file mode 100644 index 18caf2cd0..000000000 --- a/.agents/skills/opentui/SKILL.md +++ /dev/null @@ -1,202 +0,0 @@ ---- -name: opentui -description: Comprehensive OpenTUI skill for building terminal user interfaces. Covers the core imperative API, React reconciler, and Solid reconciler. Use for any TUI development task including components, layout, keyboard handling, animations, and testing. -metadata: - provider: atomic - references: core, react, solid - internal: true ---- - -# OpenTUI Platform Skill - -Consolidated skill for building terminal user interfaces with OpenTUI. Use decision trees below to find the right framework and components, then load detailed references. - -## Critical Rules - -**Follow these rules in all OpenTUI code:** - -1. **Use `create-tui` for new projects.** See framework `REFERENCE.md` quick starts. -2. **`create-tui` options must come before arguments.** `bunx create-tui -t react my-app` works, `bunx create-tui my-app -t react` does NOT. -3. **Never call `process.exit()` directly.** Use `renderer.destroy()` (see `core/gotchas.md`). -4. **Text styling requires nested tags in React/Solid.** Use modifier elements, not props (see `components/text-display.md`). - -## How to Use This Skill - -### Reference File Structure - -Framework references follow a 5-file pattern. Cross-cutting concepts are single-file guides. - -Each framework in `./references//` contains: - -| File | Purpose | When to Read | -| ------------------ | ---------------------------------- | ----------------------- | -| `REFERENCE.md` | Overview, when to use, quick start | **Always read first** | -| `api.md` | Runtime API, components, hooks | Writing code | -| `configuration.md` | Setup, tsconfig, bundling | Configuring a project | -| `patterns.md` | Common patterns, best practices | Implementation guidance | -| `gotchas.md` | Pitfalls, limitations, debugging | Troubleshooting | - -Cross-cutting concepts in `./references//` have `REFERENCE.md` as the entry point. - -### Reading Order - -1. Start with `REFERENCE.md` for your chosen framework -2. Then read additional files relevant to your task: - - Building components -> `api.md` + `components/.md` - - Setting up project -> `configuration.md` - - Layout/positioning -> `layout/REFERENCE.md` - - Keyboard/input handling -> `keyboard/REFERENCE.md` - - Animations -> `animation/REFERENCE.md` - - Troubleshooting -> `gotchas.md` + `testing/REFERENCE.md` - -### Example Paths - -``` -./references/react/REFERENCE.md # Start here for React -./references/react/api.md # React components and hooks -./references/solid/configuration.md # Solid project setup -./references/components/inputs.md # Input, Textarea, Select docs -./references/core/gotchas.md # Core debugging tips -``` - -### Runtime Notes - -OpenTUI runs on Bun and uses Zig for native builds. Read `./references/core/gotchas.md` for runtime requirements and build guidance. - -## Quick Decision Trees - -### "Which framework should I use?" - -``` -Which framework? -├─ I want full control, maximum performance, no framework overhead -│ └─ core/ (imperative API) -├─ I know React, want familiar component patterns -│ └─ react/ (React reconciler) -├─ I want fine-grained reactivity, optimal re-renders -│ └─ solid/ (Solid reconciler) -└─ I'm building a library/framework on top of OpenTUI - └─ core/ (imperative API) -``` - -### "I need to display content" - -``` -Display content? -├─ Plain or styled text -> components/text-display.md -├─ Container with borders/background -> components/containers.md -├─ Scrollable content area -> components/containers.md (scrollbox) -├─ ASCII art banner/title -> components/text-display.md (ascii-font) -├─ Data table with borders/wrapping -> components/code-diff.md (TextTable) -├─ Code with syntax highlighting -> components/code-diff.md -├─ Diff viewer (unified/split) -> components/code-diff.md -├─ Line numbers with diagnostics -> components/code-diff.md -└─ Markdown content (streaming) -> components/code-diff.md (markdown) -``` - -### "I need user input" - -``` -User input? -├─ Single-line text field -> components/inputs.md (input) -├─ Multi-line text editor -> components/inputs.md (textarea) -├─ Select from a list (vertical) -> components/inputs.md (select) -├─ Tab-based selection (horizontal) -> components/inputs.md (tab-select) -└─ Custom keyboard shortcuts -> keyboard/REFERENCE.md -``` - -### "I need layout/positioning" - -``` -Layout? -├─ Flexbox-style layouts (row, column, wrap) -> layout/REFERENCE.md -├─ Absolute positioning -> layout/patterns.md -├─ Responsive to terminal size -> layout/patterns.md -├─ Centering content -> layout/patterns.md -└─ Complex nested layouts -> layout/patterns.md -``` - -### "I need animations" - -``` -Animations? -├─ Timeline-based animations -> animation/REFERENCE.md -├─ Easing functions -> animation/REFERENCE.md -├─ Property transitions -> animation/REFERENCE.md -└─ Looping animations -> animation/REFERENCE.md -``` - -### "I need to handle input" - -``` -Input handling? -├─ Keyboard events (keypress, release) -> keyboard/REFERENCE.md -├─ Focus management -> keyboard/REFERENCE.md -├─ Paste events -> keyboard/REFERENCE.md -├─ Mouse events -> components/containers.md -├─ Text selection & copy-on-select -> keyboard/REFERENCE.md (selection) -└─ Clipboard (OSC 52) -> keyboard/REFERENCE.md (clipboard) -``` - -### "I need to test my TUI" - -``` -Testing? -├─ Snapshot testing -> testing/REFERENCE.md -├─ Interaction testing -> testing/REFERENCE.md -├─ Test renderer setup -> testing/REFERENCE.md -└─ Debugging tests -> testing/REFERENCE.md -``` - -### "I need to debug/troubleshoot" - -``` -Troubleshooting? -├─ Runtime errors, crashes -> /gotchas.md -├─ Layout issues -> layout/REFERENCE.md + layout/patterns.md -├─ Input/focus issues -> keyboard/REFERENCE.md -└─ Repro + regression tests -> testing/REFERENCE.md -``` - -### Troubleshooting Index - -- Terminal cleanup, crashes -> `core/gotchas.md` -- Text styling not applying -> `components/text-display.md` -- Input focus/shortcuts -> `keyboard/REFERENCE.md` -- Layout misalignment -> `layout/REFERENCE.md` -- Flaky snapshots -> `testing/REFERENCE.md` - -For component naming differences and text modifiers, see `components/REFERENCE.md`. - -## Product Index - -### Frameworks -| Framework | Entry File | Description | -| --------- | --------------------------------- | -------------------------------------- | -| Core | `./references/core/REFERENCE.md` | Imperative API, all primitives | -| React | `./references/react/REFERENCE.md` | React reconciler for declarative TUI | -| Solid | `./references/solid/REFERENCE.md` | SolidJS reconciler for declarative TUI | - -### Cross-Cutting Concepts -| Concept | Entry File | Description | -| ---------- | -------------------------------------- | ------------------------------- | -| Layout | `./references/layout/REFERENCE.md` | Yoga/Flexbox layout system | -| Components | `./references/components/REFERENCE.md` | Component reference by category | -| Keyboard | `./references/keyboard/REFERENCE.md` | Keyboard input handling | -| Animation | `./references/animation/REFERENCE.md` | Timeline-based animations | -| Testing | `./references/testing/REFERENCE.md` | Test renderer and snapshots | - -### Component Categories -| Category | Entry File | Components | -| -------------- | ----------------------------------------- | --------------------------------------------- | -| Text & Display | `./references/components/text-display.md` | text, ascii-font, styled text | -| Containers | `./references/components/containers.md` | box, scrollbox, borders | -| Inputs | `./references/components/inputs.md` | input, textarea, select, tab-select | -| Code & Diff | `./references/components/code-diff.md` | code, line-number, diff, markdown, text-table | - -## Resources - -**Repository**: https://github.com/anomalyco/opentui -**Core Docs**: https://github.com/anomalyco/opentui/tree/main/packages/core/docs -**Examples**: https://github.com/anomalyco/opentui/tree/main/packages/core/src/examples -**Awesome List**: https://github.com/msmps/awesome-opentui diff --git a/.agents/skills/opentui/references/animation/REFERENCE.md b/.agents/skills/opentui/references/animation/REFERENCE.md deleted file mode 100644 index 26dd95439..000000000 --- a/.agents/skills/opentui/references/animation/REFERENCE.md +++ /dev/null @@ -1,431 +0,0 @@ -# Animation System - -OpenTUI provides a timeline-based animation system for smooth property transitions. - -## Overview - -Animations in OpenTUI use: -- **Timeline**: Orchestrates multiple animations -- **Animation Engine**: Manages timelines and rendering -- **Easing Functions**: Control animation curves - -## When to Use - -Use this reference when you need timeline-driven animations, easing curves, or progressive transitions. - -## Basic Usage - -### React - -```tsx -import { useTimeline } from "@opentui/react" -import { useEffect, useState } from "react" - -function AnimatedBox() { - const [width, setWidth] = useState(0) - - const timeline = useTimeline({ - duration: 2000, - }) - - useEffect(() => { - timeline.add( - { width: 0 }, - { - width: 50, - duration: 2000, - ease: "easeOutQuad", - onUpdate: (anim) => { - setWidth(Math.round(anim.targets[0].width)) - }, - } - ) - }, []) - - return ( - - ) -} -``` - -### Solid - -```tsx -import { useTimeline } from "@opentui/solid" -import { createSignal, onMount } from "solid-js" - -function AnimatedBox() { - const [width, setWidth] = createSignal(0) - - const timeline = useTimeline({ - duration: 2000, - }) - - onMount(() => { - timeline.add( - { width: 0 }, - { - width: 50, - duration: 2000, - ease: "easeOutQuad", - onUpdate: (anim) => { - setWidth(Math.round(anim.targets[0].width)) - }, - } - ) - }) - - return ( - - ) -} -``` - -### Core - -```typescript -import { createCliRenderer, Timeline, engine } from "@opentui/core" - -const renderer = await createCliRenderer() -engine.attach(renderer) - -const timeline = new Timeline({ - duration: 2000, - autoplay: true, -}) - -timeline.add( - { x: 0 }, - { - x: 50, - duration: 2000, - ease: "easeOutQuad", - onUpdate: (anim) => { - box.setLeft(Math.round(anim.targets[0].x)) - }, - } -) - -engine.addTimeline(timeline) -``` - -## Timeline Options - -```typescript -const timeline = useTimeline({ - duration: 2000, // Total duration in ms - loop: false, // Loop the timeline - autoplay: true, // Start automatically - onComplete: () => {}, // Called when timeline completes - onPause: () => {}, // Called when timeline pauses -}) -``` - -## Timeline Methods - -```typescript -// Add animation -timeline.add(target, properties, startTime?) - -// Control playback -timeline.play() // Start/resume -timeline.pause() // Pause -timeline.restart() // Restart from beginning - -// State -timeline.progress // Current progress (0-1) -timeline.duration // Total duration -``` - -## Animation Properties - -```typescript -timeline.add( - { value: 0 }, // Target object with initial values - { - value: 100, // Final value - duration: 1000, // Animation duration in ms - ease: "linear", // Easing function - delay: 0, // Delay before starting - onUpdate: (anim) => { - // Called each frame - const current = anim.targets[0].value - }, - onComplete: () => { - // Called when this animation completes - }, - }, - 0 // Start time in timeline (optional) -) -``` - -## Easing Functions - -Available easing functions: - -### Linear - -| Name | Description | -|------|-------------| -| `linear` | Constant speed | - -### Quad (Power of 2) - -| Name | Description | -|------|-------------| -| `easeInQuad` | Slow start | -| `easeOutQuad` | Slow end | -| `easeInOutQuad` | Slow start and end | - -### Cubic (Power of 3) - -| Name | Description | -|------|-------------| -| `easeInCubic` | Slower start | -| `easeOutCubic` | Slower end | -| `easeInOutCubic` | Slower start and end | - -### Quart (Power of 4) - -| Name | Description | -|------|-------------| -| `easeInQuart` | Even slower start | -| `easeOutQuart` | Even slower end | -| `easeInOutQuart` | Even slower start and end | - -### Expo (Exponential) - -| Name | Description | -|------|-------------| -| `easeInExpo` | Exponential start | -| `easeOutExpo` | Exponential end | -| `easeInOutExpo` | Exponential start and end | - -### Back (Overshoot) - -| Name | Description | -|------|-------------| -| `easeInBack` | Pull back, then forward | -| `easeOutBack` | Overshoot, then settle | -| `easeInOutBack` | Both | - -### Elastic - -| Name | Description | -|------|-------------| -| `easeInElastic` | Elastic start | -| `easeOutElastic` | Elastic end (bouncy) | -| `easeInOutElastic` | Both | - -### Bounce - -| Name | Description | -|------|-------------| -| `easeInBounce` | Bounce at start | -| `easeOutBounce` | Bounce at end | -| `easeInOutBounce` | Both | - -## Patterns - -### Progress Bar - -```tsx -function ProgressBar({ progress }: { progress: number }) { - const [width, setWidth] = useState(0) - const maxWidth = 50 - - const timeline = useTimeline() - - useEffect(() => { - timeline.add( - { value: width }, - { - value: (progress / 100) * maxWidth, - duration: 300, - ease: "easeOutQuad", - onUpdate: (anim) => { - setWidth(Math.round(anim.targets[0].value)) - }, - } - ) - }, [progress]) - - return ( - - Progress: {progress}% - - - - - ) -} -``` - -### Fade In - -```tsx -function FadeIn({ children }) { - const [opacity, setOpacity] = useState(0) - - const timeline = useTimeline() - - useEffect(() => { - timeline.add( - { opacity: 0 }, - { - opacity: 1, - duration: 500, - ease: "easeOutQuad", - onUpdate: (anim) => { - setOpacity(anim.targets[0].opacity) - }, - } - ) - }, []) - - return ( - - {children} - - ) -} -``` - -### Looping Animation - -```tsx -function Spinner() { - const [frame, setFrame] = useState(0) - const frames = ["⠋", "⠙", "⠹", "⠸", "⠼", "⠴", "⠦", "⠧", "⠇", "⠏"] - - useEffect(() => { - const interval = setInterval(() => { - setFrame(f => (f + 1) % frames.length) - }, 80) - - return () => clearInterval(interval) - }, []) - - return {frames[frame]} Loading... -} -``` - -### Staggered Animation - -```tsx -function StaggeredList({ items }) { - const [visibleCount, setVisibleCount] = useState(0) - - useEffect(() => { - let count = 0 - const interval = setInterval(() => { - count++ - setVisibleCount(count) - if (count >= items.length) { - clearInterval(interval) - } - }, 100) - - return () => clearInterval(interval) - }, [items.length]) - - return ( - - {items.slice(0, visibleCount).map((item, i) => ( - {item} - ))} - - ) -} -``` - -### Slide In - -```tsx -function SlideIn({ children, from = "left" }) { - const [offset, setOffset] = useState(from === "left" ? -20 : 20) - - const timeline = useTimeline() - - useEffect(() => { - timeline.add( - { offset: from === "left" ? -20 : 20 }, - { - offset: 0, - duration: 300, - ease: "easeOutCubic", - onUpdate: (anim) => { - setOffset(Math.round(anim.targets[0].offset)) - }, - } - ) - }, []) - - return ( - - {children} - - ) -} -``` - -## Performance Tips - -### Batch Updates - -Timeline automatically batches updates within the render loop. - -### Use Integer Values - -Round animated values for character-based positioning: - -```typescript -onUpdate: (anim) => { - setX(Math.round(anim.targets[0].x)) -} -``` - -### Clean Up Timelines - -Hooks automatically clean up, but for core: - -```typescript -// When done with timeline -engine.removeTimeline(timeline) -``` - -## Gotchas - -### Terminal Refresh Rate - -Terminal UIs typically refresh at 60 FPS max. Very fast animations may appear choppy. - -### Character Grid - -Animations are constrained to character cells. Sub-pixel positioning isn't possible. - -### Cleanup in Effects - -Always clean up intervals and timelines: - -```tsx -useEffect(() => { - const interval = setInterval(...) - return () => clearInterval(interval) -}, []) -``` - -## See Also - -- [React API](../react/api.md) - `useTimeline` hook reference -- [Solid API](../solid/api.md) - `useTimeline` hook reference -- [Core API](../core/api.md) - `AnimationEngine` and `Timeline` classes -- [Layout Patterns](../layout/patterns.md) - Animated positioning and transitions diff --git a/.agents/skills/opentui/references/components/REFERENCE.md b/.agents/skills/opentui/references/components/REFERENCE.md deleted file mode 100644 index a28ce8dfa..000000000 --- a/.agents/skills/opentui/references/components/REFERENCE.md +++ /dev/null @@ -1,144 +0,0 @@ -# OpenTUI Components - -Reference for all OpenTUI components, organized by category. Components are available in all three frameworks (Core, React, Solid) with slight API differences. - -## When to Use - -Use this reference when you need to find the right component category or compare naming across Core, React, and Solid. - -## Component Categories - -| Category | Components | File | -|----------|------------|------| -| Text & Display | text, ascii-font, styled text | [text-display.md](./text-display.md) | -| Containers | box, scrollbox, borders | [containers.md](./containers.md) | -| Inputs | input, textarea, select, tab-select | [inputs.md](./inputs.md) | -| Code & Diff | code, line-number, diff, markdown, text-table | [code-diff.md](./code-diff.md) | - -## Component Chooser - -``` -Need a component? -├─ Styled text or ASCII art -> text-display.md -├─ Containers, borders, scrolling -> containers.md -├─ Forms or input controls -> inputs.md -└─ Code blocks, diffs, line numbers, markdown -> code-diff.md -``` - -## Component Naming - -Components have different names across frameworks: - -| Concept | Core (Class) | React (JSX) | Solid (JSX) | -|---------|--------------|-------------|-------------| -| Text | `TextRenderable` | `` | `` | -| Box | `BoxRenderable` | `` | `` | -| ScrollBox | `ScrollBoxRenderable` | `` | `` | -| Input | `InputRenderable` | `` | `` | -| Textarea | `TextareaRenderable` | ` +
+ +
+
+ + + + +
+ + + +
+
+A Toggle all +Enter Generate +Esc Cancel +
+
+ +
+
+ + + + + + + + + + + +`; +} + +const CSS = ` +*,*::before,*::after{box-sizing:border-box;margin:0;padding:0} + +:root { + --bg: #18181e; + --bg-card: #1e1e24; + --bg-elevated: #252530; + --bg-hover: #2b2b37; + --fg: #e0e0e0; + --fg-muted: #909098; + --fg-dim: #606068; + --accent: #8abeb7; + --accent-hover: #9dcec7; + --accent-muted: rgba(138, 190, 183, 0.15); + --accent-subtle: rgba(138, 190, 183, 0.08); + --border: #2a2a34; + --border-muted: #353540; + --border-checked: #8abeb7; + --check-bg: #8abeb7; + --btn-primary: #8abeb7; + --btn-primary-hover: #9dcec7; + --btn-primary-fg: #18181e; + --btn-secondary: #252530; + --btn-secondary-hover: #2b2b37; + --timer-bg: #252530; + --timer-fg: #909098; + --timer-warn-bg: rgba(240, 198, 116, 0.15); + --timer-warn-fg: #f0c674; + --timer-urgent-bg: rgba(204, 102, 102, 0.15); + --timer-urgent-fg: #cc6666; + --overlay-bg: rgba(24, 24, 30, 0.92); + --success: #b5bd68; + --warning: #f0c674; + --font: 'Outfit', system-ui, -apple-system, sans-serif; + --font-display: 'Instrument Serif', Georgia, 'Times New Roman', serif; + --font-mono: 'SF Mono', Consolas, monospace; + --radius: 10px; + --radius-sm: 6px; +} + +@media (prefers-color-scheme: light) { + :root { + --bg: #f5f5f7; + --bg-card: #ffffff; + --bg-elevated: #eeeef0; + --bg-hover: #e4e4e8; + --fg: #1a1a1e; + --fg-muted: #6c6c74; + --fg-dim: #9a9aa2; + --accent: #5f8787; + --accent-hover: #4a7272; + --accent-muted: rgba(95, 135, 135, 0.12); + --accent-subtle: rgba(95, 135, 135, 0.06); + --border: #dcdce0; + --border-muted: #c8c8d0; + --border-checked: #5f8787; + --check-bg: #5f8787; + --btn-primary: #5f8787; + --btn-primary-hover: #4a7272; + --btn-primary-fg: #ffffff; + --btn-secondary: #e4e4e8; + --btn-secondary-hover: #d4d4d8; + --timer-bg: #e4e4e8; + --timer-fg: #6c6c74; + --timer-warn-bg: rgba(217, 119, 6, 0.10); + --timer-warn-fg: #92400e; + --timer-urgent-bg: rgba(175, 95, 95, 0.10); + --timer-urgent-fg: #991b1b; + --overlay-bg: rgba(255, 255, 255, 0.92); + --success: #4d7c0f; + --warning: #b45309; + } +} + +body { + font-family: var(--font); + background: var(--bg); + background-image: radial-gradient(ellipse at 50% 0%, var(--accent-muted) 0%, transparent 60%); + color: var(--fg); + line-height: 1.5; + min-height: 100dvh; + padding-bottom: 72px; +} + +.timer-badge { + position: fixed; + top: 20px; + right: 24px; + z-index: 50; + font-family: var(--font); + font-size: 12px; + font-weight: 600; + font-variant-numeric: tabular-nums; + padding: 5px 14px; + border-radius: 999px; + background: var(--bg-elevated); + color: var(--timer-fg); + border: 1px solid var(--border); + transition: background 0.3s, color 0.3s, border-color 0.3s, opacity 0.3s; + box-shadow: 0 2px 8px rgba(0,0,0,0.2); + cursor: pointer; + user-select: none; + opacity: 0.5; +} +.timer-badge:hover { opacity: 1; } +.timer-badge.active { opacity: 1; } +.timer-badge.warn { + opacity: 1; + background: var(--timer-warn-bg); + color: var(--timer-warn-fg); + border-color: color-mix(in srgb, var(--timer-warn-fg) 30%, transparent); +} +.timer-badge.urgent { + opacity: 1; + background: var(--timer-urgent-bg); + color: var(--timer-urgent-fg); + border-color: color-mix(in srgb, var(--timer-urgent-fg) 30%, transparent); +} +.timer-adjust { + position: fixed; + top: 20px; + right: 24px; + z-index: 51; + display: none; + align-items: center; + gap: 6px; + padding: 4px 6px 4px 12px; + background: var(--bg-elevated); + border: 1px solid var(--accent); + border-radius: 999px; + box-shadow: 0 2px 12px rgba(0,0,0,0.3); +} +.timer-adjust.visible { display: flex; } +.timer-adjust input { + width: 48px; + background: transparent; + border: none; + outline: none; + color: var(--fg); + font-family: var(--font); + font-size: 13px; + font-weight: 600; + font-variant-numeric: tabular-nums; + text-align: center; +} +.timer-adjust-label { font-size: 11px; color: var(--fg-dim); } +.timer-adjust-btn { + font-family: var(--font); + font-size: 11px; + font-weight: 600; + padding: 3px 10px; + border-radius: 999px; + border: none; + background: var(--accent); + color: var(--btn-primary-fg); + cursor: pointer; +} +.timer-adjust-btn:hover { background: var(--accent-hover); } + +main { + max-width: 640px; + margin: 0 auto; + padding: 56px 24px 16px; +} + +.hero { margin-bottom: 28px; } +.hero-kicker { + font-size: 11px; + font-weight: 600; + text-transform: uppercase; + letter-spacing: 0.1em; + color: var(--accent); + margin-bottom: 8px; +} +.hero-title { + font-family: var(--font-display); + font-size: 40px; + font-weight: 400; + font-style: italic; + letter-spacing: -0.01em; + line-height: 1.1; + color: var(--fg); + margin-bottom: 10px; + text-wrap: balance; +} +.hero-desc { + font-size: 14px; + color: var(--fg-muted); + line-height: 1.5; + margin-bottom: 12px; + max-width: 480px; +} +.hero-meta { + display: flex; + align-items: center; + gap: 10px; + font-size: 13px; + color: var(--fg-dim); +} +.hero-meta-sep { + width: 3px; + height: 3px; + border-radius: 50%; + background: var(--fg-dim); + flex-shrink: 0; +} +#hero-status:empty + .hero-meta-sep { display: none; } +.provider-buttons { + display: flex; + align-items: center; + flex-wrap: wrap; + gap: 6px; +} +.summary-model-controls { + display: flex; + align-items: center; + gap: 6px; + min-width: 0; + flex-shrink: 0; +} +.summary-model-dropdown { + font-family: var(--font); + font-size: 12px; + font-weight: 600; + color: var(--fg); + background: var(--bg-elevated); + border: 1px solid var(--border-muted); + border-radius: var(--radius-sm); + padding: 4px 8px; + max-width: 220px; +} +.summary-model-dropdown:focus { + outline: none; + border-color: var(--accent); + box-shadow: 0 0 0 2px color-mix(in srgb, var(--accent) 18%, transparent); +} +.summary-model-dropdown:disabled { + opacity: 0.65; + cursor: default; +} +.provider-btn { + font-family: var(--font); + font-size: 12px; + font-weight: 600; + padding: 3px 10px; + border-radius: 999px; + border: 1px solid var(--border-muted); + background: transparent; + color: var(--fg-muted); + cursor: pointer; + transition: border-color 0.12s, background 0.12s, color 0.12s, opacity 0.12s; +} +.provider-btn.idle:hover { + color: var(--fg); + border-color: var(--accent); +} +.provider-btn.loading { + background: var(--accent-subtle); + color: var(--accent); + border-color: color-mix(in srgb, var(--accent) 35%, var(--border-muted)); + cursor: default; + pointer-events: none; + opacity: 0.85; +} +.provider-btn.loading::after { + content: " …"; + animation: provider-pulse 1.2s ease-in-out infinite; +} +.provider-btn.searched { + background: var(--btn-secondary); + color: var(--fg); + border-color: var(--border-muted); +} +.provider-btn.searched::after { + content: " ✓"; + color: var(--success); +} +.provider-btn.is-default { + box-shadow: inset 0 -2px 0 0 var(--accent); + border-color: var(--accent); +} +.provider-btn:disabled { + cursor: default; + opacity: 0.5; +} + +@keyframes provider-pulse { + 0%, 100% { opacity: 0.4; } + 50% { opacity: 1; } +} + +#result-cards { display: flex; flex-direction: column; gap: 8px; } + +.send-raw-row { + display: flex; + justify-content: flex-end; + padding: 4px 0; +} +.send-raw-row.hidden { display: none; } + +.result-loading { + border: 1px solid var(--border); + border-radius: var(--radius); + background: color-mix(in srgb, var(--bg-card) 86%, var(--accent-subtle)); + overflow: hidden; + box-shadow: 0 1px 2px rgba(0,0,0,0.06); +} +.result-loading-header { + display: flex; + align-items: center; + justify-content: space-between; + gap: 10px; + padding: 12px 14px 10px; + border-bottom: 1px solid var(--border); +} +.result-loading-title { + font-size: 12px; + font-weight: 600; + letter-spacing: 0.04em; + text-transform: uppercase; + color: var(--accent); +} +.result-loading-sub { + font-size: 12px; + color: var(--fg-dim); + font-variant-numeric: tabular-nums; +} +.result-loading-grid { + display: grid; + gap: 10px; + padding: 12px 14px 14px; +} +.loading-card { + border: 1px solid color-mix(in srgb, var(--border-muted) 80%, var(--accent-subtle)); + border-radius: var(--radius-sm); + background: var(--bg-card); + overflow: hidden; + position: relative; +} +.loading-card::after { + content: ""; + position: absolute; + inset: 0; + background: linear-gradient(105deg, transparent 10%, color-mix(in srgb, var(--accent) 18%, transparent) 45%, transparent 75%); + transform: translateX(-130%); + animation: loading-sweep 2s ease-in-out infinite; + pointer-events: none; +} +.loading-card-row { + height: 10px; + border-radius: 999px; + margin: 10px 12px; + background: color-mix(in srgb, var(--fg-dim) 35%, transparent); +} +.loading-card-row.short { width: 35%; } +.loading-card-row.mid { width: 58%; } +.loading-card-row.long { width: 78%; } + +.result-card { + background: var(--bg-card); + border: 1px solid var(--border); + border-radius: var(--radius); + overflow: hidden; + transition: border-color 0.12s; + box-shadow: 0 1px 2px rgba(0,0,0,0.06); +} +.result-card.checked { border-color: var(--border-checked); } +.result-card.searching { + opacity: 1; + border-color: color-mix(in srgb, var(--accent) 40%, var(--border)); + background: linear-gradient(180deg, color-mix(in srgb, var(--accent-subtle) 70%, var(--bg-card)) 0%, var(--bg-card) 100%); + position: relative; +} +.result-card.searching::after { + content: ""; + position: absolute; + inset: 0; + background: linear-gradient(110deg, transparent 20%, color-mix(in srgb, var(--accent) 14%, transparent) 50%, transparent 80%); + transform: translateX(-130%); + animation: loading-sweep 2.2s ease-in-out infinite; + pointer-events: none; +} +.result-card.searching .result-card-header { cursor: default; } +.result-card.searching .result-card-header:hover { background: transparent; } +.result-card.error { border-color: var(--timer-urgent-fg); } + +.result-card-header { + display: flex; + align-items: flex-start; + gap: 12px; + padding: 14px 16px; + cursor: pointer; + user-select: none; + transition: background 0.12s; +} +.result-card-header:hover { background: var(--bg-hover); } + +.result-card-header input[type="checkbox"] { + appearance: none; + width: 16px; + height: 16px; + min-width: 16px; + border: 1.5px solid var(--border-muted); + border-radius: 4px; + margin-top: 2px; + cursor: pointer; + transition: background 0.12s, border-color 0.12s; + display: grid; + place-content: center; +} +.result-card-header input[type="checkbox"]:checked { + background: var(--check-bg); + border-color: var(--check-bg); +} +.result-card-header input[type="checkbox"]:checked::after { + content: ""; + width: 9px; + height: 6px; + border-left: 2px solid var(--btn-primary-fg); + border-bottom: 2px solid var(--btn-primary-fg); + transform: rotate(-45deg); + margin-top: -1px; +} + +.result-card-info { flex: 1; min-width: 0; } + +.result-card-query-row { + display: flex; + align-items: center; + gap: 8px; + flex-wrap: wrap; + margin-bottom: 2px; +} +.result-card-query { + font-size: 14px; + font-weight: 600; + color: var(--fg); +} +.provider-tag { + display: inline-flex; + align-items: center; + padding: 1px 7px; + border-radius: 999px; + font-size: 10px; + font-weight: 700; + letter-spacing: 0.03em; + text-transform: uppercase; + border: 1px solid transparent; +} +.provider-tag.provider-exa { + color: #8dd3ff; + background: rgba(141, 211, 255, 0.14); + border-color: rgba(141, 211, 255, 0.3); +} +.provider-tag.provider-perplexity { + color: #cba6f7; + background: rgba(203, 166, 247, 0.14); + border-color: rgba(203, 166, 247, 0.3); +} +.provider-tag.provider-gemini { + color: #f5c27b; + background: rgba(245, 194, 123, 0.14); + border-color: rgba(245, 194, 123, 0.3); +} +.provider-tag.provider-unknown { + color: var(--fg-muted); + background: var(--bg-elevated); + border-color: var(--border-muted); +} +.result-card-meta { + font-size: 12px; + color: var(--fg-dim); +} +.result-card-preview { + font-size: 12.5px; + color: var(--fg-muted); + margin-top: 6px; + display: -webkit-box; + -webkit-line-clamp: 2; + -webkit-box-orient: vertical; + overflow: hidden; + line-height: 1.45; +} + +.result-card-expand { + color: var(--fg-dim); + font-size: 11px; + margin-top: 2px; + flex-shrink: 0; + padding-top: 3px; + transition: color 0.12s; +} +.result-card-header:hover .result-card-expand { color: var(--fg-muted); } + +.result-card-body { + display: none; + border-top: 1px solid var(--border); +} +.result-card-body.open { display: block; } + +.result-card-answer { + padding: 14px 16px; + font-size: 13.5px; + color: var(--fg-muted); + line-height: 1.6; + max-height: 400px; + overflow-y: auto; +} +.result-card-answer h1, +.result-card-answer h2, +.result-card-answer h3, +.result-card-answer h4 { + color: var(--fg); + font-family: var(--font); + font-weight: 600; + margin: 16px 0 6px; + line-height: 1.3; +} +.result-card-answer h1 { font-size: 16px; } +.result-card-answer h2 { font-size: 14.5px; } +.result-card-answer h3 { font-size: 13.5px; } +.result-card-answer h4 { font-size: 13px; color: var(--fg-muted); } +.result-card-answer p { margin: 0 0 10px; } +.result-card-answer p:last-child { margin-bottom: 0; } +.result-card-answer strong { color: var(--fg); font-weight: 600; } +.result-card-answer a { color: var(--accent); text-decoration: none; } +.result-card-answer a:hover { text-decoration: underline; } +.result-card-answer ul, .result-card-answer ol { + margin: 6px 0 10px; + padding-left: 20px; +} +.result-card-answer li { margin-bottom: 4px; } +.result-card-answer li::marker { color: var(--fg-dim); } +.result-card-answer code { + font-family: var(--font-mono); + font-size: 12px; + padding: 1px 5px; + background: var(--bg-elevated); + border: 1px solid var(--border); + border-radius: 3px; + color: var(--fg); +} +.result-card-answer pre { + margin: 8px 0 12px; + padding: 12px 14px; + background: var(--bg); + border: 1px solid var(--border); + border-radius: var(--radius-sm); + overflow-x: auto; + line-height: 1.45; +} +.result-card-answer pre code { + padding: 0; + background: none; + border: none; + font-size: 12px; + color: var(--fg-muted); +} +.result-card-answer blockquote { + margin: 8px 0; + padding: 8px 14px; + border-left: 3px solid var(--accent); + color: var(--fg-dim); + background: var(--accent-subtle); + border-radius: 0 var(--radius-sm) var(--radius-sm) 0; +} +.result-card-answer table { + width: 100%; + border-collapse: collapse; + margin: 8px 0 12px; + font-size: 12.5px; +} +.result-card-answer th, .result-card-answer td { + padding: 6px 10px; + border: 1px solid var(--border); + text-align: left; +} +.result-card-answer th { + background: var(--bg-elevated); + color: var(--fg); + font-weight: 600; + font-size: 11.5px; + text-transform: uppercase; + letter-spacing: 0.03em; +} +.result-card-answer hr { + border: none; + border-top: 1px solid var(--border); + margin: 14px 0; +} + +.result-card-sources { + padding: 10px 16px 14px; + border-top: 1px solid var(--border); +} +.result-card-sources-title { + font-size: 11px; + font-weight: 600; + text-transform: uppercase; + letter-spacing: 0.06em; + color: var(--fg-dim); + margin-bottom: 6px; +} +.source-link { + display: block; + padding: 4px 0; + font-size: 12.5px; + color: var(--fg-muted); + text-decoration: none; + overflow: hidden; + text-overflow: ellipsis; + white-space: nowrap; + transition: color 0.12s; +} +.source-link:hover { color: var(--accent); } +.source-domain { + color: var(--fg-dim); + margin-left: 6px; +} + +.result-card-error-msg { + padding: 12px 16px; + font-size: 13px; + color: var(--timer-urgent-fg); +} + +.card-alt-providers { + display: flex; + align-items: center; + gap: 6px; + padding: 4px 16px 8px 42px; + font-size: 11px; + color: var(--fg-dim); +} +.card-alt-chip { + font-family: var(--font); + font-size: 10px; + font-weight: 600; + padding: 2px 8px; + border-radius: 999px; + border: 1px solid var(--border-muted); + background: transparent; + color: var(--fg-muted); + cursor: pointer; + transition: border-color 0.12s, color 0.12s, background 0.12s; +} +.card-alt-chip:hover:not(:disabled) { + color: var(--accent); + border-color: var(--accent); +} +.card-alt-chip:disabled { + opacity: 0.4; + cursor: default; +} +.card-alt-chip.loading { + opacity: 0.6; + pointer-events: none; +} +.card-alt-chip.loading::after { + content: " …"; +} + +.searching-dots::after { + content: ""; + animation: dots 1.5s steps(4, end) infinite; +} +@keyframes dots { + 0% { content: ""; } + 25% { content: "."; } + 50% { content: ".."; } + 75% { content: "..."; } +} + +@keyframes loading-sweep { + 0% { transform: translateX(-130%); } + 100% { transform: translateX(130%); } +} + +@keyframes summary-pulse { + 0%, 100% { + transform: scale(0.9); + box-shadow: 0 0 0 0 color-mix(in srgb, var(--accent) 35%, transparent); + } + 50% { + transform: scale(1.15); + box-shadow: 0 0 0 6px color-mix(in srgb, var(--accent) 0%, transparent); + } +} + +@keyframes summary-sweep { + 0% { transform: translateX(-100%); } + 100% { transform: translateX(120%); } +} + +@keyframes summary-panel-sweep { + 0% { transform: translateX(-115%); } + 100% { transform: translateX(115%); } +} + +.add-search { + display: flex; + align-items: center; + gap: 10px; + margin-top: 12px; + padding: 11px 14px; + border: 1px dashed var(--border); + border-radius: var(--radius); + cursor: text; + transition: border-color 0.15s, background 0.15s; +} +.add-search:hover { + border-color: var(--border-muted); + background: var(--accent-subtle); +} +.add-search:focus-within { + border-color: var(--accent); + border-style: solid; + background: var(--accent-subtle); +} +.add-search-icon { + color: var(--fg-dim); + font-size: 16px; + font-weight: 300; + line-height: 1; + flex-shrink: 0; + transition: color 0.15s; +} +.add-search:focus-within .add-search-icon { color: var(--accent); } +.add-search input { + flex: 1; + background: transparent; + border: none; + outline: none; + color: var(--fg); + font-family: var(--font); + font-size: 13.5px; + font-weight: 500; +} +.add-search input::placeholder { + color: var(--fg-dim); + font-weight: 400; +} +.add-search-wand { + flex-shrink: 0; + width: 26px; + height: 26px; + display: flex; + align-items: center; + justify-content: center; + border: 1px solid var(--border-muted); + border-radius: 6px; + background: transparent; + color: var(--fg-dim); + font-size: 14px; + cursor: pointer; + transition: color 0.12s, border-color 0.12s, background 0.12s; +} +.add-search-wand:hover:not(:disabled) { + color: var(--accent); + border-color: var(--accent); + background: var(--accent-subtle); +} +.add-search-wand:disabled { + opacity: 0.3; + cursor: default; +} +.add-search-wand.rewriting { + pointer-events: none; + animation: wand-spin 0.8s linear infinite; +} +@keyframes wand-spin { + 0% { transform: rotate(0deg); } + 100% { transform: rotate(360deg); } +} + +.summary-panel { + margin-top: 14px; + border: 1px solid var(--border); + border-radius: var(--radius); + background: var(--bg-card); + padding: 14px; + display: flex; + flex-direction: column; + gap: 10px; +} +.summary-panel.hidden { display: none; } +.summary-header { display: flex; flex-direction: column; gap: 2px; } +.summary-header-top { display: flex; align-items: flex-start; justify-content: space-between; gap: 12px; } +.summary-title { + font-size: 14px; + font-weight: 600; + color: var(--fg); +} +.summary-subtitle { + font-size: 12px; + color: var(--fg-dim); +} +.summary-generating { + position: relative; + isolation: isolate; + overflow: hidden; + border: 1px solid color-mix(in srgb, var(--accent) 28%, var(--border)); + border-radius: var(--radius-sm); + background: linear-gradient(130deg, color-mix(in srgb, var(--accent-subtle) 78%, transparent) 0%, var(--bg-elevated) 70%); + padding: 12px; + display: flex; + flex-direction: column; + gap: 10px; +} +.summary-generating::before { + content: ""; + position: absolute; + inset: 0; + background: linear-gradient(110deg, transparent 0%, color-mix(in srgb, var(--accent) 16%, transparent) 50%, transparent 100%); + transform: translateX(-115%); + animation: summary-panel-sweep 2.4s ease-in-out infinite; + pointer-events: none; +} +.summary-generating > * { + position: relative; + z-index: 1; +} +.summary-generating.hidden { display: none; } +.summary-generating-head { + display: flex; + align-items: center; + gap: 8px; + font-size: 12px; + font-weight: 600; + color: var(--accent-hover); +} +.summary-generating-orb { + width: 10px; + height: 10px; + border-radius: 999px; + background: var(--accent); + box-shadow: 0 0 0 0 color-mix(in srgb, var(--accent) 35%, transparent); + animation: summary-pulse 1.1s ease-in-out infinite; +} +.summary-generating-bars { + display: grid; + gap: 6px; +} +.summary-generating-bar { + position: relative; + display: block; + height: 8px; + border-radius: 999px; + background: color-mix(in srgb, var(--bg) 65%, var(--bg-elevated)); + overflow: hidden; + transition: width 220ms ease; +} +.summary-generating-bar::after { + content: ""; + position: absolute; + inset: 0; + transform: translateX(-100%); + background: linear-gradient(90deg, transparent 0%, color-mix(in srgb, var(--accent) 45%, transparent) 50%, transparent 100%); + animation: summary-sweep 1.6s ease-in-out infinite; +} +.summary-generating-bar.b1 { width: 86%; } +.summary-generating-bar.b2 { width: 68%; } +.summary-generating-bar.b3 { width: 74%; } +.summary-generating[data-phase="1"] .summary-generating-bar.b1 { width: 72%; } +.summary-generating[data-phase="1"] .summary-generating-bar.b2 { width: 82%; } +.summary-generating[data-phase="1"] .summary-generating-bar.b3 { width: 60%; } +.summary-generating[data-phase="2"] .summary-generating-bar.b1 { width: 64%; } +.summary-generating[data-phase="2"] .summary-generating-bar.b2 { width: 71%; } +.summary-generating[data-phase="2"] .summary-generating-bar.b3 { width: 90%; } +.summary-generating-bar.b2::after { animation-delay: 0.15s; } +.summary-generating-bar.b3::after { animation-delay: 0.3s; } +.summary-input { + width: 100%; + min-height: 180px; + resize: vertical; + border: 1px solid var(--border-muted); + border-radius: var(--radius-sm); + padding: 10px 12px; + font-family: var(--font); + font-size: 13px; + line-height: 1.5; + color: var(--fg); + background: var(--bg-elevated); + outline: none; +} +.summary-input.hidden { display: none; } +.summary-input:focus { + border-color: var(--accent); +} +.summary-feedback-row { + display: flex; + align-items: center; + gap: 8px; + margin-top: 6px; +} +.summary-feedback { + flex: 1; + height: 32px; + border: 1px solid var(--border-muted); + border-radius: var(--radius-sm); + padding: 4px 10px; + font-family: var(--font); + font-size: 12px; + color: var(--fg); + background: var(--bg-elevated); + outline: none; +} +.summary-feedback:focus { + border-color: var(--accent); +} +.summary-feedback::placeholder { + color: var(--fg-muted); +} +.summary-actions { + display: flex; + align-items: center; + justify-content: flex-end; + gap: 8px; +} + +.action-bar { + position: fixed; + bottom: 0; + left: 0; + right: 0; + z-index: 10; + display: flex; + align-items: center; + justify-content: space-between; + padding: 12px 24px; + background: color-mix(in srgb, var(--bg) 90%, transparent); + backdrop-filter: blur(12px); + -webkit-backdrop-filter: blur(12px); + border-top: 1px solid var(--border); +} +.action-shortcuts { display: flex; align-items: center; gap: 16px; } +.shortcut { display: flex; align-items: center; gap: 5px; font-size: 11px; color: var(--fg-dim); } +.shortcut kbd { + display: inline-flex; + align-items: center; + justify-content: center; + min-width: 18px; + height: 18px; + padding: 0 4px; + font-family: var(--font-mono); + font-size: 10px; + font-weight: 500; + background: var(--bg-elevated); + border: 1px solid var(--border-muted); + border-radius: 3px; + color: var(--fg-muted); +} +.action-buttons { display: flex; gap: 8px; } + +.btn { + font-family: var(--font); + font-size: 13px; + font-weight: 500; + padding: 7px 16px; + border: none; + border-radius: var(--radius-sm); + cursor: pointer; + transition: background 0.12s, opacity 0.12s; +} +.btn:disabled { opacity: 0.35; cursor: default; } +.btn-submit { background: var(--btn-primary); color: var(--btn-primary-fg); } +.btn-submit:hover:not(:disabled) { background: var(--btn-primary-hover); } +.btn-secondary { background: var(--btn-secondary); color: var(--fg-muted); border: 1px solid var(--border); } +.btn-secondary:hover:not(:disabled) { background: var(--btn-secondary-hover); color: var(--fg); } + +.success-overlay { + position: fixed; inset: 0; z-index: 200; + background: var(--overlay-bg); + display: flex; flex-direction: column; align-items: center; justify-content: center; gap: 12px; + transition: opacity 200ms; +} +.success-overlay.hidden { display: flex !important; opacity: 0; pointer-events: none; } +.success-icon { + width: 56px; height: 56px; border-radius: 50%; + border: 2px solid var(--success); + display: flex; align-items: center; justify-content: center; + font-size: 18px; font-weight: 700; color: var(--success); +} +.success-overlay p { margin: 0; font-size: 13px; font-weight: 600; color: var(--success); letter-spacing: 0.06em; text-transform: uppercase; } + +.expired-overlay { + position: fixed; inset: 0; + background: var(--overlay-bg); + display: flex; align-items: center; justify-content: center; + opacity: 0; transition: opacity 400ms; pointer-events: none; z-index: 200; +} +.expired-overlay.visible { opacity: 1; pointer-events: auto; } +.expired-overlay.hidden { display: flex !important; opacity: 0; pointer-events: none; } +.expired-content { + text-align: center; max-width: 480px; padding: 48px 56px; + background: var(--bg-card); border: 1px solid var(--border); border-radius: 12px; +} +.expired-overlay.visible .expired-content { animation: slide-up 400ms ease-out; } +@keyframes slide-up { from { transform: translateY(20px); } to { transform: translateY(0); } } +.expired-icon { + width: 72px; height: 72px; border-radius: 50%; border: 2px solid var(--warning); + display: flex; align-items: center; justify-content: center; + font-size: 32px; font-weight: bold; color: var(--warning); margin: 0 auto 24px; +} +.expired-content h2 { color: var(--fg); margin: 0 0 16px; font-size: 22px; font-weight: 600; } +.expired-content p { color: var(--fg-muted); margin: 0 0 24px; font-size: 14px; line-height: 1.6; } +.expired-countdown { font-size: 13px; color: var(--fg-dim); font-variant-numeric: tabular-nums; } +.expired-countdown span { color: var(--warning); font-weight: 600; } + +.preview-modal { + position: fixed; inset: 0; z-index: 250; + background: var(--overlay-bg); + display: flex; align-items: center; justify-content: center; + animation: fade-in 150ms ease-out; +} +.preview-modal.hidden { display: none; } +@keyframes fade-in { from { opacity: 0; } to { opacity: 1; } } +.preview-modal-inner { + width: min(720px, calc(100% - 48px)); + max-height: calc(100vh - 80px); + background: var(--bg-card); + border: 1px solid var(--border); + border-radius: 12px; + display: flex; flex-direction: column; + animation: slide-up 200ms ease-out; +} +.preview-modal-header { + display: flex; align-items: center; justify-content: space-between; + padding: 16px 20px; + border-bottom: 1px solid var(--border); + flex-shrink: 0; +} +.preview-modal-title { font-size: 14px; font-weight: 600; color: var(--fg); margin: 0; } +.preview-modal-close { + background: none; border: none; cursor: pointer; + font-size: 22px; line-height: 1; color: var(--fg-muted); padding: 0 4px; + transition: color 0.12s; +} +.preview-modal-close:hover { color: var(--fg); } +.preview-modal-body { + position: relative; + padding: 24px 28px; + overflow-y: auto; + font-size: 14px; line-height: 1.7; color: var(--fg); +} +.preview-modal-body h1 { font-size: 20px; font-weight: 600; margin: 1.2em 0 0.5em; color: var(--fg); } +.preview-modal-body h2 { font-size: 16px; font-weight: 600; margin: 1.2em 0 0.4em; color: var(--fg); } +.preview-modal-body h3 { font-size: 14px; font-weight: 600; margin: 1em 0 0.3em; color: var(--fg); } +.preview-modal-body p { margin: 0.6em 0; } +.preview-modal-body a { color: var(--accent); } +.preview-modal-body pre { background: var(--bg-elevated); padding: 14px; border-radius: var(--radius-sm); overflow-x: auto; } +.preview-modal-body code { font-size: 0.9em; } +.preview-modal-body blockquote { border-left: 3px solid var(--border); padding-left: 14px; color: var(--fg-muted); margin: 0.6em 0; } +.preview-modal-body hr { border: none; border-top: 1px solid var(--border); margin: 1.5em 0; } +.preview-modal-body ul, .preview-modal-body ol { padding-left: 1.4em; } +.preview-modal-body li + li { margin-top: 0.25em; } +.preview-modal-body strong { color: var(--fg); } +.preview-modal-footer { + padding: 12px 20px; + border-top: 1px solid var(--border); + display: flex; align-items: center; gap: 8px; + flex-shrink: 0; +} +.preview-modal-model { + margin-right: auto; + font-family: var(--font); + font-size: 11px; + color: var(--fg-muted); + background: var(--bg-elevated); + border: 1px solid var(--border-muted); + border-radius: var(--radius-sm); + padding: 4px 8px; + max-width: 220px; + outline: none; +} +.preview-modal-model:focus { border-color: var(--accent); } + +.preview-popover { + position: absolute; + z-index: 260; + width: min(340px, calc(100% - 40px)); + background: var(--bg-elevated); + border: 1px solid var(--accent); + border-radius: var(--radius); + padding: 10px 12px; + display: flex; flex-direction: column; gap: 8px; + box-shadow: 0 8px 24px rgba(0,0,0,0.35); + animation: fade-in 100ms ease-out; +} +.preview-popover.hidden { display: none; } +.preview-popover-quote { + font-size: 12px; + color: var(--fg-muted); + font-style: italic; + border-left: 2px solid var(--accent); + padding-left: 8px; + max-height: 48px; + overflow: hidden; + display: -webkit-box; + -webkit-line-clamp: 2; + -webkit-box-orient: vertical; +} +.preview-popover-input { + font-family: var(--font); + font-size: 13px; + line-height: 1.4; + color: var(--fg); + background: var(--bg-card); + border: 1px solid var(--border-muted); + border-radius: var(--radius-sm); + padding: 6px 10px; + outline: none; + width: 100%; + resize: vertical; +} +.preview-popover-input:focus { border-color: var(--accent); } +.preview-popover-btn { align-self: flex-end; font-size: 12px; padding: 5px 14px; } + +.error-banner { + position: fixed; bottom: 64px; left: 50%; transform: translateX(-50%); z-index: 50; + padding: 10px 20px; background: var(--timer-urgent-bg); color: var(--timer-urgent-fg); + border-radius: var(--radius); font-size: 13px; font-weight: 500; +} + +.summary-panel.updating { + border-color: color-mix(in srgb, var(--accent) 35%, var(--border)); + position: relative; + overflow: hidden; +} +.summary-panel.updating::after { + content: ""; + position: absolute; + top: 0; + left: 0; + width: 30%; + height: 2px; + border-radius: var(--radius) var(--radius) 0 0; + background: linear-gradient(90deg, transparent, var(--accent), transparent); + animation: updating-bar 1.8s ease-in-out infinite; + pointer-events: none; +} +.summary-panel.updating .summary-input, +.summary-panel.updating .summary-feedback-row { + opacity: 0.45; + pointer-events: none; +} +.summary-panel.updating .summary-actions { + opacity: 0.72; +} +@keyframes updating-bar { + 0% { transform: translateX(-50%); } + 100% { transform: translateX(430%); } +} + +@media (prefers-reduced-motion: reduce) { + .loading-card::after, + .result-card.searching::after, + .provider-btn.loading::after, + .searching-dots::after, + .summary-generating::before, + .summary-generating-orb, + .summary-generating-bar::after, + .summary-panel.updating::after { + animation: none !important; + } +} + +@media (max-width: 500px) { + main { padding: 32px 16px 16px; } + .hero-title { font-size: 28px; } + .hero-desc { font-size: 13px; } + .summary-header-top { flex-direction: column; } + .summary-model-controls { flex-wrap: wrap; } + .summary-model-dropdown { max-width: 100%; } + .action-bar { padding: 10px 14px; } + .action-shortcuts { display: none; } + .result-card-header { padding: 12px 14px; } + .expired-content { padding: 32px 24px; } + .timer-badge { top: 12px; right: 16px; } +} +`; + +const SCRIPT = `(function() { + var DATA = __INLINE_DATA__; + var token = DATA.sessionToken; + var timeoutSec = DATA.timeout; + var queries = Array.isArray(DATA.queries) ? DATA.queries : []; + var providers = ["perplexity", "exa", "gemini"]; + var availProviders = DATA.availableProviders && typeof DATA.availableProviders === "object" ? DATA.availableProviders : {}; + var workflow = "summary-review"; + var initialDefaultProvider = typeof DATA.defaultProvider === "string" ? DATA.defaultProvider : "exa"; + if (providers.indexOf(initialDefaultProvider) === -1) initialDefaultProvider = "exa"; + + var summaryModels = Array.isArray(DATA.summaryModels) + ? DATA.summaryModels.filter(function(model) { + return model && typeof model === "object" && typeof model.value === "string"; + }) + : []; + var defaultSummaryModel = typeof DATA.defaultSummaryModel === "string" + ? DATA.defaultSummaryModel.trim() + : ""; + + var submitted = false; + var timerExpired = false; + var submitInFlight = false; + var searchesDone = false; + var stage = "results"; + var summaryMeta = null; + var summaryRequestSeq = 0; + var lastAutoSummarySignature = ""; + var lastInteraction = Date.now(); + var completedCount = 0; + var es = null; + + var allQueries = queries.map(function(query, slotId) { return { slotId: slotId, query: query }; }); + var nextSlotId = queries.length; + var queryIndexToSlot = new Map(); + var providerCoverage = new Map(); + + var currentProvider = initialDefaultProvider; + var initialStreamDone = queries.length === 0; + var providerBatchInFlight = false; + var batchLoadingProvider = null; + var addSearchInFlight = 0; + var isRegenerating = false; + + var timerEl = document.getElementById("timer"); + var timerAdjustEl = document.getElementById("timer-adjust"); + var timerInput = document.getElementById("timer-input"); + var timerSetBtn = document.getElementById("timer-set"); + var heroTitle = document.querySelector(".hero-title"); + var heroDesc = document.querySelector(".hero-desc"); + var resultCardsEl = document.getElementById("result-cards"); + var btnSend = document.getElementById("btn-send"); + var btnSendRaw = document.getElementById("btn-send-raw"); + var sendRawRow = document.getElementById("send-raw-row"); + var summaryPanel = document.getElementById("summary-panel"); + var summarySubtitle = document.getElementById("summary-subtitle"); + var summaryGeneratingEl = document.getElementById("summary-generating"); + var summaryGeneratingCopy = document.getElementById("summary-generating-copy"); + var summaryInput = document.getElementById("summary-input"); + var summaryFeedback = document.getElementById("summary-feedback"); + var btnSummaryBack = document.getElementById("btn-summary-back"); + var btnSummaryRegenerate = document.getElementById("btn-summary-regenerate"); + var btnSummaryPreview = document.getElementById("btn-summary-preview"); + var btnSummaryApprove = document.getElementById("btn-summary-approve"); + var successOverlay = document.getElementById("success-overlay"); + var successText = document.getElementById("success-text"); + var expiredOverlay = document.getElementById("expired-overlay"); + var expiredText = document.getElementById("expired-text"); + var closeCountdown = document.getElementById("close-countdown"); + var errorBanner = document.getElementById("error-banner"); + var addSearchInput = document.getElementById("add-search-input"); + var addSearchEl = document.getElementById("add-search"); + var addSearchWand = document.getElementById("add-search-wand"); + var heroStatus = document.getElementById("hero-status"); + var summaryProviderSelect = document.getElementById("summary-provider-select"); + var summaryModelSelect = document.getElementById("summary-model-select"); + var previewModal = document.getElementById("preview-modal"); + var previewModalBody = document.getElementById("preview-modal-body"); + var previewModalClose = document.getElementById("preview-modal-close"); + var previewModalModel = document.getElementById("preview-modal-model"); + var previewModalRegenerate = document.getElementById("preview-modal-regenerate"); + var previewModalApprove = document.getElementById("preview-modal-approve"); + var previewPopover = document.getElementById("preview-popover"); + var previewPopoverQuote = document.getElementById("preview-popover-quote"); + var previewPopoverInput = document.getElementById("preview-popover-input"); + var previewPopoverRegen = document.getElementById("preview-popover-regen"); + var providerButtons = Array.prototype.slice.call(document.querySelectorAll(".provider-btn")); + var loadingPanelEl = null; + + var summaryModelsByProvider = Object.create(null); + var summaryProviders = []; + var currentSummaryProvider = ""; + var currentSummaryModel = ""; + var summaryPendingModel = ""; + var summaryGeneratingStartedAt = 0; + var summaryGeneratingPhase = -1; + var rewriteInFlight = false; + + function escHtml(s) { + return String(s) + .replace(/&/g, "&") + .replace(//g, ">") + .replace(/\"/g, """); + } + + function sanitizeHref(url) { + var value = typeof url === "string" ? url.trim() : ""; + return /^https?:\/\//i.test(value) ? value : "#"; + } + + function sanitizeMarkdownHtml(html) { + var container = document.createElement("div"); + container.innerHTML = html; + + container.querySelectorAll("script, iframe, object, embed, form, style, link, meta, base") + .forEach(function(el) { el.remove(); }); + + var nodes = container.querySelectorAll("*"); + nodes.forEach(function(node) { + for (var i = node.attributes.length - 1; i >= 0; i--) { + var attr = node.attributes[i]; + if (/^on/i.test(attr.name)) node.removeAttribute(attr.name); + } + }); + + var anchors = container.querySelectorAll("a[href]"); + anchors.forEach(function(anchor) { + var safe = sanitizeHref(anchor.getAttribute("href") || ""); + anchor.setAttribute("href", safe); + anchor.setAttribute("rel", "noopener noreferrer"); + anchor.setAttribute("target", "_blank"); + }); + + var images = container.querySelectorAll("img[src]"); + images.forEach(function(img) { + var safe = sanitizeHref(img.getAttribute("src") || ""); + if (safe === "#") { + img.remove(); + } else { + img.setAttribute("src", safe); + } + }); + + return container.innerHTML; + } + + function post(path, body) { + return fetch(path, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify(Object.assign({ token: token }, body)), + }); + } + + function extractServerError(data) { + if (!data || typeof data !== "object") return ""; + if (typeof data.error === "string" && data.error.trim()) return data.error.trim(); + return ""; + } + + function postJson(path, body) { + return post(path, body).then(function(res) { + return res.text().then(function(raw) { + var data = null; + if (raw) { + try { + data = JSON.parse(raw); + } catch (err) { + var parseMessage = err instanceof Error ? err.message : String(err); + throw new Error("Invalid JSON response from " + path + ": " + parseMessage); + } + } + + if (!res.ok) { + throw new Error(extractServerError(data) || ("HTTP " + res.status)); + } + + return data; + }); + }); + } + + function formatTime(sec) { + var m = Math.floor(sec / 60); + var s = sec % 60; + return m + ":" + (s < 10 ? "0" : "") + s; + } + + function normalizeProvider(provider, fallback) { + if (typeof provider === "string") { + var normalized = provider.toLowerCase(); + if (providers.indexOf(normalized) !== -1) return normalized; + } + if (typeof fallback === "string") { + var fallbackNormalized = fallback.toLowerCase(); + if (providers.indexOf(fallbackNormalized) !== -1) return fallbackNormalized; + } + return ""; + } + + function providerLabel(provider) { + if (provider === "perplexity") return "Perplexity"; + if (provider === "exa") return "Exa"; + if (provider === "gemini") return "Gemini"; + return "Unknown"; + } + + function providerTagHtml(provider) { + var normalized = normalizeProvider(provider, ""); + if (!normalized) return ""; + return '' + escHtml(providerLabel(normalized)) + ""; + } + + function buildAltChipsHtml(provider, queryText) { + var normalizedProv = normalizeProvider(provider, ""); + if (!normalizedProv) return ""; + var altProviders = providers.filter(function(p) { return p !== normalizedProv && availProviders[p] === true; }); + if (altProviders.length === 0) return ""; + var html = '
Also try'; + for (var ap = 0; ap < altProviders.length; ap++) { + html += ''; + } + html += "
"; + return html; + } + + function getSummaryProvider(modelValue) { + if (typeof modelValue !== "string") return ""; + var trimmed = modelValue.trim(); + var slash = trimmed.indexOf("/"); + if (slash <= 0) return ""; + return trimmed.slice(0, slash); + } + + function summaryProviderLabel(provider) { + if (!provider) return ""; + if (provider === "openai") return "OpenAI"; + if (provider === "google") return "Google"; + if (provider === "anthropic") return "Anthropic"; + return provider.charAt(0).toUpperCase() + provider.slice(1); + } + + function buildSummaryModelState() { + summaryModelsByProvider = Object.create(null); + summaryProviders = []; + var seenValues = {}; + + for (var i = 0; i < summaryModels.length; i++) { + var model = summaryModels[i]; + if (!model || typeof model.value !== "string") continue; + var value = model.value.trim(); + if (!value || seenValues[value]) continue; + var provider = getSummaryProvider(value); + if (!provider) continue; + seenValues[value] = true; + + if (!summaryModelsByProvider[provider]) { + summaryModelsByProvider[provider] = []; + summaryProviders.push(provider); + } + + var label = typeof model.label === "string" && model.label.trim().length > 0 + ? model.label.trim() + : value; + summaryModelsByProvider[provider].push({ value: value, label: label }); + } + } + + function renderSummaryProviderSelect() { + if (!summaryProviderSelect) return; + + summaryProviderSelect.innerHTML = ""; + for (var i = 0; i < summaryProviders.length; i++) { + var provider = summaryProviders[i]; + var option = document.createElement("option"); + option.value = provider; + option.textContent = summaryProviderLabel(provider); + summaryProviderSelect.appendChild(option); + } + } + + function populateSummaryModelSelect(provider, preferredModel) { + if (!summaryModelSelect) return; + + summaryModelSelect.innerHTML = ""; + + var autoOption = document.createElement("option"); + autoOption.value = ""; + autoOption.textContent = "Auto"; + summaryModelSelect.appendChild(autoOption); + + var models = summaryModelsByProvider[provider] || []; + for (var i = 0; i < models.length; i++) { + var option = document.createElement("option"); + option.value = models[i].value; + var shortLabel = models[i].value; + var labelSlash = shortLabel.indexOf("/"); + if (labelSlash > 0) shortLabel = shortLabel.slice(labelSlash + 1); + option.textContent = shortLabel; + summaryModelSelect.appendChild(option); + } + + var hasPreferred = false; + if (preferredModel) { + for (var j = 0; j < models.length; j++) { + if (models[j].value === preferredModel) { + hasPreferred = true; + break; + } + } + } + + if (hasPreferred) { + summaryModelSelect.value = preferredModel; + } else if (models.length > 0) { + summaryModelSelect.value = models[0].value; + } else { + summaryModelSelect.value = ""; + } + + currentSummaryModel = typeof summaryModelSelect.value === "string" + ? summaryModelSelect.value.trim() + : ""; + } + + function setSummaryProvider(provider, preferredModel) { + if (summaryProviders.indexOf(provider) === -1) return; + currentSummaryProvider = provider; + + if (summaryProviderSelect) { + summaryProviderSelect.value = provider; + } + + populateSummaryModelSelect(provider, preferredModel); + } + + function initializeSummaryModelControls() { + buildSummaryModelState(); + renderSummaryProviderSelect(); + + if (summaryProviders.length === 0) { + currentSummaryProvider = ""; + currentSummaryModel = ""; + if (summaryProviderSelect) summaryProviderSelect.innerHTML = ""; + if (summaryModelSelect) { + summaryModelSelect.innerHTML = ''; + summaryModelSelect.value = ""; + } + return; + } + + var defaultProvider = getSummaryProvider(defaultSummaryModel); + if (defaultProvider && summaryProviders.indexOf(defaultProvider) !== -1) { + setSummaryProvider(defaultProvider, defaultSummaryModel); + return; + } + + setSummaryProvider(summaryProviders[0], ""); + } + + function getSelectedSummaryModel() { + if (!summaryModelSelect) return currentSummaryModel; + if (typeof summaryModelSelect.value !== "string") return currentSummaryModel; + currentSummaryModel = summaryModelSelect.value.trim(); + return currentSummaryModel; + } + + function getFeedbackText() { + if (!summaryFeedback || typeof summaryFeedback.value !== "string") return ""; + return summaryFeedback.value; + } + + function getCoverageSet(provider) { + var set = providerCoverage.get(provider); + if (set) return set; + set = new Set(); + providerCoverage.set(provider, set); + return set; + } + + function markCoverage(provider, slotId) { + if (typeof slotId !== "number") return; + var normalized = normalizeProvider(provider, ""); + if (!normalized) return; + getCoverageSet(normalized).add(slotId); + } + + function removeSlot(slotId) { + allQueries = allQueries.filter(function(slot) { return slot.slotId !== slotId; }); + + providerCoverage.forEach(function(coveredSlots) { + coveredSlots.delete(slotId); + }); + + queryIndexToSlot.forEach(function(mappedSlotId, qi) { + if (mappedSlotId === slotId) queryIndexToSlot.delete(qi); + }); + + syncLoadingPanel(); + } + + function isResultMutationLocked() { + return submitted || timerExpired || submitInFlight; + } + + function applyProviderInterlocks() { + var disableProviders = isResultMutationLocked() || providerBatchInFlight || addSearchInFlight; + for (var i = 0; i < providerButtons.length; i++) { + var btn = providerButtons[i]; + var state = btn.dataset.state || "idle"; + btn.disabled = disableProviders || state === "loading"; + } + + var disableAddSearch = isResultMutationLocked(); + if (addSearchInput) { + addSearchInput.disabled = disableAddSearch; + } + + if (addSearchEl) { + addSearchEl.style.opacity = disableAddSearch ? "0.6" : ""; + addSearchEl.style.pointerEvents = disableAddSearch ? "none" : ""; + } + + var cards = resultCardsEl ? resultCardsEl.querySelectorAll(".result-card") : []; + cards.forEach(function(card) { + var cb = card.querySelector("input[type=checkbox]"); + if (!cb) return; + var searching = card.classList.contains("searching"); + var error = card.classList.contains("error"); + cb.disabled = searching || error || isResultMutationLocked(); + }); + } + + function recomputeProviderStates() { + for (var i = 0; i < providerButtons.length; i++) { + var btn = providerButtons[i]; + var provider = normalizeProvider(btn.dataset.provider, ""); + if (!provider) continue; + + var state = "idle"; + if (providerBatchInFlight && batchLoadingProvider === provider) { + state = "loading"; + } else if (!initialStreamDone && queries.length > 0 && provider === initialDefaultProvider) { + state = "loading"; + } else if (allQueries.length > 0) { + var coveredSlots = providerCoverage.get(provider); + if (coveredSlots && coveredSlots.size >= allQueries.length) { + state = "searched"; + } + } + + btn.dataset.state = state; + btn.classList.remove("idle", "loading", "searched"); + btn.classList.add(state); + btn.classList.toggle("is-default", provider === currentProvider); + } + + applyProviderInterlocks(); + } + + function updateSummaryText() { + if (completedCount <= 0) return; + var totalCards = resultCardsEl.querySelectorAll(".result-card").length; + var searchingCount = totalCards - completedCount; + if (searchingCount > 0) { + heroTitle.textContent = completedCount + " of " + totalCards + " Searches Complete"; + } else { + heroTitle.textContent = completedCount + " Search" + (completedCount !== 1 ? "es" : "") + " Complete"; + } + heroDesc.textContent = "Check the results to include, then generate and approve a summary."; + if (heroStatus) heroStatus.textContent = completedCount + " completed" + (searchingCount > 0 ? ", " + searchingCount + " searching" : ""); + } + + function getSummaryDraftText() { + if (!summaryInput || typeof summaryInput.value !== "string") return ""; + return summaryInput.value.trim(); + } + + function clearError() { + if (!errorBanner) return; + errorBanner.hidden = true; + errorBanner.textContent = ""; + } + + function setError(text) { + if (!errorBanner) return; + errorBanner.textContent = text; + errorBanner.hidden = false; + } + + function updateSummaryGeneratingIndicator() { + if (!summaryGeneratingCopy) return; + + if (stage !== "generating-summary") { + summaryGeneratingCopy.textContent = "Generating summary draft…"; + summaryGeneratingPhase = -1; + if (summaryGeneratingEl) { + summaryGeneratingEl.removeAttribute("data-phase"); + } + return; + } + + if (summaryGeneratingStartedAt <= 0) { + summaryGeneratingStartedAt = Date.now(); + } + + var elapsedMs = Date.now() - summaryGeneratingStartedAt; + var nextPhase = Math.min(2, Math.floor(elapsedMs / 1800)); + if (nextPhase === summaryGeneratingPhase) return; + + summaryGeneratingPhase = nextPhase; + + var phaseLabel = "Planning summary"; + if (nextPhase === 1) phaseLabel = "Drafting summary"; + if (nextPhase === 2) phaseLabel = "Polishing summary"; + + summaryGeneratingCopy.textContent = summaryPendingModel + ? phaseLabel + " with " + summaryPendingModel + "…" + : phaseLabel + "…"; + + if (summaryGeneratingEl) { + summaryGeneratingEl.dataset.phase = String(nextPhase); + } + } + + function updateStageUI() { + var showSummary = stage === "summary-review" || stage === "generating-summary" || isRegenerating; + if (summaryPanel) { + summaryPanel.classList.toggle("hidden", !showSummary); + summaryPanel.classList.toggle("updating", isRegenerating); + } + if (summarySubtitle) { + var selCount = getSelectedIndices().length; + var selLabel = selCount + " selected result" + (selCount !== 1 ? "s" : ""); + if (isRegenerating && stage === "generating-summary") { + summarySubtitle.textContent = "Selection changed — regenerating summary…"; + } else if (isRegenerating) { + summarySubtitle.textContent = "Selection changed — summary will regenerate shortly…"; + } else if (stage === "generating-summary") { + summarySubtitle.textContent = summaryPendingModel + ? "Summarizing " + selLabel + " with " + summaryPendingModel + "…" + : "Summarizing " + selLabel + "…"; + } else if (summaryMeta && summaryMeta.fallbackUsed) { + summarySubtitle.textContent = "Fallback summary of " + selLabel + "."; + } else { + summarySubtitle.textContent = "Summary of " + selLabel + ". Edit directly, regenerate with feedback, or approve."; + } + } + + if (summaryGeneratingEl) { + var showGenerating = stage === "generating-summary" && !isRegenerating; + summaryGeneratingEl.classList.toggle("hidden", !showGenerating); + } + updateSummaryGeneratingIndicator(); + + if (summaryInput) { + summaryInput.classList.toggle("hidden", stage === "generating-summary" && !isRegenerating); + summaryInput.disabled = submitted || timerExpired || stage === "generating-summary" || submitInFlight || isRegenerating; + } + if (summaryFeedback) { + summaryFeedback.disabled = submitted || timerExpired || submitInFlight || stage === "generating-summary" || isRegenerating; + } + var disableSummaryModelControls = submitted || timerExpired || stage === "generating-summary" || submitInFlight || summaryProviders.length === 0; + if (summaryProviderSelect) { + summaryProviderSelect.disabled = disableSummaryModelControls; + } + if (summaryModelSelect) { + summaryModelSelect.disabled = disableSummaryModelControls; + } + + var inResults = stage === "results"; + var hasSelection = getSelectedIndices().length > 0; + var hasCompleted = getCompletedSelectableIndices().length > 0; + var canGenerate = inResults && !submitted && !timerExpired && !submitInFlight && hasCompleted; + + if (btnSend) { + if (stage === "generating-summary") { + btnSend.textContent = "Generating summary…"; + btnSend.disabled = true; + } else if (!inResults) { + btnSend.textContent = "Summary ready"; + btnSend.disabled = true; + } else if (!hasCompleted) { + btnSend.textContent = searchesDone ? "No results yet" : "Waiting for results…"; + btnSend.disabled = true; + } else { + btnSend.textContent = hasSelection ? "Generate summary" : "Select results to summarize"; + btnSend.disabled = !canGenerate || !hasSelection; + } + } + if (sendRawRow) { + sendRawRow.classList.toggle("hidden", !hasSelection || submitted || timerExpired); + } + if (btnSendRaw) { + btnSendRaw.disabled = !hasSelection || submitted || timerExpired || submitInFlight; + } + + if (btnSummaryBack) btnSummaryBack.disabled = submitted || timerExpired || submitInFlight || (stage === "generating-summary" && !isRegenerating); + if (btnSummaryRegenerate) btnSummaryRegenerate.disabled = submitted || timerExpired || submitInFlight || stage === "generating-summary" || isRegenerating; + var hasDraft = getSummaryDraftText().length > 0; + if (btnSummaryPreview) btnSummaryPreview.disabled = !hasDraft || stage === "generating-summary"; + if (btnSummaryApprove) { + btnSummaryApprove.disabled = submitted || timerExpired || submitInFlight || stage === "generating-summary" || isRegenerating || !hasSelection || !hasDraft; + } + + applyProviderInterlocks(); + } + + function shouldShowLoadingPanel() { + if (submitted || timerExpired || searchesDone) return false; + if (completedCount > 0) return false; + return allQueries.length > 0; + } + + function ensureLoadingPanel() { + if (loadingPanelEl) return loadingPanelEl; + if (!resultCardsEl) return null; + + var panel = document.createElement("div"); + panel.className = "result-loading"; + panel.innerHTML = + '
' + + '
Searching sources
' + + '
Searching\u2026
' + + '
' + + '
' + + '
' + + '
' + + '
'; + + resultCardsEl.prepend(panel); + loadingPanelEl = panel; + return panel; + } + + function updateLoadingPanelSummary() { + if (!loadingPanelEl) return; + var sub = loadingPanelEl.querySelector(".result-loading-sub"); + if (!sub) return; + + var total = allQueries.length; + if (total <= 0) { + sub.textContent = "Searching\u2026"; + return; + } + + var done = Math.min(completedCount, total); + var noun = total === 1 ? "query" : "queries"; + sub.textContent = "Searching " + done + "/" + total + " " + noun + "\u2026"; + } + + function syncLoadingPanel() { + if (shouldShowLoadingPanel()) { + if (!ensureLoadingPanel()) return; + updateLoadingPanelSummary(); + return; + } + + if (loadingPanelEl) { + loadingPanelEl.remove(); + loadingPanelEl = null; + } + } + + function renderErrorCard(card, queryText, errorText, provider) { + var tag = providerTagHtml(provider); + card.innerHTML = + '
' + + '' + + '
' + + '
' + + '
' + escHtml(queryText) + "
" + + tag + + "
" + + '
Failed
' + + "
" + + "
" + + '
' + escHtml(errorText || "Search failed") + "
"; + } + + function populateResultCard(card, data, queryText, provider) { + var sourceCount = data.results ? data.results.length : 0; + var domains = []; + if (data.results) { + for (var i = 0; i < Math.min(data.results.length, 3); i++) { + domains.push(data.results[i].domain); + } + } + var metaText = sourceCount + " source" + (sourceCount !== 1 ? "s" : ""); + if (domains.length > 0) metaText += " \u00B7 " + domains.join(", "); + if (sourceCount > 3) metaText += ", +" + (sourceCount - 3); + + var preview = ""; + if (data.answer) { + preview = data.answer.substring(0, 200).replace(/\\n+/g, " ").replace(/[#*_\\[\\]]/g, ""); + } + + var bodyHtml = ""; + if (data.answer) { + var rendered = typeof marked !== "undefined" && marked.parse + ? marked.parse(data.answer, { breaks: true }) + : "

" + escHtml(data.answer) + "

"; + bodyHtml += '
' + sanitizeMarkdownHtml(rendered) + "
"; + } + if (data.results && data.results.length > 0) { + bodyHtml += '
Sources
'; + for (var k = 0; k < data.results.length; k++) { + var r = data.results[k]; + var label = r.title && r.title.indexOf("Source ") !== 0 ? r.title : r.url; + var href = sanitizeHref(r.url); + bodyHtml += '' + escHtml(label) + '' + escHtml(r.domain) + ""; + } + bodyHtml += "
"; + } + + var altChipsHtml = buildAltChipsHtml(provider, queryText); + + card.innerHTML = + '
' + + '' + + '
' + + '
' + + '
' + escHtml(queryText) + "
" + + providerTagHtml(provider) + + "
" + + '
' + escHtml(metaText) + "
" + + (preview ? '
' + escHtml(preview) + "
" : "") + + "
" + + '
\u25BC
' + + "
" + + altChipsHtml + + '
' + bodyHtml + "
"; + } + + function applyResponseToCard(card, data, queryText, providerHint, slotHint) { + if (!card || !data) return; + if (submitted || timerExpired) return; + + var queryIndex = typeof data.queryIndex === "number" ? data.queryIndex : null; + if (queryIndex !== null) { + card.dataset.qi = String(queryIndex); + } + + var slotId = typeof slotHint === "number" ? slotHint : (queryIndex !== null ? queryIndexToSlot.get(queryIndex) : undefined); + if (typeof slotId !== "number" && queryIndex !== null) { + slotId = queryIndex; + } + if (queryIndex !== null && typeof slotId === "number") { + queryIndexToSlot.set(queryIndex, slotId); + } + + var provider = normalizeProvider(data.provider, providerHint); + + card.classList.remove("searching", "checked", "error"); + + if (data.error) { + card.classList.add("error"); + renderErrorCard(card, queryText, data.error, provider); + } else { + card.classList.add("checked"); + populateResultCard(card, data, queryText, provider); + setupCardInteraction(card); + } + + if (card.dataset.completed !== "true") { + completedCount++; + card.dataset.completed = "true"; + } + markCoverage(provider, slotId); + updateSummaryText(); + syncLoadingPanel(); + recomputeProviderStates(); + updateStageUI(); + maybeAutoGenerateSummary(); + resetTimer(); + } + + function resetTimer() { lastInteraction = Date.now(); } + + function updateTimer() { + var idleSec = Math.floor((Date.now() - lastInteraction) / 1000); + var remaining = Math.max(0, timeoutSec - idleSec); + timerEl.textContent = formatTime(remaining); + + timerEl.classList.remove("warn", "urgent", "active"); + if (remaining <= 15) timerEl.classList.add("urgent"); + else if (remaining <= 30) timerEl.classList.add("warn"); + else if (remaining < timeoutSec) timerEl.classList.add("active"); + + updateSummaryGeneratingIndicator(); + + if (remaining <= 0 && !submitted && !timerExpired) onTimeout(); + } + + setInterval(updateTimer, 1000); + updateTimer(); + + ["click", "keydown", "input", "change"].forEach(function(evt) { + document.addEventListener(evt, resetTimer, { passive: true }); + }); + document.addEventListener("scroll", resetTimer, { passive: true }); + document.addEventListener("mousemove", resetTimer, { passive: true }); + + timerEl.addEventListener("click", function(e) { + e.stopPropagation(); + timerInput.value = timeoutSec; + timerAdjustEl.classList.add("visible"); + timerEl.style.display = "none"; + timerInput.focus(); + timerInput.select(); + }); + + function applyTimerAdjust() { + var val = parseInt(timerInput.value, 10); + if (val && val > 0) timeoutSec = Math.min(val, 600); + timerAdjustEl.classList.remove("visible"); + timerEl.style.display = ""; + resetTimer(); + } + + timerSetBtn.addEventListener("click", function(e) { e.stopPropagation(); applyTimerAdjust(); }); + timerInput.addEventListener("keydown", function(e) { + if (e.key === "Enter") { e.preventDefault(); applyTimerAdjust(); } + if (e.key === "Escape") { timerAdjustEl.classList.remove("visible"); timerEl.style.display = ""; } + e.stopPropagation(); + }); + document.addEventListener("click", function() { + if (timerAdjustEl.classList.contains("visible")) { + timerAdjustEl.classList.remove("visible"); + timerEl.style.display = ""; + } + }); + + function setDefaultProvider(provider, persist) { + var normalized = normalizeProvider(provider, currentProvider); + if (!normalized) return; + currentProvider = normalized; + recomputeProviderStates(); + if (persist) { + postJson("/provider", { provider: normalized }).then(function(data) { + if (data && data.ok === false) { + throw new Error(extractServerError(data) || "request rejected"); + } + }).catch(function(err) { + var message = err instanceof Error ? err.message : String(err); + setError("Failed to save provider preference: " + (message || "unknown error")); + }); + } + } + + providerButtons.forEach(function(btn) { + btn.addEventListener("click", function() { + if (isResultMutationLocked()) return; + if (providerBatchInFlight || addSearchInFlight) return; + + var provider = normalizeProvider(btn.dataset.provider, ""); + if (!provider) return; + + var state = btn.dataset.state || "idle"; + if (state === "loading") return; + + if (state === "searched") { + if (provider === currentProvider) return; + setDefaultProvider(provider, true); + resetTimer(); + return; + } + + setDefaultProvider(provider, true); + if (allQueries.length === 0) { + resetTimer(); + return; + } + + interruptSummaryIfNeeded(); + providerBatchInFlight = true; + batchLoadingProvider = provider; + recomputeProviderStates(); + + var batchQueries = allQueries.slice(); + var inflight = batchQueries.length; + if (inflight === 0) { + providerBatchInFlight = false; + batchLoadingProvider = null; + recomputeProviderStates(); + return; + } + + var batchCards = []; + for (var bi = 0; bi < batchQueries.length; bi++) { + var bq = batchQueries[bi]; + var card = document.createElement("div"); + card.className = "result-card searching"; + card.innerHTML = + '
' + + '' + + '
' + + '
' + + '
' + escHtml(bq.query) + "
" + + providerTagHtml(provider) + + "
" + + '
Searching
' + + "
" + + "
" + + buildAltChipsHtml(provider, bq.query); + resultCardsEl.appendChild(card); + batchCards.push(card); + } + updateSummaryText(); + + batchQueries.forEach(function(slot, si) { + var searchingCard = batchCards[si]; + postJson("/search", { query: slot.query, provider: provider }) + .then(function(data) { + if (submitted || timerExpired) return; + if (!data || data.ok === false) { + applyResponseToCard(searchingCard, { + answer: "", + results: [], + error: extractServerError(data) || "Search failed", + provider: provider, + }, slot.query, provider, slot.slotId); + return; + } + applyResponseToCard(searchingCard, data, slot.query, provider, slot.slotId); + }) + .catch(function(err) { + if (submitted || timerExpired) return; + var message = err instanceof Error ? err.message : String(err); + applyResponseToCard(searchingCard, { + answer: "", + results: [], + error: message || "Search failed", + provider: provider, + }, slot.query, provider, slot.slotId); + }) + .finally(function() { + inflight -= 1; + if (inflight <= 0) { + providerBatchInFlight = false; + batchLoadingProvider = null; + recomputeProviderStates(); + updateStageUI(); + maybeAutoGenerateSummary(); + } + }); + }); + + resetTimer(); + }); + }); + + if (resultCardsEl) { + resultCardsEl.addEventListener("click", function(e) { + if (!(e.target instanceof Element)) return; + var chip = e.target.closest(".card-alt-chip"); + if (!chip) return; + if (isResultMutationLocked()) return; + + var altProvider = chip.dataset.altProvider; + var altQuery = chip.dataset.altQuery; + if (!altProvider || !altQuery) return; + + interruptSummaryIfNeeded(); + + chip.classList.add("loading"); + chip.disabled = true; + resetTimer(); + + var slotId = nextSlotId++; + allQueries.push({ slotId: slotId, query: altQuery }); + + var parentCard = chip.closest(".result-card"); + var newCard = document.createElement("div"); + newCard.className = "result-card searching"; + newCard.innerHTML = + '
' + + '' + + '
' + + '
' + + '
' + escHtml(altQuery) + "
" + + providerTagHtml(altProvider) + + "
" + + '
Searching
' + + "
" + + "
" + + buildAltChipsHtml(altProvider, altQuery); + if (parentCard && parentCard.nextSibling) { + resultCardsEl.insertBefore(newCard, parentCard.nextSibling); + } else { + resultCardsEl.appendChild(newCard); + } + updateSummaryText(); + + postJson("/search", { query: altQuery, provider: altProvider }) + .then(function(data) { + if (submitted || timerExpired) return; + if (!data || data.ok === false) { + applyResponseToCard(newCard, { + answer: "", results: [], + error: extractServerError(data) || "Search failed", + provider: altProvider, + }, altQuery, altProvider, slotId); + return; + } + applyResponseToCard(newCard, data, altQuery, altProvider, slotId); + }) + .catch(function(err) { + removeSlot(slotId); + newCard.remove(); + var message = err instanceof Error ? err.message : String(err); + setError("Re-search failed: " + (message || "Search failed")); + updateSummaryText(); + }) + .finally(function() { + chip.classList.remove("loading"); + chip.disabled = false; + recomputeProviderStates(); + updateStageUI(); + maybeAutoGenerateSummary(); + }); + }); + } + + if (addSearchInput && addSearchWand) { + addSearchInput.addEventListener("input", function() { + addSearchWand.disabled = rewriteInFlight || !addSearchInput.value.trim() || isResultMutationLocked(); + }); + + addSearchWand.addEventListener("click", function() { + var text = addSearchInput.value.trim(); + if (!text || rewriteInFlight || isResultMutationLocked()) return; + rewriteInFlight = true; + addSearchWand.disabled = true; + addSearchWand.classList.add("rewriting"); + resetTimer(); + + postJson("/rewrite", { query: text }) + .then(function(data) { + if (!data || data.ok === false) { + throw new Error(extractServerError(data) || "Rewrite failed"); + } + var rewritten = typeof data.query === "string" ? data.query.trim() : ""; + if (rewritten) { + addSearchInput.value = rewritten; + addSearchInput.focus(); + } + }) + .catch(function(err) { + var message = err instanceof Error ? err.message : String(err); + setError("Rewrite failed: " + (message || "unknown error")); + }) + .finally(function() { + rewriteInFlight = false; + addSearchWand.classList.remove("rewriting"); + addSearchWand.disabled = !addSearchInput.value.trim() || isResultMutationLocked(); + }); + }); + } + + addSearchInput.addEventListener("keydown", function(e) { + if (e.key !== "Enter") return; + var text = addSearchInput.value.trim(); + if (!text || isResultMutationLocked()) return; + interruptSummaryIfNeeded(); + e.preventDefault(); + e.stopPropagation(); + + addSearchInFlight++; + applyProviderInterlocks(); + addSearchInput.value = ""; + + var slotId = nextSlotId++; + allQueries.push({ slotId: slotId, query: text }); + syncLoadingPanel(); + recomputeProviderStates(); + + var requestedProvider = currentProvider; + + var card = document.createElement("div"); + card.className = "result-card searching"; + card.innerHTML = + '
' + + '' + + '
' + + '
' + + '
' + escHtml(text) + "
" + + providerTagHtml(requestedProvider) + + "
" + + '
Searching
' + + "
" + + "
" + + buildAltChipsHtml(requestedProvider, text); + resultCardsEl.appendChild(card); + updateSummaryText(); + resetTimer(); + + postJson("/search", { query: text, provider: requestedProvider }) + .then(function(data) { + if (!data || data.ok === false) { + removeSlot(slotId); + card.remove(); + setError("Failed to add search: " + (extractServerError(data) || "Search failed")); + recomputeProviderStates(); + updateSummaryText(); + return; + } + + if (submitted || timerExpired) return; + + applyResponseToCard(card, data, text, requestedProvider, slotId); + }) + .catch(function(err) { + removeSlot(slotId); + card.remove(); + var message = err instanceof Error ? err.message : String(err); + setError("Failed to add search: " + (message || "Search failed")); + recomputeProviderStates(); + updateSummaryText(); + }) + .finally(function() { + addSearchInFlight--; + recomputeProviderStates(); + updateStageUI(); + maybeAutoGenerateSummary(); + }); + }); + + function showSuccess(text) { + if (es) { es.close(); es = null; } + closePreviewModal(); + successText.textContent = text; + successOverlay.classList.remove("hidden"); + setTimeout(function() { window.close(); }, 800); + } + + function showExpired(text) { + if (es) { es.close(); es = null; } + closePreviewModal(); + expiredText.textContent = text; + expiredOverlay.classList.remove("hidden"); + requestAnimationFrame(function() { expiredOverlay.classList.add("visible"); }); + } + + function startOverlayCloseCountdown(seconds) { + var count = seconds; + closeCountdown.textContent = count; + var iv = setInterval(function() { + count--; + closeCountdown.textContent = count; + if (count <= 0) { + clearInterval(iv); + window.close(); + } + }, 1000); + } + + function submitPayload(payload, successText) { + if (submitInFlight) return Promise.reject(new Error("Submit already in progress")); + submitInFlight = true; + submitted = true; + syncLoadingPanel(); + updateStageUI(); + clearError(); + + return postJson("/submit", payload) + .then(function(data) { + if (data && data.ok === false) { + throw new Error(extractServerError(data) || "submit rejected"); + } + showSuccess(successText); + }) + .catch(function(err) { + submitInFlight = false; + submitted = false; + syncLoadingPanel(); + updateStageUI(); + throw err; + }); + } + + function submitWithTimeoutFallback(payload) { + if (submitInFlight) return; + submitInFlight = true; + submitted = true; + timerExpired = true; + syncLoadingPanel(); + updateStageUI(); + clearError(); + showExpired("Time\u2019s up \u2014 submitting current summary state."); + + function finalizeClose() { + submitInFlight = false; + startOverlayCloseCountdown(5); + } + + function toErrorMessage(err) { + return err instanceof Error ? err.message : String(err); + } + + function attemptCancelFallback(submitErrorMessage) { + return postJson("/cancel", { reason: "timeout" }) + .catch(function(cancelErr) { + console.error("Timeout finalize failed after submit errors:", submitErrorMessage, "| cancel:", toErrorMessage(cancelErr)); + }) + .finally(finalizeClose); + } + + postJson("/submit", payload) + .then(function(data) { + if (data && data.ok === false) { + throw new Error(extractServerError(data) || "submit rejected"); + } + finalizeClose(); + }) + .catch(function(firstErr) { + var firstMessage = toErrorMessage(firstErr); + setTimeout(function() { + postJson("/submit", payload) + .then(function(data) { + if (data && data.ok === false) { + throw new Error(extractServerError(data) || "submit rejected"); + } + finalizeClose(); + }) + .catch(function(secondErr) { + var secondMessage = toErrorMessage(secondErr); + attemptCancelFallback(firstMessage + " | " + secondMessage); + }); + }, 250); + }); + } + + function onTimeout() { + if (submitted || timerExpired) return; + var timeoutSelected = getTimeoutSelectedIndices(); + var payload = { selected: timeoutSelected }; + var draft = getSummaryDraftText(); + if (stage === "summary-review" && draft.length > 0) { + payload.summary = draft; + if (summaryMeta) payload.summaryMeta = summaryMeta; + } + submitWithTimeoutFallback(payload); + } + + if (queries.length === 0) { + heroTitle.textContent = "What do you need?"; + heroDesc.textContent = "Search for anything below, then generate and approve a summary."; + if (heroStatus) heroStatus.textContent = ""; + btnSend.textContent = "No results yet"; + } else { + for (var i = 0; i < queries.length; i++) { + queryIndexToSlot.set(i, i); + var card = document.createElement("div"); + card.className = "result-card searching"; + card.dataset.qi = i; + card.innerHTML = + '
' + + '' + + '
' + + '
' + + '
' + escHtml(queries[i]) + "
" + + providerTagHtml(initialDefaultProvider) + + "
" + + '
Searching
' + + "
" + + "
" + + buildAltChipsHtml(initialDefaultProvider, queries[i]); + resultCardsEl.appendChild(card); + } + } + + initializeSummaryModelControls(); + syncLoadingPanel(); + recomputeProviderStates(); + updateStageUI(); + + es = new EventSource("/events?session=" + encodeURIComponent(token)); + + function parseSseEventData(eventName, e) { + try { + return JSON.parse(e.data); + } catch (err) { + var message = err instanceof Error ? err.message : String(err); + setError("Invalid " + eventName + " event payload: " + (message || "unknown parse error")); + return null; + } + } + + es.addEventListener("result", function(e) { + var data = parseSseEventData("result", e); + if (!data) return; + + var card = resultCardsEl.querySelector('.result-card[data-qi="' + data.queryIndex + '"]'); + if (!card) return; + + var slotId = queryIndexToSlot.get(data.queryIndex); + if (typeof slotId !== "number") slotId = data.queryIndex; + applyResponseToCard(card, data, data.query || queries[data.queryIndex], data.provider, slotId); + }); + + es.addEventListener("search-error", function(e) { + var data = parseSseEventData("search-error", e); + if (!data) return; + + var card = resultCardsEl.querySelector('.result-card[data-qi="' + data.queryIndex + '"]'); + if (!card) return; + + var slotId = queryIndexToSlot.get(data.queryIndex); + if (typeof slotId !== "number") slotId = data.queryIndex; + applyResponseToCard(card, { + queryIndex: data.queryIndex, + answer: "", + results: [], + error: data.error || "Search failed", + provider: data.provider, + }, data.query || queries[data.queryIndex], data.provider, slotId); + }); + + es.addEventListener("done", function() { + searchesDone = true; + initialStreamDone = true; + if (completedCount > 0) { + updateSummaryText(); + } + syncLoadingPanel(); + recomputeProviderStates(); + updateStageUI(); + maybeAutoGenerateSummary(); + resetTimer(); + }); + + es.onerror = function() { + // EventSource reconnects automatically. + }; + + function setupCardInteraction(card) { + var header = card.querySelector(".result-card-header"); + var body = card.querySelector(".result-card-body"); + var cb = card.querySelector("input[type=checkbox]"); + var expandEl = card.querySelector(".result-card-expand"); + + if (!header || !cb) return; + + header.addEventListener("click", function(e) { + if (e.target.tagName === "A") return; + if (e.target === cb) { + if (isResultMutationLocked()) { + e.preventDefault(); + return; + } + card.classList.toggle("checked", cb.checked); + if (stage === "summary-review" || stage === "generating-summary") { + interruptSummaryIfNeeded(); + } + updateStageUI(); + maybeAutoGenerateSummary(); + return; + } + var isExpanded = body && body.classList.contains("open"); + if (body) body.classList.toggle("open"); + if (expandEl) expandEl.textContent = isExpanded ? "\u25BC" : "\u25B2"; + }); + + if (body) { + body.addEventListener("click", function(e) { + e.stopPropagation(); + }); + } + } + + function getSelectedIndices() { + var indices = []; + var cards = resultCardsEl.querySelectorAll(".result-card"); + cards.forEach(function(card) { + if (card.dataset.completed !== "true") return; + if (card.classList.contains("error")) return; + var cb = card.querySelector("input[type=checkbox]"); + if (!cb || !cb.checked) return; + var qi = parseInt(card.dataset.qi, 10); + if (!Number.isNaN(qi)) indices.push(qi); + }); + return indices; + } + + function getCompletedSelectableIndices() { + var indices = []; + var cards = resultCardsEl.querySelectorAll(".result-card"); + cards.forEach(function(card) { + if (card.dataset.completed !== "true") return; + if (card.classList.contains("error")) return; + var qi = parseInt(card.dataset.qi, 10); + if (!Number.isNaN(qi)) indices.push(qi); + }); + return indices; + } + + function hasPendingSearchCards() { + var cards = resultCardsEl.querySelectorAll(".result-card"); + for (var i = 0; i < cards.length; i++) { + var card = cards[i]; + if (card.dataset.completed !== "true") return true; + } + return addSearchInFlight || providerBatchInFlight; + } + + function getTimeoutSelectedIndices() { + var selected = getSelectedIndices(); + if (selected.length > 0) return selected; + return getCompletedSelectableIndices(); + } + + function normalizeSummaryMeta(meta, edited) { + if (!meta || typeof meta !== "object") { + return { + model: null, + durationMs: 0, + tokenEstimate: 0, + fallbackUsed: false, + edited: !!edited, + }; + } + + return { + model: typeof meta.model === "string" || meta.model === null ? meta.model : null, + durationMs: typeof meta.durationMs === "number" && Number.isFinite(meta.durationMs) && meta.durationMs >= 0 ? meta.durationMs : 0, + tokenEstimate: typeof meta.tokenEstimate === "number" && Number.isFinite(meta.tokenEstimate) && meta.tokenEstimate >= 0 ? meta.tokenEstimate : 0, + fallbackUsed: meta.fallbackUsed === true, + fallbackReason: typeof meta.fallbackReason === "string" ? meta.fallbackReason : undefined, + edited: !!edited, + }; + } + + function isSummaryModelSelectionError(message) { + if (typeof message !== "string") return false; + return message.indexOf("Invalid summary model") !== -1 + || message.indexOf("Summary model not found") !== -1 + || message.indexOf("No API key available for summary model") !== -1 + || message.indexOf("Invalid provider") !== -1; + } + + function resetSummaryGeneratingState() { + summaryPendingModel = ""; + summaryGeneratingStartedAt = 0; + summaryGeneratingPhase = -1; + } + + function cancelInFlightSummaryRequest() { + summaryRequestSeq += 1; + resetSummaryGeneratingState(); + } + + function interruptSummaryIfNeeded() { + if (stage !== "generating-summary" && stage !== "summary-review") return; + if (stage === "generating-summary") { + cancelInFlightSummaryRequest(); + } + clearError(); + isRegenerating = getSummaryDraftText().length > 0; + stage = "results"; + updateStageUI(); + } + + function exitRegeneratingState() { + if (!isRegenerating) return false; + if (stage === "generating-summary") { + cancelInFlightSummaryRequest(); + } + isRegenerating = false; + clearError(); + stage = "results"; + updateStageUI(); + return true; + } + + function requestSummary(indices, feedback) { + if (submitted || timerExpired || submitInFlight) return; + + if (!Array.isArray(indices) || indices.length === 0) { + setError("Select at least one result to summarize"); + stage = "results"; + updateStageUI(); + return; + } + + if (hasPendingSearchCards()) { + setError("Wait for running searches to finish before generating summary"); + stage = "results"; + updateStageUI(); + return; + } + + clearError(); + var previousStage = stage; + var wasRegenerating = isRegenerating; + var selectedSummaryModel = getSelectedSummaryModel(); + summaryPendingModel = selectedSummaryModel; + summaryGeneratingStartedAt = Date.now(); + summaryGeneratingPhase = -1; + stage = "generating-summary"; + updateStageUI(); + + var requestId = ++summaryRequestSeq; + var feedbackText = typeof feedback === "string" ? feedback.trim() : ""; + var summarizePayload = { selected: indices }; + if (selectedSummaryModel.length > 0) { + summarizePayload.model = selectedSummaryModel; + } + if (feedbackText.length > 0) { + summarizePayload.feedback = feedbackText; + } + + postJson("/summarize", summarizePayload) + .then(function(data) { + if (requestId !== summaryRequestSeq) return data; + if (!data || data.ok === false) { + throw new Error(extractServerError(data) || "summary request rejected"); + } + return data; + }) + .catch(function(err) { + if (requestId !== summaryRequestSeq) throw err; + + var firstMessage = err instanceof Error ? err.message : String(err); + if (selectedSummaryModel.length === 0 || !isSummaryModelSelectionError(firstMessage)) { + throw err; + } + + summaryPendingModel = ""; + updateStageUI(); + + var retryPayload = { selected: indices }; + if (feedbackText.length > 0) { + retryPayload.feedback = feedbackText; + } + return postJson("/summarize", retryPayload).then(function(retryData) { + if (!retryData || retryData.ok === false) { + throw new Error(extractServerError(retryData) || "summary request rejected"); + } + return retryData; + }).catch(function(retryErr) { + var retryMessage = retryErr instanceof Error ? retryErr.message : String(retryErr); + throw new Error(firstMessage + " (auto retry failed: " + (retryMessage || "unknown error") + ")"); + }); + }) + .then(function(data) { + if (requestId !== summaryRequestSeq) return; + + var summaryText = typeof data.summary === "string" ? data.summary.trim() : ""; + if (!summaryText) { + throw new Error("Summary response was empty"); + } + + if (summaryInput) { + summaryInput.value = summaryText; + } + if (summaryFeedback) { + summaryFeedback.value = ""; + } + summaryMeta = normalizeSummaryMeta(data.meta || null, false); + lastAutoSummarySignature = selectionSignature(indices); + resetSummaryGeneratingState(); + isRegenerating = false; + stage = "summary-review"; + updateStageUI(); + }) + .catch(function(err) { + if (requestId !== summaryRequestSeq) return; + var message = err instanceof Error ? err.message : String(err); + setError("Failed to generate summary — " + (message || "unknown error")); + resetSummaryGeneratingState(); + isRegenerating = false; + if (wasRegenerating && getSummaryDraftText().length > 0) { + stage = "summary-review"; + } else { + stage = previousStage === "summary-review" ? "summary-review" : "results"; + } + updateStageUI(); + }); + } + + function selectionSignature(indices) { + return indices.slice().sort(function(a, b) { return a - b; }).join(","); + } + + function maybeAutoGenerateSummary() { + if (workflow !== "summary-review") return; + if (!searchesDone) return; + if (stage !== "results") return; + if (submitted || timerExpired || submitInFlight) return; + if (hasPendingSearchCards()) return; + + var selected = getSelectedIndices(); + if (selected.length === 0) { + if (isRegenerating) { + isRegenerating = false; + updateStageUI(); + } + return; + } + + var signature = selectionSignature(selected); + if (signature === lastAutoSummarySignature) { + if (isRegenerating) { + isRegenerating = false; + if (getSummaryDraftText().length > 0) { + stage = "summary-review"; + } + updateStageUI(); + } + return; + } + + lastAutoSummarySignature = signature; + requestSummary(selected); + } + + function doApprove() { + if (submitted || timerExpired || submitInFlight || stage !== "summary-review") return; + + var selected = getSelectedIndices(); + if (selected.length === 0) { + setError("Select at least one result before approving"); + updateStageUI(); + return; + } + + var draft = getSummaryDraftText(); + var payload = { selected: selected }; + if (draft.length > 0) { + payload.summary = draft; + payload.summaryMeta = normalizeSummaryMeta(summaryMeta, summaryMeta && summaryMeta.edited === true); + } + + submitPayload(payload, "Summary approved") + .catch(function(err) { + var message = err instanceof Error ? err.message : String(err); + setError("Failed to approve summary — " + (message || "the agent may have moved on")); + }); + } + + function doCancel() { + if (submitted || timerExpired || submitInFlight) return; + submitted = true; + submitInFlight = true; + syncLoadingPanel(); + updateStageUI(); + clearError(); + + postJson("/cancel", { reason: "user" }) + .then(function(data) { + if (data && data.ok === false) { + throw new Error(extractServerError(data) || "cancel rejected"); + } + showSuccess("Skipped"); + }) + .catch(function(err) { + submitted = false; + submitInFlight = false; + syncLoadingPanel(); + updateStageUI(); + var message = err instanceof Error ? err.message : String(err); + setError("Failed to cancel — " + (message || "the agent may have moved on")); + }); + } + + btnSend.addEventListener("click", function() { + if (stage !== "results") return; + requestSummary(getSelectedIndices()); + }); + + if (btnSendRaw) { + btnSendRaw.addEventListener("click", function() { + var selected = getSelectedIndices(); + if (selected.length === 0) return; + submitPayload({ selected: selected, rawResults: true }, "Results sent") + .catch(function(err) { + var message = err instanceof Error ? err.message : String(err); + setError("Failed to send results — " + (message || "the agent may have moved on")); + }); + }); + } + + if (btnSummaryBack) { + btnSummaryBack.addEventListener("click", function() { + if (exitRegeneratingState()) { + resetTimer(); + return; + } + if (stage !== "summary-review") return; + clearError(); + stage = "results"; + updateStageUI(); + resetTimer(); + }); + } + + if (btnSummaryRegenerate) { + btnSummaryRegenerate.addEventListener("click", function() { + requestSummary(getSelectedIndices(), getFeedbackText()); + resetTimer(); + }); + } + + function openPreviewModal() { + var draft = getSummaryDraftText(); + if (!draft || !previewModal || !previewModalBody) return; + var rendered = typeof marked !== "undefined" && marked.parse + ? marked.parse(draft, { breaks: true }) + : "
" + escHtml(draft) + "
"; + previewModalBody.innerHTML = sanitizeMarkdownHtml(rendered); + if (previewModalModel) { + previewModalModel.innerHTML = ''; + for (var i = 0; i < summaryModels.length; i++) { + var m = summaryModels[i]; + var opt = document.createElement("option"); + opt.value = m.value; + opt.textContent = m.label; + previewModalModel.appendChild(opt); + } + previewModalModel.value = getSelectedSummaryModel() || ""; + } + previewModal.classList.remove("hidden"); + resetTimer(); + } + + function closePreviewModal() { + if (previewModal) previewModal.classList.add("hidden"); + if (previewModalBody) previewModalBody.innerHTML = ""; + hidePreviewPopover(); + } + + var popoverSelectedText = ""; + + function hidePreviewPopover() { + if (previewPopover) previewPopover.classList.add("hidden"); + if (previewPopoverInput) previewPopoverInput.value = ""; + popoverSelectedText = ""; + } + + function showPreviewPopover(text, rect) { + if (!previewPopover || !previewPopoverQuote || !previewModalBody) return; + popoverSelectedText = text; + var display = text.length > 120 ? text.slice(0, 117) + "\u2026" : text; + previewPopoverQuote.textContent = "\u201c" + display + "\u201d"; + if (previewPopoverInput) previewPopoverInput.value = ""; + previewPopover.classList.remove("hidden"); + + var bodyRect = previewModalBody.getBoundingClientRect(); + var popH = previewPopover.offsetHeight; + var top = rect.bottom - bodyRect.top + previewModalBody.scrollTop + 6; + if (rect.bottom + popH + 20 > bodyRect.bottom) { + top = rect.top - bodyRect.top + previewModalBody.scrollTop - popH - 6; + } + var left = Math.max(8, Math.min(rect.left - bodyRect.left, bodyRect.width - previewPopover.offsetWidth - 8)); + previewPopover.style.top = top + "px"; + previewPopover.style.left = left + "px"; + + if (previewPopoverInput) previewPopoverInput.focus(); + } + + if (btnSummaryPreview) { + btnSummaryPreview.addEventListener("click", openPreviewModal); + } + if (previewModalClose) { + previewModalClose.addEventListener("click", closePreviewModal); + } + if (previewModalRegenerate) { + previewModalRegenerate.addEventListener("click", function() { + var selectedModel = previewModalModel ? previewModalModel.value.trim() : ""; + closePreviewModal(); + var modelProvider = getSummaryProvider(selectedModel); + if (modelProvider && modelProvider !== currentSummaryProvider) { + setSummaryProvider(modelProvider, selectedModel); + } else if (summaryModelSelect) { + summaryModelSelect.value = selectedModel; + currentSummaryModel = selectedModel; + } + requestSummary(getSelectedIndices(), getFeedbackText()); + resetTimer(); + }); + } + if (previewModalApprove) { + previewModalApprove.addEventListener("click", function() { + closePreviewModal(); + doApprove(); + }); + } + if (previewModalBody) { + previewModalBody.addEventListener("mouseup", function() { + var sel = window.getSelection(); + if (!sel || sel.isCollapsed) return; + var text = sel.toString().trim(); + if (!text) return; + var range = sel.getRangeAt(0); + showPreviewPopover(text, range.getBoundingClientRect()); + }); + previewModalBody.addEventListener("mousedown", function(e) { + if (previewPopover && !previewPopover.contains(e.target)) { + hidePreviewPopover(); + } + }); + } + + if (previewPopoverRegen) { + previewPopoverRegen.addEventListener("click", function() { + var note = previewPopoverInput ? previewPopoverInput.value.trim() : ""; + var quoted = popoverSelectedText; + hidePreviewPopover(); + + var feedback = 'Regarding: "' + quoted + '"'; + if (note) feedback += " \u2014 " + note; + + var selectedModel = previewModalModel ? previewModalModel.value.trim() : ""; + closePreviewModal(); + var modelProvider = getSummaryProvider(selectedModel); + if (modelProvider && modelProvider !== currentSummaryProvider) { + setSummaryProvider(modelProvider, selectedModel); + } else if (summaryModelSelect) { + summaryModelSelect.value = selectedModel; + currentSummaryModel = selectedModel; + } + requestSummary(getSelectedIndices(), feedback); + resetTimer(); + }); + } + + if (previewPopoverInput) { + previewPopoverInput.addEventListener("keydown", function(e) { + if (e.key === "Enter" && (e.metaKey || e.ctrlKey)) { + e.preventDefault(); + if (previewPopoverRegen) previewPopoverRegen.click(); + } + if (e.key === "Escape") { + e.preventDefault(); + e.stopImmediatePropagation(); + hidePreviewPopover(); + } + }); + } + + if (previewModal) { + previewModal.addEventListener("click", function(e) { + if (e.target === previewModal) closePreviewModal(); + }); + document.addEventListener("keydown", function(e) { + if (e.key === "Escape" && !previewModal.classList.contains("hidden")) { + if (previewPopover && !previewPopover.classList.contains("hidden")) { + e.preventDefault(); + e.stopImmediatePropagation(); + hidePreviewPopover(); + return; + } + e.preventDefault(); + e.stopImmediatePropagation(); + closePreviewModal(); + } + }); + } + + if (btnSummaryApprove) { + btnSummaryApprove.addEventListener("click", function() { + doApprove(); + resetTimer(); + }); + } + + if (summaryInput) { + summaryInput.addEventListener("input", function() { + if (!summaryMeta || typeof summaryMeta !== "object") { + summaryMeta = normalizeSummaryMeta(null, true); + } + summaryMeta.edited = true; + clearError(); + updateStageUI(); + resetTimer(); + }); + } + + if (summaryProviderSelect) { + summaryProviderSelect.addEventListener("change", function() { + var provider = typeof summaryProviderSelect.value === "string" ? summaryProviderSelect.value : ""; + if (!provider || provider === currentSummaryProvider) return; + setSummaryProvider(provider, ""); + clearError(); + updateStageUI(); + resetTimer(); + }); + } + + if (summaryModelSelect) { + summaryModelSelect.addEventListener("change", function() { + currentSummaryModel = typeof summaryModelSelect.value === "string" + ? summaryModelSelect.value.trim() + : ""; + clearError(); + resetTimer(); + }); + } + + function isInteractiveTarget(target) { + if (!target || !target.tagName) return false; + var tag = target.tagName; + if (tag === "INPUT" || tag === "TEXTAREA" || tag === "SELECT" || tag === "BUTTON" || tag === "A") return true; + if (typeof target.isContentEditable === "boolean" && target.isContentEditable) return true; + if (typeof target.closest === "function") { + return !!target.closest('[contenteditable=""], [contenteditable="true"]'); + } + return false; + } + + document.addEventListener("keydown", function(e) { + if (submitted || timerExpired || submitInFlight) return; + + var isSummaryInput = summaryInput && e.target === summaryInput; + if (isSummaryInput && (e.metaKey || e.ctrlKey) && e.key === "Enter") { + e.preventDefault(); + if (stage === "summary-review") doApprove(); + return; + } + + if (e.key === "Escape") { + e.preventDefault(); + if (exitRegeneratingState()) { + return; + } + if (stage === "summary-review") { + stage = "results"; + clearError(); + updateStageUI(); + } else if (stage === "results") { + doCancel(); + } + return; + } + + if (isInteractiveTarget(e.target)) return; + + if (e.key === "Enter" && !e.metaKey && !e.ctrlKey) { + if (stage !== "results") return; + e.preventDefault(); + requestSummary(getSelectedIndices()); + return; + } + + if ((e.metaKey || e.ctrlKey) && e.key === "Enter") { + if (stage !== "summary-review") return; + e.preventDefault(); + doApprove(); + return; + } + + if (e.key.toLowerCase() === "a" && !e.metaKey && !e.ctrlKey) { + e.preventDefault(); + if (stage !== "results") return; + var boxes = resultCardsEl.querySelectorAll(".result-card input[type=checkbox]"); + var selectable = []; + boxes.forEach(function(cb) { + if (cb.disabled) return; + selectable.push(cb); + }); + if (selectable.length === 0) return; + var allChecked = true; + selectable.forEach(function(cb) { if (!cb.checked) allChecked = false; }); + selectable.forEach(function(cb) { + cb.checked = !allChecked; + var parentCard = typeof cb.closest === "function" ? cb.closest(".result-card") : null; + if (parentCard) parentCard.classList.toggle("checked", cb.checked); + }); + updateStageUI(); + maybeAutoGenerateSummary(); + resetTimer(); + } + }); + + setInterval(function() { + if (submitted) return; + postJson("/heartbeat", {}).catch(function() { + // Heartbeat is best-effort. + }); + }, 10000); + + var lastResizeHeight = 0; + function checkContentHeight() { + if (!window.glimpse || typeof window.glimpse.send !== "function") return; + var h = document.documentElement.scrollHeight || document.body.scrollHeight; + if (h > 0 && Math.abs(h - lastResizeHeight) > 30) { + lastResizeHeight = h; + window.glimpse.send({ type: "resize", height: h }); + } + } + setInterval(checkContentHeight, 500); + + if (queries.length === 0 && addSearchInput) { + addSearchInput.focus(); + } +})();`; diff --git a/packages/web-access/curator-server.ts b/packages/web-access/curator-server.ts new file mode 100644 index 000000000..888080ec9 --- /dev/null +++ b/packages/web-access/curator-server.ts @@ -0,0 +1,605 @@ +import http, { type IncomingMessage, type ServerResponse } from "node:http"; +import { generateCuratorPage } from "./curator-page.js"; +import type { SummaryMeta } from "./summary-review.js"; + +const STALE_THRESHOLD_MS = 30000; +const WATCHDOG_INTERVAL_MS = 5000; +const MAX_BODY_SIZE = 64 * 1024; + +type ServerState = "SEARCHING" | "RESULT_SELECTION" | "COMPLETED"; + +export interface CuratorServerOptions { + queries: string[]; + sessionToken: string; + timeout: number; + availableProviders: { perplexity: boolean; exa: boolean; gemini: boolean }; + defaultProvider: string; + summaryModels: Array<{ value: string; label: string }>; + defaultSummaryModel: string | null; +} + +export interface CuratorServerCallbacks { + onSubmit: (payload: { selectedQueryIndices: number[]; summary?: string; summaryMeta?: SummaryMeta; rawResults?: boolean }) => void; + onCancel: (reason: "user" | "timeout" | "stale") => void; + onProviderChange: (provider: string) => void; + onAddSearch: (query: string, queryIndex: number, provider?: string) => Promise<{ + answer: string; + results: Array<{ title: string; url: string; domain: string }>; + provider: string; + }>; + onSummarize: ( + selectedQueryIndices: number[], + signal: AbortSignal, + model?: string, + feedback?: string, + ) => Promise<{ summary: string; meta: SummaryMeta }>; + onRewriteQuery: (query: string, signal: AbortSignal) => Promise; +} + +export interface CuratorServerHandle { + server: http.Server; + url: string; + close: () => void; + pushResult: (queryIndex: number, data: { answer: string; results: Array<{ title: string; url: string; domain: string }>; provider: string }) => void; + pushError: (queryIndex: number, error: string, provider?: string) => void; + searchesDone: () => void; +} + +function sendJson(res: ServerResponse, status: number, payload: unknown): void { + res.writeHead(status, { + "Content-Type": "application/json", + "Cache-Control": "no-store", + }); + res.end(JSON.stringify(payload)); +} + +function parseJSONBody(req: IncomingMessage): Promise { + return new Promise((resolve, reject) => { + let body = ""; + let size = 0; + req.on("data", (chunk: Buffer) => { + size += chunk.length; + if (size > MAX_BODY_SIZE) { + req.destroy(); + reject(new Error("Request body too large")); + return; + } + body += chunk.toString(); + }); + req.on("end", () => { + try { + resolve(JSON.parse(body)); + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + reject(new Error(`Invalid JSON: ${message}`)); + } + }); + req.on("error", reject); + }); +} + +async function parseBodyOrSend(req: IncomingMessage, res: ServerResponse): Promise { + try { + return await parseJSONBody(req); + } catch (err) { + const message = err instanceof Error ? err.message : "Invalid body"; + const status = message === "Request body too large" ? 413 : 400; + sendJson(res, status, { ok: false, error: message }); + return null; + } +} + +function normalizeSelectedIndices( + value: unknown, + options: { allowEmpty: boolean; maxExclusive: number }, +): { ok: true; indices: number[] } | { ok: false; error: string } { + if (!Array.isArray(value)) { + return { ok: false, error: "Invalid selection" }; + } + + if (!options.allowEmpty && value.length === 0) { + return { ok: false, error: "Invalid selection" }; + } + + const normalized: number[] = []; + const seen = new Set(); + for (const item of value) { + if (typeof item !== "number" || !Number.isInteger(item) || item < 0) { + return { ok: false, error: "Invalid selection" }; + } + if (item >= options.maxExclusive) { + return { ok: false, error: "Invalid selection" }; + } + if (seen.has(item)) { + continue; + } + seen.add(item); + normalized.push(item); + } + + if (!options.allowEmpty && normalized.length === 0) { + return { ok: false, error: "Invalid selection" }; + } + + return { ok: true, indices: normalized }; +} + +function normalizeSummaryMeta(value: unknown): SummaryMeta | null { + if (!value || typeof value !== "object") return null; + const meta = value as Record; + + const model = meta.model; + if (model !== null && typeof model !== "string") return null; + + const durationMs = meta.durationMs; + if (typeof durationMs !== "number" || !Number.isFinite(durationMs) || durationMs < 0) return null; + + const tokenEstimate = meta.tokenEstimate; + if (typeof tokenEstimate !== "number" || !Number.isFinite(tokenEstimate) || tokenEstimate < 0) return null; + + const fallbackUsed = meta.fallbackUsed; + if (typeof fallbackUsed !== "boolean") return null; + + const fallbackReason = meta.fallbackReason; + if (fallbackReason !== undefined && typeof fallbackReason !== "string") return null; + + const edited = meta.edited; + if (edited !== undefined && typeof edited !== "boolean") return null; + + return { + model, + durationMs, + tokenEstimate, + fallbackUsed, + fallbackReason, + edited, + }; +} + +export function startCuratorServer( + options: CuratorServerOptions, + callbacks: CuratorServerCallbacks, +): Promise { + const { + queries, + sessionToken, + timeout, + availableProviders, + defaultProvider, + summaryModels, + defaultSummaryModel, + } = options; + let browserConnected = false; + let lastHeartbeatAt = Date.now(); + let completed = false; + let watchdog: NodeJS.Timeout | null = null; + let state: ServerState = "SEARCHING"; + let sseResponse: ServerResponse | null = null; + const sseBuffer: string[] = []; + let nextQueryIndex = queries.length; + let summarizeAbortController: AbortController | null = null; + let summarizeRequestSeq = 0; + + let sseKeepalive: NodeJS.Timeout | null = null; + + const abortInFlightSummarize = (): void => { + if (!summarizeAbortController) return; + summarizeAbortController.abort(); + summarizeAbortController = null; + }; + + const markCompleted = (): boolean => { + if (completed) return false; + completed = true; + state = "COMPLETED"; + if (watchdog) { + clearInterval(watchdog); + watchdog = null; + } + if (sseKeepalive) { + clearInterval(sseKeepalive); + sseKeepalive = null; + } + abortInFlightSummarize(); + if (sseResponse) { + try { sseResponse.end(); } catch {} + sseResponse = null; + } + return true; + }; + + const touchHeartbeat = (): void => { + lastHeartbeatAt = Date.now(); + browserConnected = true; + }; + + function validateToken(body: unknown, res: ServerResponse): boolean { + if (!body || typeof body !== "object") { + sendJson(res, 400, { ok: false, error: "Invalid body" }); + return false; + } + if ((body as { token?: string }).token !== sessionToken) { + sendJson(res, 403, { ok: false, error: "Invalid session" }); + return false; + } + return true; + } + + function isAvailableProvider(provider: string): boolean { + if (provider === "perplexity") return availableProviders.perplexity; + if (provider === "exa") return availableProviders.exa; + if (provider === "gemini") return availableProviders.gemini; + return false; + } + + function sendSSE(event: string, data: unknown): void { + const payload = `event: ${event}\ndata: ${JSON.stringify(data)}\n\n`; + const res = sseResponse; + if (res && !res.writableEnded && res.socket && !res.socket.destroyed) { + try { res.write(payload); return; } catch {} + } + sseBuffer.push(payload); + } + + const pageHtml = generateCuratorPage( + queries, + sessionToken, + timeout, + availableProviders, + defaultProvider, + summaryModels, + defaultSummaryModel, + ); + + const server = http.createServer(async (req, res) => { + try { + const method = req.method || "GET"; + const url = new URL(req.url || "/", `http://${req.headers.host || "127.0.0.1"}`); + + if (method === "GET" && url.pathname === "/") { + const token = url.searchParams.get("session"); + if (token !== sessionToken) { + res.writeHead(403, { "Content-Type": "text/plain" }); + res.end("Invalid session"); + return; + } + touchHeartbeat(); + res.writeHead(200, { + "Content-Type": "text/html; charset=utf-8", + "Cache-Control": "no-store", + }); + res.end(pageHtml); + return; + } + + if (method === "GET" && url.pathname === "/events") { + const token = url.searchParams.get("session"); + if (token !== sessionToken) { + res.writeHead(403, { "Content-Type": "text/plain" }); + res.end("Invalid session"); + return; + } + if (state === "COMPLETED") { + sendJson(res, 409, { ok: false, error: "No events available" }); + return; + } + if (sseResponse) { + try { sseResponse.end(); } catch {} + } + res.writeHead(200, { + "Content-Type": "text/event-stream", + "Cache-Control": "no-cache", + Connection: "keep-alive", + "X-Accel-Buffering": "no", + }); + res.flushHeaders(); + if (res.socket) res.socket.setNoDelay(true); + sseResponse = res; + if (sseBuffer.length > 0) { + const pending = sseBuffer.splice(0, sseBuffer.length); + for (let i = 0; i < pending.length; i++) { + const msg = pending[i]; + try { + res.write(msg); + } catch { + sseBuffer.unshift(...pending.slice(i)); + break; + } + } + } + if (sseKeepalive) clearInterval(sseKeepalive); + sseKeepalive = setInterval(() => { + if (sseResponse) { + try { sseResponse.write(":keepalive\n\n"); } catch {} + } + }, 15000); + req.on("close", () => { + if (sseResponse === res) sseResponse = null; + }); + return; + } + + if (method === "POST" && url.pathname === "/heartbeat") { + const body = await parseBodyOrSend(req, res); + if (!body) return; + if (!validateToken(body, res)) return; + touchHeartbeat(); + sendJson(res, 200, { ok: true }); + return; + } + + if (method === "POST" && url.pathname === "/provider") { + const body = await parseBodyOrSend(req, res); + if (!body) return; + if (!validateToken(body, res)) return; + const { provider } = body as { provider?: string }; + if (typeof provider !== "string" || provider.length === 0) { + sendJson(res, 400, { ok: false, error: "Invalid provider" }); + return; + } + if (!isAvailableProvider(provider)) { + sendJson(res, 400, { ok: false, error: `Provider unavailable: ${provider}` }); + return; + } + setImmediate(() => callbacks.onProviderChange(provider)); + sendJson(res, 200, { ok: true }); + return; + } + + if (method === "POST" && url.pathname === "/search") { + const body = await parseBodyOrSend(req, res); + if (!body) return; + if (!validateToken(body, res)) return; + if (state === "COMPLETED") { + sendJson(res, 409, { ok: false, error: "Session closed" }); + return; + } + const { query, provider } = body as { query?: string; provider?: string }; + if (typeof query !== "string" || query.trim().length === 0) { + sendJson(res, 400, { ok: false, error: "Invalid query" }); + return; + } + if (provider !== undefined) { + if (typeof provider !== "string" || provider.length === 0) { + sendJson(res, 400, { ok: false, error: "Invalid provider" }); + return; + } + if (!isAvailableProvider(provider)) { + sendJson(res, 400, { ok: false, error: `Provider unavailable: ${provider}` }); + return; + } + } + const qi = nextQueryIndex++; + touchHeartbeat(); + try { + const result = await callbacks.onAddSearch(query.trim(), qi, provider); + sendJson(res, 200, { + ok: true, + queryIndex: qi, + answer: result.answer, + results: result.results, + provider: result.provider, + }); + } catch (err) { + const message = err instanceof Error ? err.message : "Search failed"; + sendJson(res, 200, { + ok: true, + queryIndex: qi, + error: message, + provider: typeof provider === "string" && provider.length > 0 ? provider : undefined, + }); + } + return; + } + + if (method === "POST" && url.pathname === "/summarize") { + const body = await parseBodyOrSend(req, res); + if (!body) return; + if (!validateToken(body, res)) return; + if (state === "COMPLETED") { + sendJson(res, 409, { ok: false, error: "Session closed" }); + return; + } + + const parsed = normalizeSelectedIndices((body as { selected?: unknown }).selected, { + allowEmpty: false, + maxExclusive: nextQueryIndex, + }); + if (!parsed.ok) { + sendJson(res, 400, { ok: false, error: parsed.error }); + return; + } + + let model: string | undefined; + const bodyModel = (body as { model?: unknown }).model; + if (bodyModel !== undefined) { + if (typeof bodyModel !== "string") { + sendJson(res, 400, { ok: false, error: "Invalid model" }); + return; + } + const trimmedModel = bodyModel.trim(); + model = trimmedModel.length > 0 ? trimmedModel : undefined; + } + + const bodyFeedback = (body as { feedback?: unknown }).feedback; + const feedback = typeof bodyFeedback === "string" && bodyFeedback.trim().length > 0 + ? bodyFeedback.trim() + : undefined; + + abortInFlightSummarize(); + const controller = new AbortController(); + summarizeAbortController = controller; + const requestId = ++summarizeRequestSeq; + + try { + const result = await callbacks.onSummarize(parsed.indices, controller.signal, model, feedback); + if (requestId !== summarizeRequestSeq || state === "COMPLETED") { + sendJson(res, 409, { ok: false, error: "Summarize request superseded" }); + return; + } + sendJson(res, 200, { + ok: true, + summary: result.summary, + meta: result.meta, + }); + } catch (err) { + const message = err instanceof Error ? err.message : "Summary generation failed"; + const status = controller.signal.aborted ? 409 : 500; + sendJson(res, status, { ok: false, error: message }); + } finally { + if (summarizeAbortController === controller) { + summarizeAbortController = null; + } + } + return; + } + + if (method === "POST" && url.pathname === "/rewrite") { + const body = await parseBodyOrSend(req, res); + if (!body) return; + if (!validateToken(body, res)) return; + if (state === "COMPLETED") { + sendJson(res, 409, { ok: false, error: "Session closed" }); + return; + } + const { query } = body as { query?: unknown }; + if (typeof query !== "string" || query.trim().length === 0) { + sendJson(res, 400, { ok: false, error: "Invalid query" }); + return; + } + const controller = new AbortController(); + req.on("close", () => controller.abort()); + touchHeartbeat(); + try { + const rewritten = await callbacks.onRewriteQuery(query.trim(), controller.signal); + sendJson(res, 200, { ok: true, query: rewritten }); + } catch (err) { + const message = err instanceof Error ? err.message : "Rewrite failed"; + const status = controller.signal.aborted ? 409 : 500; + sendJson(res, status, { ok: false, error: message }); + } + return; + } + + if (method === "POST" && url.pathname === "/submit") { + const body = await parseBodyOrSend(req, res); + if (!body) return; + if (!validateToken(body, res)) return; + + const parsed = normalizeSelectedIndices((body as { selected?: unknown }).selected, { + allowEmpty: true, + maxExclusive: nextQueryIndex, + }); + if (!parsed.ok) { + sendJson(res, 400, { ok: false, error: parsed.error }); + return; + } + + let summary: string | undefined; + const bodySummary = (body as { summary?: unknown }).summary; + if (bodySummary !== undefined) { + if (typeof bodySummary !== "string") { + sendJson(res, 400, { ok: false, error: "Invalid summary" }); + return; + } + const trimmedSummary = bodySummary.trim(); + summary = trimmedSummary.length > 0 ? trimmedSummary : undefined; + } + + let summaryMeta: SummaryMeta | undefined; + const bodySummaryMeta = (body as { summaryMeta?: unknown }).summaryMeta; + if (bodySummaryMeta !== undefined) { + const parsedSummaryMeta = normalizeSummaryMeta(bodySummaryMeta); + if (!parsedSummaryMeta) { + sendJson(res, 400, { ok: false, error: "Invalid summaryMeta" }); + return; + } + summaryMeta = parsedSummaryMeta; + } + + if (state !== "SEARCHING" && state !== "RESULT_SELECTION") { + sendJson(res, 409, { ok: false, error: "Cannot submit in current state" }); + return; + } + if (!markCompleted()) { + sendJson(res, 409, { ok: false, error: "Session closed" }); + return; + } + const rawResults = (body as { rawResults?: unknown }).rawResults === true; + sendJson(res, 200, { ok: true }); + setImmediate(() => callbacks.onSubmit({ selectedQueryIndices: parsed.indices, summary, summaryMeta, rawResults })); + return; + } + + if (method === "POST" && url.pathname === "/cancel") { + const body = await parseBodyOrSend(req, res); + if (!body) return; + if (!validateToken(body, res)) return; + if (!markCompleted()) { + sendJson(res, 200, { ok: true }); + return; + } + const { reason } = body as { reason?: string }; + sendJson(res, 200, { ok: true }); + const cancelReason = reason === "timeout" ? "timeout" : "user"; + setImmediate(() => callbacks.onCancel(cancelReason)); + return; + } + + res.writeHead(404, { "Content-Type": "text/plain" }); + res.end("Not found"); + } catch (err) { + const message = err instanceof Error ? err.message : "Server error"; + sendJson(res, 500, { ok: false, error: message }); + } + }); + + return new Promise((resolve, reject) => { + const onError = (err: Error) => { + reject(new Error(`Curator server failed to start: ${err.message}`)); + }; + + server.once("error", onError); + server.listen(0, "127.0.0.1", () => { + server.off("error", onError); + const addr = server.address(); + if (!addr || typeof addr === "string") { + reject(new Error("Curator server: invalid address")); + return; + } + const url = `http://localhost:${addr.port}/?session=${sessionToken}`; + + watchdog = setInterval(() => { + if (completed || !browserConnected) return; + if (Date.now() - lastHeartbeatAt <= STALE_THRESHOLD_MS) return; + if (!markCompleted()) return; + setImmediate(() => callbacks.onCancel("stale")); + }, WATCHDOG_INTERVAL_MS); + + resolve({ + server, + url, + close: () => { + const wasOpen = markCompleted(); + try { server.close(); } catch {} + if (wasOpen) { + setImmediate(() => callbacks.onCancel("stale")); + } + }, + pushResult: (queryIndex, data) => { + if (completed) return; + sendSSE("result", { queryIndex, query: queries[queryIndex] ?? "", ...data }); + }, + pushError: (queryIndex, error, provider) => { + if (completed) return; + sendSSE("search-error", { queryIndex, query: queries[queryIndex] ?? "", error, provider }); + }, + searchesDone: () => { + if (completed) return; + sendSSE("done", {}); + state = "RESULT_SELECTION"; + }, + }); + }); + }); +} diff --git a/packages/web-access/exa.ts b/packages/web-access/exa.ts new file mode 100644 index 000000000..af1d06d7d --- /dev/null +++ b/packages/web-access/exa.ts @@ -0,0 +1,521 @@ +import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import { homedir } from "node:os"; +import { join } from "node:path"; +import { CONFIG_DIR_NAME } from "@bastani/atomic"; +import { activityMonitor } from "./activity.js"; +import type { ExtractedContent } from "./extract.js"; +import type { SearchOptions, SearchResponse } from "./perplexity.js"; + +const EXA_ANSWER_URL = "https://api.exa.ai/answer"; +const EXA_SEARCH_URL = "https://api.exa.ai/search"; +const EXA_MCP_URL = "https://mcp.exa.ai/mcp"; +const CONFIG_PATH = join(homedir(), CONFIG_DIR_NAME, "web-search.json"); +const USAGE_PATH = join(homedir(), CONFIG_DIR_NAME, "exa-usage.json"); + +const MONTHLY_LIMIT = 1000; +const WARNING_THRESHOLD = 800; + +interface WebSearchConfig { + exaApiKey?: unknown; +} + +interface ExaUsage { + month: string; + count: number; +} + +interface ExaAnswerResponse { + answer?: string; + citations?: Array<{ url?: string; title?: string; text?: string; publishedDate?: string }>; +} + +interface ExaSearchResponse { + results?: Array<{ + title?: string; + url?: string; + publishedDate?: string; + author?: string; + text?: string; + highlights?: unknown; + highlightScores?: number[]; + }>; +} + +interface ExaMcpRpcResponse { + result?: { + content?: Array<{ type?: string; text?: string }>; + isError?: boolean; + }; + error?: { + code?: number; + message?: string; + }; +} + +export type ExaSearchResult = SearchResponse | { exhausted: true } | null; + +export interface ExaSearchOptions extends SearchOptions { + includeContent?: boolean; +} + +type McpParsedResult = { title: string; url: string; content: string }; + +let cachedConfig: WebSearchConfig | null = null; +let warnedMonth: string | null = null; + +function loadConfig(): WebSearchConfig { + if (cachedConfig) return cachedConfig; + if (!existsSync(CONFIG_PATH)) { + cachedConfig = {}; + return cachedConfig; + } + + const raw = readFileSync(CONFIG_PATH, "utf-8"); + try { + cachedConfig = JSON.parse(raw) as WebSearchConfig; + return cachedConfig; + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + throw new Error(`Failed to parse ${CONFIG_PATH}: ${message}`); + } +} + +function normalizeApiKey(value: unknown): string | null { + if (typeof value !== "string") return null; + const normalized = value.trim(); + return normalized.length > 0 ? normalized : null; +} + +function getApiKey(): string | null { + return normalizeApiKey(process.env.EXA_API_KEY) ?? normalizeApiKey(loadConfig().exaApiKey); +} + +function getCurrentMonth(): string { + return new Date().toISOString().slice(0, 7); +} + +function normalizeUsage(raw: unknown): ExaUsage { + const month = getCurrentMonth(); + if (!raw || typeof raw !== "object") return { month, count: 0 }; + const data = raw as { month?: unknown; count?: unknown }; + const parsedMonth = typeof data.month === "string" ? data.month : month; + const parsedCount = typeof data.count === "number" && Number.isFinite(data.count) ? data.count : 0; + if (parsedMonth !== month) return { month, count: 0 }; + return { month: parsedMonth, count: Math.max(0, Math.floor(parsedCount)) }; +} + +function readUsage(): ExaUsage { + if (!existsSync(USAGE_PATH)) return { month: getCurrentMonth(), count: 0 }; + const raw = readFileSync(USAGE_PATH, "utf-8"); + try { + return normalizeUsage(JSON.parse(raw)); + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + throw new Error(`Failed to parse ${USAGE_PATH}: ${message}`); + } +} + +function writeUsage(usage: ExaUsage): void { + const dir = join(homedir(), CONFIG_DIR_NAME); + if (!existsSync(dir)) mkdirSync(dir, { recursive: true }); + writeFileSync(USAGE_PATH, JSON.stringify(usage, null, 2) + "\n"); +} + +function reserveRequestBudget(): { exhausted: true } | null { + const usage = readUsage(); + + if (usage.count >= MONTHLY_LIMIT) { + return { exhausted: true }; + } + + const nextCount = usage.count + 1; + if (nextCount >= WARNING_THRESHOLD && warnedMonth !== usage.month) { + warnedMonth = usage.month; + console.error(`Exa usage warning: ${nextCount}/${MONTHLY_LIMIT} monthly requests used.`); + } + + writeUsage({ month: usage.month, count: nextCount }); + return null; +} + +function requestSignal(signal?: AbortSignal): AbortSignal { + const timeout = AbortSignal.timeout(60000); + return signal ? AbortSignal.any([signal, timeout]) : timeout; +} + +function recencyToStartDate(filter: string): string { + const now = new Date(); + const offsets: Record = { + day: 1, + week: 7, + month: 30, + year: 365, + }; + const days = offsets[filter] ?? 0; + return new Date(now.getTime() - days * 86400000).toISOString(); +} + +function mapDomainFilter(domainFilter: string[] | undefined): { includeDomains?: string[]; excludeDomains?: string[] } { + if (!domainFilter?.length) return {}; + const includeDomains = domainFilter + .filter(d => !d.startsWith("-") && d.trim().length > 0) + .map(d => d.trim()); + const excludeDomains = domainFilter + .filter(d => d.startsWith("-")) + .map(d => d.slice(1).trim()) + .filter(Boolean); + return { + ...(includeDomains.length ? { includeDomains } : {}), + ...(excludeDomains.length ? { excludeDomains } : {}), + }; +} + +function normalizeHighlights(value: unknown): string[] { + if (!Array.isArray(value)) return []; + return value.filter((item): item is string => typeof item === "string" && item.trim().length > 0); +} + +function buildAnswerFromSearchResults(results: ExaSearchResponse["results"]): string { + if (!results?.length) return ""; + const parts: string[] = []; + for (let i = 0; i < results.length; i++) { + const item = results[i]; + if (!item?.url) continue; + const highlights = normalizeHighlights(item.highlights); + const content = highlights.length > 0 + ? highlights.join(" ") + : typeof item.text === "string" ? item.text.trim().slice(0, 1000) : ""; + if (!content) continue; + const sourceTitle = item.title || `Source ${i + 1}`; + parts.push(`${content}\nSource: ${sourceTitle} (${item.url})`); + } + return parts.join("\n\n"); +} + +function mapResults(results: ExaSearchResponse["results"] | ExaAnswerResponse["citations"]): SearchResponse["results"] { + if (!Array.isArray(results)) return []; + const mapped: SearchResponse["results"] = []; + for (let i = 0; i < results.length; i++) { + const item = results[i]; + if (!item?.url) continue; + mapped.push({ + title: item.title || `Source ${i + 1}`, + url: item.url, + snippet: "", + }); + } + return mapped; +} + +function mapInlineContent(results: ExaSearchResponse["results"]): ExtractedContent[] { + if (!results?.length) return []; + return results + .filter((r): r is NonNullable[number] & { url: string; text: string } => + !!r?.url && typeof r.text === "string" && r.text.length > 0) + .map(r => ({ + url: r.url, + title: r.title || "", + content: r.text, + error: null, + })); +} + +export async function callExaMcp( + toolName: string, + args: Record, + signal?: AbortSignal, +): Promise { + const response = await fetch(EXA_MCP_URL, { + method: "POST", + headers: { + "Content-Type": "application/json", + "Accept": "application/json, text/event-stream", + }, + body: JSON.stringify({ + jsonrpc: "2.0", + id: 1, + method: "tools/call", + params: { + name: toolName, + arguments: args, + }, + }), + signal: requestSignal(signal), + }); + + if (!response.ok) { + const errorText = await response.text(); + throw new Error(`Exa MCP error ${response.status}: ${errorText.slice(0, 300)}`); + } + + const body = await response.text(); + const dataLines = body.split("\n").filter(line => line.startsWith("data:")); + + let parsed: ExaMcpRpcResponse | null = null; + for (const line of dataLines) { + const payload = line.slice(5).trim(); + if (!payload) continue; + try { + const candidate = JSON.parse(payload) as ExaMcpRpcResponse; + if (candidate?.result || candidate?.error) { + parsed = candidate; + break; + } + } catch { + } + } + + if (!parsed) { + try { + const candidate = JSON.parse(body) as ExaMcpRpcResponse; + if (candidate?.result || candidate?.error) { + parsed = candidate; + } + } catch { + } + } + + if (!parsed) { + throw new Error("Exa MCP returned an empty response"); + } + + if (parsed.error) { + const code = typeof parsed.error.code === "number" ? ` ${parsed.error.code}` : ""; + const message = parsed.error.message || "Unknown error"; + throw new Error(`Exa MCP error${code}: ${message}`); + } + + if (parsed.result?.isError) { + const message = parsed.result.content + ?.find(item => item.type === "text" && typeof item.text === "string") + ?.text?.trim(); + throw new Error(message || "Exa MCP returned an error"); + } + + const text = parsed.result?.content + ?.find(item => item.type === "text" && typeof item.text === "string" && item.text.trim().length > 0) + ?.text; + + if (!text) { + throw new Error("Exa MCP returned empty content"); + } + + return text; +} + +function parseMcpResults(text: string): McpParsedResult[] | null { + const blocks = text.split(/(?=^Title: )/m).filter(block => block.trim().length > 0); + const parsed = blocks.map(block => { + const title = block.match(/^Title: (.+)/m)?.[1]?.trim() ?? ""; + const url = block.match(/^URL: (.+)/m)?.[1]?.trim() ?? ""; + let content = ""; + const textStart = block.indexOf("\nText: "); + if (textStart >= 0) { + content = block.slice(textStart + 7).trim(); + } else { + const hlMatch = block.match(/\nHighlights:\s*\n/); + if (hlMatch?.index != null) { + content = block.slice(hlMatch.index + hlMatch[0].length).trim(); + } + } + content = content.replace(/\n---\s*$/, "").trim(); + return { title, url, content }; + }).filter(result => result.url.length > 0); + return parsed.length > 0 ? parsed : null; +} + +function buildAnswerFromMcpResults(results: McpParsedResult[]): string { + if (results.length === 0) return ""; + const parts: string[] = []; + for (let i = 0; i < results.length; i++) { + const result = results[i]; + const snippet = result.content.replace(/\s+/g, " ").trim().slice(0, 500); + if (!snippet) continue; + const sourceTitle = result.title || `Source ${i + 1}`; + parts.push(`${snippet}\nSource: ${sourceTitle} (${result.url})`); + } + return parts.join("\n\n"); +} + +function mapMcpInlineContent(results: McpParsedResult[]): ExtractedContent[] { + return results + .filter(result => result.content.length > 0) + .map(result => ({ + url: result.url, + title: result.title, + content: result.content, + error: null, + })); +} + +function buildMcpQuery(query: string, options: ExaSearchOptions): string { + const parts = [query]; + if (options.domainFilter?.length) { + for (const d of options.domainFilter) { + parts.push(d.startsWith("-") ? `-site:${d.slice(1)}` : `site:${d}`); + } + } + if (options.recencyFilter) { + const now = new Date(); + switch (options.recencyFilter) { + case "day": parts.push("past 24 hours"); break; + case "week": parts.push("past week"); break; + case "month": parts.push(`${now.toLocaleString("en", { month: "long" })} ${now.getFullYear()}`); break; + case "year": parts.push(String(now.getFullYear())); break; + } + } + return parts.join(" "); +} + +async function searchWithExaMcp(query: string, options: ExaSearchOptions = {}): Promise { + const enrichedQuery = buildMcpQuery(query, options); + const activityId = activityMonitor.logStart({ type: "api", query: enrichedQuery }); + + try { + const text = await callExaMcp( + "web_search_exa", + { + query: enrichedQuery, + numResults: options.numResults ?? 5, + livecrawl: "fallback", + type: "auto", + contextMaxCharacters: options.includeContent ? 50000 : 3000, + }, + options.signal, + ); + const parsedResults = parseMcpResults(text); + activityMonitor.logComplete(activityId, 200); + + if (!parsedResults) return null; + + const response: SearchResponse = { + answer: buildAnswerFromMcpResults(parsedResults), + results: parsedResults.map((result, index) => ({ + title: result.title || `Source ${index + 1}`, + url: result.url, + snippet: "", + })), + }; + + if (options.includeContent) { + const inlineContent = mapMcpInlineContent(parsedResults); + if (inlineContent.length > 0) response.inlineContent = inlineContent; + } + + return response; + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + if (message.toLowerCase().includes("abort")) { + activityMonitor.logComplete(activityId, 0); + } else { + activityMonitor.logError(activityId, message); + } + throw err; + } +} + +export function isExaAvailable(): boolean { + if (getApiKey()) { + const usage = readUsage(); + return usage.count < MONTHLY_LIMIT; + } + return true; +} + +export function hasExaApiKey(): boolean { + return !!getApiKey(); +} + +export async function searchWithExa(query: string, options: ExaSearchOptions = {}): Promise { + const apiKey = getApiKey(); + if (!apiKey) { + return searchWithExaMcp(query, options); + } + + const budget = reserveRequestBudget(); + if (budget) return budget; + + const useSearch = options.includeContent + || !!options.recencyFilter + || !!options.domainFilter?.length + || !!(options.numResults && options.numResults !== 5); + + const activityId = activityMonitor.logStart({ type: "api", query }); + + try { + if (!useSearch) { + const response = await fetch(EXA_ANSWER_URL, { + method: "POST", + headers: { + "x-api-key": apiKey, + "Content-Type": "application/json", + }, + body: JSON.stringify({ + query, + text: true, + }), + signal: requestSignal(options.signal), + }); + + if (!response.ok) { + const errorText = await response.text(); + throw new Error(`Exa API error ${response.status}: ${errorText.slice(0, 300)}`); + } + + const data = await response.json() as ExaAnswerResponse; + activityMonitor.logComplete(activityId, response.status); + return { + answer: data.answer || "", + results: mapResults(data.citations), + }; + } + + const startDate = options.recencyFilter ? recencyToStartDate(options.recencyFilter) : null; + const domainFilters = mapDomainFilter(options.domainFilter); + const response = await fetch(EXA_SEARCH_URL, { + method: "POST", + headers: { + "x-api-key": apiKey, + "Content-Type": "application/json", + }, + body: JSON.stringify({ + query, + type: "auto", + numResults: options.numResults ?? 5, + ...domainFilters, + ...(startDate ? { startPublishedDate: startDate } : {}), + contents: { + text: options.includeContent ? true : { maxCharacters: 3000 }, + highlights: true, + }, + }), + signal: requestSignal(options.signal), + }); + + if (!response.ok) { + const errorText = await response.text(); + throw new Error(`Exa API error ${response.status}: ${errorText.slice(0, 300)}`); + } + + const data = await response.json() as ExaSearchResponse; + activityMonitor.logComplete(activityId, response.status); + + const mapped: SearchResponse = { + answer: buildAnswerFromSearchResults(data.results), + results: mapResults(data.results), + }; + if (options.includeContent) { + const inlineContent = mapInlineContent(data.results); + if (inlineContent.length > 0) mapped.inlineContent = inlineContent; + } + return mapped; + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + if (message.toLowerCase().includes("abort")) { + activityMonitor.logComplete(activityId, 0); + } else { + activityMonitor.logError(activityId, message); + } + throw err; + } +} diff --git a/packages/web-access/extract.ts b/packages/web-access/extract.ts new file mode 100644 index 000000000..68221bdaf --- /dev/null +++ b/packages/web-access/extract.ts @@ -0,0 +1,701 @@ +import { Readability } from "@mozilla/readability"; +import { parseHTML } from "linkedom"; +import TurndownService from "turndown"; +import pLimit from "p-limit"; +import { CONFIG_DIR_NAME } from "@bastani/atomic"; +import { activityMonitor } from "./activity.js"; +import { extractRSCContent } from "./rsc-extract.js"; +import { extractPDFToMarkdown, isPDF } from "./pdf-extract.js"; +import { extractGitHub } from "./github-extract.js"; +import { isYouTubeURL, isYouTubeEnabled, extractYouTube, extractYouTubeFrame, extractYouTubeFrames, getYouTubeStreamInfo } from "./youtube-extract.js"; +import { extractWithUrlContext, extractWithGeminiWeb } from "./gemini-url-context.js"; +import { isVideoFile, extractVideo, extractVideoFrame, getLocalVideoDuration } from "./video-extract.js"; +import { formatSeconds } from "./utils.js"; + +const DEFAULT_TIMEOUT_MS = 30000; +const CONCURRENT_LIMIT = 3; + +const NON_RECOVERABLE_ERRORS = ["Unsupported content type", "Response too large"]; +const MIN_USEFUL_CONTENT = 500; + +function errorMessage(err: unknown): string { + return err instanceof Error ? err.message : String(err); +} + +function isConfigParseError(err: unknown): boolean { + return errorMessage(err).startsWith("Failed to parse "); +} + +function isAbortError(err: unknown): boolean { + return errorMessage(err).toLowerCase().includes("abort"); +} + +function abortedResult(url: string): ExtractedContent { + return { url, title: "", content: "", error: "Aborted" }; +} + +const turndown = new TurndownService({ + headingStyle: "atx", + codeBlockStyle: "fenced", +}); + +const fetchLimit = pLimit(CONCURRENT_LIMIT); + +export interface VideoFrame { + data: string; + mimeType: string; + timestamp: string; +} + +export type FrameData = { data: string; mimeType: string }; +export type FrameResult = FrameData | { error: string }; + +export interface ExtractedContent { + url: string; + title: string; + content: string; + error: string | null; + thumbnail?: { data: string; mimeType: string }; + frames?: VideoFrame[]; + duration?: number; +} + +export interface ExtractOptions { + timeoutMs?: number; + forceClone?: boolean; + prompt?: string; + timestamp?: string; + frames?: number; + model?: string; +} + +const JINA_READER_BASE = "https://r.jina.ai/"; +const JINA_TIMEOUT_MS = 30000; + +async function extractWithJinaReader( + url: string, + signal?: AbortSignal, +): Promise { + const jinaUrl = JINA_READER_BASE + url; + + const activityId = activityMonitor.logStart({ type: "api", query: `jina: ${url}` }); + + try { + const res = await fetch(jinaUrl, { + headers: { + "Accept": "text/markdown", + "X-No-Cache": "true", + }, + signal: AbortSignal.any([ + AbortSignal.timeout(JINA_TIMEOUT_MS), + ...(signal ? [signal] : []), + ]), + }); + + if (!res.ok) { + activityMonitor.logComplete(activityId, res.status); + return null; + } + + const content = await res.text(); + activityMonitor.logComplete(activityId, res.status); + + const contentStart = content.indexOf("Markdown Content:"); + if (contentStart < 0) { + return null; + } + + const markdownPart = content.slice(contentStart + 17).trim(); // 17 = "Markdown Content:".length + + // Check for failed JS rendering or minimal content + if (markdownPart.length < 100 || + markdownPart.startsWith("Loading...") || + markdownPart.startsWith("Please enable JavaScript")) { + return null; + } + + const title = extractHeadingTitle(markdownPart) ?? (new URL(url).pathname.split("/").pop() || url); + return { url, title, content: markdownPart, error: null }; + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + if (message.toLowerCase().includes("abort")) { + activityMonitor.logComplete(activityId, 0); + } else { + activityMonitor.logError(activityId, message); + } + return null; + } +} + +function parseTimestamp(ts: string): number | null { + const num = Number(ts); + if (!isNaN(num) && num >= 0) return Math.floor(num); + const parts = ts.split(":").map(Number); + if (parts.some(p => isNaN(p) || p < 0)) return null; + if (parts.length === 3) return Math.floor(parts[0] * 3600 + parts[1] * 60 + parts[2]); + if (parts.length === 2) return Math.floor(parts[0] * 60 + parts[1]); + return null; +} + +type TimestampSpec = { type: "single"; seconds: number } | { type: "range"; start: number; end: number }; + +function parseTimestampSpec(ts: string): TimestampSpec | null { + const dashIdx = ts.indexOf("-", 1); + if (dashIdx > 0) { + const start = parseTimestamp(ts.slice(0, dashIdx)); + const end = parseTimestamp(ts.slice(dashIdx + 1)); + if (start !== null && end !== null && end > start) return { type: "range", start, end }; + } + const seconds = parseTimestamp(ts); + return seconds !== null ? { type: "single", seconds } : null; +} + +const DEFAULT_RANGE_FRAMES = 6; +const MIN_FRAME_INTERVAL = 5; + +function computeRangeTimestamps(start: number, end: number, maxFrames: number = DEFAULT_RANGE_FRAMES): number[] { + if (maxFrames <= 1) return [start]; + const duration = end - start; + const idealInterval = duration / (maxFrames - 1); + if (idealInterval < MIN_FRAME_INTERVAL) { + const timestamps: number[] = []; + for (let t = start; t <= end && timestamps.length < maxFrames; t += MIN_FRAME_INTERVAL) { + timestamps.push(t); + } + return timestamps; + } + return Array.from({ length: maxFrames }, (_, i) => Math.round(start + i * idealInterval)); +} + +function buildFrameResult( + url: string, label: string, requestedCount: number, + frames: VideoFrame[], error: string | null, duration?: number, +): ExtractedContent { + if (frames.length === 0) { + const msg = error ?? "Frame extraction failed"; + return { url, title: `Frames ${label} (0/${requestedCount})`, content: msg, error: msg }; + } + return { + url, + title: `Frames ${label} (${frames.length}/${requestedCount})`, + content: `${frames.length} frames extracted from ${label}`, + error: null, + frames, + duration, + }; +} + +async function extractLocalFrames( + filePath: string, timestamps: number[], +): Promise<{ frames: VideoFrame[]; error: string | null }> { + const results = await Promise.all(timestamps.map(async (t) => { + const frame = await extractVideoFrame(filePath, t); + if ("error" in frame) return { error: frame.error }; + return { ...frame, timestamp: formatSeconds(t) }; + })); + const frames = results.filter((f): f is VideoFrame => "data" in f); + const firstError = results.find((f): f is { error: string } => "error" in f); + return { frames, error: frames.length === 0 && firstError ? firstError.error : null }; +} + +function safeVideoInfo(url: string): { info: ReturnType; error?: string } { + try { + return { info: isVideoFile(url) }; + } catch (err) { + return { info: null, error: errorMessage(err) }; + } +} + +export async function extractContent( + url: string, + signal?: AbortSignal, + options?: ExtractOptions, +): Promise { + if (signal?.aborted) { + return { url, title: "", content: "", error: "Aborted" }; + } + + if (options?.frames && !options.timestamp) { + const frameCount = options.frames; + const ytInfo = isYouTubeURL(url); + if (ytInfo.isYouTube && ytInfo.videoId) { + const streamInfo = await getYouTubeStreamInfo(ytInfo.videoId); + if ("error" in streamInfo) { + return { url, title: "Frames", content: streamInfo.error, error: streamInfo.error }; + } + if (streamInfo.duration === null) { + const error = "Cannot determine video duration. Use a timestamp range instead."; + return { url, title: "Frames", content: error, error }; + } + const dur = Math.floor(streamInfo.duration); + const timestamps = computeRangeTimestamps(0, dur, frameCount); + const result = await extractYouTubeFrames(ytInfo.videoId, timestamps, streamInfo); + const label = `${formatSeconds(0)}-${formatSeconds(dur)}`; + return buildFrameResult(url, label, timestamps.length, result.frames, result.error, streamInfo.duration); + } + + const localVideo = safeVideoInfo(url); + if (localVideo.error) { + return { url, title: "", content: "", error: localVideo.error }; + } + if (localVideo.info) { + const durationResult = await getLocalVideoDuration(localVideo.info.absolutePath); + if (typeof durationResult !== "number") { + return { url, title: "Frames", content: durationResult.error, error: durationResult.error }; + } + const dur = Math.floor(durationResult); + const timestamps = computeRangeTimestamps(0, dur, frameCount); + const result = await extractLocalFrames(localVideo.info.absolutePath, timestamps); + const label = `${formatSeconds(0)}-${formatSeconds(dur)}`; + return buildFrameResult(url, label, timestamps.length, result.frames, result.error, durationResult); + } + + return { url, title: "", content: "", error: "Frame extraction only works with YouTube and local video files" }; + } + + if (options?.timestamp) { + const spec = parseTimestampSpec(options.timestamp); + if (!spec) { + return { + url, + title: "", + content: "", + error: `Invalid timestamp format: "${options.timestamp}". Use "H:MM:SS", "MM:SS", "85", or "start-end".`, + }; + } + + const frameCount = options.frames; + const ytInfo = isYouTubeURL(url); + if (ytInfo.isYouTube && ytInfo.videoId) { + const streamInfo = await getYouTubeStreamInfo(ytInfo.videoId); + if ("error" in streamInfo) { + if (spec.type === "range") { + const label = `${formatSeconds(spec.start)}-${formatSeconds(spec.end)}`; + return { url, title: `Frames ${label}`, content: streamInfo.error, error: streamInfo.error }; + } + if (frameCount) { + const end = spec.seconds + (frameCount - 1) * MIN_FRAME_INTERVAL; + const label = `${formatSeconds(spec.seconds)}-${formatSeconds(end)}`; + return { url, title: `Frames ${label}`, content: streamInfo.error, error: streamInfo.error }; + } + return { url, title: `Frame at ${options.timestamp}`, content: streamInfo.error, error: streamInfo.error }; + } + + if (spec.type === "range") { + const label = `${formatSeconds(spec.start)}-${formatSeconds(spec.end)}`; + if (streamInfo.duration !== null && spec.end > streamInfo.duration) { + const error = `Timestamp ${formatSeconds(spec.end)} exceeds video duration (${formatSeconds(Math.floor(streamInfo.duration))})`; + return { url, title: `Frames ${label}`, content: error, error }; + } + const timestamps = frameCount + ? computeRangeTimestamps(spec.start, spec.end, frameCount) + : computeRangeTimestamps(spec.start, spec.end); + const result = await extractYouTubeFrames(ytInfo.videoId, timestamps, streamInfo); + return buildFrameResult(url, label, timestamps.length, result.frames, result.error, result.duration ?? undefined); + } + + if (frameCount) { + const end = spec.seconds + (frameCount - 1) * MIN_FRAME_INTERVAL; + const label = `${formatSeconds(spec.seconds)}-${formatSeconds(end)}`; + if (streamInfo.duration !== null && end > streamInfo.duration) { + const error = `Timestamp ${formatSeconds(end)} exceeds video duration (${formatSeconds(Math.floor(streamInfo.duration))})`; + return { url, title: `Frames ${label}`, content: error, error }; + } + const timestamps = computeRangeTimestamps(spec.seconds, end, frameCount); + const result = await extractYouTubeFrames(ytInfo.videoId, timestamps, streamInfo); + return buildFrameResult(url, label, timestamps.length, result.frames, result.error, result.duration ?? undefined); + } + + if (streamInfo.duration !== null && spec.seconds > streamInfo.duration) { + const error = `Timestamp ${formatSeconds(spec.seconds)} exceeds video duration (${formatSeconds(Math.floor(streamInfo.duration))})`; + return { url, title: `Frame at ${options.timestamp}`, content: error, error }; + } + const frame = await extractYouTubeFrame(ytInfo.videoId, spec.seconds, streamInfo); + if ("error" in frame) { + return { url, title: `Frame at ${options.timestamp}`, content: frame.error, error: frame.error }; + } + return { url, title: `Frame at ${options.timestamp}`, content: `Video frame at ${options.timestamp}`, error: null, thumbnail: frame }; + } + + const localVideo = safeVideoInfo(url); + if (localVideo.error) { + return { url, title: "", content: "", error: localVideo.error }; + } + if (localVideo.info) { + if (spec.type === "range") { + const timestamps = frameCount + ? computeRangeTimestamps(spec.start, spec.end, frameCount) + : computeRangeTimestamps(spec.start, spec.end); + const result = await extractLocalFrames(localVideo.info.absolutePath, timestamps); + const label = `${formatSeconds(spec.start)}-${formatSeconds(spec.end)}`; + return buildFrameResult(url, label, timestamps.length, result.frames, result.error); + } + + if (frameCount) { + const end = spec.seconds + (frameCount - 1) * MIN_FRAME_INTERVAL; + const timestamps = computeRangeTimestamps(spec.seconds, end, frameCount); + const result = await extractLocalFrames(localVideo.info.absolutePath, timestamps); + const label = `${formatSeconds(spec.seconds)}-${formatSeconds(end)}`; + return buildFrameResult(url, label, timestamps.length, result.frames, result.error); + } + + const frame = await extractVideoFrame(localVideo.info.absolutePath, spec.seconds); + if ("error" in frame) { + return { url, title: `Frame at ${options.timestamp}`, content: frame.error, error: frame.error }; + } + return { url, title: `Frame at ${options.timestamp}`, content: `Video frame at ${options.timestamp}`, error: null, thumbnail: frame }; + } + + return { url, title: "", content: "", error: "Timestamp extraction only works with YouTube and local video files" }; + } + + const localVideo = safeVideoInfo(url); + if (localVideo.error) { + return { url, title: "", content: "", error: localVideo.error }; + } + if (localVideo.info) { + try { + const result = await extractVideo(localVideo.info, signal, options); + if (signal?.aborted) return abortedResult(url); + return result ?? { url, title: "", content: "", error: `Video analysis requires Gemini access. Either:\n 1. Sign into gemini.google.com in Chrome (free, uses cookies)\n 2. Set GEMINI_API_KEY in ~/${CONFIG_DIR_NAME}/web-search.json` }; + } catch (err) { + if (isAbortError(err)) return abortedResult(url); + return { url, title: "", content: "", error: errorMessage(err) }; + } + } + + try { + new URL(url); + } catch { + return { url, title: "", content: "", error: "Invalid URL" }; + } + + try { + const ghResult = await extractGitHub(url, signal, options?.forceClone); + if (ghResult) return ghResult; + if (signal?.aborted) return abortedResult(url); + } catch (err) { + const message = errorMessage(err); + if (isAbortError(err)) return abortedResult(url); + if (isConfigParseError(err)) { + return { url, title: "", content: "", error: message }; + } + } + + const ytInfo = isYouTubeURL(url); + let youtubeEnabled = false; + try { + youtubeEnabled = isYouTubeEnabled(); + } catch (err) { + return { url, title: "", content: "", error: errorMessage(err) }; + } + if (ytInfo.isYouTube && youtubeEnabled) { + try { + const ytResult = await extractYouTube(url, signal, options?.prompt, options?.model); + if (ytResult) return ytResult; + if (signal?.aborted) return abortedResult(url); + } catch (err) { + const message = errorMessage(err); + if (isAbortError(err)) return abortedResult(url); + if (isConfigParseError(err)) { + return { url, title: "", content: "", error: message }; + } + } + return { + url, + title: "", + content: "", + error: "Could not extract YouTube video content. Sign into Google in Chrome for automatic access, or set GEMINI_API_KEY.", + }; + } + + if (signal?.aborted) return abortedResult(url); + + const httpResult = await extractViaHttp(url, signal, options); + + if (signal?.aborted) return abortedResult(url); + if (!httpResult.error) return httpResult; + if (NON_RECOVERABLE_ERRORS.some(prefix => httpResult.error!.startsWith(prefix))) return httpResult; + + const jinaResult = await extractWithJinaReader(url, signal); + if (jinaResult) return jinaResult; + if (signal?.aborted) return abortedResult(url); + + let geminiResult: ExtractedContent | null = null; + try { + geminiResult = await extractWithUrlContext(url, signal) + ?? await extractWithGeminiWeb(url, signal); + } catch (err) { + if (isAbortError(err)) return abortedResult(url); + if (isConfigParseError(err)) { + return { ...httpResult, error: errorMessage(err) }; + } + } + + if (geminiResult) return geminiResult; + if (signal?.aborted) return abortedResult(url); + + const guidance = [ + httpResult.error, + "", + "Fallback options:", + ` \u2022 Set GEMINI_API_KEY in ~/${CONFIG_DIR_NAME}/web-search.json`, + " \u2022 Sign into gemini.google.com in Chrome", + " \u2022 Use web_search to find content about this topic", + ].join("\n"); + return { ...httpResult, error: guidance }; +} + +function stripElementBlocks(html: string, tagName: "script" | "style"): string { + let output = ""; + let cursor = 0; + const lower = html.toLowerCase(); + const startNeedle = `<${tagName}`; + const endNeedle = `", end + endNeedle.length); + if (endClose === -1) { + break; + } + cursor = endClose + 1; + } + + return output; +} + +function stripTags(html: string): string { + let output = ""; + let insideTag = false; + for (const char of html) { + if (char === "<") { + insideTag = true; + continue; + } + if (char === ">") { + insideTag = false; + continue; + } + if (!insideTag) output += char; + } + return output; +} + +function collapseWhitespace(text: string): string { + let output = ""; + let pendingSpace = false; + for (const char of text) { + if (/\s/.test(char)) { + pendingSpace = output.length > 0; + continue; + } + if (pendingSpace) { + output += " "; + pendingSpace = false; + } + output += char; + } + return output.trim(); +} + +function isLikelyJSRendered(html: string): boolean { + // Extract body content + const bodyMatch = html.match(/]*>([\s\S]*?)<\/body>/i); + if (!bodyMatch) return false; + + const bodyHtml = bodyMatch[1]; + + // Strip tags to get text content + const textContent = collapseWhitespace(stripTags(stripElementBlocks(stripElementBlocks(bodyHtml, "script"), "style"))); + + // Count scripts + const scriptCount = (html.match(/ 3; +} + +async function extractViaHttp( + url: string, + signal?: AbortSignal, + options?: ExtractOptions, +): Promise { + const timeoutMs = options?.timeoutMs ?? DEFAULT_TIMEOUT_MS; + const activityId = activityMonitor.logStart({ type: "fetch", url }); + + const controller = new AbortController(); + const timeoutId = setTimeout(() => controller.abort(), timeoutMs); + + const onAbort = () => controller.abort(); + signal?.addEventListener("abort", onAbort); + + try { + const response = await fetch(url, { + signal: controller.signal, + headers: { + "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36", + "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8", + "Accept-Language": "en-US,en;q=0.9", + "Cache-Control": "no-cache", + "Sec-Fetch-Dest": "document", + "Sec-Fetch-Mode": "navigate", + "Sec-Fetch-Site": "none", + "Sec-Fetch-User": "?1", + "Upgrade-Insecure-Requests": "1", + }, + }); + + if (!response.ok) { + activityMonitor.logComplete(activityId, response.status); + return { + url, + title: "", + content: "", + error: `HTTP ${response.status}: ${response.statusText}`, + }; + } + + const contentLengthHeader = response.headers.get("content-length"); + const contentType = response.headers.get("content-type") || ""; + const isPDFContent = isPDF(url, contentType); + const maxResponseSize = isPDFContent ? 20 * 1024 * 1024 : 5 * 1024 * 1024; + if (contentLengthHeader) { + const contentLength = parseInt(contentLengthHeader, 10); + if (contentLength > maxResponseSize) { + activityMonitor.logComplete(activityId, response.status); + return { + url, + title: "", + content: "", + error: `Response too large (${Math.round(contentLength / 1024 / 1024)}MB)`, + }; + } + } + + if (isPDFContent) { + try { + const buffer = await response.arrayBuffer(); + const result = await extractPDFToMarkdown(buffer, url); + activityMonitor.logComplete(activityId, response.status); + return { + url, + title: result.title, + content: `PDF extracted and saved to: ${result.outputPath}\n\nPages: ${result.pages}\nCharacters: ${result.chars}`, + error: null, + }; + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + activityMonitor.logError(activityId, message); + return { url, title: "", content: "", error: `PDF extraction failed: ${message}` }; + } + } + + if (contentType.includes("application/octet-stream") || + contentType.includes("image/") || + contentType.includes("audio/") || + contentType.includes("video/") || + contentType.includes("application/zip")) { + activityMonitor.logComplete(activityId, response.status); + return { + url, + title: "", + content: "", + error: `Unsupported content type: ${contentType.split(";")[0]}`, + }; + } + + const text = await response.text(); + const isHTML = contentType.includes("text/html") || contentType.includes("application/xhtml+xml"); + + if (!isHTML) { + activityMonitor.logComplete(activityId, response.status); + const title = extractTextTitle(text, url); + return { url, title, content: text, error: null }; + } + + const { document } = parseHTML(text); + const reader = new Readability(document as unknown as Document); + const article = reader.parse(); + + if (!article) { + const rscResult = extractRSCContent(text); + if (rscResult) { + activityMonitor.logComplete(activityId, response.status); + return { url, title: rscResult.title, content: rscResult.content, error: null }; + } + + activityMonitor.logComplete(activityId, response.status); + + // Provide more specific error message + const jsRendered = isLikelyJSRendered(text); + const errorMsg = jsRendered + ? "Page appears to be JavaScript-rendered (content loads dynamically)" + : "Could not extract readable content from HTML structure"; + + return { + url, + title: "", + content: "", + error: errorMsg, + }; + } + + const markdown = turndown.turndown(article.content); + activityMonitor.logComplete(activityId, response.status); + + if (markdown.length < MIN_USEFUL_CONTENT) { + return { + url, + title: article.title || "", + content: markdown, + error: isLikelyJSRendered(text) + ? "Page appears to be JavaScript-rendered (content loads dynamically)" + : "Extracted content appears incomplete", + }; + } + + return { url, title: article.title || "", content: markdown, error: null }; + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + if (message.toLowerCase().includes("abort")) { + activityMonitor.logComplete(activityId, 0); + } else { + activityMonitor.logError(activityId, message); + } + return { url, title: "", content: "", error: message }; + } finally { + clearTimeout(timeoutId); + signal?.removeEventListener("abort", onAbort); + } +} + +export function extractHeadingTitle(text: string): string | null { + const match = text.match(/^#{1,2}\s+(.+)/m); + if (!match) return null; + const cleaned = match[1].replace(/\*+/g, "").trim(); + return cleaned || null; +} + +function extractTextTitle(text: string, url: string): string { + return extractHeadingTitle(text) ?? (new URL(url).pathname.split("/").pop() || url); +} + +export async function fetchAllContent( + urls: string[], + signal?: AbortSignal, + options?: ExtractOptions, +): Promise { + return Promise.all(urls.map((url) => fetchLimit(() => extractContent(url, signal, options)))); +} diff --git a/packages/web-access/gemini-api.ts b/packages/web-access/gemini-api.ts new file mode 100644 index 000000000..2647d14d9 --- /dev/null +++ b/packages/web-access/gemini-api.ts @@ -0,0 +1,113 @@ +import { existsSync, readFileSync } from "node:fs"; +import { homedir } from "node:os"; +import { join } from "node:path"; +import { CONFIG_DIR_NAME } from "@bastani/atomic"; + +export const API_BASE = "https://generativelanguage.googleapis.com/v1beta"; +const CONFIG_PATH = join(homedir(), CONFIG_DIR_NAME, "web-search.json"); +export const DEFAULT_MODEL = "gemini-3-flash-preview"; + +interface GeminiApiConfig { + geminiApiKey?: unknown; +} + +let cachedConfig: GeminiApiConfig | null = null; + +function loadConfig(): GeminiApiConfig { + if (cachedConfig) return cachedConfig; + if (!existsSync(CONFIG_PATH)) { + cachedConfig = {}; + return cachedConfig; + } + + const raw = readFileSync(CONFIG_PATH, "utf-8"); + try { + cachedConfig = JSON.parse(raw) as GeminiApiConfig; + return cachedConfig; + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + throw new Error(`Failed to parse ${CONFIG_PATH}: ${message}`); + } +} + +function withTimeout(signal: AbortSignal | undefined, timeoutMs: number): AbortSignal { + const timeout = AbortSignal.timeout(timeoutMs); + return signal ? AbortSignal.any([signal, timeout]) : timeout; +} + +function normalizeApiKey(value: unknown): string | null { + if (typeof value !== "string") return null; + const normalized = value.trim(); + return normalized.length > 0 ? normalized : null; +} + +export function getApiKey(): string | null { + return normalizeApiKey(process.env.GEMINI_API_KEY) ?? normalizeApiKey(loadConfig().geminiApiKey); +} + +export function isGeminiApiAvailable(): boolean { + return getApiKey() !== null; +} + +export interface GeminiApiOptions { + model?: string; + mimeType?: string; + signal?: AbortSignal; + timeoutMs?: number; +} + +export async function queryGeminiApiWithVideo( + prompt: string, + videoUri: string, + options: GeminiApiOptions = {}, +): Promise { + const apiKey = getApiKey(); + if (!apiKey) throw new Error("GEMINI_API_KEY not configured"); + + const model = options.model ?? DEFAULT_MODEL; + const signal = withTimeout(options.signal, options.timeoutMs ?? 120000); + const url = `${API_BASE}/models/${model}:generateContent?key=${apiKey}`; + + const fileData: Record = { fileUri: videoUri }; + if (options.mimeType) fileData.mimeType = options.mimeType; + + const body = { + contents: [ + { + parts: [ + { fileData }, + { text: prompt }, + ], + }, + ], + }; + + const res = await fetch(url, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify(body), + signal, + }); + + if (!res.ok) { + const errorText = await res.text(); + throw new Error(`Gemini API error ${res.status}: ${errorText.slice(0, 300)}`); + } + + const data = (await res.json()) as GenerateContentResponse; + const text = data.candidates?.[0]?.content?.parts + ?.map((p) => p.text) + .filter(Boolean) + .join("\n"); + + if (!text) throw new Error("Gemini API returned empty response"); + return text; +} + +interface GenerateContentResponse { + candidates?: Array<{ + content?: { + parts?: Array<{ text?: string }>; + }; + }>; +} diff --git a/packages/web-access/gemini-search.ts b/packages/web-access/gemini-search.ts new file mode 100644 index 000000000..2f6559321 --- /dev/null +++ b/packages/web-access/gemini-search.ts @@ -0,0 +1,362 @@ +import { existsSync, readFileSync } from "node:fs"; +import { homedir } from "node:os"; +import { join } from "node:path"; +import { CONFIG_DIR_NAME } from "@bastani/atomic"; +import { activityMonitor } from "./activity.js"; +import { getApiKey, API_BASE, DEFAULT_MODEL } from "./gemini-api.js"; +import { isGeminiWebAvailable, queryWithCookies } from "./gemini-web.js"; +import { isPerplexityAvailable, searchWithPerplexity, type SearchResult, type SearchResponse, type SearchOptions } from "./perplexity.js"; +import { hasExaApiKey, isExaAvailable, searchWithExa } from "./exa.js"; + +export type SearchProvider = "auto" | "perplexity" | "gemini" | "exa"; +export type ResolvedSearchProvider = Exclude; + +export interface AttributedSearchResponse extends SearchResponse { + provider: ResolvedSearchProvider; +} + +const CONFIG_PATH = join(homedir(), CONFIG_DIR_NAME, "web-search.json"); + +let cachedSearchConfig: { searchProvider: SearchProvider; searchModel?: string } | null = null; + +function getSearchConfig(): { searchProvider: SearchProvider; searchModel?: string } { + if (cachedSearchConfig) return cachedSearchConfig; + if (!existsSync(CONFIG_PATH)) { + cachedSearchConfig = { searchProvider: "auto", searchModel: undefined }; + return cachedSearchConfig; + } + + const rawText = readFileSync(CONFIG_PATH, "utf-8"); + let raw: { + searchProvider?: SearchProvider; + provider?: SearchProvider; + searchModel?: unknown; + }; + try { + raw = JSON.parse(rawText) as { + searchProvider?: SearchProvider; + provider?: SearchProvider; + searchModel?: unknown; + }; + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + throw new Error(`Failed to parse ${CONFIG_PATH}: ${message}`); + } + + cachedSearchConfig = { + searchProvider: normalizeSearchProvider(raw.searchProvider ?? raw.provider), + searchModel: normalizeSearchModel(raw.searchModel), + }; + return cachedSearchConfig; +} + +function normalizeSearchModel(value: unknown): string | undefined { + if (typeof value !== "string") return undefined; + const normalized = value.trim(); + return normalized.length > 0 ? normalized : undefined; +} + +function normalizeSearchProvider(value: unknown): SearchProvider { + const normalized = typeof value === "string" ? value.trim().toLowerCase() : ""; + return normalized === "auto" || normalized === "perplexity" || normalized === "gemini" || normalized === "exa" + ? normalized + : "auto"; +} + +export interface FullSearchOptions extends SearchOptions { + provider?: SearchProvider; + includeContent?: boolean; +} + +function errorMessage(err: unknown): string { + return err instanceof Error ? err.message : String(err); +} + +function isAbortError(err: unknown): boolean { + return errorMessage(err).toLowerCase().includes("abort"); +} + +async function searchWithGemini( + query: string, + options: SearchOptions, + strictErrors: boolean, +): Promise { + const errors: string[] = []; + + try { + const apiResult = await searchWithGeminiApi(query, options); + if (apiResult) return apiResult; + } catch (err) { + if (isAbortError(err)) throw err; + errors.push(`Gemini API: ${errorMessage(err)}`); + } + + try { + const webResult = await searchWithGeminiWeb(query, options); + if (webResult) return webResult; + } catch (err) { + if (isAbortError(err)) throw err; + errors.push(`Gemini Web: ${errorMessage(err)}`); + } + + if (strictErrors && errors.length > 0) { + throw new Error(`Gemini search failed:\n - ${errors.join("\n - ")}`); + } + + return null; +} + +export async function search(query: string, options: FullSearchOptions = {}): Promise { + const config = getSearchConfig(); + const provider = options.provider ?? config.searchProvider; + + if (provider === "perplexity") { + const result = await searchWithPerplexity(query, options); + return { ...result, provider: "perplexity" }; + } + + if (provider === "gemini") { + const result = await searchWithGemini(query, options, true); + if (result) return { ...result, provider: "gemini" }; + throw new Error( + "Gemini search unavailable. Either:\n" + + ` 1. Set GEMINI_API_KEY in ~/${CONFIG_DIR_NAME}/web-search.json\n` + + " 2. Sign into gemini.google.com in a supported Chromium-based browser" + ); + } + + if (provider === "exa") { + const exaApiKeyConfigured = hasExaApiKey(); + try { + const result = await searchWithExa(query, options); + if (result && "exhausted" in result) { + throw new Error( + "Exa monthly free tier exhausted (1,000 requests). Resets next month.\n" + + " Use provider: 'perplexity' or 'gemini', or upgrade at exa.ai/pricing" + ); + } + if (result && "answer" in result) return { ...result, provider: "exa" }; + if (exaApiKeyConfigured) { + throw new Error("Exa search returned no results."); + } + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + if (message.toLowerCase().includes("abort")) throw err; + if (exaApiKeyConfigured) throw err; + // No API key: allow provider fallback. + } + } + + const fallbackErrors: string[] = []; + + if (provider !== "exa" && isExaAvailable()) { + try { + const result = await searchWithExa(query, options); + if (result && "answer" in result) return { ...result, provider: "exa" }; + } catch (err) { + if (isAbortError(err)) throw err; + fallbackErrors.push(`Exa: ${errorMessage(err)}`); + } + } + + if (isPerplexityAvailable()) { + try { + const result = await searchWithPerplexity(query, options); + return { ...result, provider: "perplexity" }; + } catch (err) { + if (isAbortError(err)) throw err; + fallbackErrors.push(`Perplexity: ${errorMessage(err)}`); + } + } + + try { + const geminiResult = await searchWithGemini(query, options, false); + if (geminiResult) return { ...geminiResult, provider: "gemini" }; + } catch (err) { + if (isAbortError(err)) throw err; + fallbackErrors.push(`Gemini: ${errorMessage(err)}`); + } + + if (fallbackErrors.length > 0) { + throw new Error(`Auto provider search failed:\n - ${fallbackErrors.join("\n - ")}`); + } + + throw new Error( + "No search provider available. Either:\n" + + ` 1. Set perplexityApiKey in ~/${CONFIG_DIR_NAME}/web-search.json\n` + + ` 2. Set EXA_API_KEY (or exaApiKey) in ~/${CONFIG_DIR_NAME}/web-search.json\n` + + ` 3. Set GEMINI_API_KEY in ~/${CONFIG_DIR_NAME}/web-search.json\n` + + " 4. Sign into gemini.google.com in a supported Chromium-based browser" + ); +} + +async function searchWithGeminiApi(query: string, options: SearchOptions = {}): Promise { + const apiKey = getApiKey(); + if (!apiKey) return null; + + const activityId = activityMonitor.logStart({ type: "api", query }); + + try { + const model = getSearchConfig().searchModel ?? DEFAULT_MODEL; + const body = { + contents: [{ parts: [{ text: query }] }], + tools: [{ google_search: {} }], + }; + + const res = await fetch(`${API_BASE}/models/${model}:generateContent?key=${apiKey}`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify(body), + signal: AbortSignal.any([ + AbortSignal.timeout(60000), + ...(options.signal ? [options.signal] : []), + ]), + }); + + if (!res.ok) { + const errorText = await res.text(); + throw new Error(`Gemini API error ${res.status}: ${errorText.slice(0, 300)}`); + } + + const data = await res.json() as GeminiSearchResponse; + activityMonitor.logComplete(activityId, res.status); + + const answer = data.candidates?.[0]?.content?.parts + ?.map(p => p.text).filter(Boolean).join("\n") ?? ""; + + const metadata = data.candidates?.[0]?.groundingMetadata; + const results = await resolveGroundingChunks(metadata?.groundingChunks, options.signal); + + if (!answer && results.length === 0) return null; + return { answer, results }; + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + if (message.toLowerCase().includes("abort")) { + activityMonitor.logComplete(activityId, 0); + } else { + activityMonitor.logError(activityId, message); + } + throw err; + } +} + +async function searchWithGeminiWeb(query: string, options: SearchOptions = {}): Promise { + const cookies = await isGeminiWebAvailable(); + if (!cookies) return null; + + const prompt = buildSearchPrompt(query, options); + const activityId = activityMonitor.logStart({ type: "api", query }); + + try { + const text = await queryWithCookies(prompt, cookies, { + model: "gemini-3-flash-preview", + signal: options.signal, + timeoutMs: 60000, + }); + + activityMonitor.logComplete(activityId, 200); + + const results = extractSourceUrls(text); + return { answer: text, results }; + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + if (message.toLowerCase().includes("abort")) { + activityMonitor.logComplete(activityId, 0); + } else { + activityMonitor.logError(activityId, message); + } + throw err; + } +} + +function buildSearchPrompt(query: string, options: SearchOptions): string { + let prompt = `Search the web and answer the following question. Include source URLs for your claims.\nFormat your response as:\n1. A direct answer to the question\n2. Cited sources as markdown links\n\nQuestion: ${query}`; + + if (options.recencyFilter) { + const labels: Record = { + day: "past 24 hours", + week: "past week", + month: "past month", + year: "past year", + }; + prompt += `\n\nOnly include results from the ${labels[options.recencyFilter]}.`; + } + + if (options.domainFilter?.length) { + const includes = options.domainFilter.filter(d => !d.startsWith("-")); + const excludes = options.domainFilter.filter(d => d.startsWith("-")).map(d => d.slice(1)); + if (includes.length) prompt += `\n\nOnly cite sources from: ${includes.join(", ")}`; + if (excludes.length) prompt += `\n\nDo not cite sources from: ${excludes.join(", ")}`; + } + + return prompt; +} + +function extractSourceUrls(markdown: string): SearchResult[] { + const results: SearchResult[] = []; + const seen = new Set(); + const linkRegex = /\[([^\]]+)\]\((https?:\/\/[^)]+)\)/g; + for (const match of markdown.matchAll(linkRegex)) { + const url = match[2]; + if (seen.has(url)) continue; + seen.add(url); + results.push({ title: match[1], url, snippet: "" }); + } + return results; +} + +async function resolveGroundingChunks( + chunks: GroundingChunk[] | undefined, + signal?: AbortSignal, +): Promise { + if (!chunks?.length) return []; + + const results: SearchResult[] = []; + for (const chunk of chunks) { + if (!chunk.web) continue; + const title = chunk.web.title || ""; + let url = chunk.web.uri || ""; + + if (url.includes("vertexaisearch.cloud.google.com/grounding-api-redirect")) { + const resolved = await resolveRedirect(url, signal); + if (resolved) url = resolved; + } + + if (url) results.push({ title, url, snippet: "" }); + } + return results; +} + +async function resolveRedirect(proxyUrl: string, signal?: AbortSignal): Promise { + try { + const res = await fetch(proxyUrl, { + method: "HEAD", + redirect: "manual", + signal: AbortSignal.any([ + AbortSignal.timeout(5000), + ...(signal ? [signal] : []), + ]), + }); + return res.headers.get("location") || null; + } catch { + return null; + } +} + +interface GeminiSearchResponse { + candidates?: Array<{ + content?: { parts?: Array<{ text?: string }> }; + groundingMetadata?: { + webSearchQueries?: string[]; + groundingChunks?: GroundingChunk[]; + groundingSupports?: Array<{ + segment?: { startIndex?: number; endIndex?: number; text?: string }; + groundingChunkIndices?: number[]; + }>; + }; + }>; +} + +interface GroundingChunk { + web?: { uri?: string; title?: string }; +} diff --git a/packages/web-access/gemini-url-context.ts b/packages/web-access/gemini-url-context.ts new file mode 100644 index 000000000..e0219b482 --- /dev/null +++ b/packages/web-access/gemini-url-context.ts @@ -0,0 +1,126 @@ +import { activityMonitor } from "./activity.js"; +import { getApiKey, API_BASE, DEFAULT_MODEL } from "./gemini-api.js"; +import { isGeminiWebAvailable, queryWithCookies } from "./gemini-web.js"; +import { extractHeadingTitle, type ExtractedContent } from "./extract.js"; + +const EXTRACTION_PROMPT = `Extract the complete readable content from this URL as clean markdown. +Include the page title, all text content, code blocks, and tables. +Do not summarize — extract the full content. + +URL: `; + +function shouldRethrow(err: unknown): boolean { + const message = err instanceof Error ? err.message : String(err); + return message.startsWith("Failed to parse "); +} + +export async function extractWithUrlContext( + url: string, + signal?: AbortSignal, +): Promise { + const apiKey = getApiKey(); + if (!apiKey) return null; + + const activityId = activityMonitor.logStart({ type: "api", query: `url_context: ${url}` }); + + try { + const model = DEFAULT_MODEL; + const body = { + contents: [{ parts: [{ text: EXTRACTION_PROMPT + url }] }], + tools: [{ url_context: {} }], + }; + + const res = await fetch(`${API_BASE}/models/${model}:generateContent?key=${apiKey}`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify(body), + signal: AbortSignal.any([ + AbortSignal.timeout(60000), + ...(signal ? [signal] : []), + ]), + }); + + if (!res.ok) { + activityMonitor.logComplete(activityId, res.status); + return null; + } + + const data = await res.json() as UrlContextResponse; + activityMonitor.logComplete(activityId, res.status); + + const metadata = data.candidates?.[0]?.url_context_metadata; + if (metadata?.url_metadata?.length) { + const status = metadata.url_metadata[0].url_retrieval_status; + if (status === "URL_RETRIEVAL_STATUS_UNSAFE" || status === "URL_RETRIEVAL_STATUS_ERROR") { + return null; + } + } + + const content = data.candidates?.[0]?.content?.parts + ?.map(p => p.text).filter(Boolean).join("\n") ?? ""; + + if (!content || content.length < 50) return null; + + const title = extractTitleFromContent(content, url); + return { url, title, content, error: null }; + } catch (err) { + if (shouldRethrow(err)) throw err; + const message = err instanceof Error ? err.message : String(err); + if (message.toLowerCase().includes("abort")) { + activityMonitor.logComplete(activityId, 0); + } else { + activityMonitor.logError(activityId, message); + } + return null; + } +} + +export async function extractWithGeminiWeb( + url: string, + signal?: AbortSignal, +): Promise { + const cookies = await isGeminiWebAvailable(); + if (!cookies) return null; + + const activityId = activityMonitor.logStart({ type: "api", query: `gemini_web: ${url}` }); + + try { + const text = await queryWithCookies(EXTRACTION_PROMPT + url, cookies, { + model: "gemini-3-flash-preview", + signal, + timeoutMs: 60000, + }); + + activityMonitor.logComplete(activityId, 200); + + if (!text || text.length < 50) return null; + + const title = extractTitleFromContent(text, url); + return { url, title, content: text, error: null }; + } catch (err) { + if (shouldRethrow(err)) throw err; + const message = err instanceof Error ? err.message : String(err); + if (message.toLowerCase().includes("abort")) { + activityMonitor.logComplete(activityId, 0); + } else { + activityMonitor.logError(activityId, message); + } + return null; + } +} + +function extractTitleFromContent(text: string, url: string): string { + return extractHeadingTitle(text) ?? (new URL(url).pathname.split("/").pop() || url); +} + +interface UrlContextResponse { + candidates?: Array<{ + content?: { parts?: Array<{ text?: string }> }; + url_context_metadata?: { + url_metadata?: Array<{ + retrieved_url?: string; + url_retrieval_status?: string; + }>; + }; + }>; +} diff --git a/packages/web-access/gemini-web-config.ts b/packages/web-access/gemini-web-config.ts new file mode 100644 index 000000000..9750bacf2 --- /dev/null +++ b/packages/web-access/gemini-web-config.ts @@ -0,0 +1,54 @@ +import { existsSync, readFileSync } from "node:fs"; +import { homedir } from "node:os"; +import { join } from "node:path"; +import { APP_NAME, CONFIG_DIR_NAME } from "@bastani/atomic"; + +const CONFIG_PATH = join(homedir(), CONFIG_DIR_NAME, "web-search.json"); +const ALLOW_BROWSER_COOKIES_ENV = `${APP_NAME.toUpperCase()}_ALLOW_BROWSER_COOKIES`; + +interface GeminiWebConfig { + chromeProfile?: string; + allowBrowserCookies?: boolean; +} + +let cachedConfig: GeminiWebConfig | null = null; + +export function normalizeChromeProfile(value: unknown): string | undefined { + if (typeof value !== "string") return undefined; + const normalized = value.trim(); + return normalized.length > 0 ? normalized : undefined; +} + +function loadConfig(): GeminiWebConfig { + if (cachedConfig) return cachedConfig; + if (!existsSync(CONFIG_PATH)) { + cachedConfig = {}; + return cachedConfig; + } + + const rawText = readFileSync(CONFIG_PATH, "utf-8"); + let raw: { chromeProfile?: unknown; allowBrowserCookies?: unknown }; + try { + raw = JSON.parse(rawText) as { chromeProfile?: unknown; allowBrowserCookies?: unknown }; + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + throw new Error(`Failed to parse ${CONFIG_PATH}: ${message}`); + } + + cachedConfig = { + chromeProfile: normalizeChromeProfile(raw.chromeProfile), + allowBrowserCookies: raw.allowBrowserCookies === true, + }; + return cachedConfig; +} + +export function getChromeProfileFromConfig(): string | undefined { + return loadConfig().chromeProfile; +} + +export function isBrowserCookieAccessAllowed(): boolean { + if (process.env[ALLOW_BROWSER_COOKIES_ENV] === "1") { + return true; + } + return loadConfig().allowBrowserCookies === true; +} diff --git a/packages/web-access/gemini-web.ts b/packages/web-access/gemini-web.ts new file mode 100644 index 000000000..3dc070938 --- /dev/null +++ b/packages/web-access/gemini-web.ts @@ -0,0 +1,396 @@ +import { basename } from "node:path"; +import { type CookieMap, getGoogleCookies } from "./chrome-cookies.js"; +import { getChromeProfileFromConfig, isBrowserCookieAccessAllowed, normalizeChromeProfile } from "./gemini-web-config.ts"; + +const GEMINI_APP_URL = "https://gemini.google.com/app"; +const GEMINI_STREAM_GENERATE_URL = + "https://gemini.google.com/_/BardChatUi/data/assistant.lamda.BardFrontendService/StreamGenerate"; +const GEMINI_UPLOAD_URL = "https://content-push.googleapis.com/upload"; +const GEMINI_UPLOAD_PUSH_ID = "feeds/mcudyrk2a4khkz"; +const GOOGLE_LIST_ACCOUNTS_URL = + "https://accounts.google.com/ListAccounts?gpsia=1&source=ChromiumBrowser&laf=b64bin&json=standard"; + +const USER_AGENT = + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"; + +const MODEL_HEADER_NAME = "x-goog-ext-525001261-jspb"; +const MODEL_HEADERS: Record = { + "gemini-3-pro": '[1,null,null,null,"9d8ca3786ebdfbea",null,null,0,[4]]', + "gemini-2.5-pro": '[1,null,null,null,"4af6c7f5da75d65d",null,null,0,[4]]', + "gemini-2.5-flash": '[1,null,null,null,"9ec249fc9ad08861",null,null,0,[4]]', +}; + +const REQUIRED_COOKIES = ["__Secure-1PSID", "__Secure-1PSIDTS"]; + +export interface GeminiWebOptions { + youtubeUrl?: string; + model?: string; + files?: string[]; + signal?: AbortSignal; + timeoutMs?: number; +} + +export async function isGeminiWebAvailable(chromeProfile?: string): Promise { + if (!isBrowserCookieAccessAllowed()) return null; + + const result = await getGoogleCookies({ + profile: normalizeChromeProfile(chromeProfile) ?? getChromeProfileFromConfig(), + requiredCookies: REQUIRED_COOKIES, + }); + if (!result) return null; + return result.cookies; +} + +export async function getActiveGoogleEmail(cookies: CookieMap): Promise { + const cookieHeader = buildCookieHeader(cookies); + if (!cookieHeader) return null; + + try { + const html = await fetchWithCookieRedirects( + GEMINI_APP_URL, + cookieHeader, + 10, + AbortSignal.timeout(10000), + ); + const email = extractEmailFromGeminiHtml(html); + if (email) return email; + } catch { + } + + try { + const response = await fetchWithCookieRedirects( + GOOGLE_LIST_ACCOUNTS_URL, + cookieHeader, + 10, + AbortSignal.timeout(10000), + ); + return extractEmailFromListAccounts(response); + } catch { + return null; + } +} + +export async function queryWithCookies( + prompt: string, + cookieMap: CookieMap, + options: GeminiWebOptions = {}, +): Promise { + const model = options.model && MODEL_HEADERS[options.model] ? options.model : "gemini-2.5-flash"; + const timeoutMs = options.timeoutMs ?? 120000; + + let fullPrompt = prompt; + if (options.youtubeUrl) { + fullPrompt = `${fullPrompt}\n\nYouTube video: ${options.youtubeUrl}`; + } + + const result = await runGeminiWebOnce(fullPrompt, cookieMap, model, options.files, timeoutMs, options.signal); + + if (isModelUnavailable(result.errorCode) && model !== "gemini-2.5-flash") { + const fallback = await runGeminiWebOnce(fullPrompt, cookieMap, "gemini-2.5-flash", options.files, timeoutMs, options.signal); + if (fallback.errorMessage) throw new Error(fallback.errorMessage); + if (!fallback.text) throw new Error("Gemini Web returned empty response (fallback model)"); + return fallback.text; + } + + if (result.errorMessage) throw new Error(result.errorMessage); + if (!result.text) throw new Error("Gemini Web returned empty response"); + return result.text; +} + +interface GeminiWebResult { + text: string; + errorCode?: number; + errorMessage?: string; +} + +async function runGeminiWebOnce( + prompt: string, + cookieMap: CookieMap, + model: string, + files: string[] | undefined, + timeoutMs: number, + signal?: AbortSignal, +): Promise { + const effectiveSignal = withTimeout(signal, timeoutMs); + const cookieHeader = buildCookieHeader(cookieMap); + const accessToken = await fetchAccessToken(cookieHeader, effectiveSignal); + + const uploaded: Array<{ id: string; name: string }> = []; + if (files) { + for (const filePath of files) { + uploaded.push(await uploadFile(filePath, cookieHeader, effectiveSignal)); + } + } + + const fReq = buildFReqPayload(prompt, uploaded); + const params = new URLSearchParams(); + params.set("at", accessToken); + params.set("f.req", fReq); + + const res = await fetch(GEMINI_STREAM_GENERATE_URL, { + method: "POST", + headers: { + "content-type": "application/x-www-form-urlencoded;charset=utf-8", + host: "gemini.google.com", + origin: "https://gemini.google.com", + referer: "https://gemini.google.com/", + "x-same-domain": "1", + "user-agent": USER_AGENT, + cookie: cookieHeader, + [MODEL_HEADER_NAME]: MODEL_HEADERS[model], + }, + body: params.toString(), + signal: effectiveSignal, + }); + + const rawText = await res.text(); + + if (!res.ok) { + return { text: "", errorMessage: `Gemini request failed: ${res.status}` }; + } + + try { + return parseStreamGenerateResponse(rawText); + } catch (err) { + let errorCode: number | undefined; + try { + const json = JSON.parse(trimJsonEnvelope(rawText)); + errorCode = extractErrorCode(json); + } catch { + } + return { + text: "", + errorCode, + errorMessage: err instanceof Error ? err.message : String(err), + }; + } +} + +async function fetchAccessToken( + cookieHeader: string, + signal: AbortSignal, +): Promise { + const html = await fetchWithCookieRedirects(GEMINI_APP_URL, cookieHeader, 10, signal); + + for (const key of ["SNlM0e", "thykhd"]) { + const match = html.match(new RegExp(`"${key}":"(.*?)"`)); + if (match?.[1]) return match[1]; + } + + throw new Error("Unable to authenticate with Gemini. Make sure you're signed into gemini.google.com in a supported Chromium-based browser."); +} + +async function fetchWithCookieRedirects( + url: string, + cookieHeader: string, + maxRedirects: number, + signal: AbortSignal, +): Promise { + let current = url; + for (let i = 0; i <= maxRedirects; i++) { + const res = await fetch(current, { + headers: { "user-agent": USER_AGENT, cookie: cookieHeader }, + redirect: "manual", + signal, + }); + if (res.status >= 300 && res.status < 400) { + const location = res.headers.get("location"); + if (location) { + current = new URL(location, current).toString(); + continue; + } + } + return await res.text(); + } + throw new Error(`Too many redirects (>${maxRedirects})`); +} + +function extractEmailFromGeminiHtml(html: string): string | null { + const patterns = [ + /"email"\s*:\s*"([^"]+)"/, + /"displayEmail"\s*:\s*"([^"]+)"/, + /"identifier"\s*:\s*"([^"]+)"/, + /"defaultEmail"\s*:\s*"([^"]+)"/, + /"gaiaIdentifier"\s*:\s*"([^"]+)"/, + ]; + + for (const pattern of patterns) { + const match = html.match(pattern); + const email = normalizeEmail(match?.[1]); + if (email) return email; + } + + return findFirstEmail(html); +} + +function extractEmailFromListAccounts(text: string): string | null { + const trimmed = text.replace(/^\)\]\}'\s*/, ""); + try { + return findEmailInValue(JSON.parse(trimmed)) ?? findFirstEmail(trimmed); + } catch { + return findFirstEmail(trimmed); + } +} + +function findEmailInValue(value: unknown): string | null { + if (typeof value === "string") return normalizeEmail(value); + if (Array.isArray(value)) { + for (const item of value) { + const email = findEmailInValue(item); + if (email) return email; + } + return null; + } + if (value && typeof value === "object") { + for (const item of Object.values(value as Record)) { + const email = findEmailInValue(item); + if (email) return email; + } + } + return null; +} + +function findFirstEmail(text: string): string | null { + const normalized = decodeEmailEscapes(text); + const match = normalized.match(/\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}\b/i); + return match?.[0] ?? null; +} + +function normalizeEmail(value: string | undefined): string | null { + if (!value) return null; + const normalized = decodeEmailEscapes(value.trim()); + return /^[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}$/i.test(normalized) ? normalized : null; +} + +function decodeEmailEscapes(value: string): string { + return value + .replace(/\\u0040/gi, "@") + .replace(/\\x40/gi, "@") + .replace(/@/gi, "@") + .replace(/@/gi, "@") + .replace(/\\"/g, "\"") + .replace(/\\\\/g, "\\"); +} + +async function uploadFile( + filePath: string, + cookieHeader: string, + signal: AbortSignal, +): Promise<{ id: string; name: string }> { + const data = readFileSync(filePath); + const fileName = basename(filePath); + const boundary = "----FormBoundary" + Math.random().toString(36).slice(2); + const header = `--${boundary}\r\nContent-Disposition: form-data; name="file"; filename="${fileName}"\r\nContent-Type: application/octet-stream\r\n\r\n`; + const footer = `\r\n--${boundary}--\r\n`; + + const body = Buffer.concat([ + Buffer.from(header, "utf-8"), + data, + Buffer.from(footer, "utf-8"), + ]); + + const res = await fetch(GEMINI_UPLOAD_URL, { + method: "POST", + headers: { + "content-type": `multipart/form-data; boundary=${boundary}`, + "push-id": GEMINI_UPLOAD_PUSH_ID, + "user-agent": USER_AGENT, + cookie: cookieHeader, + }, + body, + signal, + }); + + if (!res.ok) { + const text = await res.text(); + throw new Error(`File upload failed: ${res.status} (${text.slice(0, 200)})`); + } + + return { id: await res.text(), name: fileName }; +} + +function buildFReqPayload( + prompt: string, + uploaded: Array<{ id: string; name: string }>, +): string { + const promptPayload = + uploaded.length > 0 + ? [prompt, 0, null, uploaded.map((file) => [[file.id, 1]])] + : [prompt]; + const innerList = [promptPayload, null, null]; + return JSON.stringify([null, JSON.stringify(innerList)]); +} + +function withTimeout(signal: AbortSignal | undefined, timeoutMs: number): AbortSignal { + const timeout = AbortSignal.timeout(timeoutMs); + return signal ? AbortSignal.any([signal, timeout]) : timeout; +} + +function buildCookieHeader(cookieMap: CookieMap): string { + return Object.entries(cookieMap) + .filter(([, value]) => typeof value === "string" && value.length > 0) + .map(([name, value]) => `${name}=${value}`) + .join("; "); +} + +function getNestedValue(value: unknown, pathParts: number[]): unknown { + let current: unknown = value; + for (const part of pathParts) { + if (current == null) return undefined; + if (!Array.isArray(current)) return undefined; + current = (current as unknown[])[part]; + } + return current; +} + +function trimJsonEnvelope(text: string): string { + const start = text.indexOf("["); + const end = text.lastIndexOf("]"); + if (start === -1 || end === -1 || end <= start) { + throw new Error("Gemini response did not contain a JSON payload."); + } + return text.slice(start, end + 1); +} + +function extractErrorCode(responseJson: unknown): number | undefined { + const code = getNestedValue(responseJson, [0, 5, 2, 0, 1, 0]); + return typeof code === "number" && code >= 0 ? code : undefined; +} + +function isModelUnavailable(errorCode: number | undefined): boolean { + return errorCode === 1052; +} + +function parseStreamGenerateResponse(rawText: string): GeminiWebResult { + const responseJson = JSON.parse(trimJsonEnvelope(rawText)); + const errorCode = extractErrorCode(responseJson); + + const parts = Array.isArray(responseJson) ? responseJson : []; + let body: unknown = null; + + for (let i = 0; i < parts.length; i++) { + const partBody = getNestedValue(parts[i], [2]); + if (!partBody || typeof partBody !== "string") continue; + try { + const parsed = JSON.parse(partBody); + const candidateList = getNestedValue(parsed, [4]); + if (Array.isArray(candidateList) && candidateList.length > 0) { + body = parsed; + break; + } + } catch { + } + } + + const candidateList = getNestedValue(body, [4]); + const firstCandidate = Array.isArray(candidateList) ? (candidateList as unknown[])[0] : undefined; + const textRaw = getNestedValue(firstCandidate, [1, 0]) as string | undefined; + + let text = textRaw ?? ""; + if (/^http:\/\/googleusercontent\.com\/card_content\/\d+/.test(text)) { + const alt = getNestedValue(firstCandidate, [22, 0]) as string | undefined; + if (alt) text = alt; + } + + return { text, errorCode }; +} diff --git a/packages/web-access/github-api.ts b/packages/web-access/github-api.ts new file mode 100644 index 000000000..a613b0383 --- /dev/null +++ b/packages/web-access/github-api.ts @@ -0,0 +1,196 @@ +import { execFile } from "node:child_process"; +import type { ExtractedContent } from "./extract.js"; +import type { GitHubUrlInfo } from "./github-extract.js"; + +const MAX_TREE_ENTRIES = 200; +const MAX_INLINE_FILE_CHARS = 100_000; + +let ghAvailable: boolean | null = null; +let ghHintShown = false; + +export async function checkGhAvailable(): Promise { + if (ghAvailable !== null) return ghAvailable; + + return new Promise((resolve) => { + execFile("gh", ["--version"], { timeout: 5000 }, (err) => { + ghAvailable = !err; + resolve(ghAvailable); + }); + }); +} + +export function showGhHint(): void { + if (!ghHintShown) { + ghHintShown = true; + console.error("[pi-web-access] Install `gh` CLI for better GitHub repo access including private repos."); + } +} + +export async function checkRepoSize(owner: string, repo: string): Promise { + if (!(await checkGhAvailable())) return null; + + return new Promise((resolve) => { + execFile("gh", ["api", `repos/${owner}/${repo}`, "--jq", ".size"], { timeout: 10000 }, (err, stdout) => { + if (err) { + resolve(null); + return; + } + const kb = parseInt(stdout.trim(), 10); + resolve(Number.isNaN(kb) ? null : kb); + }); + }); +} + +async function getDefaultBranch(owner: string, repo: string): Promise { + if (!(await checkGhAvailable())) return null; + + return new Promise((resolve) => { + execFile("gh", ["api", `repos/${owner}/${repo}`, "--jq", ".default_branch"], { timeout: 10000 }, (err, stdout) => { + if (err) { + resolve(null); + return; + } + const branch = stdout.trim(); + resolve(branch || null); + }); + }); +} + +async function fetchTreeViaApi(owner: string, repo: string, ref: string): Promise { + if (!(await checkGhAvailable())) return null; + + return new Promise((resolve) => { + execFile( + "gh", + ["api", `repos/${owner}/${repo}/git/trees/${ref}?recursive=1`, "--jq", ".tree[].path"], + { timeout: 15000, maxBuffer: 5 * 1024 * 1024 }, + (err, stdout) => { + if (err) { + resolve(null); + return; + } + const paths = stdout.trim().split("\n").filter(Boolean); + if (paths.length === 0) { + resolve(null); + return; + } + const truncated = paths.length > MAX_TREE_ENTRIES; + const display = paths.slice(0, MAX_TREE_ENTRIES).join("\n"); + resolve(truncated ? display + `\n... (${paths.length} total entries)` : display); + }, + ); + }); +} + +async function fetchReadmeViaApi(owner: string, repo: string, ref: string): Promise { + if (!(await checkGhAvailable())) return null; + + return new Promise((resolve) => { + execFile( + "gh", + ["api", `repos/${owner}/${repo}/readme?ref=${ref}`, "--jq", ".content"], + { timeout: 10000 }, + (err, stdout) => { + if (err) { + resolve(null); + return; + } + try { + const decoded = Buffer.from(stdout.trim(), "base64").toString("utf-8"); + resolve(decoded.length > 8192 ? decoded.slice(0, 8192) + "\n\n[README truncated at 8K chars]" : decoded); + } catch { + resolve(null); + } + }, + ); + }); +} + +async function fetchFileViaApi(owner: string, repo: string, path: string, ref: string): Promise { + if (!(await checkGhAvailable())) return null; + + return new Promise((resolve) => { + execFile( + "gh", + ["api", `repos/${owner}/${repo}/contents/${path}?ref=${ref}`, "--jq", ".content"], + { timeout: 10000, maxBuffer: 2 * 1024 * 1024 }, + (err, stdout) => { + if (err) { + resolve(null); + return; + } + try { + resolve(Buffer.from(stdout.trim(), "base64").toString("utf-8")); + } catch { + resolve(null); + } + }, + ); + }); +} + +export async function fetchViaApi( + url: string, + owner: string, + repo: string, + info: GitHubUrlInfo, + sizeNote?: string, +): Promise { + const ref = info.ref || (await getDefaultBranch(owner, repo)); + if (!ref) return null; + + const lines: string[] = []; + if (sizeNote) { + lines.push(sizeNote); + lines.push(""); + } + + if (info.type === "blob" && info.path) { + const content = await fetchFileViaApi(owner, repo, info.path, ref); + if (!content) return null; + + lines.push(`## ${info.path}`); + if (content.length > MAX_INLINE_FILE_CHARS) { + lines.push(content.slice(0, MAX_INLINE_FILE_CHARS)); + lines.push(`\n[File truncated at 100K chars]`); + } else { + lines.push(content); + } + + return { + url, + title: `${owner}/${repo} - ${info.path}`, + content: lines.join("\n"), + error: null, + }; + } + + const [tree, readme] = await Promise.all([ + fetchTreeViaApi(owner, repo, ref), + fetchReadmeViaApi(owner, repo, ref), + ]); + + if (!tree && !readme) return null; + + if (tree) { + lines.push("## Structure"); + lines.push(tree); + lines.push(""); + } + + if (readme) { + lines.push("## README.md"); + lines.push(readme); + lines.push(""); + } + + lines.push("This is an API-only view. Clone the repo or use `read`/`bash` for deeper exploration."); + + const title = info.path ? `${owner}/${repo} - ${info.path}` : `${owner}/${repo}`; + return { + url, + title, + content: lines.join("\n"), + error: null, + }; +} diff --git a/packages/web-access/github-extract.ts b/packages/web-access/github-extract.ts new file mode 100644 index 000000000..238b237ee --- /dev/null +++ b/packages/web-access/github-extract.ts @@ -0,0 +1,635 @@ +import { existsSync, readFileSync, rmSync, statSync, readdirSync, openSync, readSync, closeSync, realpathSync } from "node:fs"; +import { execFile } from "node:child_process"; +import { homedir } from "node:os"; +import { extname, join, resolve as resolvePath, sep as pathSep } from "node:path"; +import { CONFIG_DIR_NAME } from "@bastani/atomic"; +import { activityMonitor } from "./activity.js"; +import type { ExtractedContent } from "./extract.js"; +import { checkGhAvailable, checkRepoSize, fetchViaApi, showGhHint } from "./github-api.js"; + +const CONFIG_PATH = join(homedir(), CONFIG_DIR_NAME, "web-search.json"); + +const BINARY_EXTENSIONS = new Set([ + ".png", ".jpg", ".jpeg", ".gif", ".bmp", ".ico", ".webp", ".svg", ".tiff", ".tif", + ".mp3", ".mp4", ".avi", ".mov", ".mkv", ".flv", ".wmv", ".wav", ".ogg", ".webm", ".flac", ".aac", + ".zip", ".tar", ".gz", ".bz2", ".xz", ".7z", ".rar", ".zst", + ".exe", ".dll", ".so", ".dylib", ".bin", ".o", ".a", ".lib", + ".woff", ".woff2", ".ttf", ".otf", ".eot", + ".pdf", ".doc", ".docx", ".xls", ".xlsx", ".ppt", ".pptx", + ".sqlite", ".db", ".sqlite3", + ".pyc", ".pyo", ".class", ".jar", ".war", + ".iso", ".img", ".dmg", +]); + +const NOISE_DIRS = new Set([ + "node_modules", "vendor", ".next", "dist", "build", "__pycache__", + ".venv", "venv", ".tox", ".mypy_cache", ".pytest_cache", + "target", ".gradle", ".idea", ".vscode", +]); + +const MAX_INLINE_FILE_CHARS = 100_000; +const MAX_TREE_ENTRIES = 200; + +export interface GitHubUrlInfo { + owner: string; + repo: string; + ref?: string; + refIsFullSha: boolean; + path?: string; + type: "root" | "blob" | "tree"; +} + +interface CachedClone { + localPath: string; + clonePromise: Promise; +} + +interface GitHubCloneConfig { + enabled: boolean; + maxRepoSizeMB: number; + cloneTimeoutSeconds: number; + clonePath: string; +} + +const cloneCache = new Map(); + +let cachedConfig: GitHubCloneConfig | null = null; + +function normalizeEnabled(value: unknown, fallback: boolean): boolean { + return typeof value === "boolean" ? value : fallback; +} + +function normalizePositiveNumber(value: unknown, fallback: number): number { + if (typeof value !== "number" || !Number.isFinite(value)) return fallback; + return value > 0 ? value : fallback; +} + +function normalizeClonePath(value: unknown, fallback: string): string { + if (typeof value !== "string") return fallback; + const normalized = value.trim(); + return normalized.length > 0 ? normalized : fallback; +} + +function loadGitHubConfig(): GitHubCloneConfig { + if (cachedConfig) return cachedConfig; + + const defaults: GitHubCloneConfig = { + enabled: true, + maxRepoSizeMB: 350, + cloneTimeoutSeconds: 30, + clonePath: "/tmp/pi-github-repos", + }; + + if (!existsSync(CONFIG_PATH)) { + cachedConfig = defaults; + return cachedConfig; + } + + const rawText = readFileSync(CONFIG_PATH, "utf-8"); + let raw: { githubClone?: { enabled?: unknown; maxRepoSizeMB?: unknown; cloneTimeoutSeconds?: unknown; clonePath?: unknown } }; + try { + raw = JSON.parse(rawText) as { githubClone?: { enabled?: unknown; maxRepoSizeMB?: unknown; cloneTimeoutSeconds?: unknown; clonePath?: unknown } }; + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + throw new Error(`Failed to parse ${CONFIG_PATH}: ${message}`); + } + + const gc = raw.githubClone ?? {}; + cachedConfig = { + enabled: normalizeEnabled(gc.enabled, defaults.enabled), + maxRepoSizeMB: normalizePositiveNumber(gc.maxRepoSizeMB, defaults.maxRepoSizeMB), + cloneTimeoutSeconds: normalizePositiveNumber(gc.cloneTimeoutSeconds, defaults.cloneTimeoutSeconds), + clonePath: normalizeClonePath(gc.clonePath, defaults.clonePath), + }; + return cachedConfig; +} + +const NON_CODE_SEGMENTS = new Set([ + "issues", "pull", "pulls", "discussions", "releases", "wiki", + "actions", "settings", "security", "projects", "graphs", + "compare", "commits", "tags", "branches", "stargazers", + "watchers", "network", "forks", "milestone", "labels", + "packages", "codespaces", "contribute", "community", + "sponsors", "invitations", "notifications", "insights", +]); + +export function parseGitHubUrl(url: string): GitHubUrlInfo | null { + let parsed: URL; + try { + parsed = new URL(url); + } catch { + return null; + } + + const host = parsed.hostname.toLowerCase(); + if (host !== "github.com" && host !== "www.github.com") return null; + + const segments = parsed.pathname + .split("/") + .filter(Boolean) + .map((segment) => { + try { + return decodeURIComponent(segment); + } catch { + return segment; + } + }); + if (segments.length < 2) return null; + + const owner = segments[0]; + const repo = segments[1].replace(/\.git$/, ""); + + if (NON_CODE_SEGMENTS.has(segments[2]?.toLowerCase())) return null; + + if (segments.length === 2) { + return { owner, repo, refIsFullSha: false, type: "root" }; + } + + const action = segments[2]; + if (action !== "blob" && action !== "tree") return null; + if (segments.length < 4) return null; + + const ref = segments[3]; + const refIsFullSha = /^[0-9a-f]{40}$/.test(ref); + const pathParts = segments.slice(4); + const path = pathParts.length > 0 ? pathParts.join("/") : ""; + + return { + owner, + repo, + ref, + refIsFullSha, + path, + type: action as "blob" | "tree", + }; +} + +function cacheKey(owner: string, repo: string, ref?: string): string { + return ref ? `${owner}/${repo}@${ref}` : `${owner}/${repo}`; +} + +function cloneDir(config: GitHubCloneConfig, owner: string, repo: string, ref?: string): string { + const dirName = ref ? `${repo}@${ref}` : repo; + return join(config.clonePath, owner, dirName); +} + +function execClone(args: string[], localPath: string, timeoutMs: number, signal?: AbortSignal): Promise { + return new Promise((resolve) => { + const child = execFile(args[0], args.slice(1), { timeout: timeoutMs }, (err) => { + if (err) { + try { + rmSync(localPath, { recursive: true, force: true }); + } catch { + } + resolve(null); + return; + } + resolve(localPath); + }); + + if (signal) { + const onAbort = () => child.kill(); + signal.addEventListener("abort", onAbort, { once: true }); + child.on("exit", () => signal.removeEventListener("abort", onAbort)); + } + }); +} + +async function cloneRepo( + owner: string, + repo: string, + ref: string | undefined, + config: GitHubCloneConfig, + signal?: AbortSignal, +): Promise { + const localPath = cloneDir(config, owner, repo, ref); + + try { + rmSync(localPath, { recursive: true, force: true }); + } catch { + } + + const timeoutMs = config.cloneTimeoutSeconds * 1000; + const hasGh = await checkGhAvailable(); + + if (hasGh) { + const args = ["gh", "repo", "clone", `${owner}/${repo}`, localPath, "--", "--depth", "1", "--single-branch"]; + if (ref) args.push("--branch", ref); + return execClone(args, localPath, timeoutMs, signal); + } + + showGhHint(); + + const gitUrl = `https://github.com/${owner}/${repo}.git`; + const args = ["git", "clone", "--depth", "1", "--single-branch"]; + if (ref) args.push("--branch", ref); + args.push(gitUrl, localPath); + return execClone(args, localPath, timeoutMs, signal); +} + +function isBinaryFile(filePath: string): boolean { + const ext = extname(filePath).toLowerCase(); + if (BINARY_EXTENSIONS.has(ext)) return true; + + let fd: number; + try { + fd = openSync(filePath, "r"); + } catch { + return false; + } + try { + const buf = Buffer.alloc(512); + const bytesRead = readSync(fd, buf, 0, 512, 0); + for (let i = 0; i < bytesRead; i++) { + if (buf[i] === 0) return true; + } + } catch { + return false; + } finally { + closeSync(fd); + } + + return false; +} + +function formatFileSize(bytes: number): string { + if (bytes < 1024) return `${bytes} B`; + if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)} KB`; + return `${(bytes / (1024 * 1024)).toFixed(1)} MB`; +} + +function resolveWithinRepo(rootPath: string, relativePath: string): string | null { + const normalizedRoot = resolvePath(rootPath); + const candidate = resolvePath(normalizedRoot, relativePath); + if (candidate !== normalizedRoot) { + const rootPrefix = normalizedRoot.endsWith(pathSep) ? normalizedRoot : normalizedRoot + pathSep; + if (!candidate.startsWith(rootPrefix)) return null; + } + + if (!existsSync(candidate)) return candidate; + + try { + const realRoot = realpathSync(normalizedRoot); + const realCandidate = realpathSync(candidate); + if (realCandidate === realRoot) return candidate; + const realRootPrefix = realRoot.endsWith(pathSep) ? realRoot : realRoot + pathSep; + return realCandidate.startsWith(realRootPrefix) ? candidate : null; + } catch { + return null; + } +} + +function readTextFile(path: string): string | null { + try { + return readFileSync(path, "utf-8"); + } catch { + return null; + } +} + +function buildTree(rootPath: string): string { + const entries: string[] = []; + + function walk(dir: string, relPath: string): void { + if (entries.length >= MAX_TREE_ENTRIES) return; + + let items: string[]; + try { + items = readdirSync(dir).sort(); + } catch { + return; + } + + for (const item of items) { + if (entries.length >= MAX_TREE_ENTRIES) return; + if (item === ".git") continue; + + const rel = relPath ? `${relPath}/${item}` : item; + const safePath = resolveWithinRepo(rootPath, rel); + if (!safePath) { + entries.push(`${rel} [outside repo skipped]`); + continue; + } + + let stat; + try { + stat = statSync(safePath); + } catch { + continue; + } + + if (stat.isDirectory()) { + if (NOISE_DIRS.has(item)) { + entries.push(`${rel}/ [skipped]`); + continue; + } + entries.push(`${rel}/`); + walk(safePath, rel); + } else { + entries.push(rel); + } + } + } + + walk(rootPath, ""); + + if (entries.length >= MAX_TREE_ENTRIES) { + entries.push(`... (truncated at ${MAX_TREE_ENTRIES} entries)`); + } + + return entries.join("\n"); +} + +function buildDirListing(rootPath: string, subPath: string): string { + const targetPath = resolveWithinRepo(rootPath, subPath); + if (!targetPath) return "(path escapes repository root)"; + const lines: string[] = []; + + let items: string[]; + try { + items = readdirSync(targetPath).sort(); + } catch { + return "(directory not readable)"; + } + + for (const item of items) { + if (item === ".git") continue; + const rel = subPath ? `${subPath}/${item}` : item; + const safePath = resolveWithinRepo(rootPath, rel); + if (!safePath) { + lines.push(` ${item} (outside repo)`); + continue; + } + try { + const stat = statSync(safePath); + if (stat.isDirectory()) { + lines.push(` ${item}/`); + } else { + lines.push(` ${item} (${formatFileSize(stat.size)})`); + } + } catch { + lines.push(` ${item} (unreadable)`); + } + } + + return lines.join("\n"); +} + +function readReadme(localPath: string): string | null { + const candidates = ["README.md", "readme.md", "README", "README.txt", "README.rst"]; + for (const name of candidates) { + const readmePath = join(localPath, name); + if (existsSync(readmePath)) { + try { + const content = readFileSync(readmePath, "utf-8"); + return content.length > 8192 ? content.slice(0, 8192) + "\n\n[README truncated at 8K chars]" : content; + } catch { + continue; + } + } + } + return null; +} + +function generateContent(localPath: string, info: GitHubUrlInfo): string { + const lines: string[] = []; + lines.push(`Repository cloned to: ${localPath}`); + lines.push(""); + + if (info.type === "root") { + lines.push("## Structure"); + lines.push(buildTree(localPath)); + lines.push(""); + + const readme = readReadme(localPath); + if (readme) { + lines.push("## README.md"); + lines.push(readme); + lines.push(""); + } + + lines.push("Use `read` and `bash` tools at the path above to explore further."); + return lines.join("\n"); + } + + if (info.type === "tree") { + const dirPath = info.path || ""; + const fullDirPath = resolveWithinRepo(localPath, dirPath); + + if (!fullDirPath || !existsSync(fullDirPath)) { + lines.push(`Path \`${dirPath}\` not found in clone. Showing repository root instead.`); + lines.push(""); + lines.push("## Structure"); + lines.push(buildTree(localPath)); + } else { + lines.push(`## ${dirPath || "/"}`); + lines.push(buildDirListing(localPath, dirPath)); + } + + lines.push(""); + lines.push("Use `read` and `bash` tools at the path above to explore further."); + return lines.join("\n"); + } + + if (info.type === "blob") { + const filePath = info.path || ""; + const fullFilePath = resolveWithinRepo(localPath, filePath); + + if (!fullFilePath || !existsSync(fullFilePath)) { + lines.push(`Path \`${filePath}\` not found in clone. Showing repository root instead.`); + lines.push(""); + lines.push("## Structure"); + lines.push(buildTree(localPath)); + lines.push(""); + lines.push("Use `read` and `bash` tools at the path above to explore further."); + return lines.join("\n"); + } + + let stat: ReturnType; + try { + stat = statSync(fullFilePath); + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + lines.push(`Could not inspect \`${filePath}\`: ${message}`); + lines.push(""); + lines.push("Use `read` and `bash` tools at the path above to explore further."); + return lines.join("\n"); + } + + if (stat.isDirectory()) { + lines.push(`## ${filePath || "/"}`); + lines.push(buildDirListing(localPath, filePath)); + lines.push(""); + lines.push("Use `read` and `bash` tools at the path above to explore further."); + return lines.join("\n"); + } + + if (isBinaryFile(fullFilePath)) { + const ext = extname(filePath).replace(".", ""); + lines.push(`## ${filePath}`); + lines.push(`Binary file (${ext}, ${formatFileSize(stat.size)}). Use \`read\` or \`bash\` tools at the path above to inspect.`); + return lines.join("\n"); + } + + const content = readTextFile(fullFilePath); + if (content === null) { + lines.push(`Could not read \`${filePath}\` as UTF-8 text.`); + lines.push(""); + lines.push("Use `read` and `bash` tools at the path above to explore further."); + return lines.join("\n"); + } + lines.push(`## ${filePath}`); + + if (content.length > MAX_INLINE_FILE_CHARS) { + lines.push(content.slice(0, MAX_INLINE_FILE_CHARS)); + lines.push(""); + lines.push(`[File truncated at 100K chars. Full file: ${fullFilePath}]`); + } else { + lines.push(content); + } + + lines.push(""); + lines.push("Use `read` and `bash` tools at the path above to explore further."); + return lines.join("\n"); + } + + return lines.join("\n"); +} + +async function awaitCachedClone( + cached: CachedClone, + url: string, + owner: string, + repo: string, + info: GitHubUrlInfo, + signal?: AbortSignal, +): Promise { + if (signal?.aborted) return null; + const result = await cached.clonePromise; + if (signal?.aborted) return null; + if (result) { + const content = generateContent(result, info); + const title = info.path ? `${owner}/${repo} - ${info.path}` : `${owner}/${repo}`; + return { url, title, content, error: null }; + } + return fetchViaApi(url, owner, repo, info); +} + +export async function extractGitHub( + url: string, + signal?: AbortSignal, + forceClone?: boolean, +): Promise { + const info = parseGitHubUrl(url); + if (!info) return null; + + if (signal?.aborted) return null; + + const config = loadGitHubConfig(); + if (!config.enabled) return null; + + const { owner, repo } = info; + const key = cacheKey(owner, repo, info.ref); + + const cached = cloneCache.get(key); + if (cached) return awaitCachedClone(cached, url, owner, repo, info, signal); + + if (info.refIsFullSha) { + if (signal?.aborted) return null; + const sizeNote = `Note: Commit SHA URLs use the GitHub API instead of cloning.`; + return fetchViaApi(url, owner, repo, info, sizeNote); + } + + const activityId = activityMonitor.logStart({ type: "fetch", url: `github.com/${owner}/${repo}` }); + + if (!forceClone) { + const sizeKB = await checkRepoSize(owner, repo); + if (signal?.aborted) { + activityMonitor.logComplete(activityId, 0); + return null; + } + if (sizeKB !== null) { + const sizeMB = sizeKB / 1024; + if (sizeMB > config.maxRepoSizeMB) { + if (signal?.aborted) { + activityMonitor.logComplete(activityId, 0); + return null; + } + const sizeNote = + `Note: Repository is ${Math.round(sizeMB)}MB (threshold: ${config.maxRepoSizeMB}MB). ` + + `Showing API-fetched content instead of full clone. Ask the user if they'd like to clone the full repo -- ` + + `if yes, call fetch_content again with the same URL and add forceClone: true to the params.`; + const apiView = await fetchViaApi(url, owner, repo, info, sizeNote); + if (apiView) { + activityMonitor.logComplete(activityId, 200); + return apiView; + } + activityMonitor.logError(activityId, "api fallback unavailable for oversized repository"); + return null; + } + } + } + + if (signal?.aborted) { + activityMonitor.logComplete(activityId, 0); + return null; + } + + // Re-check: another concurrent caller may have started a clone while we awaited the size check + const cachedAfterSizeCheck = cloneCache.get(key); + if (cachedAfterSizeCheck) { + const cachedResult = await awaitCachedClone(cachedAfterSizeCheck, url, owner, repo, info, signal); + if (signal?.aborted) { + activityMonitor.logComplete(activityId, 0); + } else if (cachedResult) { + activityMonitor.logComplete(activityId, 200); + } else { + activityMonitor.logError(activityId, "clone failed"); + } + return cachedResult; + } + + const clonePromise = cloneRepo(owner, repo, info.ref, config, signal); + const localPath = cloneDir(config, owner, repo, info.ref); + cloneCache.set(key, { localPath, clonePromise }); + + const result = await clonePromise; + if (signal?.aborted) { + if (!result) cloneCache.delete(key); + activityMonitor.logComplete(activityId, 0); + return null; + } + + if (!result) { + cloneCache.delete(key); + if (signal?.aborted) { + activityMonitor.logComplete(activityId, 0); + return null; + } + + const apiFallback = await fetchViaApi(url, owner, repo, info); + if (apiFallback) { + activityMonitor.logComplete(activityId, 200); + return apiFallback; + } + + activityMonitor.logError(activityId, "clone and API fallback failed"); + return null; + } + + activityMonitor.logComplete(activityId, 200); + const content = generateContent(result, info); + const title = info.path ? `${owner}/${repo} - ${info.path}` : `${owner}/${repo}`; + return { url, title, content, error: null }; +} + +export function clearCloneCache(): void { + for (const entry of cloneCache.values()) { + try { + rmSync(entry.localPath, { recursive: true, force: true }); + } catch { + } + } + cloneCache.clear(); + cachedConfig = null; +} diff --git a/packages/web-access/index.ts b/packages/web-access/index.ts new file mode 100644 index 000000000..47f493fdc --- /dev/null +++ b/packages/web-access/index.ts @@ -0,0 +1,2347 @@ +import type { ExtensionAPI, ExtensionContext } from "@mariozechner/pi-coding-agent"; +import { Box, Text, truncateToWidth } from "@mariozechner/pi-tui"; +import { Type } from "typebox"; +import { StringEnum, complete, getModel, type Model } from "@mariozechner/pi-ai"; +import { fetchAllContent, type ExtractedContent } from "./extract.js"; +import { clearCloneCache } from "./github-extract.js"; +import { search, type SearchProvider, type ResolvedSearchProvider } from "./gemini-search.js"; +import { executeCodeSearch } from "./code-search.js"; +import type { SearchResult } from "./perplexity.js"; +import { formatSeconds } from "./utils.js"; +import { + clearResults, + deleteResult, + generateId, + getAllResults, + getResult, + restoreFromSession, + storeResult, + type QueryResultData, + type StoredSearchData, +} from "./storage.js"; +import { activityMonitor, type ActivityEntry } from "./activity.js"; +import { startCuratorServer, type CuratorServerHandle } from "./curator-server.js"; +import { + buildDeterministicSummary, + generateSummaryDraft, + type SummaryGenerationContext, + type SummaryMeta, +} from "./summary-review.js"; +import { randomUUID } from "node:crypto"; +import { execFileSync } from "node:child_process"; +import { createRequire } from "node:module"; +import { platform, homedir } from "node:os"; +import { existsSync, readFileSync, writeFileSync, mkdirSync } from "node:fs"; +import { join } from "node:path"; +import { CONFIG_DIR_NAME } from "@bastani/atomic"; +import { isPerplexityAvailable } from "./perplexity.js"; +import { isExaAvailable } from "./exa.js"; +import { isGeminiApiAvailable } from "./gemini-api.js"; +import { getActiveGoogleEmail, isGeminiWebAvailable } from "./gemini-web.js"; +import { isBrowserCookieAccessAllowed } from "./gemini-web-config.ts"; + +const WEB_SEARCH_CONFIG_PATH = join(homedir(), CONFIG_DIR_NAME, "web-search.json"); + +interface WebSearchConfig { + provider?: string; + workflow?: string; + curatorTimeoutSeconds?: unknown; + summaryModel?: string; + shortcuts?: { + curate?: string; + activity?: string; + }; +} + +interface ProviderAvailability { + perplexity: boolean; + exa: boolean; + gemini: boolean; +} + +type WebSearchWorkflow = "none" | "summary-review"; +type CuratorWorkflow = "summary-review"; + +interface CuratorBootstrap { + availableProviders: ProviderAvailability; + defaultProvider: ResolvedSearchProvider; + timeoutSeconds: number; +} + +function loadConfig(): WebSearchConfig { + if (!existsSync(WEB_SEARCH_CONFIG_PATH)) return {}; + const raw = readFileSync(WEB_SEARCH_CONFIG_PATH, "utf-8"); + try { + return JSON.parse(raw) as WebSearchConfig; + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + throw new Error(`Failed to parse ${WEB_SEARCH_CONFIG_PATH}: ${message}`); + } +} + +function saveConfig(updates: Partial): void { + let config: Record = {}; + if (existsSync(WEB_SEARCH_CONFIG_PATH)) { + const raw = readFileSync(WEB_SEARCH_CONFIG_PATH, "utf-8"); + try { + config = JSON.parse(raw) as Record; + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + throw new Error(`Failed to parse ${WEB_SEARCH_CONFIG_PATH}: ${message}`); + } + } + + Object.assign(config, updates); + const dir = join(homedir(), CONFIG_DIR_NAME); + if (!existsSync(dir)) mkdirSync(dir, { recursive: true }); + writeFileSync(WEB_SEARCH_CONFIG_PATH, JSON.stringify(config, null, 2) + "\n"); +} + +const DEFAULT_SHORTCUTS = { curate: "ctrl+shift+s", activity: "ctrl+shift+w" }; +const DEFAULT_CURATOR_TIMEOUT_SECONDS = 20; +const MAX_CURATOR_TIMEOUT_SECONDS = 600; + +function loadConfigForExtensionInit(): WebSearchConfig { + try { + return loadConfig(); + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + console.error(`[pi-web-access] ${message}`); + return {}; + } +} + +function normalizeProviderInput(value: unknown): SearchProvider | undefined { + if (value === undefined) return undefined; + if (typeof value !== "string") return "auto"; + const normalized = value.trim().toLowerCase(); + if (normalized === "auto" || normalized === "exa" || normalized === "perplexity" || normalized === "gemini") { + return normalized; + } + return "auto"; +} + +function normalizeCuratorTimeoutSeconds(value: unknown): number | undefined { + if (typeof value !== "number" || !Number.isFinite(value)) return undefined; + const normalized = Math.floor(value); + if (normalized < 1) return undefined; + return Math.min(normalized, MAX_CURATOR_TIMEOUT_SECONDS); +} + +function resolveWorkflow(input: unknown, hasUI: boolean): WebSearchWorkflow { + if (!hasUI) return "none"; + if (typeof input === "string" && input.trim().toLowerCase() === "none") return "none"; + return "summary-review"; +} + +function normalizeQueryList(queryList: unknown[]): string[] { + const normalized: string[] = []; + for (const query of queryList) { + if (typeof query !== "string") continue; + const trimmed = query.trim(); + if (trimmed.length > 0) normalized.push(trimmed); + } + return normalized; +} + +function getCuratorTimeoutSeconds(): number { + const source = loadConfig(); + return normalizeCuratorTimeoutSeconds(source.curatorTimeoutSeconds) ?? DEFAULT_CURATOR_TIMEOUT_SECONDS; +} + +async function getProviderAvailability(): Promise { + const geminiWebAvail = await isGeminiWebAvailable(); + return { + perplexity: isPerplexityAvailable(), + exa: isExaAvailable(), + gemini: isGeminiApiAvailable() || !!geminiWebAvail, + }; +} + +async function loadCuratorBootstrap(requestedProvider: unknown): Promise { + const availableProviders = await getProviderAvailability(); + return { + availableProviders, + defaultProvider: resolveProvider(requestedProvider, availableProviders), + timeoutSeconds: getCuratorTimeoutSeconds(), + }; +} + +function resolveProvider( + requested: unknown, + available: ProviderAvailability, +): ResolvedSearchProvider { + const provider = normalizeProviderInput(requested ?? loadConfig().provider ?? "auto") ?? "auto"; + + if (provider === "auto") { + if (available.exa) return "exa"; + if (available.perplexity) return "perplexity"; + if (available.gemini) return "gemini"; + return "exa"; + } + if (provider === "exa" && !available.exa) { + if (available.perplexity) return "perplexity"; + return available.gemini ? "gemini" : "exa"; + } + if (provider === "perplexity" && !available.perplexity) { + if (available.exa) return "exa"; + return available.gemini ? "gemini" : "perplexity"; + } + if (provider === "gemini" && !available.gemini) { + if (available.exa) return "exa"; + return available.perplexity ? "perplexity" : "gemini"; + } + return provider; +} + +const pendingFetches = new Map(); +let sessionActive = false; +let widgetVisible = false; +let widgetUnsubscribe: (() => void) | null = null; +let activeCurator: CuratorServerHandle | null = null; +let glimpseWin: GlimpseWindow | null = null; + +interface PendingCurate { + phase: "searching" | "curating"; + workflow: CuratorWorkflow; + summaryContext: SummaryGenerationContext; + searchResults: Map; + allInlineContent: ExtractedContent[]; + queryList: string[]; + includeContent: boolean; + numResults?: number; + recencyFilter?: "day" | "week" | "month" | "year"; + domainFilter?: string[]; + availableProviders: ProviderAvailability; + defaultProvider: ResolvedSearchProvider; + summaryModels: Array<{ value: string; label: string }>; + defaultSummaryModel: string | null; + timeoutSeconds: number; + onUpdate: ((update: { content: Array<{ type: string; text: string }>; details?: Record }) => void) | undefined; + signal: AbortSignal | undefined; + abortSearches: () => void; + finish: (value: unknown) => void; + cancel: (reason?: "user" | "stale") => void; + browserPromise?: Promise; +} + +let pendingCurate: PendingCurate | null = null; + +function cancelPendingCurate(reason: "user" | "stale" = "stale"): void { + pendingCurate?.cancel(reason); +} + +const MAX_INLINE_CONTENT = 30000; // Content returned directly to agent + +function stripThumbnails(results: ExtractedContent[]): ExtractedContent[] { + return results.map(({ thumbnail, frames, ...rest }) => rest); +} + +function formatSearchSummary(results: SearchResult[], answer: string): string { + let output = answer ? `${answer}\n\n---\n\n**Sources:**\n` : ""; + output += results.map((r, i) => `${i + 1}. ${r.title}\n ${r.url}`).join("\n\n"); + return output; +} + +function duplicateQuerySet(results: QueryResultData[]): Set { + const counts = new Map(); + for (const result of results) { + counts.set(result.query, (counts.get(result.query) ?? 0) + 1); + } + const duplicates = new Set(); + for (const [query, count] of counts) { + if (count > 1) duplicates.add(query); + } + return duplicates; +} + +function formatQueryHeader(query: string, provider: string | undefined, duplicateQueries: Set): string { + const suffix = duplicateQueries.has(query) && provider ? ` (${provider})` : ""; + return `## Query: "${query}"${suffix}\n\n`; +} + +function hasFullInlineCoverage(urls: string[], inlineContent: ExtractedContent[] | undefined): boolean { + if (!inlineContent || inlineContent.length === 0) return false; + const coveredUrls = new Set(inlineContent.map(c => c.url)); + return urls.every(url => coveredUrls.has(url)); +} + +function formatFullResults(queryData: QueryResultData): string { + let output = `## Results for: "${queryData.query}"\n\n`; + if (queryData.answer) { + output += `${queryData.answer}\n\n---\n\n`; + } + for (const r of queryData.results) { + output += `### ${r.title}\n${r.url}\n\n`; + } + return output; +} + +function abortPendingFetches(): void { + for (const controller of pendingFetches.values()) { + controller.abort(); + } + pendingFetches.clear(); +} + +function closeCurator(): void { + const win = glimpseWin; + glimpseWin = null; + try { win?.close(); } catch {} + cancelPendingCurate(); + if (activeCurator) { + activeCurator.close(); + activeCurator = null; + } +} + +async function openInBrowser(pi: ExtensionAPI, url: string): Promise { + const plat = platform(); + const result = plat === "darwin" + ? await pi.exec("open", [url]) + : plat === "win32" + ? await pi.exec("cmd", ["/c", "start", "", url]) + : await pi.exec("xdg-open", [url]); + if (result.code !== 0) { + throw new Error(result.stderr || `Failed to open browser (exit code ${result.code})`); + } +} + +interface GlimpseWindow { + on(event: "closed", handler: () => void): void; + on(event: "message", handler: (data: unknown) => void): void; + on(event: "ready", handler: (info: { screen?: { visibleHeight?: number } }) => void): void; + close(): void; + _write(obj: Record): void; +} + +let glimpseOpen: ((html: string, opts: Record) => GlimpseWindow) | null | undefined; + +function findGlimpseMjs(): string | null { + try { + const req = createRequire(import.meta.url); + return req.resolve("glimpseui"); + } catch { + // Optional dependency. + } + try { + const globalRoot = execFileSync("npm", ["root", "-g"], { encoding: "utf-8" }).trim(); + const entry = join(globalRoot, "glimpseui", "src", "glimpse.mjs"); + if (existsSync(entry)) return entry; + } catch { + // npm may be unavailable. + } + return null; +} + +async function getGlimpseOpen() { + if (glimpseOpen !== undefined) return glimpseOpen; + const resolved = findGlimpseMjs(); + if (resolved) { + try { + glimpseOpen = (await import(resolved)).open; + return glimpseOpen; + } catch {} + } + glimpseOpen = null; + return glimpseOpen; +} + +function openInGlimpse( + open: (html: string, opts: Record) => GlimpseWindow, + url: string, + title: string, +): GlimpseWindow { + const shellHTML = ` + +${title} + + + +`; + const win = open(shellHTML, { + width: 800, + height: 900, + title, + }); + + let maxHeight = 1200; + win.on("ready", (info) => { + const visibleHeight = info?.screen?.visibleHeight; + if (typeof visibleHeight === "number" && visibleHeight > 0) { + maxHeight = Math.floor(visibleHeight * 0.85); + } + }); + win.on("message", (data) => { + if (!data || typeof data !== "object") return; + const msg = data as Record; + if (msg.type !== "resize" || typeof msg.height !== "number") return; + const clamped = Math.max(400, Math.min(Math.round(msg.height), maxHeight)); + win._write({ type: "resize", width: 800, height: clamped }); + }); + + return win; +} + +function extractDomain(url: string): string { + try { return new URL(url).hostname; } + catch { return url; } +} + +function updateWidget(ctx: ExtensionContext): void { + const theme = ctx.ui.theme; + const entries = activityMonitor.getEntries(); + const lines: string[] = []; + + lines.push(theme.fg("accent", "─── Web Search Activity " + "─".repeat(36))); + + if (entries.length === 0) { + lines.push(theme.fg("muted", " No activity yet")); + } else { + for (const e of entries) { + lines.push(" " + formatEntryLine(e, theme)); + } + } + + lines.push(theme.fg("accent", "─".repeat(60))); + + const rateInfo = activityMonitor.getRateLimitInfo(); + const resetMs = rateInfo.oldestTimestamp ? Math.max(0, rateInfo.oldestTimestamp + rateInfo.windowMs - Date.now()) : 0; + const resetSec = Math.ceil(resetMs / 1000); + lines.push( + theme.fg("muted", `Rate: ${rateInfo.used}/${rateInfo.max}`) + + (resetMs > 0 ? theme.fg("dim", ` (resets in ${resetSec}s)`) : ""), + ); + + ctx.ui.setWidget("web-activity", new Text(lines.join("\n"), 0, 0)); +} + +function formatEntryLine( + entry: ActivityEntry, + theme: { fg: (color: string, text: string) => string }, +): string { + const typeStr = entry.type === "api" ? "API" : "GET"; + const target = + entry.type === "api" + ? `"${truncateToWidth(entry.query || "", 28, "")}"` + : truncateToWidth(entry.url?.replace(/^https?:\/\//, "") || "", 30, ""); + + const duration = entry.endTime + ? `${((entry.endTime - entry.startTime) / 1000).toFixed(1)}s` + : `${((Date.now() - entry.startTime) / 1000).toFixed(1)}s`; + + let statusStr: string; + let indicator: string; + if (entry.error) { + statusStr = "err"; + indicator = theme.fg("error", "✗"); + } else if (entry.status === null) { + statusStr = "..."; + indicator = theme.fg("warning", "⋯"); + } else if (entry.status === 0) { + statusStr = "abort"; + indicator = theme.fg("muted", "○"); + } else { + statusStr = String(entry.status); + indicator = entry.status >= 200 && entry.status < 300 ? theme.fg("success", "✓") : theme.fg("error", "✗"); + } + + return `${typeStr.padEnd(4)} ${target.padEnd(32)} ${statusStr.padStart(5)} ${duration.padStart(5)} ${indicator}`; +} + +function handleSessionChange(ctx: ExtensionContext): void { + abortPendingFetches(); + closeCurator(); + clearCloneCache(); + sessionActive = true; + restoreFromSession(ctx); + // Unsubscribe before clear() to avoid callback with stale ctx + widgetUnsubscribe?.(); + widgetUnsubscribe = null; + activityMonitor.clear(); + if (widgetVisible) { + // Re-subscribe with new ctx + widgetUnsubscribe = activityMonitor.onUpdate(() => updateWidget(ctx)); + updateWidget(ctx); + } +} + +export default function (pi: ExtensionAPI) { + const initConfig = loadConfigForExtensionInit(); + const curateKey = initConfig.shortcuts?.curate || DEFAULT_SHORTCUTS.curate; + const activityKey = initConfig.shortcuts?.activity || DEFAULT_SHORTCUTS.activity; + + function startBackgroundFetch(urls: string[]): string | null { + if (urls.length === 0) return null; + const fetchId = generateId(); + const controller = new AbortController(); + pendingFetches.set(fetchId, controller); + fetchAllContent(urls, controller.signal) + .then((fetched) => { + if (!sessionActive || !pendingFetches.has(fetchId)) return; + const data: StoredSearchData = { + id: fetchId, + type: "fetch", + timestamp: Date.now(), + urls: stripThumbnails(fetched), + }; + storeResult(fetchId, data); + pi.appendEntry("web-search-results", data); + const ok = fetched.filter(f => !f.error).length; + pi.sendMessage( + { + customType: "web-search-content-ready", + content: `Content fetched for ${ok}/${fetched.length} URLs [${fetchId}]. Full page content now available.`, + display: true, + }, + { triggerTurn: true }, + ); + }) + .catch((err) => { + if (!sessionActive || !pendingFetches.has(fetchId)) return; + const message = err instanceof Error ? err.message : String(err); + const isAbort = (err instanceof Error && err.name === "AbortError") || message.toLowerCase().includes("abort"); + if (!isAbort) { + pi.sendMessage( + { + customType: "web-search-error", + content: `Content fetch failed [${fetchId}]: ${message}`, + display: true, + }, + { triggerTurn: false }, + ); + } + }) + .finally(() => { pendingFetches.delete(fetchId); }); + return fetchId; + } + + function storeAndPublishSearch(results: QueryResultData[]): string { + const id = generateId(); + const data: StoredSearchData = { + id, type: "search", timestamp: Date.now(), queries: results, + }; + storeResult(id, data); + pi.appendEntry("web-search-results", data); + return id; + } + + interface SearchReturnOptions { + queryList: string[]; + results: QueryResultData[]; + urls: string[]; + includeContent: boolean; + inlineContent?: ExtractedContent[]; + curated?: boolean; + curatedFrom?: number; + workflow?: CuratorWorkflow; + approvedSummary?: string; + summaryMeta?: SummaryMeta; + } + + function normalizeSummaryMeta(meta: SummaryMeta | undefined, summaryText: string): SummaryMeta { + const normalizedText = summaryText.trim(); + if (!meta) { + return { + model: null, + durationMs: 0, + tokenEstimate: normalizedText.length > 0 ? Math.max(1, Math.ceil(normalizedText.length / 4)) : 0, + fallbackUsed: false, + edited: false, + }; + } + + return { + model: meta.model, + durationMs: Number.isFinite(meta.durationMs) && meta.durationMs >= 0 ? meta.durationMs : 0, + tokenEstimate: Number.isFinite(meta.tokenEstimate) && meta.tokenEstimate >= 0 + ? meta.tokenEstimate + : (normalizedText.length > 0 ? Math.max(1, Math.ceil(normalizedText.length / 4)) : 0), + fallbackUsed: meta.fallbackUsed === true, + fallbackReason: meta.fallbackReason, + edited: meta.edited === true, + }; + } + + function buildCurationCancelledReturn(reason: "user" | "stale") { + const message = `Search curation cancelled (${reason}).`; + return { + content: [{ type: "text", text: message }], + details: { + error: message, + cancelled: true, + cancelReason: reason, + }, + }; + } + + async function resolveFirstAvailableModel( + ctx: SummaryGenerationContext, + candidates: Array<{ provider: string; id: string }>, + ): Promise<{ model: Model; apiKey: string; headers?: Record }> { + for (const { provider, id } of candidates) { + const model = getModel(provider, id); + if (!model) continue; + const auth = await ctx.modelRegistry.getApiKeyAndHeaders(model); + if (auth.ok && auth.apiKey) return { model, apiKey: auth.apiKey, headers: auth.headers }; + } + throw new Error(`No model available: ${candidates.map(c => `${c.provider}/${c.id}`).join(", ")}`); + } + + async function rewriteSearchQuery(query: string, ctx: SummaryGenerationContext, signal: AbortSignal): Promise { + const { model, apiKey, headers } = await resolveFirstAvailableModel(ctx, [ + { provider: "anthropic", id: "claude-haiku-4-5" }, + { provider: "google", id: "gemini-2.5-flash" }, + { provider: "openai", id: "gpt-4.1-mini" }, + ]); + const response = await complete( + model, + { + messages: [{ + role: "user", + content: [{ type: "text", text: `Rewrite this web search query to get better, more specific results. Add relevant year qualifiers, precise technical terms, and specificity. Return ONLY the improved query text, nothing else.\n\nQuery: ${query}` }], + timestamp: Date.now(), + }], + }, + { apiKey, headers, signal }, + ); + if (response.stopReason === "aborted") throw new Error("Aborted"); + const contentParts = Array.isArray(response.content) ? response.content : []; + const text = contentParts + .map(p => { + if (!p || typeof p !== "object") return ""; + const part = p as Record; + return typeof part.text === "string" ? part.text : ""; + }) + .join("") + .trim(); + if (!text) throw new Error("Rewrite returned empty response"); + return text; + } + + async function generateSummaryForSelectedIndices( + selectedQueryIndices: number[], + resultsByIndex: Map, + summaryContext: SummaryGenerationContext, + signal?: AbortSignal, + modelOverride?: string, + feedback?: string, + ): Promise<{ summary: string; meta: SummaryMeta }> { + const selectedResults: QueryResultData[] = []; + for (const qi of selectedQueryIndices) { + const result = resultsByIndex.get(qi); + if (result) selectedResults.push(result); + } + if (selectedResults.length === 0) { + throw new Error("No selected results available for summary generation"); + } + try { + return await generateSummaryDraft(selectedResults, summaryContext, signal, modelOverride, feedback); + } catch (err) { + const isEmptyResponse = err instanceof Error && err.message.includes("Summary model returned empty response"); + if (!isEmptyResponse) throw err; + const deterministic = buildDeterministicSummary(selectedResults); + return { + summary: deterministic.summary, + meta: { + ...deterministic.meta, + fallbackReason: "summary-model-empty-response", + }, + }; + } + } + + async function loadSummaryModelChoices( + summaryContext: SummaryGenerationContext, + ): Promise<{ summaryModels: Array<{ value: string; label: string }>; defaultSummaryModel: string | null }> { + const summaryModels: Array<{ value: string; label: string }> = []; + const seen = new Set(); + const availableValues = new Set(); + + const addModel = (provider: string, id: string) => { + const value = `${provider}/${id}`; + if (seen.has(value)) return; + seen.add(value); + summaryModels.push({ value, label: value }); + }; + + try { + const availableModels = summaryContext.modelRegistry.getAvailable(); + for (const model of availableModels) { + const value = `${model.provider}/${model.id}`; + availableValues.add(value); + addModel(model.provider, model.id); + } + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + console.error(`Failed to load summary models: ${message}`); + } + + const currentModelValue = summaryContext.model + ? `${summaryContext.model.provider}/${summaryContext.model.id}` + : null; + if (summaryContext.model && currentModelValue && !seen.has(currentModelValue)) { + addModel(summaryContext.model.provider, summaryContext.model.id); + } + + const config = loadConfig(); + const configuredSummaryModel = typeof config.summaryModel === "string" ? config.summaryModel.trim() : ""; + const preferredDefaults = [ + "anthropic/claude-haiku-4-5", + "openai-codex/gpt-5.3-codex-spark", + ]; + + let defaultSummaryModel: string | null = null; + if (configuredSummaryModel.length > 0 && availableValues.has(configuredSummaryModel)) { + defaultSummaryModel = configuredSummaryModel; + } + if (!defaultSummaryModel) { + for (const preferred of preferredDefaults) { + if (availableValues.has(preferred)) { + defaultSummaryModel = preferred; + break; + } + } + } + if (!defaultSummaryModel && summaryModels.length > 0) { + defaultSummaryModel = summaryModels[0].value; + } + + return { summaryModels, defaultSummaryModel }; + } + + function resolveSummaryForSubmit( + payload: { selectedQueryIndices: number[]; summary?: string; summaryMeta?: SummaryMeta }, + resultsByIndex: Map, + ): { approvedSummary: string; summaryMeta: SummaryMeta } { + const submittedSummary = typeof payload.summary === "string" ? payload.summary.trim() : ""; + if (submittedSummary.length > 0) { + return { + approvedSummary: submittedSummary, + summaryMeta: normalizeSummaryMeta(payload.summaryMeta, submittedSummary), + }; + } + + const selected = filterByQueryIndices(payload.selectedQueryIndices, resultsByIndex).results; + const fallbackResults = selected.length > 0 ? selected : [...resultsByIndex.values()]; + const deterministic = buildDeterministicSummary(fallbackResults); + return { + approvedSummary: deterministic.summary, + summaryMeta: deterministic.meta, + }; + } + + function buildSearchReturn(opts: SearchReturnOptions) { + const sc = opts.results.filter(r => !r.error).length; + const tr = opts.results.reduce((sum, r) => sum + r.results.length, 0); + + const hasApprovedSummary = typeof opts.approvedSummary === "string" && opts.approvedSummary.trim().length > 0; + let output = ""; + if (hasApprovedSummary) { + output = opts.approvedSummary!.trim(); + } else { + if (opts.curated) { + output += "[These results were manually curated by the user in the browser. Use them as-is — do not re-search or discard.]\n\n"; + } + const duplicateQueries = opts.curated ? duplicateQuerySet(opts.results) : new Set(); + for (const { query, answer, results, error, provider } of opts.results) { + if (opts.queryList.length > 1) { + output += opts.curated + ? formatQueryHeader(query, provider, duplicateQueries) + : `## Query: "${query}"\n\n`; + } + if (error) output += `Error: ${error}\n\n`; + else if (results.length === 0) output += "No results found.\n\n"; + else output += formatSearchSummary(results, answer) + "\n\n"; + } + } + + const hasInlineReady = hasFullInlineCoverage(opts.urls, opts.inlineContent); + let fetchId: string | null = null; + if (hasInlineReady && opts.inlineContent) { + fetchId = generateId(); + const data: StoredSearchData = { + id: fetchId, + type: "fetch", + timestamp: Date.now(), + urls: opts.inlineContent, + }; + storeResult(fetchId, data); + pi.appendEntry("web-search-results", data); + if (!hasApprovedSummary) { + output += `---\nFull content for ${opts.inlineContent.length} sources available [${fetchId}].`; + } + } else if (opts.includeContent) { + fetchId = startBackgroundFetch(opts.urls); + if (fetchId && !hasApprovedSummary) { + output += `---\nContent fetching in background [${fetchId}]. Will notify when ready.`; + } + } + + const searchId = storeAndPublishSearch(opts.results); + const isBackgroundFetch = fetchId !== null && !hasInlineReady; + + return { + content: [{ type: "text", text: output.trim() }], + details: { + queries: opts.queryList, + queryCount: opts.queryList.length, + successfulQueries: sc, + totalResults: tr, + includeContent: opts.includeContent, + fetchId, + fetchUrls: isBackgroundFetch ? opts.urls : undefined, + searchId, + ...(opts.curated ? { + curated: true, + curatedFrom: opts.curatedFrom, + curatedQueries: opts.results.map(r => ({ + query: r.query, + provider: r.provider || null, + answer: r.answer || null, + sources: r.results.map(s => ({ title: s.title, url: s.url })), + error: r.error, + })), + } : {}), + ...((opts.workflow && hasApprovedSummary) + ? { + summary: { + text: opts.approvedSummary!.trim(), + workflow: opts.workflow, + model: opts.summaryMeta?.model ?? null, + durationMs: opts.summaryMeta?.durationMs ?? 0, + tokenEstimate: opts.summaryMeta?.tokenEstimate ?? 0, + fallbackUsed: opts.summaryMeta?.fallbackUsed === true, + fallbackReason: opts.summaryMeta?.fallbackReason, + edited: opts.summaryMeta?.edited === true, + }, + } + : {}), + }, + }; + } + + function filterByQueryIndices(selectedQueryIndices: number[], results: Map) { + const filteredResults: QueryResultData[] = []; + const filteredUrls: string[] = []; + for (const qi of selectedQueryIndices) { + const r = results.get(qi); + if (r) { + filteredResults.push(r); + for (const res of r.results) { + if (!filteredUrls.includes(res.url)) filteredUrls.push(res.url); + } + } + } + return { results: filteredResults, urls: filteredUrls }; + } + + function collectAllResultsAndUrls(resultsByIndex: Map) { + const results = [...resultsByIndex.values()]; + const urls: string[] = []; + for (const result of results) { + for (const source of result.results) { + if (!urls.includes(source.url)) urls.push(source.url); + } + } + return { results, urls }; + } + + async function openCuratorBrowser(pc: PendingCurate, searchesComplete = true): Promise { + let handle: CuratorServerHandle | null = null; + try { + pc.phase = "curating"; + + const searchAbort = new AbortController(); + const addSearchSignal = pc.signal + ? AbortSignal.any([pc.signal, searchAbort.signal]) + : searchAbort.signal; + + const sessionToken = randomUUID(); + handle = await startCuratorServer( + { + queries: pc.queryList, + sessionToken, + timeout: pc.timeoutSeconds, + availableProviders: pc.availableProviders, + defaultProvider: pc.defaultProvider, + summaryModels: pc.summaryModels, + defaultSummaryModel: pc.defaultSummaryModel, + }, + { + async onSummarize(selectedQueryIndices, summarizeSignal, model, feedback) { + if (pendingCurate !== pc) throw new Error("Curator session is no longer active."); + pc.onUpdate?.({ + content: [{ type: "text", text: "Generating summary draft..." }], + details: { phase: "generating-summary", progress: 0.9 }, + }); + const draft = await generateSummaryForSelectedIndices( + selectedQueryIndices, + pc.searchResults, + pc.summaryContext, + summarizeSignal, + model, + feedback, + ); + if (pendingCurate !== pc) throw new Error("Curator session is no longer active."); + pc.onUpdate?.({ + content: [{ type: "text", text: "Summary draft ready — waiting for approval..." }], + details: { phase: "waiting-for-approval", progress: 1 }, + }); + return draft; + }, + onSubmit(payload) { + if (pendingCurate !== pc) return; + searchAbort.abort(); + const filtered = payload.selectedQueryIndices.length > 0 + ? filterByQueryIndices(payload.selectedQueryIndices, pc.searchResults) + : collectAllResultsAndUrls(pc.searchResults); + const filteredInline = pc.allInlineContent.filter(c => filtered.urls.includes(c.url)); + const base: SearchReturnOptions = { + queryList: filtered.results.map(r => r.query), + results: filtered.results, + urls: filtered.urls, + includeContent: pc.includeContent, + inlineContent: filteredInline.length > 0 ? filteredInline : undefined, + curated: true, + curatedFrom: pc.searchResults.size, + }; + if (!payload.rawResults) { + const resolvedSummary = resolveSummaryForSubmit(payload, pc.searchResults); + base.workflow = pc.workflow; + base.approvedSummary = resolvedSummary.approvedSummary; + base.summaryMeta = resolvedSummary.summaryMeta; + } + pc.finish(buildSearchReturn(base)); + closeCurator(); + }, + onCancel(reason) { + if (pendingCurate !== pc) return; + searchAbort.abort(); + if (reason === "timeout") { + const resolvedSummary = resolveSummaryForSubmit({ selectedQueryIndices: [], summary: undefined, summaryMeta: undefined }, pc.searchResults); + const all = collectAllResultsAndUrls(pc.searchResults); + const filteredInline = pc.allInlineContent.filter(c => all.urls.includes(c.url)); + pc.finish(buildSearchReturn({ + queryList: all.results.map(r => r.query), + results: all.results, + urls: all.urls, + includeContent: pc.includeContent, + inlineContent: filteredInline.length > 0 ? filteredInline : undefined, + curated: true, + curatedFrom: pc.searchResults.size, + workflow: pc.workflow, + approvedSummary: resolvedSummary.approvedSummary, + summaryMeta: resolvedSummary.summaryMeta, + })); + } else { + pc.finish(buildCurationCancelledReturn(reason)); + } + closeCurator(); + }, + onProviderChange(provider) { + if (pendingCurate !== pc) return; + const normalized = normalizeProviderInput(provider); + if (!normalized || normalized === "auto") return; + pc.defaultProvider = normalized; + try { + saveConfig({ provider: normalized }); + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + console.error(`Failed to persist default provider: ${message}`); + } + }, + async onAddSearch(query, queryIndex, provider) { + if (pendingCurate !== pc) throw new Error("Curator session is no longer active."); + const normalizedProvider = normalizeProviderInput(provider); + const requestedProvider = !normalizedProvider || normalizedProvider === "auto" + ? pc.defaultProvider + : normalizedProvider; + try { + const { answer, results, inlineContent, provider: actualProvider } = await search(query, { + provider: requestedProvider, + numResults: pc.numResults, + recencyFilter: pc.recencyFilter, + domainFilter: pc.domainFilter, + includeContent: pc.includeContent, + signal: addSearchSignal, + }); + if (pendingCurate !== pc) throw new Error("Curator session is no longer active."); + pc.searchResults.set(queryIndex, { query, answer, results, error: null, provider: actualProvider }); + if (inlineContent) pc.allInlineContent.push(...inlineContent); + return { + answer, + results: results.map(r => ({ title: r.title, url: r.url, domain: extractDomain(r.url) })), + provider: actualProvider, + }; + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + if (pendingCurate === pc) { + pc.searchResults.set(queryIndex, { query, answer: "", results: [], error: message, provider: requestedProvider }); + } + throw err; + } + }, + async onRewriteQuery(query, rewriteSignal) { + if (pendingCurate !== pc) throw new Error("Curator session is no longer active."); + return rewriteSearchQuery(query, pc.summaryContext, rewriteSignal); + }, + }, + ); + + if (pendingCurate !== pc) { + handle.close(); + return; + } + + activeCurator = handle; + + for (const [qi, data] of pc.searchResults) { + if (data.error) { + handle.pushError(qi, data.error, data.provider); + } else { + handle.pushResult(qi, { + answer: data.answer, + results: data.results.map(r => ({ title: r.title, url: r.url, domain: extractDomain(r.url) })), + provider: data.provider || pc.defaultProvider, + }); + } + } + if (searchesComplete) handle.searchesDone(); + + pc.onUpdate?.({ + content: [{ type: "text", text: searchesComplete ? "Waiting for summary approval in browser..." : "Searches streaming to browser..." }], + details: { phase: "curating", progress: searchesComplete ? 1 : 0.5 }, + }); + + const open = platform() === "darwin" ? await getGlimpseOpen() : null; + if (open) { + try { + const win = openInGlimpse(open, handle.url, "Search Curator"); + glimpseWin = win; + win.on("closed", () => { + if (glimpseWin === win) { + glimpseWin = null; + closeCurator(); + } + }); + return; + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + console.error(`Failed to open Glimpse curator window: ${message}`); + glimpseWin = null; + } + } + await openInBrowser(pi, handle.url); + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + console.error(`Failed to open curator UI: ${message}`); + if (pendingCurate === pc || (handle && activeCurator === handle)) { + closeCurator(); + } + } + } + + pi.registerShortcut(curateKey, { + description: "Review search results", + handler: async (ctx) => { + if (!pendingCurate) return; + + if (pendingCurate.phase === "searching") { + pendingCurate.browserPromise = openCuratorBrowser(pendingCurate, false); + ctx.ui.notify("Opening curator — remaining searches will stream in", "info"); + return; + } + }, + }); + + pi.registerShortcut(activityKey, { + description: "Toggle web search activity", + handler: async (ctx) => { + widgetVisible = !widgetVisible; + if (widgetVisible) { + widgetUnsubscribe = activityMonitor.onUpdate(() => updateWidget(ctx)); + updateWidget(ctx); + } else { + widgetUnsubscribe?.(); + widgetUnsubscribe = null; + ctx.ui.setWidget("web-activity", null); + } + }, + }); + + pi.on("session_start", async (_event, ctx) => handleSessionChange(ctx)); + pi.on("session_tree", async (_event, ctx) => handleSessionChange(ctx)); + + pi.on("session_shutdown", () => { + sessionActive = false; + abortPendingFetches(); + closeCurator(); + clearCloneCache(); + clearResults(); + // Unsubscribe before clear() to avoid callback with stale ctx + widgetUnsubscribe?.(); + widgetUnsubscribe = null; + activityMonitor.clear(); + widgetVisible = false; + }); + + pi.registerTool({ + name: "web_search", + label: "Web Search", + description: + `Search the web using Perplexity AI, Exa, or Gemini. Returns an AI-synthesized answer with source citations. For comprehensive research, prefer queries (plural) with 2-4 varied angles over a single query — each query gets its own synthesized answer, so varying phrasing and scope gives much broader coverage. When includeContent is true, full page content is fetched in the background. Searches auto-open the interactive browser curator and stream results live; set workflow to "none" to skip curation. Provider auto-selects: Exa (direct API with key, MCP fallback without), else Perplexity (needs key), else Gemini API (needs key), else Gemini Web (needs a supported Chromium-based browser login).`, + promptSnippet: + "Use for web research questions. Prefer {queries:[...]} with 2-4 varied angles over a single query for broader coverage.", + parameters: Type.Object({ + query: Type.Optional(Type.String({ description: "Single search query. For research tasks, prefer 'queries' with multiple varied angles instead." })), + queries: Type.Optional(Type.Array(Type.String(), { description: "Multiple queries searched in sequence, each returning its own synthesized answer. Prefer this for research — vary phrasing, scope, and angle across 2-4 queries to maximize coverage. Good: ['React vs Vue performance benchmarks 2026', 'React vs Vue developer experience comparison', 'React ecosystem size vs Vue ecosystem']. Bad: ['React vs Vue', 'React vs Vue comparison', 'React vs Vue review'] (too similar, redundant results)." })), + numResults: Type.Optional(Type.Number({ description: "Results per query (default: 5, max: 20)" })), + includeContent: Type.Optional(Type.Boolean({ description: "Fetch full page content (async)" })), + recencyFilter: Type.Optional( + StringEnum(["day", "week", "month", "year"], { description: "Filter by recency" }), + ), + domainFilter: Type.Optional(Type.Array(Type.String(), { description: "Limit to domains (prefix with - to exclude)" })), + provider: Type.Optional( + StringEnum(["auto", "perplexity", "gemini", "exa"], { description: "Search provider (default: auto)" }), + ), + workflow: Type.Optional( + StringEnum(["none", "summary-review"], { + description: "Search workflow mode: none = no curator, summary-review = open curator with auto summary draft (default)", + }), + ), + }), + + async execute(_toolCallId, params, signal, onUpdate, ctx) { + const rawQueryList: unknown[] = Array.isArray(params.queries) + ? params.queries + : (params.query !== undefined ? [params.query] : []); + const queryList = normalizeQueryList(rawQueryList); + const configWorkflow = loadConfigForExtensionInit().workflow; + const workflow = resolveWorkflow(params.workflow ?? configWorkflow, ctx?.hasUI !== false); + const shouldCurate = workflow !== "none"; + + if (queryList.length === 0) { + return { + content: [{ type: "text", text: "Error: No query provided. Use 'query' or 'queries' parameter." }], + details: { error: "No query provided" }, + }; + } + + if (shouldCurate && !ctx) { + return { + content: [{ type: "text", text: "Error: Curation requires an active extension context." }], + details: { error: "Missing extension context" }, + }; + } + + if (shouldCurate) { + closeCurator(); + + let resolvePromise: (value: unknown) => void = () => {}; + const promise = new Promise((resolve) => { + resolvePromise = resolve; + }); + const includeContent = params.includeContent ?? false; + const searchResults = new Map(); + const allInlineContent: ExtractedContent[] = []; + const searchAbort = new AbortController(); + const searchSignal = signal + ? AbortSignal.any([signal, searchAbort.signal]) + : searchAbort.signal; + let cancelled = false; + + const bootstrap = await loadCuratorBootstrap(params.provider); + const availableProviders = bootstrap.availableProviders; + const defaultProvider = bootstrap.defaultProvider; + const curatorTimeoutSeconds = bootstrap.timeoutSeconds; + const curatorWorkflow: CuratorWorkflow = "summary-review"; + + const summaryContext: SummaryGenerationContext = { + model: ctx.model, + modelRegistry: ctx.modelRegistry, + }; + const summaryModelChoices = await loadSummaryModelChoices(summaryContext); + + const pc: PendingCurate = { + phase: "searching", + workflow: curatorWorkflow, + summaryContext, + searchResults, + allInlineContent, + queryList, + includeContent, + numResults: params.numResults, + recencyFilter: params.recencyFilter, + domainFilter: params.domainFilter, + availableProviders, + defaultProvider, + summaryModels: summaryModelChoices.summaryModels, + defaultSummaryModel: summaryModelChoices.defaultSummaryModel, + timeoutSeconds: curatorTimeoutSeconds, + onUpdate: onUpdate as PendingCurate["onUpdate"], + signal, + abortSearches: () => { + if (!searchAbort.signal.aborted) searchAbort.abort(); + }, + finish: () => {}, + cancel: () => {}, + }; + + const finish = (value: unknown) => { + if (cancelled) return; + cancelled = true; + pc.abortSearches(); + signal?.removeEventListener("abort", onAbort); + pendingCurate = null; + resolvePromise(value); + }; + + const cancel = (reason: "user" | "stale" = "stale") => { + if (cancelled) return; + finish(buildCurationCancelledReturn(reason)); + }; + + pc.finish = finish; + pc.cancel = cancel; + + const onAbort = () => closeCurator(); + pendingCurate = pc; + signal?.addEventListener("abort", onAbort, { once: true }); + pc.browserPromise = openCuratorBrowser(pc, false); + + for (let qi = 0; qi < queryList.length; qi++) { + if (signal?.aborted || cancelled || searchAbort.signal.aborted) break; + onUpdate?.({ + content: [{ type: "text", text: `Searching ${qi + 1}/${queryList.length}: "${queryList[qi]}"...` }], + details: { phase: "searching", progress: qi / queryList.length, currentQuery: queryList[qi] }, + }); + const requestedProvider = pc.defaultProvider; + try { + const { answer, results, inlineContent, provider } = await search(queryList[qi], { + provider: requestedProvider, + numResults: params.numResults, + recencyFilter: params.recencyFilter, + domainFilter: params.domainFilter, + includeContent: params.includeContent, + signal: searchSignal, + }); + if (signal?.aborted || cancelled || searchAbort.signal.aborted) break; + searchResults.set(qi, { query: queryList[qi], answer, results, error: null, provider }); + if (inlineContent) allInlineContent.push(...inlineContent); + if (activeCurator) { + activeCurator.pushResult(qi, { + answer, + results: results.map(r => ({ title: r.title, url: r.url, domain: extractDomain(r.url) })), + provider, + }); + } + } catch (err) { + if (signal?.aborted || cancelled || searchAbort.signal.aborted) break; + const message = err instanceof Error ? err.message : String(err); + searchResults.set(qi, { query: queryList[qi], answer: "", results: [], error: message, provider: requestedProvider }); + if (activeCurator) { + activeCurator.pushError(qi, message, requestedProvider); + } + } + } + + if (signal?.aborted || cancelled || searchAbort.signal.aborted) { + cancel(); + return promise; + } + + await pc.browserPromise; + if (activeCurator && !cancelled) { + activeCurator.searchesDone(); + pc.onUpdate?.({ + content: [{ type: "text", text: "All searches complete — waiting for summary approval in browser..." }], + details: { phase: "curating", progress: 1 }, + }); + } + + return promise; + } + + const searchResults: QueryResultData[] = []; + const allUrls: string[] = []; + const allInlineContent: ExtractedContent[] = []; + const resolvedProvider = normalizeProviderInput(params.provider ?? loadConfig().provider); + + for (let i = 0; i < queryList.length; i++) { + const query = queryList[i]; + + onUpdate?.({ + content: [{ type: "text", text: `Searching ${i + 1}/${queryList.length}: "${query}"...` }], + details: { phase: "search", progress: i / queryList.length, currentQuery: query }, + }); + + try { + const { answer, results, inlineContent, provider } = await search(query, { + provider: resolvedProvider, + numResults: params.numResults, + recencyFilter: params.recencyFilter, + domainFilter: params.domainFilter, + includeContent: params.includeContent, + signal, + }); + + searchResults.push({ query, answer, results, error: null, provider }); + for (const r of results) { + if (!allUrls.includes(r.url)) { + allUrls.push(r.url); + } + } + if (inlineContent) allInlineContent.push(...inlineContent); + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + const requestedProvider = typeof resolvedProvider === "string" && resolvedProvider !== "auto" + ? resolvedProvider + : undefined; + searchResults.push({ query, answer: "", results: [], error: message, provider: requestedProvider }); + } + } + + return buildSearchReturn({ + queryList, + results: searchResults, + urls: allUrls, + includeContent: params.includeContent ?? false, + inlineContent: allInlineContent.length > 0 ? allInlineContent : undefined, + }); + }, + + renderCall(args, theme) { + const input = args as { query?: unknown; queries?: unknown }; + const rawQueryList: unknown[] = Array.isArray(input.queries) + ? input.queries + : (input.query !== undefined ? [input.query] : []); + const queryList = normalizeQueryList(rawQueryList); + if (queryList.length === 0) { + return new Text(theme.fg("toolTitle", theme.bold("search ")) + theme.fg("error", "(no query)"), 0, 0); + } + if (queryList.length === 1) { + const q = queryList[0]; + const display = q.length > 60 ? q.slice(0, 57) + "..." : q; + return new Text(theme.fg("toolTitle", theme.bold("search ")) + theme.fg("accent", `"${display}"`), 0, 0); + } + const lines = [theme.fg("toolTitle", theme.bold("search ")) + theme.fg("accent", `${queryList.length} queries`)]; + for (const q of queryList.slice(0, 5)) { + const display = q.length > 50 ? q.slice(0, 47) + "..." : q; + lines.push(theme.fg("muted", ` "${display}"`)); + } + if (queryList.length > 5) { + lines.push(theme.fg("muted", ` ... and ${queryList.length - 5} more`)); + } + return new Text(lines.join("\n"), 0, 0); + }, + + renderResult(result, { expanded, isPartial }, theme) { + type QueryDetail = { + query: string; + provider: string | null; + answer: string | null; + sources: Array<{ title: string; url: string }>; + error: string | null; + }; + const details = result.details as { + queryCount?: number; + successfulQueries?: number; + totalResults?: number; + error?: string; + fetchId?: string; + fetchUrls?: string[]; + phase?: string; + progress?: number; + currentQuery?: string; + curated?: boolean; + curatedFrom?: number; + curatedQueries?: QueryDetail[]; + cancelled?: boolean; + cancelReason?: string; + summary?: { + text: string; + workflow: CuratorWorkflow; + model: string | null; + durationMs: number; + tokenEstimate: number; + fallbackUsed: boolean; + fallbackReason?: string; + edited?: boolean; + }; + }; + + if (isPartial) { + if (details?.phase === "curating") { + return new Text(theme.fg("accent", "waiting for summary approval..."), 0, 0); + } + if (details?.phase === "searching") { + const progress = details?.progress ?? 0; + const bar = "\u2588".repeat(Math.floor(progress * 10)) + "\u2591".repeat(10 - Math.floor(progress * 10)); + const query = details?.currentQuery || ""; + const display = query.length > 40 ? query.slice(0, 37) + "..." : query; + return new Text(theme.fg("accent", `[${bar}] ${display}`), 0, 0); + } + const progress = details?.progress ?? 0; + const bar = "\u2588".repeat(Math.floor(progress * 10)) + "\u2591".repeat(10 - Math.floor(progress * 10)); + return new Text(theme.fg("accent", `[${bar}] ${details?.phase || "searching"}`), 0, 0); + } + + if (details?.error) { + return new Text(theme.fg("error", `Error: ${details.error}`), 0, 0); + } + + let statusLine: string; + const queryInfo = details?.queryCount === 1 ? "" : `${details?.successfulQueries}/${details?.queryCount} queries, `; + statusLine = theme.fg("success", `${queryInfo}${details?.totalResults ?? 0} sources`); + if (details?.curated && details?.curatedFrom) { + statusLine += theme.fg("muted", ` (${details.queryCount}/${details.curatedFrom} queries curated)`); + } + if (details?.fetchId && details?.fetchUrls) { + statusLine += theme.fg("muted", ` (fetching ${details.fetchUrls.length} URLs)`); + } else if (details?.fetchId) { + statusLine += theme.fg("muted", " (content ready)"); + } + + // Build expanded lines first so collapsed view can reference total count + const lines = [statusLine]; + if (details?.summary?.text) { + lines.push(""); + lines.push(theme.fg("accent", `── Summary (${details.summary.workflow}) ` + "─".repeat(32))); + lines.push(""); + for (const line of details.summary.text.split("\n")) { + lines.push(` ${line}`); + } + lines.push(""); + const metaParts = [ + details.summary.model ? `model=${details.summary.model}` : "model=deterministic", + `duration=${details.summary.durationMs}ms`, + `tokens~${details.summary.tokenEstimate}`, + details.summary.fallbackUsed ? "fallback=true" : "fallback=false", + details.summary.edited ? "edited=true" : "edited=false", + ]; + if (details.summary.fallbackReason) { + metaParts.push(`reason=${details.summary.fallbackReason}`); + } + lines.push(theme.fg("dim", " " + metaParts.join(" · "))); + } + + const queryDetails = details?.curatedQueries; + if (queryDetails?.length) { + const kept = queryDetails.length; + const from = details?.curatedFrom ?? kept; + lines.push(""); + lines.push(theme.fg("accent", `\u2500\u2500 Curated Results (${kept} of ${from} queries kept) ` + "\u2500".repeat(24))); + + for (const cq of queryDetails) { + lines.push(""); + const dq = cq.query.length > 65 ? cq.query.slice(0, 62) + "..." : cq.query; + const providerLabel = cq.provider ? ` (${cq.provider})` : ""; + lines.push(theme.fg("accent", ` "${dq}"${providerLabel}`)); + + if (cq.error) { + lines.push(theme.fg("error", ` ${cq.error}`)); + } else if (cq.answer) { + lines.push(""); + for (const line of cq.answer.split("\n")) { + lines.push(` ${line}`); + } + } + + if (cq.sources.length > 0) { + lines.push(""); + for (const s of cq.sources) { + const domain = s.url.replace(/^https?:\/\//, "").replace(/\/.*$/, ""); + const title = s.title.length > 50 ? s.title.slice(0, 47) + "..." : s.title; + lines.push(theme.fg("muted", ` \u25b8 ${title}`) + theme.fg("dim", ` \u00b7 ${domain}`)); + } + } + } + lines.push(""); + } else { + const textContent = result.content.find((c) => c.type === "text")?.text || ""; + const preview = textContent.length > 500 ? textContent.slice(0, 500) + "..." : textContent; + for (const line of preview.split("\n")) { + lines.push(theme.fg("dim", line)); + } + } + + if (details?.fetchUrls && details.fetchUrls.length > 0) { + if (details.curated) { + lines.push(theme.fg("muted", `Fetching ${details.fetchUrls.length} URLs in background`)); + } else { + lines.push(theme.fg("muted", "Fetching:")); + for (const u of details.fetchUrls.slice(0, 5)) { + const display = u.length > 60 ? u.slice(0, 57) + "..." : u; + lines.push(theme.fg("dim", " " + display)); + } + if (details.fetchUrls.length > 5) { + lines.push(theme.fg("dim", ` ... and ${details.fetchUrls.length - 5} more`)); + } + } + } + + const totalLines = lines.length; + + if (!expanded) { + const box = new Box(1, 0, (t) => theme.bg("toolSuccessBg", t)); + box.addChild(new Text(statusLine, 0, 0)); + + let collapsedLines = 1; // statusLine + const summaryPreview = details?.summary?.text?.trim() || ""; + if (summaryPreview) { + const preview = summaryPreview.length > 120 ? summaryPreview.slice(0, 117) + "..." : summaryPreview; + box.addChild(new Text(theme.fg("dim", preview), 0, 0)); + collapsedLines++; + } else if (details?.curatedQueries?.length) { + for (const cq of details.curatedQueries.slice(0, 3)) { + const dq = cq.query.length > 55 ? cq.query.slice(0, 52) + "..." : cq.query; + const srcCount = cq.sources?.length ?? 0; + const suffix = cq.error ? theme.fg("error", " (error)") : theme.fg("dim", ` · ${srcCount} sources`); + box.addChild(new Text(theme.fg("accent", ` "${dq}"`) + suffix, 0, 0)); + collapsedLines++; + } + if (details.curatedQueries.length > 3) { + box.addChild(new Text(theme.fg("dim", ` ... and ${details.curatedQueries.length - 3} more`), 0, 0)); + collapsedLines++; + } + } else { + const textContent = result.content.find((c) => c.type === "text")?.text || ""; + const firstContentLine = textContent.split("\n").find(l => { + const t = l.trim(); + return t && !t.startsWith("[") && !t.startsWith("#") && !t.startsWith("---"); + }); + const fallbackLine = (firstContentLine?.trim() || "").replace(/\*\*/g, ""); + if (fallbackLine) { + const preview = fallbackLine.length > 120 ? fallbackLine.slice(0, 117) + "..." : fallbackLine; + box.addChild(new Text(theme.fg("dim", preview), 0, 0)); + collapsedLines++; + } + } + const moreLines = Math.max(0, totalLines - collapsedLines); + if (moreLines > 0) { + box.addChild(new Text(theme.fg("muted", `\n... (${moreLines} more lines, ${totalLines} total, ctrl+o to expand)`), 0, 0)); + } + return box; + } + + return new Text(lines.join("\n"), 0, 0); + }, + }); + + pi.registerTool({ + name: "code_search", + label: "Code Search", + description: "Search for code examples, documentation, and API references. Returns relevant code snippets and docs from GitHub, Stack Overflow, and official documentation. Use for any programming question — API usage, library examples, debugging help.", + promptSnippet: + "Use for programming/API/library questions to retrieve concrete examples and docs before implementing or debugging code.", + parameters: Type.Object({ + query: Type.String({ description: "Programming question, API, library, or debugging topic to search for" }), + maxTokens: Type.Optional(Type.Integer({ + minimum: 1000, + maximum: 50000, + description: "Maximum tokens of code/documentation context to return (default: 5000)", + })), + }), + + async execute(toolCallId, params, signal) { + return executeCodeSearch(toolCallId, params, signal); + }, + + renderCall(args, theme) { + const { query } = args as { query?: string }; + const display = !query + ? "(no query)" + : query.length > 70 ? query.slice(0, 67) + "..." : query; + return new Text(theme.fg("toolTitle", theme.bold("code_search ")) + theme.fg("accent", display), 0, 0); + }, + + renderResult(result, { expanded }, theme) { + const details = result.details as { query?: string; maxTokens?: number; error?: string }; + if (details?.error) { + return new Text(theme.fg("error", `Error: ${details.error}`), 0, 0); + } + + const summary = theme.fg("success", "code context returned") + + theme.fg("muted", ` (${details?.maxTokens ?? 5000} tokens max)`); + if (!expanded) return new Text(summary, 0, 0); + + const textContent = result.content.find((c) => c.type === "text")?.text || ""; + const preview = textContent.length > 500 ? textContent.slice(0, 500) + "..." : textContent; + return new Text(summary + "\n" + theme.fg("dim", preview), 0, 0); + }, + }); + + pi.registerTool({ + name: "fetch_content", + label: "Fetch Content", + description: "Fetch URL(s) and extract readable content as markdown. Supports YouTube video transcripts (with thumbnail), GitHub repository contents, and local video files (with frame thumbnail). Video frames can be extracted via timestamp/range or sampled across the entire video with frames alone. Falls back to Gemini for pages that block bots or fail Readability extraction. For YouTube and video files: ALWAYS pass the user's specific question via the prompt parameter — this directs the AI to focus on that aspect of the video, producing much better results than a generic extraction. Content is always stored and can be retrieved with get_search_content.", + promptSnippet: + "Use to extract readable content from URL(s), YouTube, GitHub repos, or local videos. For video questions, pass the user's exact question in prompt.", + parameters: Type.Object({ + url: Type.Optional(Type.String({ description: "Single URL to fetch" })), + urls: Type.Optional(Type.Array(Type.String(), { description: "Multiple URLs (parallel)" })), + forceClone: Type.Optional(Type.Boolean({ + description: "Force cloning large GitHub repositories that exceed the size threshold", + })), + prompt: Type.Optional(Type.String({ + description: "Question or instruction for video analysis (YouTube and video files). Pass the user's specific question here — e.g. 'describe the book shown at the advice for beginners section'. Without this, a generic transcript extraction is used which may miss what the user is asking about.", + })), + timestamp: Type.Optional(Type.String({ + description: "Extract video frame(s) at a timestamp or time range. Single: '1:23:45', '23:45', or '85' (seconds). Range: '23:41-25:00' extracts evenly-spaced frames across that span (default 6). Use frames with ranges to control density; single+frames uses a fixed 5s interval. YouTube requires yt-dlp + ffmpeg; local videos require ffmpeg. Use a range when you know the approximate area but not the exact moment — you'll get a contact sheet to visually identify the right frame.", + })), + frames: Type.Optional(Type.Integer({ + minimum: 1, + maximum: 12, + description: "Number of frames to extract. Use with timestamp range for custom density, with single timestamp to get N frames at 5s intervals, or alone to sample across the entire video. Requires yt-dlp + ffmpeg for YouTube, ffmpeg for local video.", + })), + model: Type.Optional(Type.String({ + description: "Override the Gemini model for video/YouTube analysis (e.g. 'gemini-2.5-flash', 'gemini-3-flash-preview'). Defaults to config or gemini-3-flash-preview.", + })), + }), + + async execute(_toolCallId, params, signal, onUpdate) { + const urlList = params.urls ?? (params.url ? [params.url] : []); + if (urlList.length === 0) { + return { + content: [{ type: "text", text: "Error: No URL provided." }], + details: { error: "No URL provided" }, + }; + } + + onUpdate?.({ + content: [{ type: "text", text: `Fetching ${urlList.length} URL(s)...` }], + details: { phase: "fetch", progress: 0 }, + }); + + const fetchResults = await fetchAllContent(urlList, signal, { + forceClone: params.forceClone, + prompt: params.prompt, + timestamp: params.timestamp, + frames: params.frames, + model: params.model, + }); + const successful = fetchResults.filter((r) => !r.error).length; + const totalChars = fetchResults.reduce((sum, r) => sum + r.content.length, 0); + + // ALWAYS store results (even for single URL) + const responseId = generateId(); + const data: StoredSearchData = { + id: responseId, + type: "fetch", + timestamp: Date.now(), + urls: stripThumbnails(fetchResults), + }; + storeResult(responseId, data); + pi.appendEntry("web-search-results", data); + + // Single URL: return content directly (possibly truncated) with responseId + if (urlList.length === 1) { + const result = fetchResults[0]; + if (result.error) { + return { + content: [{ type: "text", text: `Error: ${result.error}` }], + details: { urls: urlList, urlCount: 1, successful: 0, error: result.error, responseId, prompt: params.prompt, timestamp: params.timestamp, frames: params.frames }, + }; + } + + const fullLength = result.content.length; + const truncated = fullLength > MAX_INLINE_CONTENT; + let output = truncated + ? result.content.slice(0, MAX_INLINE_CONTENT) + "\n\n[Content truncated...]" + : result.content; + + if (truncated) { + output += `\n\n---\nShowing ${MAX_INLINE_CONTENT} of ${fullLength} chars. ` + + `Use get_search_content({ responseId: "${responseId}", urlIndex: 0 }) for full content.`; + } + + const content: Array<{ type: string; text?: string; data?: string; mimeType?: string }> = []; + if (result.frames?.length) { + for (const frame of result.frames) { + content.push({ type: "image", data: frame.data, mimeType: frame.mimeType }); + content.push({ type: "text", text: `Frame at ${frame.timestamp}` }); + } + } else if (result.thumbnail) { + content.push({ type: "image", data: result.thumbnail.data, mimeType: result.thumbnail.mimeType }); + } + content.push({ type: "text", text: output }); + + const imageCount = (result.frames?.length ?? 0) + (result.thumbnail ? 1 : 0); + return { + content, + details: { + urls: urlList, + urlCount: 1, + successful: 1, + totalChars: fullLength, + title: result.title, + responseId, + truncated, + hasImage: imageCount > 0, + imageCount, + prompt: params.prompt, + timestamp: params.timestamp, + frames: params.frames, + duration: result.duration, + }, + }; + } + + // Multi-URL: existing behavior (summary + responseId) + let output = "## Fetched URLs\n\n"; + for (const { url, title, content, error } of fetchResults) { + if (error) { + output += `- ${url}: Error - ${error}\n`; + } else { + output += `- ${title || url} (${content.length} chars)\n`; + } + } + output += `\n---\nUse get_search_content({ responseId: "${responseId}", urlIndex: 0 }) to retrieve full content.`; + + return { + content: [{ type: "text", text: output }], + details: { urls: urlList, urlCount: urlList.length, successful, totalChars, responseId }, + }; + }, + + renderCall(args, theme) { + const { url, urls, prompt, timestamp, frames, model } = args as { url?: string; urls?: string[]; prompt?: string; timestamp?: string; frames?: number; model?: string }; + const urlList = urls ?? (url ? [url] : []); + if (urlList.length === 0) { + return new Text(theme.fg("toolTitle", theme.bold("fetch ")) + theme.fg("error", "(no URL)"), 0, 0); + } + const lines: string[] = []; + if (urlList.length === 1) { + const display = urlList[0].length > 60 ? urlList[0].slice(0, 57) + "..." : urlList[0]; + lines.push(theme.fg("toolTitle", theme.bold("fetch ")) + theme.fg("accent", display)); + } else { + lines.push(theme.fg("toolTitle", theme.bold("fetch ")) + theme.fg("accent", `${urlList.length} URLs`)); + for (const u of urlList.slice(0, 5)) { + const display = u.length > 60 ? u.slice(0, 57) + "..." : u; + lines.push(theme.fg("muted", " " + display)); + } + if (urlList.length > 5) { + lines.push(theme.fg("muted", ` ... and ${urlList.length - 5} more`)); + } + } + if (timestamp) { + lines.push(theme.fg("dim", " timestamp: ") + theme.fg("warning", timestamp)); + } + if (typeof frames === "number") { + lines.push(theme.fg("dim", " frames: ") + theme.fg("warning", String(frames))); + } + if (prompt) { + const display = prompt.length > 250 ? prompt.slice(0, 247) + "..." : prompt; + lines.push(theme.fg("dim", " prompt: ") + theme.fg("muted", `"${display}"`)); + } + if (model) { + lines.push(theme.fg("dim", " model: ") + theme.fg("warning", model)); + } + return new Text(lines.join("\n"), 0, 0); + }, + + renderResult(result, { expanded, isPartial }, theme) { + const details = result.details as { + urlCount?: number; + successful?: number; + totalChars?: number; + error?: string; + title?: string; + truncated?: boolean; + responseId?: string; + phase?: string; + progress?: number; + hasImage?: boolean; + imageCount?: number; + prompt?: string; + timestamp?: string; + frames?: number; + duration?: number; + }; + + if (isPartial) { + const progress = details?.progress ?? 0; + const bar = "\u2588".repeat(Math.floor(progress * 10)) + "\u2591".repeat(10 - Math.floor(progress * 10)); + return new Text(theme.fg("accent", `[${bar}] ${details?.phase || "fetching"}`), 0, 0); + } + + if (details?.error) { + return new Text(theme.fg("error", `Error: ${details.error}`), 0, 0); + } + + if (details?.urlCount === 1) { + const title = details?.title || "Untitled"; + const imgCount = details?.imageCount ?? (details?.hasImage ? 1 : 0); + const imageBadge = imgCount > 1 + ? theme.fg("accent", ` [${imgCount} images]`) + : imgCount === 1 + ? theme.fg("accent", " [image]") + : ""; + let statusLine = theme.fg("success", title) + theme.fg("muted", ` (${details?.totalChars ?? 0} chars)`) + imageBadge; + if (details?.truncated) { + statusLine += theme.fg("warning", " [truncated]"); + } + if (typeof details?.duration === "number") { + statusLine += theme.fg("muted", ` | ${formatSeconds(Math.floor(details.duration))} total`); + } + const textContent = result.content.find((c) => c.type === "text")?.text || ""; + if (!expanded) { + const brief = textContent.length > 200 ? textContent.slice(0, 200) + "..." : textContent; + return new Text(statusLine + "\n" + theme.fg("dim", brief), 0, 0); + } + const lines = [statusLine]; + if (details?.prompt) { + const display = details.prompt.length > 250 ? details.prompt.slice(0, 247) + "..." : details.prompt; + lines.push(theme.fg("dim", ` prompt: "${display}"`)); + } + if (details?.timestamp) { + lines.push(theme.fg("dim", ` timestamp: ${details.timestamp}`)); + } + if (typeof details?.frames === "number") { + lines.push(theme.fg("dim", ` frames: ${details.frames}`)); + } + const preview = textContent.length > 500 ? textContent.slice(0, 500) + "..." : textContent; + lines.push(theme.fg("dim", preview)); + return new Text(lines.join("\n"), 0, 0); + } + + const countColor = (details?.successful ?? 0) > 0 ? "success" : "error"; + const statusLine = theme.fg(countColor, `${details?.successful}/${details?.urlCount} URLs`) + theme.fg("muted", " (content stored)"); + if (!expanded) { + return new Text(statusLine, 0, 0); + } + const textContent = result.content.find((c) => c.type === "text")?.text || ""; + const preview = textContent.length > 500 ? textContent.slice(0, 500) + "..." : textContent; + return new Text(statusLine + "\n" + theme.fg("dim", preview), 0, 0); + }, + }); + + pi.registerTool({ + name: "get_search_content", + label: "Get Search Content", + description: "Retrieve full content from a previous web_search or fetch_content call.", + promptSnippet: + "Use after web_search/fetch_content when full stored content is needed via responseId plus query/url selectors.", + parameters: Type.Object({ + responseId: Type.String({ description: "The responseId from web_search or fetch_content" }), + query: Type.Optional(Type.String({ description: "Get content for this query (web_search)" })), + queryIndex: Type.Optional(Type.Number({ description: "Get content for query at index" })), + url: Type.Optional(Type.String({ description: "Get content for this URL" })), + urlIndex: Type.Optional(Type.Number({ description: "Get content for URL at index" })), + }), + + async execute(_toolCallId, params) { + const data = getResult(params.responseId); + if (!data) { + return { + content: [{ type: "text", text: `Error: No stored results for "${params.responseId}"` }], + details: { error: "Not found", responseId: params.responseId }, + }; + } + + if (data.type === "search" && data.queries) { + let queryData: QueryResultData | undefined; + + if (params.query !== undefined) { + queryData = data.queries.find((q) => q.query === params.query); + if (!queryData) { + const available = data.queries.map((q) => `"${q.query}"`).join(", "); + return { + content: [{ type: "text", text: `Query "${params.query}" not found. Available: ${available}` }], + details: { error: "Query not found" }, + }; + } + } else if (params.queryIndex !== undefined) { + queryData = data.queries[params.queryIndex]; + if (!queryData) { + return { + content: [{ type: "text", text: `Index ${params.queryIndex} out of range (0-${data.queries.length - 1})` }], + details: { error: "Index out of range" }, + }; + } + } else { + const available = data.queries.map((q, i) => `${i}: "${q.query}"`).join(", "); + return { + content: [{ type: "text", text: `Specify query or queryIndex. Available: ${available}` }], + details: { error: "No query specified" }, + }; + } + + if (queryData.error) { + return { + content: [{ type: "text", text: `Error for "${queryData.query}": ${queryData.error}` }], + details: { error: queryData.error, query: queryData.query }, + }; + } + + return { + content: [{ type: "text", text: formatFullResults(queryData) }], + details: { query: queryData.query, resultCount: queryData.results.length }, + }; + } + + if (data.type === "fetch" && data.urls) { + let urlData: ExtractedContent | undefined; + + if (params.url !== undefined) { + urlData = data.urls.find((u) => u.url === params.url); + if (!urlData) { + const available = data.urls.map((u) => u.url).join("\n "); + return { + content: [{ type: "text", text: `URL not found. Available:\n ${available}` }], + details: { error: "URL not found" }, + }; + } + } else if (params.urlIndex !== undefined) { + urlData = data.urls[params.urlIndex]; + if (!urlData) { + return { + content: [{ type: "text", text: `Index ${params.urlIndex} out of range (0-${data.urls.length - 1})` }], + details: { error: "Index out of range" }, + }; + } + } else { + const available = data.urls.map((u, i) => `${i}: ${u.url}`).join("\n "); + return { + content: [{ type: "text", text: `Specify url or urlIndex. Available:\n ${available}` }], + details: { error: "No URL specified" }, + }; + } + + if (urlData.error) { + return { + content: [{ type: "text", text: `Error for ${urlData.url}: ${urlData.error}` }], + details: { error: urlData.error, url: urlData.url }, + }; + } + + return { + content: [{ type: "text", text: `# ${urlData.title}\n\n${urlData.content}` }], + details: { url: urlData.url, title: urlData.title, contentLength: urlData.content.length }, + }; + } + + return { + content: [{ type: "text", text: "Invalid stored data format" }], + details: { error: "Invalid data" }, + }; + }, + + renderCall(args, theme) { + const { responseId, query, queryIndex, url, urlIndex } = args as { + responseId: string; + query?: string; + queryIndex?: number; + url?: string; + urlIndex?: number; + }; + let target = ""; + if (query) target = `query="${query}"`; + else if (queryIndex !== undefined) target = `queryIndex=${queryIndex}`; + else if (url) target = url.length > 30 ? url.slice(0, 27) + "..." : url; + else if (urlIndex !== undefined) target = `urlIndex=${urlIndex}`; + return new Text(theme.fg("toolTitle", theme.bold("get_content ")) + theme.fg("accent", target || responseId.slice(0, 8)), 0, 0); + }, + + renderResult(result, { expanded }, theme) { + const details = result.details as { + error?: string; + query?: string; + url?: string; + title?: string; + resultCount?: number; + contentLength?: number; + }; + + if (details?.error) { + return new Text(theme.fg("error", `Error: ${details.error}`), 0, 0); + } + + let statusLine: string; + if (details?.query) { + statusLine = theme.fg("success", `"${details.query}"`) + theme.fg("muted", ` (${details.resultCount} results)`); + } else { + statusLine = theme.fg("success", details?.title || "Content") + theme.fg("muted", ` (${details?.contentLength ?? 0} chars)`); + } + + if (!expanded) { + return new Text(statusLine, 0, 0); + } + + const textContent = result.content.find((c) => c.type === "text")?.text || ""; + const preview = textContent.length > 500 ? textContent.slice(0, 500) + "..." : textContent; + return new Text(statusLine + "\n" + theme.fg("dim", preview), 0, 0); + }, + }); + + pi.registerCommand("websearch", { + description: "Open web search curator", + handler: async (args, ctx) => { + closeCurator(); + const sessionToken = randomUUID(); + + const raw = args.trim(); + const queries = raw.length > 0 + ? normalizeQueryList(raw.split(",")) + : []; + + let bootstrap: CuratorBootstrap; + try { + bootstrap = await loadCuratorBootstrap(undefined); + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + ctx.ui.notify(`Failed to load web search config: ${message}`, "error"); + return; + } + const availableProviders = bootstrap.availableProviders; + const initialProvider = bootstrap.defaultProvider; + const curatorTimeoutSeconds = bootstrap.timeoutSeconds; + let currentProvider = initialProvider; + const summaryContext: SummaryGenerationContext = { + model: ctx.model, + modelRegistry: ctx.modelRegistry, + }; + const summaryModelChoices = await loadSummaryModelChoices(summaryContext); + + ctx.ui.notify("Opening web search curator...", "info"); + + const collected = new Map(); + const searchAbort = new AbortController(); + let aborted = false; + let commandHandle: CuratorServerHandle | null = null; + + function sendFollowUpFromReturn(payload: ReturnType) { + pi.sendMessage({ + customType: "web-search-results", + content: payload.content, + display: "tool", + details: payload.details, + }, { triggerTurn: true, deliverAs: "followUp" }); + } + + try { + const handle = await startCuratorServer( + { + queries, + sessionToken, + timeout: curatorTimeoutSeconds, + availableProviders, + defaultProvider: initialProvider, + summaryModels: summaryModelChoices.summaryModels, + defaultSummaryModel: summaryModelChoices.defaultSummaryModel, + }, + { + async onSummarize(selectedQueryIndices, summarizeSignal, model, feedback) { + if (commandHandle && activeCurator !== commandHandle) { + throw new Error("Curator session is no longer active."); + } + return generateSummaryForSelectedIndices( + selectedQueryIndices, + collected, + summaryContext, + summarizeSignal, + model, + feedback, + ); + }, + onSubmit(payload) { + if (commandHandle && activeCurator !== commandHandle) return; + aborted = true; + searchAbort.abort(); + const filtered = payload.selectedQueryIndices.length > 0 + ? filterByQueryIndices(payload.selectedQueryIndices, collected) + : collectAllResultsAndUrls(collected); + const base: SearchReturnOptions = { + queryList: filtered.results.map(r => r.query), + results: filtered.results, + urls: filtered.urls, + includeContent: false, + curated: true, + curatedFrom: collected.size, + }; + if (!payload.rawResults) { + const resolvedSummary = resolveSummaryForSubmit(payload, collected); + base.workflow = "summary-review"; + base.approvedSummary = resolvedSummary.approvedSummary; + base.summaryMeta = resolvedSummary.summaryMeta; + } + sendFollowUpFromReturn(buildSearchReturn(base)); + closeCurator(); + }, + onCancel(reason) { + if (commandHandle && activeCurator !== commandHandle) return; + aborted = true; + searchAbort.abort(); + if (reason === "timeout") { + const all = collectAllResultsAndUrls(collected); + const resolvedSummary = resolveSummaryForSubmit({ selectedQueryIndices: [], summary: undefined, summaryMeta: undefined }, collected); + sendFollowUpFromReturn(buildSearchReturn({ + queryList: all.results.map(r => r.query), + results: all.results, + urls: all.urls, + includeContent: false, + curated: true, + curatedFrom: collected.size, + workflow: "summary-review", + approvedSummary: resolvedSummary.approvedSummary, + summaryMeta: resolvedSummary.summaryMeta, + })); + } + closeCurator(); + }, + onProviderChange(provider) { + if (commandHandle && activeCurator !== commandHandle) return; + const normalized = normalizeProviderInput(provider); + if (!normalized || normalized === "auto") return; + currentProvider = normalized; + try { + saveConfig({ provider: normalized }); + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + console.error(`Failed to persist default provider: ${message}`); + } + }, + async onAddSearch(query, queryIndex, provider) { + if (commandHandle && activeCurator !== commandHandle) { + throw new Error("Curator session is no longer active."); + } + const normalizedProvider = normalizeProviderInput(provider); + const requestedProvider = !normalizedProvider || normalizedProvider === "auto" + ? currentProvider + : normalizedProvider; + try { + const { answer, results, provider: actualProvider } = await search(query, { + provider: requestedProvider, + signal: searchAbort.signal, + }); + if (commandHandle && activeCurator !== commandHandle) { + throw new Error("Curator session is no longer active."); + } + collected.set(queryIndex, { query, answer, results, error: null, provider: actualProvider }); + return { + answer, + results: results.map(r => ({ title: r.title, url: r.url, domain: extractDomain(r.url) })), + provider: actualProvider, + }; + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + if (!commandHandle || activeCurator === commandHandle) { + collected.set(queryIndex, { query, answer: "", results: [], error: message, provider: requestedProvider }); + } + throw err; + } + }, + async onRewriteQuery(query, rewriteSignal) { + if (commandHandle && activeCurator !== commandHandle) { + throw new Error("Curator session is no longer active."); + } + return rewriteSearchQuery(query, summaryContext, rewriteSignal); + }, + }, + ); + + commandHandle = handle; + activeCurator = handle; + const open = platform() === "darwin" ? await getGlimpseOpen() : null; + if (open) { + try { + const win = openInGlimpse(open, handle.url, "Search Curator"); + glimpseWin = win; + win.on("closed", () => { + if (glimpseWin === win) { + glimpseWin = null; + closeCurator(); + } + }); + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + console.error(`Failed to open Glimpse curator window: ${message}`); + glimpseWin = null; + await openInBrowser(pi, handle.url); + } + } else { + await openInBrowser(pi, handle.url); + } + + if (queries.length > 0) { + (async () => { + for (let qi = 0; qi < queries.length; qi++) { + if (aborted || activeCurator !== handle) break; + const requestedProvider = currentProvider; + try { + const { answer, results, provider } = await search(queries[qi], { + provider: requestedProvider, + signal: searchAbort.signal, + }); + if (aborted || activeCurator !== handle) break; + handle.pushResult(qi, { + answer, + results: results.map(r => ({ title: r.title, url: r.url, domain: extractDomain(r.url) })), + provider, + }); + collected.set(qi, { query: queries[qi], answer, results, error: null, provider }); + } catch (err) { + if (aborted || activeCurator !== handle) break; + const message = err instanceof Error ? err.message : String(err); + handle.pushError(qi, message, requestedProvider); + collected.set(qi, { query: queries[qi], answer: "", results: [], error: message, provider: requestedProvider }); + } + } + if (!aborted && activeCurator === handle) handle.searchesDone(); + })(); + } else { + if (activeCurator === handle) handle.searchesDone(); + } + } catch (err) { + closeCurator(); + const message = err instanceof Error ? err.message : String(err); + ctx.ui.notify(`Failed to open curator: ${message}`, "error"); + } + }, + }); + + pi.registerCommand("curator", { + description: "Toggle or configure the search curator workflow", + handler: async (args, ctx) => { + const arg = args.trim().toLowerCase(); + + let newWorkflow: WebSearchWorkflow; + if (arg.length === 0) { + const current = resolveWorkflow(loadConfigForExtensionInit().workflow, true); + newWorkflow = current === "none" ? "summary-review" : "none"; + } else if (arg === "on") { + newWorkflow = "summary-review"; + } else if (arg === "off") { + newWorkflow = "none"; + } else if (arg === "none" || arg === "summary-review") { + newWorkflow = arg; + } else { + ctx.ui.notify(`Unknown option: ${arg}. Use on, off, or summary-review.`, "error"); + return; + } + + try { + saveConfig({ workflow: newWorkflow }); + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + ctx.ui.notify(`Failed to save config: ${message}`, "error"); + return; + } + + const label = newWorkflow === "none" + ? "Curator disabled — web_search will return raw results" + : "Curator enabled — web_search will open curator and auto-generate a summary draft"; + pi.sendMessage({ + customType: "curator-config", + content: [{ type: "text", text: label }], + display: "tool", + details: { workflow: newWorkflow }, + }, { triggerTurn: false, deliverAs: "followUp" }); + }, + }); + + pi.registerCommand("google-account", { + description: "Show the active Google account for Gemini Web", + handler: async () => { + if (!isBrowserCookieAccessAllowed()) { + pi.sendMessage({ + customType: "google-account", + content: [{ type: "text", text: `Gemini Web browser cookie access is disabled. Set allowBrowserCookies: true in ~/${CONFIG_DIR_NAME}/web-search.json to enable it.` }], + display: "tool", + details: { available: false, cookieAccessAllowed: false }, + }, { triggerTurn: true, deliverAs: "followUp" }); + return; + } + + const cookies = await isGeminiWebAvailable(); + if (!cookies) { + pi.sendMessage({ + customType: "google-account", + content: [{ type: "text", text: "Gemini Web is unavailable. Sign into gemini.google.com in a supported Chromium-based browser." }], + display: "tool", + details: { available: false, cookieAccessAllowed: true }, + }, { triggerTurn: true, deliverAs: "followUp" }); + return; + } + + const email = await getActiveGoogleEmail(cookies); + const text = email + ? `Active Google account: ${email}` + : "Gemini Web is available, but the active Google account could not be determined."; + + pi.sendMessage({ + customType: "google-account", + content: [{ type: "text", text }], + display: "tool", + details: { available: true, email: email ?? null }, + }, { triggerTurn: true, deliverAs: "followUp" }); + }, + }); + + pi.registerCommand("search", { + description: "Browse stored web search results", + handler: async (_args, ctx) => { + const results = getAllResults(); + + if (results.length === 0) { + ctx.ui.notify("No stored search results", "info"); + return; + } + + const options = results.map((r) => { + const age = Math.floor((Date.now() - r.timestamp) / 60000); + const ageStr = age < 60 ? `${age}m ago` : `${Math.floor(age / 60)}h ago`; + if (r.type === "search" && r.queries) { + const query = r.queries[0]?.query || "unknown"; + return `[${r.id.slice(0, 6)}] "${query}" (${r.queries.length} queries) - ${ageStr}`; + } + if (r.type === "fetch" && r.urls) { + return `[${r.id.slice(0, 6)}] ${r.urls.length} URLs fetched - ${ageStr}`; + } + return `[${r.id.slice(0, 6)}] ${r.type} - ${ageStr}`; + }); + + const choice = await ctx.ui.select("Stored Search Results", options); + if (!choice) return; + + const match = choice.match(/^\[([a-z0-9]+)\]/); + if (!match) return; + + const selected = results.find((r) => r.id.startsWith(match[1])); + if (!selected) return; + + const actions = ["View details", "Delete"]; + const action = await ctx.ui.select(`Result ${selected.id.slice(0, 6)}`, actions); + + if (action === "Delete") { + deleteResult(selected.id); + ctx.ui.notify(`Deleted ${selected.id.slice(0, 6)}`, "info"); + } else if (action === "View details") { + let info = `ID: ${selected.id}\nType: ${selected.type}\nAge: ${Math.floor((Date.now() - selected.timestamp) / 60000)}m\n\n`; + if (selected.type === "search" && selected.queries) { + info += "Queries:\n"; + const queries = selected.queries.slice(0, 10); + for (const q of queries) { + info += `- "${q.query}" (${q.results.length} results)\n`; + } + if (selected.queries.length > 10) { + info += `... and ${selected.queries.length - 10} more\n`; + } + } + if (selected.type === "fetch" && selected.urls) { + info += "URLs:\n"; + const urls = selected.urls.slice(0, 10); + for (const u of urls) { + const urlDisplay = u.url.length > 50 ? u.url.slice(0, 47) + "..." : u.url; + info += `- ${urlDisplay} (${u.error || `${u.content.length} chars`})\n`; + } + if (selected.urls.length > 10) { + info += `... and ${selected.urls.length - 10} more\n`; + } + } + ctx.ui.notify(info, "info"); + } + }, + }); +} diff --git a/packages/web-access/package.json b/packages/web-access/package.json new file mode 100644 index 000000000..cf4ed2c23 --- /dev/null +++ b/packages/web-access/package.json @@ -0,0 +1,54 @@ +{ + "name": "@bastani/web-access", + "version": "0.8.0", + "private": true, + "description": "Atomic extension for web search, URL fetching, GitHub repo cloning, PDF/video extraction.", + "contributors": [ + "Nico Bailon", + "Alex Lavaee" + ], + "license": "MIT", + "type": "module", + "engines": { + "bun": ">=1.3.14" + }, + "main": "./index.ts", + "types": "./index.ts", + "exports": { + ".": "./index.ts" + }, + "files": [ + "*.ts", + "README.md", + "CHANGELOG.md", + "LICENSE" + ], + "pi": { + "extensions": [ + "./index.ts" + ] + }, + "peerDependencies": { + "@bastani/atomic": "*", + "@earendil-works/pi-coding-agent": "*", + "@earendil-works/pi-tui": "*" + }, + "peerDependenciesMeta": { + "@bastani/atomic": { + "optional": true + }, + "@earendil-works/pi-coding-agent": { + "optional": true + }, + "@earendil-works/pi-tui": { + "optional": true + } + }, + "dependencies": { + "@mozilla/readability": "^0.5.0", + "linkedom": "^0.16.0", + "p-limit": "^6.1.0", + "turndown": "^7.2.0", + "unpdf": "^1.6.2" + } +} diff --git a/packages/web-access/pdf-extract.ts b/packages/web-access/pdf-extract.ts new file mode 100644 index 000000000..23cb6a367 --- /dev/null +++ b/packages/web-access/pdf-extract.ts @@ -0,0 +1,192 @@ +/** + * PDF Content Extractor + * + * Extracts text from PDF files and saves to markdown. + * Uses unpdf (pdfjs-dist wrapper) for text extraction. + */ + +import { getDocumentProxy } from "unpdf"; +import { writeFile, mkdir } from "node:fs/promises"; +import { join, basename } from "node:path"; +import { homedir } from "node:os"; + +export interface PDFExtractResult { + title: string; + pages: number; + chars: number; + outputPath: string; +} + +export interface PDFExtractOptions { + maxPages?: number; + outputDir?: string; + filename?: string; +} + +const DEFAULT_MAX_PAGES = 100; +const DEFAULT_OUTPUT_DIR = join(homedir(), "Downloads"); + +/** + * Extract text from a PDF buffer and save to markdown file + */ +export async function extractPDFToMarkdown( + buffer: ArrayBuffer, + url: string, + options: PDFExtractOptions = {} +): Promise { + const { + maxPages = DEFAULT_MAX_PAGES, + outputDir = DEFAULT_OUTPUT_DIR, + filename, + } = options; + + const safeMaxPages = Number.isFinite(maxPages) + ? Math.max(1, Math.floor(maxPages)) + : DEFAULT_MAX_PAGES; + + const pdf = await getDocumentProxy(new Uint8Array(buffer)); + const metadata = await pdf.getMetadata(); + const metadataInfo = metadata.info && typeof metadata.info === "object" + ? metadata.info as Record + : null; + + // Extract title from metadata or URL + const metaTitle = typeof metadataInfo?.Title === "string" ? metadataInfo.Title : undefined; + const metaAuthor = typeof metadataInfo?.Author === "string" ? metadataInfo.Author : undefined; + const urlTitle = extractTitleFromURL(url); + const title = metaTitle?.trim() || urlTitle; + + // Determine pages to extract + const pagesToExtract = Math.min(pdf.numPages, safeMaxPages); + const truncated = pdf.numPages > safeMaxPages; + + // Extract text page by page for better structure + const pages: { pageNum: number; text: string }[] = []; + for (let i = 1; i <= pagesToExtract; i++) { + const page = await pdf.getPage(i); + const textContent = await page.getTextContent(); + const pageText = textContent.items + .map((item: unknown) => { + const textItem = item as { str?: string }; + return textItem.str || ""; + }) + .join(" ") + .replace(/\s+/g, " ") + .trim(); + + if (pageText) { + pages.push({ pageNum: i, text: pageText }); + } + } + + // Build markdown content + const lines: string[] = []; + + // Header with metadata + lines.push(`# ${title}`); + lines.push(""); + lines.push(`> Source: ${url}`); + lines.push(`> Pages: ${pdf.numPages}${truncated ? ` (extracted first ${pagesToExtract})` : ""}`); + if (metaAuthor) { + lines.push(`> Author: ${metaAuthor}`); + } + lines.push(""); + lines.push("---"); + lines.push(""); + + // Content with page markers + for (let i = 0; i < pages.length; i++) { + if (i > 0) { + lines.push(""); + lines.push(``); + lines.push(""); + } + lines.push(pages[i].text); + } + + if (truncated) { + lines.push(""); + lines.push("---"); + lines.push(""); + lines.push(`*[Truncated: Only first ${pagesToExtract} of ${pdf.numPages} pages extracted]*`); + } + + const content = lines.join("\n"); + + // Generate output filename + const outputFilename = filename || sanitizeFilename(title) + ".md"; + const outputPath = join(outputDir, outputFilename); + + // Ensure output directory exists + await mkdir(outputDir, { recursive: true }); + + // Write file + await writeFile(outputPath, content, "utf-8"); + + return { + title, + pages: pdf.numPages, + chars: content.length, + outputPath, + }; +} + +/** + * Extract a reasonable title from URL + */ +function extractTitleFromURL(url: string): string { + try { + const urlObj = new URL(url); + const pathname = urlObj.pathname; + + // Get filename without extension + let filename = basename(pathname, ".pdf"); + + // Handle arxiv URLs: /pdf/1706.03762 → "arxiv-1706.03762" + if (urlObj.hostname === "arxiv.org" || urlObj.hostname.endsWith(".arxiv.org")) { + const match = pathname.match(/\/(?:pdf|abs)\/(\d+\.\d+)/); + if (match) { + filename = `arxiv-${match[1]}`; + } + } + + // Clean up filename + filename = filename + .replace(/[_-]+/g, " ") + .replace(/\s+/g, " ") + .trim(); + + return filename || "document"; + } catch { + return "document"; + } +} + +/** + * Sanitize string for use as filename + */ +function sanitizeFilename(name: string): string { + return name + .toLowerCase() + .replace(/[^a-z0-9\s-]/g, "") + .replace(/\s+/g, "-") + .replace(/-+/g, "-") + .slice(0, 100) + .replace(/^-|-$/g, "") + || "document"; +} + +/** + * Check if URL or content-type indicates a PDF + */ +export function isPDF(url: string, contentType?: string): boolean { + if (contentType?.includes("application/pdf")) { + return true; + } + try { + const urlObj = new URL(url); + return urlObj.pathname.toLowerCase().endsWith(".pdf"); + } catch { + return false; + } +} diff --git a/packages/web-access/perplexity.ts b/packages/web-access/perplexity.ts new file mode 100644 index 000000000..77ed69c78 --- /dev/null +++ b/packages/web-access/perplexity.ts @@ -0,0 +1,196 @@ +import { existsSync, readFileSync } from "node:fs"; +import { homedir } from "node:os"; +import { join } from "node:path"; +import { CONFIG_DIR_NAME } from "@bastani/atomic"; +import { activityMonitor } from "./activity.js"; +import type { ExtractedContent } from "./extract.js"; + +const PERPLEXITY_API_URL = "https://api.perplexity.ai/chat/completions"; +const CONFIG_PATH = join(homedir(), CONFIG_DIR_NAME, "web-search.json"); + +const RATE_LIMIT = { + maxRequests: 10, + windowMs: 60 * 1000, +}; + +const requestTimestamps: number[] = []; + +export interface SearchResult { + title: string; + url: string; + snippet: string; +} + +export interface SearchResponse { + answer: string; + results: SearchResult[]; + inlineContent?: ExtractedContent[]; +} + +export interface SearchOptions { + numResults?: number; + recencyFilter?: "day" | "week" | "month" | "year"; + domainFilter?: string[]; + signal?: AbortSignal; +} + +interface WebSearchConfig { + perplexityApiKey?: unknown; +} + +let cachedConfig: WebSearchConfig | null = null; + +function loadConfig(): WebSearchConfig { + if (cachedConfig) return cachedConfig; + if (!existsSync(CONFIG_PATH)) { + cachedConfig = {}; + return cachedConfig; + } + + const content = readFileSync(CONFIG_PATH, "utf-8"); + try { + cachedConfig = JSON.parse(content) as WebSearchConfig; + return cachedConfig; + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + throw new Error(`Failed to parse ${CONFIG_PATH}: ${message}`); + } +} + +function normalizeApiKey(value: unknown): string | null { + if (typeof value !== "string") return null; + const normalized = value.trim(); + return normalized.length > 0 ? normalized : null; +} + +function getApiKey(): string { + const config = loadConfig(); + const key = normalizeApiKey(process.env.PERPLEXITY_API_KEY) ?? normalizeApiKey(config.perplexityApiKey); + if (!key) { + throw new Error( + "Perplexity API key not found. Either:\n" + + ` 1. Create ${CONFIG_PATH} with { "perplexityApiKey": "your-key" }\n` + + " 2. Set PERPLEXITY_API_KEY environment variable\n" + + "Get a key at https://perplexity.ai/settings/api" + ); + } + return key; +} + +function checkRateLimit(): void { + const now = Date.now(); + const windowStart = now - RATE_LIMIT.windowMs; + + while (requestTimestamps.length > 0 && requestTimestamps[0] < windowStart) { + requestTimestamps.shift(); + } + + if (requestTimestamps.length >= RATE_LIMIT.maxRequests) { + const waitMs = requestTimestamps[0] + RATE_LIMIT.windowMs - now; + throw new Error(`Rate limited. Try again in ${Math.ceil(waitMs / 1000)}s`); + } + + requestTimestamps.push(now); +} + +function validateDomainFilter(domains: string[]): string[] { + return domains.filter((d) => { + const domain = d.startsWith("-") ? d.slice(1) : d; + return /^[a-zA-Z0-9][a-zA-Z0-9-_.]*\.[a-zA-Z]{2,}$/.test(domain); + }); +} + +export function isPerplexityAvailable(): boolean { + const config = loadConfig(); + return !!(normalizeApiKey(process.env.PERPLEXITY_API_KEY) ?? normalizeApiKey(config.perplexityApiKey)); +} + +export async function searchWithPerplexity(query: string, options: SearchOptions = {}): Promise { + checkRateLimit(); + + const activityId = activityMonitor.logStart({ type: "api", query }); + + activityMonitor.updateRateLimit({ + used: requestTimestamps.length, + max: RATE_LIMIT.maxRequests, + oldestTimestamp: requestTimestamps[0] ?? null, + windowMs: RATE_LIMIT.windowMs, + }); + + const apiKey = getApiKey(); + const numResults = Math.min(options.numResults ?? 5, 20); + + const requestBody: Record = { + model: "sonar", + messages: [{ role: "user", content: query }], + max_tokens: 1024, + return_related_questions: false, + }; + + if (options.recencyFilter) { + requestBody.search_recency_filter = options.recencyFilter; + } + + if (options.domainFilter && options.domainFilter.length > 0) { + const validated = validateDomainFilter(options.domainFilter); + if (validated.length > 0) { + requestBody.search_domain_filter = validated; + } + } + + let response: Response; + try { + response = await fetch(PERPLEXITY_API_URL, { + method: "POST", + headers: { + Authorization: `Bearer ${apiKey}`, + "Content-Type": "application/json", + }, + body: JSON.stringify(requestBody), + signal: options.signal, + }); + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + if (message.toLowerCase().includes("abort")) { + activityMonitor.logComplete(activityId, 0); + } else { + activityMonitor.logError(activityId, message); + } + throw err; + } + + if (!response.ok) { + activityMonitor.logComplete(activityId, response.status); + const errorText = await response.text(); + throw new Error(`Perplexity API error ${response.status}: ${errorText}`); + } + + let data: Record; + try { + data = await response.json(); + } catch (err) { + activityMonitor.logComplete(activityId, response.status); + const message = err instanceof Error ? err.message : String(err); + throw new Error(`Perplexity API returned invalid JSON: ${message}`); + } + + const answer = (data.choices as Array<{ message?: { content?: string } }>)?.[0]?.message?.content || ""; + const citations = Array.isArray(data.citations) ? data.citations : []; + + const results: SearchResult[] = []; + for (let i = 0; i < Math.min(citations.length, numResults); i++) { + const citation = citations[i]; + if (typeof citation === "string") { + results.push({ title: `Source ${i + 1}`, url: citation, snippet: "" }); + } else if (citation && typeof citation === "object" && typeof citation.url === "string") { + results.push({ + title: citation.title || `Source ${i + 1}`, + url: citation.url, + snippet: "", + }); + } + } + + activityMonitor.logComplete(activityId, response.status); + return { answer, results }; +} diff --git a/packages/web-access/rsc-extract.ts b/packages/web-access/rsc-extract.ts new file mode 100644 index 000000000..44d22fe60 --- /dev/null +++ b/packages/web-access/rsc-extract.ts @@ -0,0 +1,338 @@ +/** + * RSC Content Extractor + * + * Extracts readable content from Next.js React Server Components (RSC) flight payloads. + * RSC pages embed content as JSON in tags. + */ + +export interface RSCExtractResult { + title: string; + content: string; +} + +export function extractRSCContent(html: string): RSCExtractResult | null { + if (!html.includes("self.__next_f.push")) { + return null; + } + + // Parse all RSC chunks into a map + const chunkMap = new Map(); + const scriptRegex = / + + diff --git a/ui/dispatch-mockup.html b/ui/dispatch-mockup.html new file mode 100644 index 000000000..30fae464c --- /dev/null +++ b/ui/dispatch-mockup.html @@ -0,0 +1,390 @@ + + + + + @bastani/workflows — dispatch confirmation, compact + + + + +
+

@bastani/workflows · dispatch confirmation, compact

+

+ After typing /workflow deep-research-codebase prompt="explore the codebase" max_partitions=4, the current chat surface emits seven lines of chrome for a single dispatch event. The run identity, workflow name, status, and inputs are each restated 2–3 times across a band, a tagged card, and two hint rows. Below: a side-by-side with the current rendering on the left and a compressed two-line replacement on the right, plus three edge cases (no inputs, long inputs, narrow terminal) showing the new layout holds shape. +

+ + +

§1 /workflow run · primary comparison

+ +
+ +
+
▼ before — 7 chat-surface rows, 4 restatements
+
+ 
✓ submitted  ·  /workflow deep-research-codebase prompt="explore the codebase" max_partitions=4
+
+✓ submitted  ·  /workflow deep-research-codebase
+ [ DISPATCHED ] deep-research-codebase ● running +
+ +
be3181c1run id + inputs prompt="explore the codebase" · max_partitions=4 + status starting…
+
+
+
▸ /workflow connect be3181c1 attach & watch
+
▸ /workflow status list in-flight runs
+
+
+ 4× the same dispatch is restated four times +
    +
  • line 1 — pi already echoes the user's slash command, with inputs
    • +
  • line 3 — our renderer adds a second ✓ submitted line that drops the inputs (a strict subset of line 1)
    • +
  • line 4 — [ DISPATCHED ] band repeats the workflow name a third time
    • +
  • lines 5–7 — card repeats run identity (tag + "run id"), inputs (already on line 1), and status (already on line 4 as ● running)
    • +
+
+
+

why it hurts · seven lines of chrome for one event · run identity, status, and inputs are each said 2–3 times · scrolls the actual conversation out of view after a couple of dispatches

+
+ + +
+
▶ after — 2 chat-surface rows, identity said once
+
+ 
✓ submitted  ·  /workflow deep-research-codebase prompt="explore the codebase" max_partitions=4
+
+ +
be3181c1deep-research-codebase · prompt="explore the codebase" · max_partitions=4 ● running
+
+
+
▸ /workflow connect be3181c1 attach & watch
+
+
+

why it works · pi's echo is the confirmation; we don't restate it · one stripe-card carries id + name + inputs + status in a single row · one hint, because /workflow status is a separate intent and already discoverable from an empty /workflow · scales to many dispatches without burying the conversation

+
+
+ + +

§2 edge cases — the after layout holds

+

+ Three stress tests on the compressed row: (a) no inputs at all, (b) more inputs than fit on a row, (c) a 60-column terminal. The card body shrinks gracefully; truncation uses the canonical … rule from ui/mockups.html §4. +

+ +
+ +
+
▶ a · workflow with no inputs
+
+ 
✓ submitted  ·  /workflow primer
+
+ +
5b91ee54primer ● running
+
+
+
▸ /workflow connect 5b91ee54 attach & watch
+
+
+

policy · drop the inputs segment entirely (no (none) placeholder) · the badge anchors the right edge so the row stays visually balanced

+
+ + +
+
▶ b · long inputs (overflow)
+
+ 
✓ submitted  ·  /workflow enterprise-research prompt="map every TypeScript file…" model="claude-opus-4" max_partitions=12 …
+
+ +
7c4a91bfenterprise-deep-research-codebase-with-multi-stage-fan… ● running + prompt="map every TypeScript file in the codebase, focu…" · model="claude-opus-4" · +3 more
+
+
+
▸ /workflow connect 7c4a91bf attach & watch
+
+
+

policy · row 1 carries identity + status only when inputs would overflow · row 2 holds the inputs, end-truncating long values inside the quotes and collapsing past 2 pairs to +N more · max 2 rows + 1 hint regardless of input count · full inputs reachable via /workflow inputs <id>

+
+
+ +
+ +
+
▶ c · narrow terminal (60 columns)
+
+ 
✓ submitted  ·  /workflow deep-research-codebase prompt="explore the c…"
+
+ +
be3181c1deep-research-codebase ● running + prompt="explore the codeb…" · max_partitions=4
+
+
+
▸ /workflow connect be3181c1 attach
+
+
+

policy · width responds to process.stdout.columns · inputs wrap to a second body row when row 1 cannot hold both name + status badge · hint verb-phrase shortens when the wide form would clip

+
+ + +
+
▶ d · plain mode (NO_COLOR / logs)
+
+ 
✓ submitted  ·  /workflow deep-research-codebase prompt="explore the codebase" max_partitions=4
+
+│ be3181c1  deep-research-codebase  ·  prompt="explore the codebase"  ·  max_partitions=4    ● running
+▸ /workflow connect be3181c1   attach & watch
+
+

policy · stripe degrades to │; same shape, no color · pi's chat echo prints once; we print one row + one hint · same 2-line footprint

+
+
+ +
+ + +

§3 what changes — line-by-line

+
    +
  • + remove · the ✓ submitted · /workflow <name> echo emitted by submittedLine() in src/tui/dispatch-confirm.ts. Pi's slash-input chat row is already that confirmation and carries the full original input. +
    • +
  • + remove · the [ DISPATCHED ] band (renderFlatBand call). A band makes sense for ambient surfaces like BACKGROUND or WORKFLOWS that frame multiple cards; here it frames exactly one card and duplicates the workflow name into the subtitle. +
    • +
  • + remove · the statusRow() body row (status starting…). The yellow ● running badge says the same thing, in a slot that's already visible. +
    • +
  • + remove · the run id caption next to the tag. The tag's surface0 chip + 8-char hex visually communicates "identifier" without a label. +
    • +
  • + remove · the second /workflow status hint row. It is a separate intent (list other in-flight runs) and is already discoverable from a bare /workflow command and from the picker's confirm panel. +
    • +
  • + keep · the stripe-card vocabulary. It is the same shape used by /workflow status and /workflow list; consistency across the three chat surfaces is load-bearing. +
    • +
  • + keep · the yellow status colour for in-flight runs. Status-Is-Truth still holds — yellow stripe means "this run is running." +
    • +
  • + keep · the connect hint. It carries the canonical /workflow connect <id> command with the freshly generated runId pre-filled — the most common next action. +
    • +
  • + keep · the inputs echo, but inline on the identity row when it fits, and falling back to a second row only on overflow (with the existing +N more rule from ui/mockups.html §4). +
    • +
+ +

§4 rendering contract — proposed

+
    +
  • Shape. Exactly one renderTaggedCard + one renderHintRows with a single entry. No band, no leading echo line.
    • +
  • Tag. 8-char prefix of the run UUID (unchanged). No trailing "run id" caption.
    • +
  • Body row 1. <workflow name> in text bold, then inputs as k=v pairs joined by · , then a right-aligned ● running badge in yellow + textMuted.
    • +
  • Body row 2 (conditional). Emitted only when row 1's inline-input span would exceed the interior budget. Carries the inputs in the same k=v · k=v · +N more form, with row 1 reduced to identity + status badge.
    • +
  • Hint row. One entry: { command: "/workflow connect <id>", hint: "attach & watch" }.
    • +
  • Plain mode. Stripe → │; badge → ● running; same shape. Tag and bold degrade to plain text.
    • +
  • Tests. The existing dispatch-confirm snapshot tests in test/unit/dispatch-confirm.test.ts need to update to the new shape; behaviour assertions (run id present, workflow name present, connect hint present) carry over unchanged.
    • +
+
+ + diff --git a/ui/hil1.png b/ui/hil1.png new file mode 100644 index 000000000..86be81c7c Binary files /dev/null and b/ui/hil1.png differ diff --git a/ui/hil2.png b/ui/hil2.png new file mode 100644 index 000000000..5866dea21 Binary files /dev/null and b/ui/hil2.png differ diff --git a/ui/mockups.html b/ui/mockups.html new file mode 100644 index 000000000..4b2e82969 --- /dev/null +++ b/ui/mockups.html @@ -0,0 +1,554 @@ + + + + + @bastani/workflows — chat-surface mockups + + + + +
+

@bastani/workflows · chat-surface redesign

+

+ Three chat surfaces — the post-dispatch start toast, /workflow status, and /workflow list — rebuilt around the orchestrator panel's two load-bearing patterns: a full-width surface0 band with an outlined pill, subtitle, and right-aligned status badges; and a status-colored card with an identity tab on the top edge for each item below. The list view collapses per-stage detail into a single horizontal progress strip so the surface scales to many stages without losing scannability. +

+ +
+ base (canvas) + surface0 (band) + blue (live) + mauve (catalogue) + success + running + error + killed / idle +
+ + +

§1 after /workflow run — start confirmation

+

+ Current output stacks three restatements (a ✓ Workflow started banner with plain-text hints, then a one-row BACKGROUND band that re-shows the same run). Redesign folds the confirmation into a single full-width DISPATCHED band — same pattern as the orchestrator panel header — with inputs in their canonical form and two next-step hints below. +

+ +
+
+
▼ before — 6 lines, three restatements
+
+ 
✓ submitted  ·  /workflow deep-research-codebase prompt="map the codebase" max_partitions=4
+
+✓ Workflow "deep-research-codebase" started   runId 0391c9c1
+  attach   /workflow connect 0391c9c1
+  monitor  /workflow status
+ [ BACKGROUND ] 1 run ● 1 + 
   ●  0391c9  deep-research-codebase  running   single · 0/1 · 0s
+
+

issues · echoes the command we just typed · duplicates run identity across three places · plain-text hint lines drift away from the brand chrome

+
+ +
+
▶ after — one full-width band, one card
+
+ 
✓ submitted  ·  /workflow deep-research-codebase
+ [ DISPATCHED ] deep-research-codebase ● running +
+ +
0391c9c1run id + inputs prompt="map the codebase" · max_partitions=4 + mode chain · starting…
+
+
+
▸ /workflow connect 0391c9c1 attach & watch
+
▸ /workflow status list in-flight runs
+
+
+

why · one event = one band · run identity rendered exactly once (the runId tab on the card) · hint grammar matches the orchestrator's bottom toolbar

+
+
+ + +

§2 /workflow status — list of runs

+

+ Current layout uses indented stage rows under each run header. Five runs × six stages = thirty stage rows mixed in with five run rows; the eye cannot tell where one run ends and the next begins. Redesign replaces the indented stage list with a per-run card (status-colored stripe, runId tab, two body rows) and a horizontal progress strip; full per-stage detail moves into /workflow status <id>. +

+ +
+
+
▼ before — 2 runs, 3 stages, already busy
+
+ 
/workflow status
+ [ BACKGROUND ] 2 runs ⊘ 2 + 
+   ⊘  939629  deep-research-codebase  killed    single · 8s ago
+     ✗ scout         failed              6s
+   ⊘  0391c9  deep-research-codebase  killed    chain · 2/2 · 4m24s ago
+     ✓ scout         completed           22s
+     ✗ partition     failed              0s
+
+   ▸ workflow status id=939629  for detail
+
+

issues · run header and stages share the same indent + glyph alphabet → no visual grouping · scales linearly with stage count · five 8-stage chains = 40+ rows

+
+ +
+
▶ after — one card per run, status-coloured stripe
+
+ 
/workflow status
+ [ BACKGROUND ] 2 runs ⊘ 2 +
+ +
939629deep-research-codebase ⊘ killed + single [✗] scout · 6s · 8s ago
+
+
+ +
0391c9deep-research-codebase ⊘ killed + chain [✓][✗] failed at partition · 4m24s ago
+
+
+
▸ /workflow status 939629 drill into a run
+
+
+

why · each card is two rows + a colored stripe — identity, then progress strip + meta · runId sits in a surface0 tab on the stripe, mirroring the orchestrator's stage-tab pattern · status color is carried by frame + badge, not text shading

+
+
+ +

§2b stress test — many stages, mixed states

+

+ Where the current layout falls apart. Same surface, four runs, two of them with 8-stage chains. +

+ +
+
+
▼ before — 4 runs × up to 8 stages = wall
+
+ 
/workflow status
+ [ BACKGROUND ] 4 runs ✓ 1 ● 1 ⊘ 2 + 
+   ●  a18f30  ship-feature          running   chain · 3/8 · 1m42s
+     ✓ planner       completed           2.1s
+     ✓ build         completed           12.4s
+     ✓ test          completed           31.2s
+     ● review-a      running   codex · 18s   20.4s
+     ● review-b      running   claude · 14s  17.9s
+     ○ merge         pending             —
+     ○ deploy        pending             —
+     ○ smoke         pending             —
+   ✓  5b91ee  open-claude-design    completed single · 18s ago
+     ✓ primer        completed           11.3s
+   ⊘  939629  deep-research-codeb…  killed    single · 8s ago
+     ✗ scout         failed              6s
+   ⊘  0391c9  deep-research-codeb…  killed    chain · 2/2 · 4m24s
+     ✓ scout         completed           22s
+     ✗ partition     failed              0s
+
+

issues · 17 stage rows compete with 4 run rows · runs no longer stand out · more than one viewport tall

+
+ +
+
▶ after — 4 cards, fits one viewport
+
+ 
/workflow status
+ [ BACKGROUND ] 4 runs ✓ 1 ● 1 ⊘ 2 +
+ +
a18f30ship-feature ● running + chain [✓][✓][✓][●][●][○][○][○] 3/8 · review-a, review-b · 1m42s
+
+
+ +
5b91eeopen-claude-design ✓ completed + single [✓] primer · 11.3s · 18s ago
+
+
+ +
939629deep-research-codebase ⊘ killed + single [✗] scout · 6s · 8s ago
+
+
+ +
0391c9deep-research-codebase ⊘ killed + chain [✓][✗] failed at partition · 4m24s ago
+
+
+
▸ /workflow status a18f30 drill into a run
+
▸ /workflow status --all include ended runs older than 1h
+
+
+

why · two lines / card regardless of stage count · the colored stripe gives an instant aggregate read for the surface · the strip uses ASCII bracket cells so the plain-text fallback ([✓][✗][○]) carries the same shape, just without color

+
+
+ + +

§3 /workflow list — registered workflows

+

+ A single comma-joined sentence buries the only thing that matters: what does each workflow do, and what does it expect? Redesign reuses the band-card frame, lists each workflow as a card with name tab, one-line description, and an input signature row. The mauve accent separates catalogue chrome from the blue live-run chrome in §1 / §2. +

+ +
+
+
▼ before — comma-joined sentence
+
+ 
/workflow list
+
+Registered workflows: deep-research-codebase, open-claude-design, ralph
+
+

issues · no description · no input hints · no entry path to inputs or run

+
+ +
+
▶ after — branded list, callable
+
+ 
/workflow list
+ [ WORKFLOWS ] 3 registered +
+ +
deep-research-codebase + Partitioned, parallel research across a codebase. + inputs prompt · max_partitions?
+
+
+ +
open-claude-design + Open Claude Code primed with the impeccable design skill. + inputs target
+
+
+ +
ralph + Ralph-the-rabbit improvement loop until an exit condition trips. + inputs prompt · iterations?
+
+
+
▸ /workflow <name> … run a workflow
+
▸ /workflow inputs <name> inspect input schema
+
+
+

why · description and signature on the surface, not buried behind a second command · mauve stripe = catalogue (vs blue live runs) so the surfaces stay distinct when stacked in chat history

+
+
+ + +

§4 edge cases — truncation

+

+ Inputs, descriptions, and identifiers can be arbitrarily long; the card must hold its shape. All overflow uses the single-character ellipsis … (never ...) and never wraps onto a second line of the same field. Truncation policy varies by field so the surviving portion is the most useful slice. +

+ +
+
+
▶ §1 dispatched — long & many inputs
+
+ 
✓ submitted  ·  /workflow ship-feature
+ [ DISPATCHED ] enterprise-deep-research-codebase-with-multi-stage-fan… ● running +
+ +
7c4a91bfrun id + inputs prompt="map every TypeScript file in the codebase, focu…" + model="claude-opus-4" · max_partitions=12 · +3 more + mode chain · starting…
+
+
+
▸ /workflow inputs 7c4a91bf show full input values
+
▸ /workflow connect 7c4a91bf attach & watch
+
+
+

policy · long workflow name → end-ellipsis in subtitle · long single value → truncate inside the quotes, keep closing " · >3 inputs → first 3 inline, rest collapsed to +N more · drill via /workflow inputs <id>

+
+ +
+
▶ §2 status — long workflow + stage names
+
+ 
/workflow status
+ [ BACKGROUND ] 2 runs ● 1 ⊘ 1 +
+ +
7c4a91enterprise-deep-research-codebase-with-mu… ● running + chain [✓][✓][●][○][○][○][○][○][○][○][○][○] 3/12 · cross-repo-analy… · 1m42s
+
+
+ +
939629deep-research-codebase ⊘ killed + single [✗] scout-and-partition… · 6s · 8s ago
+
+
+
▸ /workflow status 7c4a91 full stage names + durations
+
+
+

policy · workflow name and stage label both end-truncate · progress strip never truncates — the cells are the truth · runId tag is fixed at 6 chars, so the tag width is constant regardless of input

+
+
+ +
+
+
▶ §3 list — long descriptions
+
+ 
/workflow list
+ [ WORKFLOWS ] 2 registered +
+ +
enterprise-deep-research-codebase-with-multi-stage… + Partitioned, parallel research across a large codebase with specialist sub-agents fanning… + inputs prompt · max_partitions? · model? · +2 more
+
+
+ +
ralph + Ralph-the-rabbit improvement loop until an exit condition trips. + inputs prompt · iterations?
+
+
+
▸ /workflow inputs <name> full description + input schema
+
+
+

policy · description truncates at end-of-line · workflow name in tag also end-truncates · >4 inputs → first 3 inline, rest as +N more · full text always reachable via /workflow inputs <name>

+
+ +
+
▶ narrow terminal — 60 columns
+
+ 
/workflow status
+ [ BACKGROUND ] 2 runs ● 1 ⊘ 1 +
+ +
7c4a91enterprise-deep-resear… ● running + chain [✓][✓][●][○][○]… 3/12 · 1m42s
+
+
+ +
939629deep-research-codebase ⊘ killed + single [✗] scout · 6s ago
+
+
+
▸ /workflow status 7c4a91 drill in
+
+
+

policy · width responds to process.stdout.columns · progress strip itself truncates with a trailing … when the run has more stages than fit · meta label drops second-tier detail first (loses stage names, keeps fraction + duration)

+
+
+ +
+ + +

§5 notes & implementation

+
    +
  • + Full-width bands — the chat surface in pi paints each line with customMessageBg; the redesigned band emits a surface0 fill that spans the line and a left-anchored outline pill. Implementation: a single renderBand({ label, subtitle, badges, totalWidth }) helper replaces the current 64-cell renderBandHeader. Total width is read from process.stdout.columns (or a passed width on overlay surfaces). +
    • +
  • + Status cards — each card has a 1-cell coloured stripe (status family) on the left, a surface0 tag on the top edge carrying the runId / workflow name, and a body of one or two indented rows. No top/bottom frame chars, so cards stack densely without becoming a wall of ╭─╮. +
    • +
  • + Progress strip — one bracketed cell per stage [✓] [●] [○], coloured by state. At very wide fan-outs the strip wraps; the trailing dim italic always names the meaningful stage (current activity, failure point), so the wrap stays decorative. +
    • +
  • + Truncation — uses truncateToWidth(str, n, "…") from @earendil-works/pi-tui (already a dep). Per-field budgets: workflow name 40 cells; description the rest of the row after tag + 2 + status badge; single input value half the row, with the closing quote preserved; input list 3 inline pairs then +N more; stage activity label 24 cells. Budgets shrink proportionally when process.stdout.columns < 80. +
    • +
  • + Hint rows — one grammar everywhere: ▸  /slash command  verb-phrase hint. Echoes the orchestrator's bottom toolbar (↵ attach · / stages · q quit). +
    • +
  • + Plain-text fallback — bands collapse to ▎ BACKGROUND 2 runs ⊘ 2 on uncoloured terminals; the card stripe degrades to │; the strip uses bare ASCII [✓][✗][○]. Same shape, no color. +
    • +
  • + Surfaces — bands and cards are emitted from renderResult() in src/extension/render-result.ts, which already returns a multi-line string. No new surface contract is required to land this — only the renderer functions in src/tui/status-list.ts and src/tui/run-detail.ts change. +
    • +
+
+ + diff --git a/ui/stage-chat-mockup.html b/ui/stage-chat-mockup.html new file mode 100644 index 000000000..600e3edc4 --- /dev/null +++ b/ui/stage-chat-mockup.html @@ -0,0 +1,1079 @@ + + + + + @bastani/workflows — stage chat (Pi-box UI) + + + + +
+

@bastani/workflows · stage chat (Pi-box UI)

+

+ Pressing Enter on a graph node attaches the orchestrator overlay to that stage's live pi AgentSession. Same popup chrome, swapped interior. The chat surface here is intentionally minimal — a steering surface, not a second Pi shell. It reuses Pi's box primitives verbatim: filled user bars, unboxed assistant Markdown, state-coloured tool bars, italic thinking, bordered loader, rounded editor, two-line dim footer, key-hint strip. Every SDK operation the workflow performs on the stage session (setModel, setThinkingLevel, compact, …) surfaces here so the user can see what the workflow is doing without leaving the overlay. +

+ + +

§1 idle · just attached, editor empty

+
▼ stage chat — pending first message
+ +
+ pi-workflows/ review-fix-loop ›review-a +
+ Ctrl+D←back to graph + +
+ ▎ + STAGE + review-fix-loop/review-a +
+ just attached · session 9e2a47c1 + idle +
+
+ +
+
+
▎
+

Attached to review-fix-loop / review-a

+

This stage is idle. Press ↵ to send the first prompt — the SDK session will be created on submit. The workflow body keeps running in the background; closing this overlay does not kill the run.

+
+
modelclaude-sonnet-4-5
+
thinkingmedium
+
parentssetup
+
depth1
+
mcp scopeall
+
toolsread · grep · edit · bash · workflow
+
+
+
+ +
+ ❯ + type a message… + + medium thinking +
+ +
+
+ ~/atomic (release/v0.1.1) • review-fix-loop-run-9e2a + (anthropic) claude-sonnet-4-5 • medium +
+
+ ↑0 ↓0 R0 W0 $0.000 0.0%/200k (auto) + stage session not yet realized +
+
+ +
+ ↵send + · + Ctrl+Ffollow-up + · + Ctrl+Ppause + · + Ctrl+Dback + · + Escclose +
+
+
+

why this shape · the popup interior keeps Pi's vocabulary exactly — same editor band, same footer composition, same hint strip. The welcome panel is the only addition (Pi never starts blank because its editor is permanent; we need an explicit landing for first attach). All the SDK state Pi shows in its footer is shown here, sourced from the stage's own AgentSession — not the host chat.

+ +
+ + +

§2 streaming · assistant turn live, tools mid-flight

+
▼ stage chat — turn in progress
+ +
+ pi-workflows/ review-fix-loop ›review-a +
+ Ctrl+D←back to graph + +
+ ▎ + STAGE + review-fix-loop/review-a +
+ running 38s · session 9e2a47c1 + running +
+
+ +
+
+ +
Review the auth module for security issues, especially around setRuntimeApiKey lifecycle. Be thorough.
+ +
+ +
Looking at the auth surface. I'll start with core/auth-storage.ts and trace runtime overrides, then check dispose paths.
+ +
+ +
I'll walk through the auth files and check the usual suspects: storage lifecycle, dispose hygiene, env-var precedence, runtime-override clearing. Starting with the file you mentioned.
+ +
+ +
+
+ read + path=packages/coding-agent/src/core/auth-storage.ts + ✓ 412 lines +
+
+ +
+
+ grep + pattern="setRuntimeApiKey|clearRuntime" + ✓ 6 matches in 3 files +
+
142:setRuntimeApiKey(provider: string, key: string): void { +156:clearRuntimeApiKey(provider: string): void { +312:dispose(): void { /* no clearRuntime — see issue */
+
+ +
+
+ read + path=packages/coding-agent/src/core/agent-session.ts + READING +
+
+ +
+ +
+ ## Findings +

+ Runtime API key overrides set via setRuntimeApiKey are not cleared on AgentSession.dispose. If the host process reuses the same AuthStorage instance across sessions (which the SDK encourages), a previous session's key continues to win against env vars in the next session. +

+ The resolution chain is runtime > agent.db > env > custom-provider; that's correct but means a stale runtime override is the highest-priority slot and silently masks the configured key. +

+ Inspecting the session dispose path next to confirm the leak +
+ +
+
+ +
+
+
+ + Working · 38s · 12.3k in, 1.8k out, 4.3% ctx + Esc interrupt +
+
+
+ +
+ ❯ + type to steer the current turn… (queues with ↵) + + medium thinking +
+ +
+
+ ~/atomic (release/v0.1.1) • review-fix-loop-run-9e2a + (anthropic) claude-sonnet-4-5 • medium +
+
+ ↑12.3k ↓1.8k R1.1k W342 $0.034 4.3%/200k (auto) + streaming · 38s +
+
+ +
+ ↵steer + · + Ctrl+Ffollow-up + · + Ctrl+Ppause + · + Ctrl+Dback + · + Escinterrupt +
+
+
+

contract · Enter steers the active turn (interrupt mid-stream — handle.steer(text)), Ctrl+F follows up after the turn finishes (handle.followUp(text)). Ctrl+P pauses cleanly without aborting; the original prompt awaiter is held until resume. The loader mirrors Pi's BorderedLoader exactly: top rule + spinner + message + cancel hint + bottom rule, no surrounding box. The footer always reflects the stage's own session state, not the host chat's.

+ +
+ + +

§3 paused · controlled pause via Ctrl+P

+
▼ stage chat — paused, awaiting resume message
+ +
+ pi-workflows/ review-fix-loop ›review-a +
+ Ctrl+D←back to graph + +
+ ▎ + STAGE + review-fix-loop/review-a +
+ paused 12s ago · session 9e2a47c1 + paused +
+
+ +
+
+ ❚❚ + PAUSED + stopped between turns · type a message to resume, or Ctrl+P to release without input +
+ +
+ +
Review the auth module for security issues, especially around setRuntimeApiKey lifecycle. Be thorough.
+ +
+ +
I started with core/auth-storage.ts — found one leak around setRuntimeApiKey not being cleared on dispose. Working through the session dispose path next.
+ +
+ +
+
+ read + path=packages/coding-agent/src/core/auth-storage.ts + ✓ 412 lines +
+
+ +
+
+ grep + pattern="setRuntimeApiKey|clearRuntime" + ✓ 6 matches in 3 files +
+
+ +
+
+ +
+ ❯ + also check pi-tui auth boundary while you're paused + + medium thinking +
+ +
+
+ ~/atomic (release/v0.1.1) • review-fix-loop-run-9e2a + (anthropic) claude-sonnet-4-5 • medium +
+
+ ↑12.3k ↓1.8k R1.1k W342 $0.034 4.3%/200k (auto) + paused · ready to resume +
+
+ +
+ ↵resume with message + · + Ctrl+Presume empty + · + Ctrl+Dback + · + Escclose +
+
+
+

contract · the paused banner is full-width and color-coded yellow to match Mocha's warning vocabulary. Enter resumes with the typed text fed back into the SDK session as the next user message (handle.resume(text)); Ctrl+P resumes empty (handle.resume()). The previous transcript stays visible so the user can read what the stage already produced. Footer right-rail flips to paused · ready to resume so the SDK state is unambiguous.

+ +
+ + +

§4 observability · every SDK op the workflow performs surfaces inline

+
▼ stage chat — workflow steered the stage from inside the SDK
+ +
+ pi-workflows/ review-fix-loop ›review-a +
+ Ctrl+D←back to graph + +
+ ▎ + STAGE + review-fix-loop/review-a +
+ running 2m 14s + running +
+
+ +
+
+ +
Run a first pass quickly, then deepen on anything suspicious.
+ +
+ +
~thinking→lowwas medium · workflow body called stage.setThinkingLevel("low")
+
~model→claude-haiku-4was claude-sonnet-4-5 · stage.setModel(haiku)
+ +
+ +
Scanning quickly. Will surface the highest-priority findings first.
+ +
+ +
+
+ find + glob=packages/coding-agent/src/core/**/*.ts + ✓ 47 files +
+
+ +
+ +
~compaction→summarized 14 turns14.2k → 1.4k tokens · stage.compact() · auto-triggered at 78%
+ +
+ +
~model→claude-sonnet-4-5workflow restored deeper model for findings pass
+
~thinking→highstage.setThinkingLevel("high")
+ +
+ +
Re-examining with deeper analysis. The earlier scan flagged three sites worth detailing.
+ +
+ +
Three findings, ranked by severity
+ +
+
+ +
+
+
+ + Working · 2m 14s · 1.4k in (post-compact), 312 out + Esc interrupt +
+
+
+ +
+ ❯ + type to steer the current turn… + + high thinking +
+ +
+
+ ~/atomic (release/v0.1.1) • review-fix-loop-run-9e2a + (anthropic) claude-sonnet-4-5 • high +
+
+ ↑14.6k ↓2.1k R1.4k W612 $0.062 7.8%/200k (auto) + streaming · post-compact +
+
+ +
+ ↵steer + · + Ctrl+Ffollow-up + · + Ctrl+Ppause + · + Ctrl+Dback + · + Escinterrupt +
+
+
+

this is the load-bearing decision · every state-changing SDK call the workflow body performs against this stage (setModel, setThinkingLevel, compact, cycleModel, navigateTree, model auto-fallback, MCP scope changes) emits a one-line dim notice in the transcript and updates the footer immediately. The user attaches, glances, and instantly sees not just what the stage said but what the workflow steered it to do. This is what makes the overlay a steering surface rather than a log viewer.

+ +
+ + +

§5 settled · completed stage, read-only post-mortem

+
▼ stage chat — stage completed, no editor
+ +
+ pi-workflows/ review-fix-loop ›review-a +
+ Ctrl+D←back to graph + +
+ ▎ + STAGE + review-fix-loop/review-a +
+ completed 3m 42s · session 9e2a47c1 + completed +
+
+ +
+
+ ✓ + COMPLETED + stage settled · transcript is read-only · session file persisted at .pi/sessions/…/9e2a47c1.jsonl +
+ +
+ +
Review the auth module for security issues.
+ +
+ +
+ ## Findings (final) +

+ Runtime API key not cleared on dispose — setRuntimeApiKey overrides survive across sessions when AuthStorage is reused. Suggest clearing in AgentSession.dispose. +

+ Env-var precedence is masked by stale runtime overrides — same root cause; documenting the resolution order in the SDK README would help embedders avoid the foot-gun. +

+ No automated cleanup on process exit — minor; runtime overrides are in-memory so this is a same-process leak only. +
+ +
+ +
+
+ edit + path=packages/coding-agent/src/core/agent-session.ts + ✓ 1 edit applied +
+
312 dispose(): void { +313+ this.auth.clearRuntimeApiKey(this.model?.provider ?? ""); +314 this.eventBus.removeAllListeners();
+
+ +
+ +
Wrote one defensive clear-on-dispose. The other two findings are doc-only and surfaced for the next stage in the loop.
+ +
+
+ +
+ ❯ + read-only · stage has no live handle + medium thinking +
+ +
+
+ ~/atomic (release/v0.1.1) • review-fix-loop-run-9e2a + (anthropic) claude-sonnet-4-5 • medium +
+
+ ↑18.4k ↓3.6k R1.4k W612 $0.087 9.2%/200k + completed · settled at 3m 42s +
+
+ +
+ Ctrl+Dback to graph + · + Escclose + · + /workflow inspect 9e2a opens this session in a fresh Pi shell when you want a full editor +
+
+
+

contract · settled stages drop the live editor (no StageControlHandle means nothing to send to). We keep the transcript fully scrollable and the footer numbers final-frozen. The hints strip drops the send-side keys and replaces them with a single forward-pointer: power users who want a full Pi editor on the persisted session can launch one with /workflow inspect — an opt-in separate command, not the default node-Enter path.

+ +
+ + +

§6 what we are not doing

+
    +
  • no chat bubbles · Pi is a full-width continuous column. Bubbles imply two participants on the same plane; this is an SDK session in a single column.
    • +
  • no left-stripe accents on messages · user identity is the filled background; we don't need a colored bar on the side too.
    • +
  • no nested cards · the omp-window is the only frame. Boxes inside are background fills, not card-on-card.
    • +
  • no gradient text, no glass blur, no decorative icons · Pi's chat is sober terminal-native. We match.
    • +
  • no second Pi shell · we deliberately do not instantiate InteractiveMode here. The host Pi owns the only terminal loop. This overlay reuses Pi's components (UserMessage, AssistantMessage, ToolExecution, BorderedLoader, CustomEditor, Footer) but not its mode controller.
    • +
  • no nested session manager · the stage's AgentSession is created by the workflow runtime via createAgentSession; the chat just attaches to its handle. There is exactly one session per stage, persisted to one file.
    • +
  • no model-picker / login / tree / fork commands inside the overlay · those are full-Pi flows. From the overlay you can see the model/thinking state (footer) and see when the workflow changes them (inline notices). To change them yourself, you go back to the host Pi.
    • +
+ +

§7 what gets surfaced when the workflow steers the stage

+
    +
  • stage.setModel(model) · inline notice ~ model → <new-id> + footer model field updates immediately
    • +
  • stage.setThinkingLevel(level) · inline notice ~ thinking → <level> + footer thinking field + editor border tint updates
    • +
  • stage.cycleModel() / stage.cycleThinkingLevel() · same as above
    • +
  • stage.compact() · inline notice ~ compaction → summarized N turns (Xk → Yk tokens) + footer token deltas reflect the new baseline
    • +
  • stage.navigateTree(targetId) · inline notice ~ tree → navigated to <label> + transcript rebuilds from the new branch
    • +
  • stage.abort() / kill · transcript appends ~ aborted (reason: …) and footer status flips to aborted
    • +
  • handle.pause() / handle.resume() · paused banner + status pill swap; resume keeps transcript
    • +
  • model auto-fallback (provider returned 429, host swapped model) · same shape as explicit setModel notice but tagged (fallback)
    • +
  • MCP scope changes (mcp.scope.set) · inline notice ~ mcp → allow=… deny=… when scope is non-trivial
    • +
+ +

summary · the attach UI is a minimal steering surface that mirrors Pi's component vocabulary and the stage's SDK session state. Boxed bars for user/tool, unboxed Markdown for assistant, italic dim for thinking, bordered loader, rounded editor, two-line dim footer, key-hint strip. Everything the workflow author can do to a stage via the SDK shows up here without leaving the overlay.

+
+ + diff --git a/verdaccio.yaml b/verdaccio.yaml deleted file mode 100644 index f968a2abe..000000000 --- a/verdaccio.yaml +++ /dev/null @@ -1,46 +0,0 @@ -# Verdaccio config for the publish-validation job. -# Lets the validate workflow exercise the real publish + install code path -# against a throwaway local registry — catches publish-layout regressions -# (synthesised wrapper, optionalDependencies resolution) before -# we touch the public registry. -# -# Anonymous publish is intentional: the validate job runs in a hermetic CI -# step and the registry is destroyed when the runner exits. - -# Paths are config-relative so the same file works on every CI runner OS -# (Windows /tmp doesn't map to a node-readable path). -storage: ./.verdaccio-storage -listen: 0.0.0.0:4873 - -# Per-platform binary tarballs are ~50MB compiled; bump from the 10MB default. -max_body_size: 200mb - -auth: - htpasswd: - file: ./.verdaccio-htpasswd - max_users: -1 - -uplinks: - npmjs: - url: https://registry.npmjs.org/ - -packages: - # `@bastani/*` MUST NOT proxy to npmjs — otherwise verdaccio could serve a - # previously-released version when our local publish is missing a tarball, - # silently masking regressions. - "@bastani/*": - access: $all - publish: $anonymous - unpublish: $anonymous - - # Everything else (e.g. transitive deps pulled by `bun install -g`) proxies - # through to npmjs as a read-only mirror. - "**": - access: $all - publish: $authenticated - proxy: npmjs - -log: - type: stdout - format: pretty - level: warn