Skip to content

Skills and agents layer, trial day 1: upstream/skills.sh delta, 2 trial skills, agent frontmatter test - #328

Merged
seathatflowsinourveins merged 4 commits into
mainfrom
claude/skills-agents-layer-20260926
Sep 26, 2026
Merged

Skills and agents layer, trial day 1: upstream/skills.sh delta, 2 trial skills, agent frontmatter test#328
seathatflowsinourveins merged 4 commits into
mainfrom
claude/skills-agents-layer-20260926

Conversation

@seathatflowsinourveins

@seathatflowsinourveins seathatflowsinourveins commented Sep 26, 2026

Copy link
Copy Markdown
Collaborator

The user asked on 2026-09-26 to strengthen the Claude Code native workflow layer (skills and agents) from SOTA upstream, using skills.sh repositories, installed seamlessly and backed by evidence. This PR is day 1 of the 30-day skills trial (2026-09-25 to 10-25): a delta against upstream and skills.sh, with changes only where the trial's budget rules allow them.

Results (details in evidence/artifacts/skills-agents-layer-20260926/README.md and delta.json)

Question Result Evidence class
Has upstream moved past any of the 26 pins? No. Every pinned folder has the same git tree SHA at its source's HEAD. Source review (upstream at read time)
Did any skills.sh audit label change? No, all 26 labels are as on 09-25. The audit API rates agentic-actions-auditor Socket critical while the page shows Warn. Independent observation
Invoke rate, day 1 Top skills by uses: verification-before-completion 32, search-first 29, supply-chain-risk-auditor 29, iterative-retrieval 25, fp-check 10. Native /skill-doctor (0 turns, $0)
Installed folders vs pins 25 of 26 match. supply-chain-risk-auditor's own uv run wrote runtime files into its folder, and the new status check reports that. Local check
Agent frontmatter vs 2.1.283 docs No defect. All 7 agents use documented fields, declare effort: max with an explicit model, and are byte-identical to ~/.claude/agents. Source review and local check
Does claude plugin validate --strict catch bad agent frontmatter? No: it passed unparseable YAML, misspelled fields and undocumented values. A repository test now does that check. Native execution with negative controls
New skills Two trial additions and one deferred after review; 23 candidates reviewed individually. Source review, repository measurement, native runs on synthetic fixtures

Changes

  • adoption/skills/manifest.json:
    • Two trial additions: variant-analysis (trailofbits@0cc1c73) and writing-for-agents (mattpocock@c55ee46, limited to the AGENTS.md/CLAUDE.md gap).
    • Budget: 7,409 of 8,000 listed characters.
    • 14 new excluded entries, including skill-security. It was deferred because both pinned scanners follow symlinks outside the skill and print what they read, so neither local scanner is adopted.
    • The settings template mirrors skillOverrides.
  • scripts/skills_status.py: an informational per-skill git tree check (ok, runtime_artifacts or drift), with tests that use git write-tree as the oracle.
  • tests/test_install_claude_profile.py: shipped agents must parse as YAML and use only documented fields, types and values, since the native validator does not check them. A mutation of one agent fails only this test.
  • The trial decision record gets a 2026-09-26 addendum, and adoption/update.md and the native-skill-practice README are updated.

Tests: tests.test_install_claude_profile, test_skills_manifest, test_skills_status, test_install_skills and test_skill_usage ran 189, all OK. validate.py passed after registration.

Reviews: Claude Opus (6 findings, 2 medium) and GPT-6-Astra (2 medium). All real findings were fixed in one repair round, with no rejections.

SOTA sources

Open

  • The gap "no local content scan before a pin" stays open (P9). Upstream issues for the two scanners' symlink reads were not filed; filing them is an external action left for a decision.
  • No deterministic scanner flags memory poisoning written only as instructions.
  • After merge, the coordinator installs the two trial skills on this host through adoption/update.md's skills step.

🤖 Generated with Claude Code

Scout and others added 4 commits September 26, 2026 02:43
Scout @claude
Skills status: recompute each installed folder's git tree; test agent…
a73bc55 
… frontmatter against the docs

scripts/skills_status.py now hashes every canonical ~/.agents/skills/<name>
folder the way git hashes a tree and reports, per skill and informationally,
ok, runtime_artifacts (matches once __pycache__, .venv, node_modules and tool
caches are left out) or drift. The lock's skillFolderHash is install-time
metadata and never recomputed: on this host supply-chain-risk-auditor's own
`uv run` created scripts/.venv and rewrote the pinned scripts/uv.lock while
the status check passed. Tests use `git write-tree` as the oracle (exec bit,
symlink, nested and empty directories, git's directory sort order).

tests/test_install_claude_profile.py adds ShippedAgentFrontmatterTests: each
shipped agent uses only the fields and values of the sub-agents docs'
Frontmatter reference (read 2026-09-26, Claude Code 2.1.283) and names an
explicit model. Claude Code ignores an unknown field silently, and
`claude plugin validate --strict` passed an agents directory with an unknown
field and unparseable YAML, so this test is the check that sees a typo.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Scout @claude
Skills layer delta 2026-09-26: three trial pins, corrected counts, re…
226b676 
…viewed candidates, evidence

Day 1 of the skills trial. All 26 pinned folders still hash to their pins at
upstream HEAD and their skills.sh audit labels are unchanged, so no pin moves.

adoption/skills/manifest.json, under the trial record's selection rule and
8,000-character listing cap:
- trial: variant-analysis (trailofbits@0cc1c73, on), writing-for-agents
  (mattpocock@c55ee46, on, Codex on), skill-security (superagent-ai@0da315b,
  name-only), each with its demonstrated gap, license and audit reading;
  skill-security beat getsentry skill-scanner 5 of 5 to 3 of 5 on synthetic
  malicious fixtures
- description_chars follows one stated method; semgrep, codeql and
  sarif-parsing corrected; on-listed 7,409 of 8,000, Codex-enabled 3,054
- 13 excluded groups cover every other candidate reviewed 2026-09-26
- cli.notes records that skills.sh /api/v1 needs a Vercel OIDC token
The settings template mirrors skillOverrides. The trial record gains a dated
addendum (additions, counts, the install-time lock-hash limitation).

evidence/artifacts/skills-agents-layer-20260926/ holds the sanitized readings:
upstream heads, audit pages and API, /skill-doctor and Codex invoke counts
(0 turns, $0), agent frontmatter and installed-copy sha256, native validator
negative controls, the scanner comparison fixtures and results, and proposals
P1-P8 for the verdict wave. Nothing was installed on the host.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Scout @claude
Skills layer review repair: defer skill-security, correct the scanner…
adb4665 
… claims, narrow writing-for-agents, parse agent YAML

Four review findings, each verified against source and native execution:

- skill-security follows symlinks out of its target (analyzers.py discover()
  and read_text() at 0da315b). A new m7 fixture, run under an open() audit
  trace, shows it reads a file outside the skill, lists it as the skill's
  own and prints a line of it. getsentry skill-scanner reports the symlink
  as critical but reads through it too. skill-security moves from the trial
  pins to excluded, with an overturn condition, and the settings template
  drops it. Proposal P9 keeps the local-scan gap open.
- The scanner win was misstated. m2 was caught only through its
  settings.json permission-grant clause; a new m6 fixture without that
  clause gets no finding from either scanner (skill-security scores it 0,
  LIKELY SAFE). The metric is now "at least one high or critical finding"
  (6 of 7 against 4 of 7), recorded beside skill-security's own score band
  and default verdict for each fixture.
- writing-for-agents' gap is narrowed to AGENTS.md and CLAUDE.md. A rule 4/6
  disposition records the overlap on skill editing with
  anthropic-skills:skill-creator and Codex .system/skill-creator. Proposal
  P10 records the trial confound. All three upstream files are now read.
- ShippedAgentFrontmatterTests now parses the frontmatter with
  yaml.safe_load (skipped only where PyYAML is absent), requires a mapping
  with documented field types and keys that match the text reader's, and
  rejects the unclosed-YAML control that the text reader alone passes.

The budget is unchanged (7,409 on-listed characters, 3,054 Codex-enabled).
scanner_compare.py reproduces scanner_comparison from delta.json.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Scout @claude
Register the skills and agents layer files (hot-file commit)
9ed75d3 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 26, 2026

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@seathatflowsinourveins
seathatflowsinourveins enabled auto-merge (squash) September 26, 2026 06:45
@seathatflowsinourveins
seathatflowsinourveins merged commit b5c6313 into main Sep 26, 2026
24 checks passed
@seathatflowsinourveins
seathatflowsinourveins deleted the claude/skills-agents-layer-20260926 branch September 26, 2026 07:00
@seathatflowsinourveins seathatflowsinourveins added the lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers label Sep 26, 2026
@seathatflowsinourveins seathatflowsinourveins mentioned this pull request Sep 26, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@seathatflowsinourveins