feat: Enterprise Behavioral Testing certification check

[csoceanu]

Summary

Implements the ENTERPRISE_BEHAVIORAL_TESTING certification check (Certified level), which validates that skills behave correctly under stress, edge cases, and unusual conditions — not just the happy path.

• Consistency check: flags skills with high trial variance (std_reward > 0.3), using data already captured in every evaluation run
• Edge case testing: new edge_cases/ submission directory with per-edge-case ASE evaluation, EdgeCaseGate with new BEHAVIORAL gate type
• Stability check: get_historical_variance() and _check_stability() implemented and tested, staged for wiring when upstream data producer exists
• Failure mode analysis: failure_handling and uncertainty_acknowledgment criteria added to LLMJudgeVerifier, _check_failure_mode() ready
• Certification wiring: enabled in CERTIFIED_CHECKS and skill profile, weighted composite with minimum-2 sub-check coverage requirement
• Pipeline wiring: separate ASE run per edge case, aggregation, merge into report.json

Breaking change: Certified level for skill profile

This PR intentionally raises the bar for Certified level in the skill profile. Skills now require an edge_cases/ directory with behavioral edge case tests to reach Certified. Without edge cases, the behavioral testing check fails due to the minimum-2 sub-check requirement (only consistency has data).

• Foundational and Trusted levels are unaffected
• Agent and full profiles are unaffected (behavioral testing is commented out for those)
• Skills without edge_cases/ will fail the Certified check — this is the intended behavior

Design decisions

• Composite pass/fail uses all_passed only — each sub-check enforces its own threshold. The composite score is a weighted average for reporting, not a gate.
• Minimum 2 sub-checks with real data required to pass — prevents trivial certification on a single sub-check.
• Single data path for edge cases — EdgeCaseGate is the source of truth; certification consumes its result rather than re-reading report.json.
• ASE for edge cases — each edge case runs as a separate ASE invocation, avoiding expensive Harbor container builds per edge case.
• Stability and failure mode are staged — sub-check functions are implemented and tested, extraction will be added when upstream producers exist.

[GuyZivRH]

Consolidated review: PR #51

Verdict: Request changes (not “approve with one fix”).
Library design and tests are strong (745 tests green; CI green), but enabling Certified without end-to-end producers is a product bug, and there are silent Compass/scorecard naming issues plus policy/process blockers.

---

Blocking

1. Check enabled; pipeline not wired → Certified broken in prod
ENTERPRISE_BEHAVIORAL_TESTING is in CERTIFIED_CHECKS and skill/agent/full profiles, with min-2 active sub-checks. This PR does not change pipeline/**. Harbor still runs treatment/control only; analyze never gets --edge-case-results-dir. _extract_behavioral_data typically only gets std_reward → min-2 fails → Certified always fails.

Not “backward compatible” for Certified outcomes (scaffold/validate without edge_cases/ are fine; Certified is not).

Fix: keep check out of CERTIFIED_CHECKS/profiles until Harbor build/eval + analyze are wired; or ship that wiring here; or temporarily min-1 with an explicit “partial” profile flag.

2. EdgeCaseGate breaks gate_name / policy_key convention
Docstring + every other gate: gate_name = category, policy_key = implementation. This gate sets both to "edge-case" while reading policy via get_gate_policy("behavioral").

Concrete fallout:

• Compass: should_push_fact("edge-case") ignores gates.behavioral.push_fact
• WEIGHTED scorecard: weight looked up under "edge-case", not "behavioral"

Fix: gate_name="behavioral", keep policy_key=self.name.

3. Missing edge trial output silently dropped
analyze_edge_cases does if not trials: continue, so empty/failed edge dirs never count as failures → inflated pass rates. Treat missing trial output as failure before calling this Certified-complete.

4. Submitter-owned edge_cases/ + min-2 is gameable
Only consistency + edge are extractable today. Author-written trivial edges can satisfy min-2. Need required packs / stricter rules (or don’t enable Certified yet).

---

Medium / follow-up

Issue	Notes
Stability / failure_mode / get_historical_variance	Implemented & tested as library stubs, not wired in _extract_behavioral_data. Don’t oversell as staged product.
EdgeCaseGate fail-open on [] / missing	passed=True, score=1.0 — unlike quality BLOCK fail-closed; confusing vs certification min-2 fail
Consistency ignores absolute performance	Low mean + low variance still passes
Dead DEFAULT_THRESHOLDS[…]=0.6	Unused after “pass = all_passed only”
Policy key split	is_enabled("edge-case") vs get_gate_policy("behavioral") (same smell as #3)
Scaffold returns only (treatment, control)	Edge dirs are log-only for callers
Threshold drift	Gate uses gates.behavioral.threshold; cert sub-check hardcodes DEFAULT_EDGE_CASE_PASS_THRESHOLD=0.5 — can diverge later
First-match gate scan in extract	Fine with one behavioral gate; fragile if more are added
get_historical_variance untested	Cheap to add when wiring stability
LLM new criteria	Opt-in via self.criteria — good for judge backward compat

---

What looks good

• Min-2 + all_passed as the only gate; composite score reporting-only (avoids “all pass / composite fail”).
• no_data sentinel so skipped sub-checks don’t false-fail.
• Single path: EdgeCaseGate → extract → certification (no second parse of report.json for cert).
• Scaffold DRY helpers; optional edge_cases/ validation; score clamping.
• Broad unit + scaffold→analyze→certify coverage; ruff clean on touched files; CI green.

---

Merge bar

1. Strip Claude co-author trailers.
2. Either wire Harbor build/eval + analyze for edge tasks end-to-end, or keep the check out of CERTIFIED_CHECKS/profiles until that exists.
3. Fix gate_name="behavioral".
4. Treat missing edge trial output as failure.
5. Be explicit in PR text: scaffold/validate compat ≠ Certified outcome change; stability/failure_mode are stubs until extract + producers land.

Until (2)–(4) land, this should stay Request changes, not approve-with-nits.

[GuyZivRH]

PR #51 Review: Enterprise Behavioral Testing

Verdict: Request changes
PR: #51 | CI green | 6 commits | +2,026/−43

What looks good

• ASE-per-edge-case wiring (ase-eval-edge-cases + aggregate_edge_case_evals.py + merge into report.json) is the right shape
• gate_name="behavioral" / policy_key=self.name matches Compass/scorecard conventions
• Fail-closed missing report.json in BLOCK mode
• Missing grading.json counted as failure (not silently skipped)
• Composite score is reporting-only; per-sub-check thresholds gate pass/fail
• Strong unit coverage (813 lines) for sub-checks and certification integration
• Scaffold no longer pretends Harbor owns edge cases
• Clean architecture with new GateType.BEHAVIORAL following existing gate registry pattern

---

Blocking (must fix before merge)

1. False backward-compatibility claim

PR says: "Backward compatible — submissions without edge_cases/ work exactly as before."
This is incorrect for skill Certified level:

• ENTERPRISE_BEHAVIORAL_TESTING is enabled in the skill Certified profile
• Check requires min 2 sub-checks with real data
• Only two producers exist today:
  | Sub-check | Wired? |
  |-----------|--------|
  | consistency (std_reward) | Yes |
  | edge_case | Only if edge_cases/ exists and ASE ran |
  | stability | No (TODO) |
  | failure_mode | No (TODO) |
  Without edge_cases/, a submission gets n_active == 1 → "Insufficient behavioral coverage… need at least 2" → Certified always fails.
  Fix options:
• Keep the check commented out of Certified/skill until more producers exist or edge_cases/ is a required asset
• Drop min-2 for MVP (consistency alone can pass)
• Make edge_cases/ a required Certified asset and document this as an intentional breaking change

2. Edge-case ASE steps skip eval-engine=ase

The condition only allows harbor / both:

if [ "$EVAL_ENGINE" != "harbor" ] && [ "$EVAL_ENGINE" != "both" ]; then
  echo "Skipping edge case eval (eval-engine=$EVAL_ENGINE)"
  exit 0
fi

But the implementation is ASE-based. Pure ase runs never evaluate edge cases → same Certified failure due to min-2.
Fix: Run whenever edge_cases/ exists, at minimum for ase and both.

Major (should fix)

3. Stale profile comments

full and agent profiles say "Implemented but not wired into pipeline yet" while skill has it enabled. Update comments to reflect current state.

4. Gate policy key mismatch

• is_enabled(behavioral_gate.name) checks "edge-case"
• EdgeCaseGate.evaluate() calls get_gate_policy("behavioral")
  Disabling via gates.behavioral vs gates.edge-case behaves differently. Align with security/quality convention (one key for both enable check and evaluate).

5. Runtime npm install --global agent-skills-eval on :latest

Same reproducibility/airgap concern as other pipeline steps. Prefer a pinned ASE image or digest for production.

Minor (non-blocking)

• Unquoted variable: for MD_FILE in $MD_FILES breaks on filenames with spaces
• || true on ASE: Hides infra failures (aggregate marks fail — acceptable but noisy)
• Empty edge_case_results: Returns pass score 1.0 in BLOCK mode (scorecard noise)
• Generic ASE assertions: Hardcoded defaults are weak for "behavioral" claims — fine for MVP
• Stability/failure_mode stubs: OK if documented as future work, not if marketed as Certified coverage

---

Required before merge

1. Fix Certified enablement vs min-2 / optional edge_cases/ (product decision + code alignment)
2. Run edge-case ASE for eval-engine=ase, not only harbor/both

[csoceanu]

Validated end-to-end on the cluster.

With edge cases: hello-world-behavioral (ASE)

• Run: test-behavioral-ase-dbgj9
• Submission: skill-submissions branch eval/hello-world-behavioral — includes edge_cases/ with 3 edge cases (adversarial_input, boundary_values, empty_input)
• Result: Behavioral testing passed (2 sub-checks, score 1.00)
  • Consistency: passed (std_reward=0.000)
  • Edge cases: 3/3 passed
• Certification: certification.json shows enterprise_behavioral_testing: passed=true

Without edge cases: hello-world-full (ASE)

• Run: test-hello-world-no-edge-44ss6
• Submission: skill-submissions branch eval/hello-world-full — no edge_cases/ directory
• Result: Behavioral testing failed — "Insufficient behavioral coverage: 1 sub-check(s) with data, need at least 2"
  • Edge case ASE step skipped cleanly: "No edge_cases/ directory"
• Certification: certification.json shows enterprise_behavioral_testing: passed=false

Both runs confirm the behavioral testing check works as intended. Foundational and Trusted levels are unaffected.