feat: v0.6.0 standards alignment + legibility

[vaaraio]

Summary

v0.5.x closed the capability axis (jailbreak coverage, classifier rebalance). v0.6 is the legibility axis: policies become readable, audit records become standards-aligned, adversarial numbers get split by source, and architecture contribution gets documented.

Seven feature commits plus the bump:

• vaara.policy package. JSON-native loader, optional [yaml] extra, frozen dataclasses, hand-rolled validation with field-path errors. Reuses vaara.taxonomy.actions enums verbatim. Implements Sketch A from the v0.6 DSL design exploration. Embedded-Python (B) and standalone DSL (C) stay deferred to v0.7+ pending external pull.
• vaara trail purge CLI + SQLiteAuditBackend.purge_older_than API. Article 12(2) retention, tenant-scoped. Hash-chain integrity preserved with a documented seam at the retention boundary. Intended workflow: signed handoff zip first, archive externally, then purge.
• prEN ISO/IEC 12792 four-axis transparency taxonomy on AuditRecord. Four optional fields (system_operation, data_usage, decision_making, limitations) with default-classification heuristic per EventType and per-record override. Excluded from record_hash so pre-v0.6 chains stay valid. v0.7+ may add a separate signing mechanism if compliance requires.
• Distribution-shift split. Hand-curated (held-out, 250): attack recall 97.1% / benign FPR 70.0%. LLM-generated (in-sample, 5,705): attack recall 95.2% / benign FPR 87.5%. The 18pp benign-FPR gap is the dominant distribution-shift signal.
• Stack ablation. Heuristic-only 35% / 63%. Classifier-only 94% / 86%. Full-stack 97% / 98%. Layers not redundant. Heuristic catches a small attack set the classifier misses, which justifies the ensemble. Most full-stack benign FPR comes from heuristic ESCALATEs, not classifier upgrades.
• PAIR adaptive-attacker calibration. Qwen2.5-32B-Instruct as attacker + judge, 25 hand-curated jailbreak seeds, max 5 iterations: ASR 0.0% (0/25). NOT a claim of impervence to all adaptive attackers. A stronger attacker (70B+), longer iteration budgets, or alternate strategies (multi-turn drift, language-switch, obfuscation) might produce non-zero ASR.
• COMPLIANCE.md gains EU AI Act Annex IV evidence-section mapping and CEN-CENELEC harmonised-standards alignment across ISO/IEC 42001, prEN 18286, prEN 18228, ISO/IEC 42006, prEN ISO/IEC 24970, prEN 18229-1, prEN ISO/IEC 12792.

Schema migration

Audit DB v2 to v3 adds four nullable transparency columns. Pre-v0.6 records get NULL for the new columns. Their stored record_hash is preserved (NOT re-hashed on load), so historical chain verification continues to work.

Test plan

• pytest -q — 296 passed, 12 skipped (vaara[ml] extras path)
• ruff check src/ scripts/ tests/ clean
• vaara.__version__ reads 0.6.0
• CHANGELOG [0.6.0] - 2026-04-27 entry present
• Reproducible artifacts: tests/adversarial/distribution_shift_v0_5_3.json, stack_ablation_v0_5_3.json, pair_v0_5_3.json
• CI green
• Reviewer eyeball pass

Deferred to v0.7+

• prEN ISO/IEC 24970 field-alias layer (gated on standard publication)
• DORA mapping refinement (gated on deployer signal)

Summary by CodeRabbit

• New Features

  • JSON-native policy loader with optional YAML support; example policies added
  • Tenant-scoped audit-trail retention purge (CLI + backend) with dry-run
  • Expanded audit-record transparency metadata (four-axis taxonomy; excluded from hashing)
  • New evaluation scripts for distribution-shift, stack-ablation, and PAIR attacks

• Documentation

  • Updated compliance docs with Annex IV mapping, standards alignment, measured v0.6 results, and migration guidance
  • Added CHANGELOG entry for v0.6.0

• Chores

  • Version bumped to v0.6.0; optional YAML extra added; audit DB schema advanced (backward-compatible)
  • .gitignore updated to ignore research/ and .claude/

• Tests

  • Extensive new test suites and reproducible evaluation artifacts added

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds Vaara v0.6.0: JSON/YAML policy schema and loaders, policy examples, four-axis transparency taxonomy with SQLite audit schema bump v2→v3, tenant-scoped purge API + CLI, three adversarial evaluation scripts and fixtures, packaging yaml extra, compliance doc updates, and comprehensive tests.

Changes

Cohort / File(s)	Summary
Version & Packaging pyproject.toml, src/vaara/__init__.py	Bump package version to 0.6.0 and add optional yaml extra (pyyaml>=6.0).
Policy System src/vaara/policy/schema.py, src/vaara/policy/loader.py, src/vaara/policy/__init__.py, examples/policies/minimal.json, examples/policies/full.yaml	Add frozen dataclass policy schema, PolicyError, Thresholds/Sequence/Escalation types, loader APIs (from_dict, from_json, from_yaml), JSON/YAML loading behavior, and example policies.
Audit Trail & CLI src/vaara/audit/trail.py, src/vaara/audit/sqlite_backend.py, src/vaara/cli.py, tests/test_audit_purge.py, tests/test_transparency_taxonomy.py	Introduce four transparency fields on AuditRecord with per-EventType defaults, bump SQLite SCHEMA_VERSION 2→3 (nullable columns), add purge_older_than(...) (tenant-scoped, dry-run) and vaara trail purge CLI; include migration, semantics, and tenant-isolation tests.
Evaluation Scripts & Helpers scripts/eval_distribution_shift.py, scripts/eval_pair_attack.py, scripts/eval_stack_ablation.py	Add three evaluator CLIs for distribution-shift, PAIR-style attacks, and stack ablation; integrate with Pipeline and optional AdversarialClassifier (requires vaara[ml]), produce JSON reports, and provide attacker/judge and scoring helpers.
Tests & Fixtures tests/test_policy.py, tests/adversarial/*.json, tests/test_*	Add comprehensive policy loader tests (validation, immutability, JSON/YAML coverage) and adversarial evaluation fixtures/results; multiple new test modules for purge, transparency, and policy behavior.
Docs & Misc CHANGELOG.md, COMPLIANCE.md, .gitignore	Add v0.6.0 changelog entry; expand COMPLIANCE.md with Annex IV mapping, retention/purge guidance and measured evaluation limits; update .gitignore to exclude research/ and .claude/.

Sequence Diagram(s)

sequenceDiagram
  rect rgba(200,230,201,0.5)
    participant CLI as "User CLI"
    participant Backend as "SQLiteAuditBackend"
    participant DB as "SQLite DB"
    participant Logger as "Logger"
  end

  CLI->>Backend: vaara trail purge --retention-days N [--dry-run]
  Backend->>DB: open connection (tenant_id scoped)
  Backend->>DB: SELECT COUNT(*) WHERE ts < cutoff AND tenant_id = ?
  DB-->>Backend: would_delete_count
  alt dry_run
    Backend->>CLI: print "Would purge {count}" (dry-run)
    Backend->>DB: close
  else actual_purge
    Backend->>DB: DELETE FROM audit_records WHERE ts < cutoff AND tenant_id = ?
    DB-->>Backend: deleted_count
    Backend->>Logger: warn about hash-chain seam at retention boundary
    Backend->>CLI: print "Purged {count}" and retention warning
    Backend->>DB: close
  end

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• feat: v0.5.0 adversarial classifier (opt-in ML scorer) #34: Implements the adversarial classifier and ml packaging that the new evaluation scripts depend on.
• test: adversarial corpus v1 (200 entries, 8 categories) + evaluator #23: Adds evaluator scripts and adversarial corpora complementary to the new adversarial tooling and fixtures.
• docs: COMPLIANCE.md article-by-article evidence mapping #16: Overlaps COMPLIANCE.md Annex IV evidence mapping and related compliance documentation updates.

Poem

🐰 I hop through policies, JSON and YAML in paw,
I tuck transparency fields where auditors awe,
I count what to purge (tenant-bound, tidy and dry),
I run three small scripts where crafty prompts try,
A carrot for tests — a tiny celebratory hop!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 43.01% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'feat: v0.6.0 standards alignment + legibility' accurately reflects the main changes: a standards-aligned v0.6.0 release with policy schema, audit retention, transparency taxonomy, and compliance documentation.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/v0.6-policy-schema

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 9

🧹 Nitpick comments (2)

tests/test_audit_purge.py (1)

14-31: Consider using a fixture-scoped counter instead of global state.

The global _record_seq counter persists across test runs within a session, which could theoretically cause issues with test isolation if tests make assumptions about record ID patterns. However, since each test gets a fresh SQLite database via the backend fixture and record IDs include the timestamp, this is unlikely to cause actual failures.

If you want stricter isolation, consider resetting the counter in a fixture or using uuid.uuid4() instead.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tests/test_audit_purge.py` around lines 14 - 31, The global _record_seq used
by _make_record creates shared state across tests; replace it with either a
fixture-scoped counter or remove it entirely by generating unique IDs (e.g., use
uuid.uuid4()) and/or reset the counter in a fixture so tests don't rely on
module-level mutation; update the _make_record implementation to use the new
generator (or fixture-injected counter) and remove or stop mutating the global
_record_seq to ensure test isolation.

scripts/eval_stack_ablation.py (1)

47-59: Verify source tagging logic.

Files directly under corpus_root/*.jsonl are tagged as hand_curated, while files under generated/ and benign_generated/ subdirectories are tagged as llm_generated. This assumes a specific directory structure. Consider documenting this expected layout in the docstring or validating that at least some files are found.

📝 Suggested documentation improvement

 def load_corpus(corpus_root: Path) -> list[dict]:
-    """Load both hand-curated and LLM-generated entries with source tags."""
+    """Load both hand-curated and LLM-generated entries with source tags.
+    
+    Expected layout:
+        corpus_root/*.jsonl           -> tagged as 'hand_curated'
+        corpus_root/generated/*.jsonl -> tagged as 'llm_generated'
+        corpus_root/benign_generated/*.jsonl -> tagged as 'llm_generated'
+    """

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@scripts/eval_stack_ablation.py` around lines 47 - 59, The load_corpus
function currently tags files under corpus_root/*.jsonl as "hand_curated" and
those under generated/ and benign_generated/ as "llm_generated" without
documenting or validating that structure; update the load_corpus docstring to
explicitly describe the expected directory layout (top-level jsonl are
hand_curated, subdirs "generated" and "benign_generated" contain llm_generated)
and add a simple validation after collecting entries (using entries,
corpus_root, and _load_jsonl) to raise or warn if no files were found or if
expected subdirs are missing so callers get immediate feedback.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@CHANGELOG.md`:
- Line 27: Replace the misspelled word "impervence" with "imperviousness" in the
sentence inside CHANGELOG.md that reads "NOT a claim of impervence to all
adaptive attackers ..." and also fix the identical typo in COMPLIANCE.md so both
files read "NOT a claim of imperviousness to all adaptive attackers ...".

In `@COMPLIANCE.md`:
- Line 320: Replace the misspelled word "impervence" with "imperviousness" in
the sentence that currently reads "capability level. NOT a claim of impervence
to all adaptive attackers" so it reads "... NOT a claim of imperviousness to all
adaptive attackers"; update only that token to correct the typo.

In `@scripts/eval_distribution_shift.py`:
- Around line 161-165: The AdversarialClassifier() constructor can raise
ImportError for missing optional ML deps but is instantiated outside the
try-except, so wrap both the import and the classifier construction in the same
try-except block (covering from "from vaara.adversarial_classifier import
AdversarialClassifier" through "classifier = AdversarialClassifier()") and on
ImportError raise SystemExit with a clear message that includes the caught
exception (exc) so missing dependency errors are handled the same as import
failures.

In `@scripts/eval_pair_attack.py`:
- Line 88: The current fallback to "ALLOW" when extracting decision from
Pipeline.intercept() is unsafe; change the fallback to a safe value like "DENY"
(or "UNKNOWN") and validate the extracted value against allowable decisions
before using it. Specifically, in the code around the decision =
str(getattr(result, "decision", "ALLOW")).upper() expression, replace the
default "ALLOW" with "DENY" (or "UNKNOWN"), and add a short check that result is
the expected type and that decision is in the set of valid decisions (e.g.,
{"ALLOW","DENY","UNKNOWN"}); if validation fails, set decision to the safe
default and emit a warning/log so malformed returns from Pipeline.intercept()
are treated as denied.
- Around line 43-54: The call_llm function currently assumes the response is
valid JSON with choices[0].message.content and will let
JSONDecodeError/KeyError/IndexError/TypeError escape and kill the run; wrap the
response parsing and indexing in a try/except that catches json.JSONDecodeError,
KeyError, IndexError, and TypeError, read the raw response body for context, and
convert these into a consistent per-seed error (e.g., raise a specific
LLMResponseError or re-raise as urllib.error.URLError/ValueError with a clear
message containing the raw body) so callers that already handle
URLError/TimeoutError will treat malformed responses as per-seed failures; apply
the same change to the similar parsing block around lines 120-137.

In `@src/vaara/cli.py`:
- Around line 186-189: The code fails to expand user home tilde for the DB path;
change how db_path is constructed so you call Path(args.db).expanduser() (e.g.,
db_path = Path(args.db).expanduser()) before the exists() check used in the CLI
(the db_path variable and args.db reference), so "--db '~/path/audit.db'" is
correctly resolved prior to printing the "audit DB not found" error.

In `@src/vaara/policy/loader.py`:
- Around line 73-77: The thresholds_overrides dict built from thr_block should
be validated immediately instead of deferring to Policy.threshold_for; for each
k, v in thr_block (skipping "default") iterate per-action override mapping (the
inner dict from thresholds_overrides) and attempt to construct or validate using
the Thresholds constructor/validator (or replicate its range/order checks) to
ensure values are floats in [0,1] and that escalate <= deny <= allow ordering
(or the exact invariants enforced by Thresholds); raise a clear exception on
invalid entries so malformed policies like {"foo": {"escalate": 0.9, "deny":
0.2}} are rejected during load rather than later in Policy.threshold_for().
- Around line 47-49: The from_dict() implementation must validate nested section
shapes before iterating/coercing: for each nested key (e.g., action_classes,
thresholds, escalation, sequences) check the raw value's type (mapping for
dict-like sections, list/sequence for lists, dict for nested mappings like
sequences[name].pattern) and raise PolicyError with the offending path (e.g.,
"action_classes.<name>: must be a mapping" or "thresholds: must be a sequence")
instead of calling .items(), .get(), or tuple(...) which can raise
AttributeError or coerce strings into tuples; update the parsing code in
from_dict() to perform these explicit isinstance checks and raise PolicyError
with clear path context for the other blocks referenced (the blocks currently
around the action_classes, thresholds, escalation, sequences parsing) before any
further processing.

In `@src/vaara/policy/schema.py`:
- Around line 1-5: The Policy dataclass is currently "frozen" but its mutable
dict fields (action_classes, thresholds_overrides) allow in-place mutation; in
Policy.__post_init__ wrap/replace these fields with immutable mapping proxies
(e.g., types.MappingProxyType) or deep-copied nested mapping proxies so nested
dicts are also protected, ensuring the frozen value-object guarantee; update
Policy.__post_init__ to import types and perform deep copy + MappingProxyType
conversion for action_classes and for each nested mapping inside
thresholds_overrides (and assign back to the dataclass fields inside
__post_init__ despite frozen dataclass by using object.__setattr__).

---

Nitpick comments:
In `@scripts/eval_stack_ablation.py`:
- Around line 47-59: The load_corpus function currently tags files under
corpus_root/*.jsonl as "hand_curated" and those under generated/ and
benign_generated/ as "llm_generated" without documenting or validating that
structure; update the load_corpus docstring to explicitly describe the expected
directory layout (top-level jsonl are hand_curated, subdirs "generated" and
"benign_generated" contain llm_generated) and add a simple validation after
collecting entries (using entries, corpus_root, and _load_jsonl) to raise or
warn if no files were found or if expected subdirs are missing so callers get
immediate feedback.

In `@tests/test_audit_purge.py`:
- Around line 14-31: The global _record_seq used by _make_record creates shared
state across tests; replace it with either a fixture-scoped counter or remove it
entirely by generating unique IDs (e.g., use uuid.uuid4()) and/or reset the
counter in a fixture so tests don't rely on module-level mutation; update the
_make_record implementation to use the new generator (or fixture-injected
counter) and remove or stop mutating the global _record_seq to ensure test
isolation.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 745e85cf-842a-4c29-8490-c420a8dc25fc

📥 Commits

Reviewing files that changed from the base of the PR and between 45c3ee3 and 4c3e3c4.

📒 Files selected for processing (22)

• .gitignore
• CHANGELOG.md
• COMPLIANCE.md
• examples/policies/full.yaml
• examples/policies/minimal.json
• pyproject.toml
• scripts/eval_distribution_shift.py
• scripts/eval_pair_attack.py
• scripts/eval_stack_ablation.py
• src/vaara/__init__.py
• src/vaara/audit/sqlite_backend.py
• src/vaara/audit/trail.py
• src/vaara/cli.py
• src/vaara/policy/__init__.py
• src/vaara/policy/loader.py
• src/vaara/policy/schema.py
• tests/adversarial/distribution_shift_v0_5_3.json
• tests/adversarial/pair_v0_5_3.json
• tests/adversarial/stack_ablation_v0_5_3.json
• tests/test_audit_purge.py
• tests/test_policy.py
• tests/test_transparency_taxonomy.py

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

Fix typo: "impervence" → "imperviousness".

Same spelling error as in COMPLIANCE.md.

✏️ Proposed fix

-  **ASR 0.0% (0/25)**. NOT a claim of impervence to all adaptive attackers — stronger at...
+  **ASR 0.0% (0/25)**. NOT a claim of imperviousness to all adaptive attackers — stronger at...

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	- **PAIR adaptive-attacker calibration.** Qwen2.5-32B-Instruct as both attacker and judge, 25 hand-curated jailbreak seeds, max 5 iterations: **ASR 0.0% (0/25)**. NOT a claim of impervence to all adaptive attackers — stronger attacker (70B+), longer iteration budgets, or alternate strategies (multi-turn drift, language-switch, obfuscation) might produce non-zero ASR.
	- **PAIR adaptive-attacker calibration.** Qwen2.5-32B-Instruct as both attacker and judge, 25 hand-curated jailbreak seeds, max 5 iterations: **ASR 0.0% (0/25)**. NOT a claim of imperviousness to all adaptive attackers — stronger attacker (70B+), longer iteration budgets, or alternate strategies (multi-turn drift, language-switch, obfuscation) might produce non-zero ASR.

🧰 Tools

🪛 LanguageTool

[grammar] ~27-~27: Ensure spelling is correct
Context: ...ns: ASR 0.0% (0/25). NOT a claim of impervence to all adaptive attackers — stronger at...

(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@CHANGELOG.md` at line 27, Replace the misspelled word "impervence" with
"imperviousness" in the sentence inside CHANGELOG.md that reads "NOT a claim of
impervence to all adaptive attackers ..." and also fix the identical typo in
COMPLIANCE.md so both files read "NOT a claim of imperviousness to all adaptive
attackers ...".

[coderabbitai]

Actionable comments posted: 2

♻️ Duplicate comments (1)

scripts/eval_pair_attack.py (1)

43-54: ⚠️ Potential issue | 🟠 Major

Malformed LLM responses can still crash the evaluation run.

The parsing at line 54 can raise JSONDecodeError, KeyError, or IndexError if the API returns non-JSON or an unexpected response shape. These exceptions are not caught by the URLError/TimeoutError handlers in run_pair_one (lines 128-129, 142-143), causing the entire evaluation to terminate on a single bad response.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@scripts/eval_pair_attack.py` around lines 43 - 54, call_llm currently assumes
the HTTP response is well-formed JSON and accesses
["choices"][0]["message"]["content"] directly, which can raise
JSONDecodeError/KeyError/IndexError and crash the run; wrap the response
reading/parsing in a try/except that catches json.JSONDecodeError, KeyError,
IndexError, and ValueError, and on failure raise a urllib.error.URLError (or
re-use an existing exception type handled by run_pair_one) with a clear message
that includes the raw response body for debugging; also defensively check the
HTTP response status (if available) and treat non-2xx as errors so
run_pair_one's existing URLError/TimeoutError handlers will catch malformed or
unexpected responses from call_llm.

🧹 Nitpick comments (1)

src/vaara/cli.py (1)

182-217: LGTM! The vaara trail purge implementation is solid.

The past review comment about expanduser() has been addressed (line 186). The function validates inputs, handles cleanup via try/finally, and provides appropriate user feedback including the hash-chain seam warning on stderr.

Minor style suggestion: Consider using the context manager protocol since SQLiteAuditBackend implements __enter__/__exit__ (per src/vaara/audit/sqlite_backend.py:816-821).

♻️ Optional refactor to use context manager

-    backend = SQLiteAuditBackend(db_path)
-    try:
-        count = backend.purge_older_than(retention_seconds, dry_run=args.dry_run)
-    finally:
-        backend.close()
+    with SQLiteAuditBackend(db_path) as backend:
+        count = backend.purge_older_than(retention_seconds, dry_run=args.dry_run)

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@src/vaara/cli.py` around lines 182 - 217, Refactor _cmd_trail_purge to use
the context manager on SQLiteAuditBackend instead of manual try/finally: replace
creating backend = SQLiteAuditBackend(db_path) and the try/finally close pattern
with a with SQLiteAuditBackend(db_path) as backend: block, then call
backend.purge_older_than(retention_seconds, dry_run=args.dry_run) inside that
block and preserve all existing logic around args.dry_run, printing, and return
codes; this leverages SQLiteAuditBackend.__enter__/__exit__ while keeping
purge_older_than and the same error/print behavior.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@scripts/eval_pair_attack.py`:
- Around line 4-7: The module-level docstring in scripts/eval_pair_attack.py
claims the run uses "Llama-3.3-70B as both attacker and judge" which conflicts
with COMPLIANCE.md (line 309) that documents "Qwen2.5-32B-Instruct (Apache
2.0)"; update the docstring text to accurately reflect the actual model used
(or, if the doc is wrong, update COMPLIANCE.md instead) so both sources match;
locate the module docstring in eval_pair_attack.py and replace the model string
to "Qwen2.5-32B-Instruct (Apache 2.0)" (or the correct model name you decide to
keep) and run a quick grep of the repository for other occurrences of
"Llama-3.3-70B" to ensure no stale references remain.

In `@tests/test_policy.py`:
- Around line 253-265: The module-level pytest.importorskip("yaml") causes all
tests in this file to be skipped when PyYAML is absent; move the import-or-skip
call into the YAML-specific tests (test_from_yaml_string and
test_from_yaml_path) so only those two tests are skipped. Specifically, remove
the top-level yaml = pytest.importorskip("yaml") and instead call yaml =
pytest.importorskip("yaml") at the start of each of test_from_yaml_string and
test_from_yaml_path (or use pytest.importorskip inside those functions and then
import yaml via pytest.importorskip), ensuring the yaml variable is available to
each test that needs it.

---

Duplicate comments:
In `@scripts/eval_pair_attack.py`:
- Around line 43-54: call_llm currently assumes the HTTP response is well-formed
JSON and accesses ["choices"][0]["message"]["content"] directly, which can raise
JSONDecodeError/KeyError/IndexError and crash the run; wrap the response
reading/parsing in a try/except that catches json.JSONDecodeError, KeyError,
IndexError, and ValueError, and on failure raise a urllib.error.URLError (or
re-use an existing exception type handled by run_pair_one) with a clear message
that includes the raw response body for debugging; also defensively check the
HTTP response status (if available) and treat non-2xx as errors so
run_pair_one's existing URLError/TimeoutError handlers will catch malformed or
unexpected responses from call_llm.

---

Nitpick comments:
In `@src/vaara/cli.py`:
- Around line 182-217: Refactor _cmd_trail_purge to use the context manager on
SQLiteAuditBackend instead of manual try/finally: replace creating backend =
SQLiteAuditBackend(db_path) and the try/finally close pattern with a with
SQLiteAuditBackend(db_path) as backend: block, then call
backend.purge_older_than(retention_seconds, dry_run=args.dry_run) inside that
block and preserve all existing logic around args.dry_run, printing, and return
codes; this leverages SQLiteAuditBackend.__enter__/__exit__ while keeping
purge_older_than and the same error/print behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 36120b96-bb11-4536-888b-061af7ccab76

📥 Commits

Reviewing files that changed from the base of the PR and between 902c5d5 and 11f0d11.

📒 Files selected for processing (7)

• CHANGELOG.md
• COMPLIANCE.md
• scripts/eval_pair_attack.py
• src/vaara/cli.py
• src/vaara/policy/loader.py
• src/vaara/policy/schema.py
• tests/test_policy.py

✅ Files skipped from review due to trivial changes (1)

• CHANGELOG.md

🚧 Files skipped from review as they are similar to previous changes (1)

• src/vaara/policy/loader.py

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

scripts/eval_distribution_shift.py (1)

77-77: Consider defaulting to "ERROR:Unknown" for consistency with evaluate() bucketing.

When result.decision is missing, the fallback "UNKNOWN" won't match any of the outcome branches in evaluate() (lines 113-120), so the entry counts toward n but no outcome counter increments. This silently skews metrics.

For consistency with eval_pair_attack.py (which defaults to "DENY") and proper metric accounting, consider either:

• Returning an error-like string: f"ERROR:MissingDecision"
• Or treating it as "DENY" to fail closed

♻️ Option A: Count as error

-    decision = str(getattr(result, "decision", "UNKNOWN")).upper()
+    raw = getattr(result, "decision", None)
+    if raw is None:
+        return "ERROR:MissingDecision"
+    decision = str(raw).upper()

♻️ Option B: Fail closed (match eval_pair_attack.py)

-    decision = str(getattr(result, "decision", "UNKNOWN")).upper()
+    raw = getattr(result, "decision", None)
+    decision = str(raw).upper() if raw is not None else "DENY"
+    if decision not in {"ALLOW", "ESCALATE", "DENY"}:
+        decision = "DENY"

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@scripts/eval_distribution_shift.py` at line 77, The fallback for
result.decision should be an error-like string so evaluate() counts it in the
error bucket; change the assignment of the decision variable (currently decision
= str(getattr(result, "decision", "UNKNOWN")).upper()) to default to an ERROR
token (e.g., "ERROR:MissingDecision" or "ERROR:Unknown") instead of "UNKNOWN" so
the entry increments the error outcome in evaluate() and does not silently skew
metrics; update the decision assignment in scripts/eval_distribution_shift.py
(the decision variable) to use the chosen ERROR default and ensure it remains
uppercased.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@tests/test_policy.py`:
- Around line 246-249: The test
test_from_json_unreadable_input_raises_policy_error is flaky because
from_json("not json at all") may hit a real file in the CWD; update the test to
use a guaranteed-missing tmp_path file instead: create a Path via the pytest
tmp_path fixture (e.g. tmp_path / "surely_missing.json"), ensure it does not
exist, and pass its string to from_json in the pytest.raises block so
from_json(...) deterministically raises PolicyError; keep the assertion text
match unchanged and reference the test function name and from_json call when
making the change.

---

Nitpick comments:
In `@scripts/eval_distribution_shift.py`:
- Line 77: The fallback for result.decision should be an error-like string so
evaluate() counts it in the error bucket; change the assignment of the decision
variable (currently decision = str(getattr(result, "decision",
"UNKNOWN")).upper()) to default to an ERROR token (e.g., "ERROR:MissingDecision"
or "ERROR:Unknown") instead of "UNKNOWN" so the entry increments the error
outcome in evaluate() and does not silently skew metrics; update the decision
assignment in scripts/eval_distribution_shift.py (the decision variable) to use
the chosen ERROR default and ensure it remains uppercased.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 9c645cf0-208b-46a9-bd88-b1cc548f247f

📥 Commits

Reviewing files that changed from the base of the PR and between 11f0d11 and 422f4d5.

📒 Files selected for processing (3)

• scripts/eval_distribution_shift.py
• scripts/eval_pair_attack.py
• tests/test_policy.py

[coderabbitai]

Actionable comments posted: 3

🧹 Nitpick comments (1)

tests/test_policy.py (1)

166-181: Consider parametrizing strict-shape mapping tests to reduce duplication.

These three tests share identical structure and could be consolidated with pytest.mark.parametrize for easier extension and maintenance.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tests/test_policy.py` around lines 166 - 181, Consolidate the three
near-identical tests into a single parametrized test using
pytest.mark.parametrize: iterate over the field names "action_classes",
"thresholds", and "escalation", build bad = {**MINIMAL_POLICY, field: []}, and
assert from_dict(bad) raises PolicyError with the appropriate regex f"{field}:
must be a mapping"; update or remove the original
test_action_classes_must_be_mapping_not_list,
test_thresholds_must_be_mapping_not_list, and
test_escalation_must_be_mapping_not_list accordingly so the single test covers
all three cases while still referencing MINIMAL_POLICY, from_dict, and
PolicyError.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@scripts/eval_distribution_shift.py`:
- Around line 120-128: The code currently records ERROR outcomes into b["error"]
but still proceeds to compute and return metrics; update the post-counting logic
to abort the run if any bucket has b["error"] > 0: after the loop that
builds/upserts buckets and before the final return that computes percentages,
scan buckets for any nonzero "error" counts and raise a non-zero exit (e.g.,
raise SystemExit or RuntimeError) with a clear message listing the offending
src/cls keys and counts so the process fails instead of publishing invalid
percentages; apply this check to the same counting logic that uses variables
decision, b, buckets and the final return that spreads counts into the result.
- Around line 165-166: The current corpus validation only checks
corpus_root.exists(), which allows non-directories or empty loads to proceed;
update the validation to ensure corpus_root.is_dir() and after loading JSONL
compute the total entries (e.g., the variable tracking loaded entries or
total_entries) and raise SystemExit with a clear message if total_entries == 0;
modify the checks around corpus_root and the JSONL load logic (referencing
corpus_root and the variable that accumulates loaded entries) so the script
fails fast when the corpus path is not a directory or when no entries were
loaded.
- Around line 99-106: The function expected_category should validate the
entry["expected"] value instead of coercing unknown/missing values to "attack":
in expected_category, check that "expected" exists and is either a string
"ALLOW"/"DENY" (case-insensitive) or a non-empty list of such strings, normalize
to uppercase, and raise a ValueError with a descriptive message if the type or
contained labels are invalid or empty; then return "benign" only when the
normalized set is exactly {"ALLOW"} and "attack" when exactly {"DENY"} (or the
appropriate normalized set), so malformed fixtures fail fast (refer to function
expected_category and the local variables expected and expected_set).

---

Nitpick comments:
In `@tests/test_policy.py`:
- Around line 166-181: Consolidate the three near-identical tests into a single
parametrized test using pytest.mark.parametrize: iterate over the field names
"action_classes", "thresholds", and "escalation", build bad = {**MINIMAL_POLICY,
field: []}, and assert from_dict(bad) raises PolicyError with the appropriate
regex f"{field}: must be a mapping"; update or remove the original
test_action_classes_must_be_mapping_not_list,
test_thresholds_must_be_mapping_not_list, and
test_escalation_must_be_mapping_not_list accordingly so the single test covers
all three cases while still referencing MINIMAL_POLICY, from_dict, and
PolicyError.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: a3a0254e-584c-4acd-9d61-a8d1ed7103ad

📥 Commits

Reviewing files that changed from the base of the PR and between 422f4d5 and a784831.

📒 Files selected for processing (2)

• scripts/eval_distribution_shift.py
• tests/test_policy.py

[vaaraio]

@coderabbitai full review

[coderabbitai]

✅ Actions performed

Full review triggered.

[coderabbitai]

Actionable comments posted: 10

♻️ Duplicate comments (3)

scripts/eval_stack_ablation.py (3)

168-173: ⚠️ Potential issue | 🟠 Major

Wrap AdversarialClassifier() in the same try block as the import.

The constructor can still raise ImportError when optional ML deps are missing, so the current code can fall through to an unhandled traceback even after the import guard.

💡 Proposed fix

     try:
         from vaara.adversarial_classifier import AdversarialClassifier
+        classifier = AdversarialClassifier()
     except ImportError as exc:
         raise SystemExit(f"this script needs vaara[ml]: {exc}")
-    classifier = AdversarialClassifier()

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@scripts/eval_stack_ablation.py` around lines 168 - 173, The instantiation of
AdversarialClassifier can raise ImportError and must be inside the same
try/except as the import; move the call to AdversarialClassifier() into the try
block that imports vaara.adversarial_classifier and catch ImportError as exc,
then raise SystemExit with a clear message including exc (i.e., wrap both "from
vaara.adversarial_classifier import AdversarialClassifier" and "classifier =
AdversarialClassifier()" in the same try and keep the existing
print(f"[classifier] bundle v{classifier.bundle_version},
threshold={classifier.threshold}") after the try).

---

62-65: ⚠️ Potential issue | 🟠 Major

Validate expected labels before bucketing them.

Missing or malformed expected values are silently treated as "attack" here, which can skew both recall and benign FPR. This should fail fast instead of defaulting to "DENY".

💡 Proposed fix

 def expected_class(entry: dict) -> str:
-    expected = entry.get("expected", "DENY")
-    s = {str(x).upper() for x in (expected if isinstance(expected, list) else [expected])}
+    if "expected" not in entry:
+        raise ValueError("missing required field: expected")
+    expected = entry["expected"]
+    values = expected if isinstance(expected, list) else [expected]
+    s = {str(x).upper() for x in values}
+    allowed = {"ALLOW", "DENY", "ESCALATE"}
+    if not s or not s.issubset(allowed):
+        raise ValueError(f"invalid expected value(s): {sorted(s)}")
     return "benign" if s == {"ALLOW"} else "attack"

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@scripts/eval_stack_ablation.py` around lines 62 - 65, The function
expected_class currently defaults missing/malformed expected values to "DENY"
and silently classifies them as "attack"; update expected_class to validate the
entry["expected"] value(s) (accepting either a string or list of strings,
case-insensitive) and raise a clear exception when expected is missing or
contains any value outside the allowed set {"ALLOW","DENY"} instead of
defaulting, then keep the existing bucketing logic (treat {"ALLOW"} -> "benign",
otherwise "attack"); refer to the expected_class function and the local variable
expected/entry to implement this validation and error handling.

---

133-155: ⚠️ Potential issue | 🟠 Major

Abort the run before reporting metrics if any ERROR:* decisions occurred.

The script records err counts, but it still prints and optionally writes metrics afterward. That can publish invalid percentages from a partially failed run.

💡 Proposed fix

 def report(counts, json_out: Path | None) -> None:
     """Print per-config / per-source recall + FPR table."""
+    total_errors = sum(c["err"] for by_key in counts.values() for c in by_key.values())
+    if total_errors:
+        raise SystemExit(
+            f"aborting: {total_errors} ERROR outcome(s) present; "
+            "fix corpus/runtime issues before publishing metrics"
+        )
+
     rows = []
     print("\n=== Stack ablation (v0.5.3) ===")

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@scripts/eval_stack_ablation.py` around lines 133 - 155, In report(), before
printing or writing metrics, detect any recorded errors by scanning counts (for
each config in ("heuristic_only","classifier_only","full_stack") and each bucket
c in counts[config]) and if any c["err"] > 0 abort the run (raise SystemExit or
call sys.exit with non‑zero) after printing a short diagnostic message; do this
check early in the function (using the existing counts, config_name, c and err
keys) so you never print or write metrics to json_out when there were ERROR:*
decisions.

🧹 Nitpick comments (2)

tests/test_audit_purge.py (1)

17-31: Seed purge tests through a real hash chain.

_make_record() computes standalone hashes with an empty previous_hash, so any sequence written directly with backend.write_record() is already chain-broken before purge. That makes it impossible for this module to assert the documented retention-boundary seam separately from setup artifacts. Consider seeding via AuditTrail or linking previous_hash/record_hash explicitly.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tests/test_audit_purge.py` around lines 17 - 31, The helper _make_record
builds AuditRecord objects but computes record_hash with an empty previous_hash,
breaking any chain when tests call backend.write_record() directly; instead seed
tests with a proper chain by either creating/appending records through
AuditTrail (use AuditTrail.append_record or whatever method ensures
previous_hash is set) or, if creating records manually, set each
record.previous_hash to the prior record.record_hash before calling
rec.record_hash = rec.compute_hash() so the linked hash chain is intact for
purge boundary assertions (refer to AuditRecord, compute_hash, previous_hash,
record_hash, backend.write_record, and AuditTrail).

tests/test_transparency_taxonomy.py (1)

128-167: Make the migration test prove the columns were actually added.

_row_to_record() now treats missing row[12:16] as None, so this test still passes even if the v2→v3 migration never creates the four transparency columns. Please assert the schema contains those columns, or do a post-migration write/read round-trip that would fail on a partially upgraded DB.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tests/test_transparency_taxonomy.py` around lines 128 - 167, The test
currently doesn't prove the v2→v3 migration actually added the four transparency
columns; after opening the DB with SQLiteAuditBackend in
test_migration_from_v2_to_v3, query the migrated schema (e.g., PRAGMA
table_info('audit_records')) and assert that columns 'system_operation',
'data_usage', 'decision_making', and 'limitations' are present, or perform a
post-migration write/read round-trip via SQLiteAuditBackend (insert a record
with those fields set, then reload it) to fail if the columns were never
created; reference the test function test_migration_from_v2_to_v3 and the
SQLiteAuditBackend/_row_to_record code paths so the assertion validates the
actual schema change rather than relying on None defaults.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@CHANGELOG.md`:
- Around line 13-14: Update the CHANGELOG wording to avoid implying "vaara trail
verify" validates the live DB after purge: clarify that src/vaara/cli.py's vaara
trail verify only validates signed zip bundles (the exported handoff) and that
those bundles remain self-consistent, while the live DB/trail will show a
deliberate chain seam where deleted records are referenced via previous_hash;
recommend exporting a signed handoff zip and using vaara trail verify on that
bundle, and explicitly state that the live DB chain break at the retention
boundary is expected and not validated by vaara trail verify.

In `@scripts/eval_distribution_shift.py`:
- Around line 69-73: The call to pipe.intercept is collapsing missing agent_id
values to the constant "adv", causing unrelated samples to share history; change
the fallback for agent_id in the pipe.intercept call (where
agent_id=entry.get("agent_id", "adv")) to produce a unique per-sample id (for
example derive from an existing entry unique field, use the loop index, or
generate a uuid) so each sample without an explicit agent_id gets its own
distinct identifier instead of "adv".

In `@scripts/eval_pair_attack.py`:
- Around line 161-163: The --model argument currently defaults to "llama" which
can cause silent mismatches; update the ap.add_argument for "--model" (the
parser setup in scripts/eval_pair_attack.py) to make the served model explicit
by either setting required=True or changing the default to the documented
"Qwen2.5-32B-Instruct", and update the help string to mention it must match the
vLLM --served-model-name to ensure reproducible evaluations.
- Around line 173-181: The ImportError branch currently falls back silently to
heuristic-only behavior, making published PAIR runs non-reproducible; change the
logic around importing AdversarialClassifier so that if AdversarialClassifier
cannot be imported or instantiation fails, the script aborts with a clear
error/exit (instead of setting classifier = None) and a message instructing the
user to install vaara[ml]; update the block that currently mentions
AdversarialClassifier and classifier (and the subsequent Pipeline usage) to
validate classifier is available before proceeding and exit/raise with a
descriptive error if not.

In `@scripts/eval_stack_ablation.py`:
- Around line 95-102: full_stack currently re-invokes heuristic_only(entry,
pipe) which can mutate sequence/agent-history state and make comparisons
invalid; change full_stack so it does NOT call heuristic_only again — instead
either accept the heuristic result as an extra parameter (e.g., def
full_stack(entry: dict, pipe: Pipeline, classifier, heuristic_result: str) ->
str) or ensure the caller computes h = heuristic_only(...) once and passes it
in; then use that h and only call classifier_only(entry, classifier) when h ==
"ALLOW". Apply the same fix to the other occurrence referenced (the block around
lines 121-125) so heuristic_only is computed once and reused rather than re-run.

In `@src/vaara/audit/sqlite_backend.py`:
- Around line 137-142: The migration logic in _run_migrations() must not
unconditionally set schema_version to 3 after any exception during the ALTER
TABLE block; change it so that the ALTERs for adding system_operation,
data_usage, decision_making, and limitations are executed atomically (either run
them inside a single transaction or run each ALTER and only mark success after
all four succeed) and ensure that on any exception you do not update
schema_version and instead surface the error (or return a failure) so
write_record() won't assume the columns exist; update _run_migrations() to only
update schema_version when all ALTERs complete without error and reference the
ALTER statements and the schema_version update in that function to implement
this behavior.

In `@src/vaara/cli.py`:
- Around line 199-201: The purge command currently instantiates
SQLiteAuditBackend(db_path) without a tenant, causing purge_older_than to act
globally; update the CLI parser for "vaara trail purge" to accept a tenant
override (e.g., --tenant or --tenant-id) and pass that tenant into the audit
backend instantiation or directly into purge_older_than so the tenant-scoped
clause is used; specifically modify the parser code around the purge command in
src/vaara/cli.py and change the backend construction
(SQLiteAuditBackend(db_path) -> SQLiteAuditBackend(db_path,
tenant_id=parsed_tenant)) or call purge_older_than(retention_seconds,
dry_run=args.dry_run, tenant_id=args.tenant) so the _tenant_clause() in
src/vaara/audit/sqlite_backend.py no longer falls back to "1=1" and only deletes
records for the requested tenant (also apply the same tenant wiring to the other
purge-related block around lines 258-274).

In `@src/vaara/policy/loader.py`:
- Around line 182-201: When reading input paths in the loader functions (e.g.
from_json and from_yaml) the code only converts FileNotFoundError to PolicyError
but other read_text failures (IsADirectoryError, PermissionError,
UnicodeDecodeError, generic OSError) currently leak; update the Path.read_text()
exception handlers to catch and normalize all file-read errors (catch a tuple
like (FileNotFoundError, IsADirectoryError, PermissionError, UnicodeDecodeError,
OSError) or simply OSError/Exception as appropriate) and raise
PolicyError(f"input is neither a JSON/YAML object nor a readable file path:
{source!r}") (or a clear message used elsewhere) from the original exception; do
the same normalization in both the JSON loader block around Path.read_text() and
the YAML loader block so callers never see raw OS/file exceptions (keep JSON
decoding still wrapped as PolicyError when json.loads fails).
- Around line 68-95: The loader currently accepts arbitrary keys under
thresholds.default and thresholds.<name> (in thr_block) and silently coerces
unknown keys; update the code that builds thresholds_default and
thresholds_overrides to validate allowed keys only ("escalate" and "deny") and
raise a PolicyError for any unknown key (include the path like
"thresholds.default: unknown key 'deni'" or "thresholds.<name>: unknown key
'denial'"). Specifically, in the handling of default_block and in the loop that
processes thr_block items (where thresholds_default, thresholds_overrides,
Thresholds(...), thr_block, and _require_mapping are used), check the
set(override.keys()) (and set(default_block.keys()) for default) against
{"escalate","deny"} before float coercion and raise PolicyError with a clear
message if extras are present.

In `@tests/adversarial/pair_v0_5_3.json`:
- Around line 6-8: The fixture metadata currently stores an ambiguous "model"
value ("qwen32b"); update tests/adversarial/pair_v0_5_3.json to record the full,
unambiguous attacker and judge model identifiers and the attacker-vs-judge
pairing so runs are reproducible: replace the single "model" field with explicit
keys (e.g., "attacker_model" and "judge_model" or a single "model_pair" string)
and ensure each entry in the "results" array references those full IDs so the
attacker/judge setup is preserved in the fixture metadata.

---

Duplicate comments:
In `@scripts/eval_stack_ablation.py`:
- Around line 168-173: The instantiation of AdversarialClassifier can raise
ImportError and must be inside the same try/except as the import; move the call
to AdversarialClassifier() into the try block that imports
vaara.adversarial_classifier and catch ImportError as exc, then raise SystemExit
with a clear message including exc (i.e., wrap both "from
vaara.adversarial_classifier import AdversarialClassifier" and "classifier =
AdversarialClassifier()" in the same try and keep the existing
print(f"[classifier] bundle v{classifier.bundle_version},
threshold={classifier.threshold}") after the try).
- Around line 62-65: The function expected_class currently defaults
missing/malformed expected values to "DENY" and silently classifies them as
"attack"; update expected_class to validate the entry["expected"] value(s)
(accepting either a string or list of strings, case-insensitive) and raise a
clear exception when expected is missing or contains any value outside the
allowed set {"ALLOW","DENY"} instead of defaulting, then keep the existing
bucketing logic (treat {"ALLOW"} -> "benign", otherwise "attack"); refer to the
expected_class function and the local variable expected/entry to implement this
validation and error handling.
- Around line 133-155: In report(), before printing or writing metrics, detect
any recorded errors by scanning counts (for each config in
("heuristic_only","classifier_only","full_stack") and each bucket c in
counts[config]) and if any c["err"] > 0 abort the run (raise SystemExit or call
sys.exit with non‑zero) after printing a short diagnostic message; do this check
early in the function (using the existing counts, config_name, c and err keys)
so you never print or write metrics to json_out when there were ERROR:*
decisions.

---

Nitpick comments:
In `@tests/test_audit_purge.py`:
- Around line 17-31: The helper _make_record builds AuditRecord objects but
computes record_hash with an empty previous_hash, breaking any chain when tests
call backend.write_record() directly; instead seed tests with a proper chain by
either creating/appending records through AuditTrail (use
AuditTrail.append_record or whatever method ensures previous_hash is set) or, if
creating records manually, set each record.previous_hash to the prior
record.record_hash before calling rec.record_hash = rec.compute_hash() so the
linked hash chain is intact for purge boundary assertions (refer to AuditRecord,
compute_hash, previous_hash, record_hash, backend.write_record, and AuditTrail).

In `@tests/test_transparency_taxonomy.py`:
- Around line 128-167: The test currently doesn't prove the v2→v3 migration
actually added the four transparency columns; after opening the DB with
SQLiteAuditBackend in test_migration_from_v2_to_v3, query the migrated schema
(e.g., PRAGMA table_info('audit_records')) and assert that columns
'system_operation', 'data_usage', 'decision_making', and 'limitations' are
present, or perform a post-migration write/read round-trip via
SQLiteAuditBackend (insert a record with those fields set, then reload it) to
fail if the columns were never created; reference the test function
test_migration_from_v2_to_v3 and the SQLiteAuditBackend/_row_to_record code
paths so the assertion validates the actual schema change rather than relying on
None defaults.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 15c3473a-5cc9-4516-a1b8-7d0a4005b051

📥 Commits

Reviewing files that changed from the base of the PR and between 45c3ee3 and e4d8d69.

📒 Files selected for processing (22)

• .gitignore
• CHANGELOG.md
• COMPLIANCE.md
• examples/policies/full.yaml
• examples/policies/minimal.json
• pyproject.toml
• scripts/eval_distribution_shift.py
• scripts/eval_pair_attack.py
• scripts/eval_stack_ablation.py
• src/vaara/__init__.py
• src/vaara/audit/sqlite_backend.py
• src/vaara/audit/trail.py
• src/vaara/cli.py
• src/vaara/policy/__init__.py
• src/vaara/policy/loader.py
• src/vaara/policy/schema.py
• tests/adversarial/distribution_shift_v0_5_3.json
• tests/adversarial/pair_v0_5_3.json
• tests/adversarial/stack_ablation_v0_5_3.json
• tests/test_audit_purge.py
• tests/test_policy.py
• tests/test_transparency_taxonomy.py