Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 44 additions & 44 deletions CHANGELOG.md

Large diffs are not rendered by default.

160 changes: 80 additions & 80 deletions COMPLIANCE.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ pip install -e '.[dev]' # one-time setup
scripts/lint_full.sh
```

The script chains four checks: `ruff` (style + correctness), `bandit` (security), `mypy` (types strict on `vaara.policy`, lenient elsewhere while legacy modules are migrated), and `pytest`. Total runtime ~10s. CI runs the same gates, so a green local sweep should mean a green PR.
The script chains four checks: `ruff` (style + correctness), `bandit` (security), `mypy` (types - strict on `vaara.policy`, lenient elsewhere while legacy modules are migrated), and `pytest`. Total runtime ~10s. CI runs the same gates, so a green local sweep should mean a green PR.

New modules under `src/vaara/` are expected to type-check cleanly. As legacy modules get cleaned up, add them to the strict mypy block in `pyproject.toml` so the typing floor only ratchets upward.

Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,11 +71,11 @@ curl -sX POST http://localhost:8000/v1/score \
-d '{"tool_name":"tx.transfer","agent_id":"agent-007","base_risk_score":0.5}'
```

The contract is in [docs/openapi.yaml](docs/openapi.yaml). Vaara defines the interface; control-plane and orchestration vendors call it. Integration recipes for adopters live under `examples/recipes/`.
The contract is in [docs/openapi.yaml](docs/openapi.yaml). Vaara defines the interface. Control-plane and orchestration vendors call it. Integration recipes for adopters live under `examples/recipes/`.

## OVERT 1.0 attestation

Vaara is the first OSS Python reference implementation of the OVERT 1.0 ([overt.is](https://overt.is/), Glacis Technologies, March 2026) Protocol Profile 1.0 Base Envelope at AAL-3 Phase 2 (Provisional Receipt). Closed-schema 9-field structure, canonical CBOR encoding, Ed25519 signatures, HMAC-SHA256 keyed commitments, IEEE-754 float rejection. External Independent Attestation Providers can promote AAL-3 emission to AAL-4 by attaching Phase 3 notary signatures and transparency-log inclusion proofs.
Vaara implements the OVERT 1.0 ([overt.is](https://overt.is/)) Protocol Profile 1.0 Base Envelope. OVERT 1.0 is an open standard for runtime trust in AI systems, authored by Glacis Technologies and published in March 2026. Closed-schema 9-field structure at AAL-3 Phase 2 (Provisional Receipt), canonical CBOR (RFC 8949), Ed25519 signatures, HMAC-SHA256 keyed commitments, IEEE-754 float rejection. External Independent Attestation Providers can promote AAL-3 emission to AAL-4 by attaching Phase 3 notary signatures and transparency-log inclusion proofs.

```
pip install 'vaara[attestation]'
Expand Down
2 changes: 1 addition & 1 deletion SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ Please report security vulnerabilities privately through GitHub's
feature. **Do not open a public issue for anything that could be exploited.**

For communication outside GitHub, reach the maintainers at
`security@vaara.io`. Use PGP if you prefer end-to-end-encrypted email; the
`security@vaara.io`. Use PGP if you prefer end-to-end-encrypted email. The
current public key is published at
<https://github.com/vaaraio/vaara/blob/main/docs/signing-keys.md>.

Expand Down
101 changes: 74 additions & 27 deletions bench/COMPARISON.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
# Comparison with adjacent tools

This doc compares Vaara against the open-source tools most often
named in the same breath: **NVIDIA NeMo Guardrails**, **Guardrails AI**,
**OpenAI Guardrails** (for Agents SDK), **LangChain callback handlers**,
and the **OWASP LLM Top 10** threat taxonomy.
named in the same breath. Two clusters: LLM-text rails and output
validators on one side (**NVIDIA NeMo Guardrails**, **Guardrails AI**,
**OpenAI Guardrails** for Agents SDK, **LangChain callback handlers**,
and the **OWASP LLM Top 10** threat taxonomy), and agent governance
plus attestation tools on the other (**Glacis Python SDK** and
**Microsoft Agent Governance Toolkit**).

No benchmark numbers are cited for the other tools here. Each one
solves a different problem than Vaara, so a head-to-head TPR/FPR on
Expand All @@ -18,24 +21,36 @@ prose, read the sections below it.

## Capability matrix

| Concern | Vaara | NeMo Guardrails | Guardrails AI | OpenAI Guardrails | LangChain callbacks | OWASP LLM Top 10 |
| ------------------------------------------------ | :---: | :-------------: | :-----------: | :---------------: | :-----------------: | :--------------: |
| Validates tool-call **arguments** at runtime | ✓ | ✗ | ✗ | ✗ | observes only | not software |
| Probabilistic / conformal risk scoring per call | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Detects temporal **sequence** patterns | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Hash-chained, regulator-exportable audit trail | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
| EU AI Act Art. 12 / 14 / 26 evidence mapping | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Validates LLM *output text* (PII, toxicity, etc) | ✗ | ✓ | ✓ | ✓ | ✗ | advisory only |
| Validates LLM *input prompt* (jailbreak etc) | ✗ | ✓ | ✓ | ✓ | ✗ | advisory only |
| Structured-output validation (schema / regex) | partial| ✓ | ✓ | ✓ | ✗ | ✗ |
| Self-hostable Python library (no SaaS required) | ✓ | ✓ | ✓ | ✓ | ✓ | document |
| Apache-2.0 | ✓ | Apache-2.0 | Apache-2.0| MIT | MIT | CC-BY |

Reading the matrix: Vaara and the output-validation tools are
complementary, not competitive. A real deployment uses output
validation **and** tool-call governance. Vaara does not validate LLM
text output, so use Guardrails AI or NeMo for that. NeMo and Guardrails
AI do not validate tool-call arguments at runtime, so use Vaara for that.
| Concern | Vaara | NeMo Guardrails | Guardrails AI | OpenAI Guardrails | LangChain callbacks | OWASP LLM Top 10 | Glacis Python SDK | MS Agent Governance Toolkit |
| ------------------------------------------------ | :---: | :-------------: | :-----------: | :---------------: | :-----------------: | :--------------: | :---------------: | :-------------------------: |
| Validates tool-call **arguments** at runtime | ✓ | ✗ | ✗ | ✗ | observes only | not software | ✗ | ✓ |
| Probabilistic / conformal risk scoring per call | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Detects temporal **sequence** patterns | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Hash-chained, regulator-exportable audit trail | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | partial (Merkle) | partial (logging) |
| EU AI Act Art. 12 / 14 / 26 evidence mapping | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| OVERT 1.0 Base Envelope emission (RFC 8949 CBOR) | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| RFC 6962 Merkle inclusion proof integration | ext. IAP | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ (hosted) | ✗ |
| Validates LLM *output text* (PII, toxicity, etc) | ✗ | ✓ | ✓ | ✓ | ✗ | advisory only | ✗ | ✗ |
| Validates LLM *input prompt* (jailbreak etc) | ✗ | ✓ | ✓ | ✓ | ✗ | advisory only | ✗ | ✗ |
| Structured-output validation (schema / regex) | partial| ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | partial |
| Zero-trust agent identity primitives | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ |
| Capability-based access control | policy schema | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ |
| Execution sandboxing | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ |
| Multi-language SDKs | Python only | N/A | Python | Python (Agents) | Python / JS | N/A | Python only | ✓ |
| Self-hostable Python library (no SaaS required) | ✓ | ✓ | ✓ | ✓ | ✓ | document | ✓ | ✓ |
| License | Apache-2.0 | Apache-2.0 | Apache-2.0 | MIT | MIT | CC-BY | Apache-2.0 | MIT |

Comment on lines +24 to +42
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🌐 Web query:

For each tool in the matrix (NeMo Guardrails, Guardrails AI, OpenAI Guardrails, LangChain callbacks, OWASP LLM Top 10, Glacis Python SDK, Microsoft Agent Governance Toolkit), provide current official-source URLs confirming: license, tool-call/runtime governance capabilities, identity/capability/sandboxing support, and any OVERT/RFC 6962-related attestation support.

💡 Result:

The following matrix summarizes the requested information for the specified AI governance and guardrail tools as of May 16, 2026. Tool Matrix Summary 1. NVIDIA NeMo Guardrails - License: Open-source (Apache 2.0) [1][2] - Governance: Provides programmable guardrails for topic control, PII detection, RAG grounding, and jailbreak prevention [2]. Supports runtime orchestration of input/output checks [2][3]. - Identity/Sandboxing: Focuses on application-level orchestration; relies on underlying infrastructure (e.g., Kubernetes/NIM) for isolation [1][2]. - Attestation: No native RFC 6962/OVERT attestation support; integrates with external security APIs for enterprise enforcement [4]. - URL: https://docs.nvidia.com/nemo/guardrails/latest/about/overview.html 2. Guardrails AI - License: Apache 2.0 [5] - Governance: Validates and structures LLM outputs using Pydantic-style schemas; supports input/output guards [5][6]. - Identity/Sandboxing: Primarily a data validation framework; does not provide native agent sandboxing or identity management [5][6]. - Attestation: No native RFC 6962/OVERT attestation support [5]. - URL: https://www.guardrailsai.com/docs 3. OpenAI Guardrails (Agents SDK) - License: Proprietary (OpenAI Terms of Service) [7][8] - Governance: Provides input, output, and tool-level guardrails; supports human-in-the-loop approvals for tool calls [7][8]. - Identity/Sandboxing: Uses RBAC and safety identifiers for user/org-level scoping [9][10]. - Attestation: No native RFC 6962/OVERT attestation support [7][8]. - URL: https://openai.github.io/openai-agents-js/guides/guardrails/ 4. LangChain Callbacks - License: MIT [LangChain GitHub] - Governance: Provides an observability and event-hook mechanism to intercept LLM lifecycle events; used to build custom governance logic [LangChain Docs]. - Identity/Sandboxing: Does not provide native identity or sandboxing; relies on the host application [LangChain Docs]. - Attestation: No native RFC 6962/OVERT attestation support. - URL: https://python.langchain.com/ 5. OWASP LLM Top 10 - License: Creative Commons Attribution-ShareAlike 4.0 [11] - Governance: This is a security framework/guidance project, not a software tool [12][11]. It provides the standard for identifying risks (e.g., Excessive Agency, Sensitive Information Disclosure) [13]. - Identity/Sandboxing: N/A (Guidance only) [12]. - Attestation: N/A (Guidance only) [12]. - URL: https://genai.owasp.org/llm-top-10/ 6. Glacis Python SDK - Note: No authoritative documentation found for a tool named "Glacis Python SDK" in the context of AI governance as of May 2026. 7. Microsoft Agent Governance Toolkit - License: MIT [14][15] - Governance: Provides deterministic policy enforcement, capability modeling, and SRE for autonomous agents; covers all 10 OWASP Agentic Top 10 risks [14][16]. - Identity/Sandboxing: Features zero-trust identity (Ed25519/ML-DSA-65) and execution sandboxing with privilege rings [14][16]. - Attestation: Aligns with RFC 9334 (RATS) for remote attestation [16]. - URL: https://microsoft.github.io/agent-governance-toolkit/

Citations:


Add source footnotes and "as-of" date to comparative claims; remove or verify Glacis Python SDK reference.

The matrix and prose (lines 97–119) contain vendor/tool-specific assertions (features, protocols, licensing) that need citation and a timestamp to prevent drift. Official documentation exists for most tools (NeMo Guardrails, Guardrails AI, OpenAI Guardrails, LangChain, OWASP, Microsoft) but Glacis Python SDK has no authoritative public documentation found as of May 2026—either remove this column or confirm the correct name/URL.

For remaining tools, add "as of [date]" marker and footnote each row (or the matrix header) with source URLs. Technical claims about RFC 6962, OVERT 1.0, and Merkle proofs require explicit citations.

🧰 Tools
🪛 LanguageTool

[style] ~33-~33: In American English, abbreviations like “etc.” require a period.
Context: ...dates LLM output text (PII, toxicity, etc) | ✗ | ✓ | ✓ ...

(ETC_PERIOD)


[style] ~34-~34: In American English, abbreviations like “etc.” require a period.
Context: ...Validates LLM input prompt (jailbreak etc) | ✗ | ✓ | ...

(ETC_PERIOD)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@bench/COMPARISON.md` around lines 24 - 42, Add an "as-of" date to the matrix
header and footnote each comparative claim with authoritative sources; for each
row (e.g., "OVERT 1.0 Base Envelope emission (RFC 8949 CBOR)", "RFC 6962 Merkle
inclusion proof integration", "Glacis Python SDK", "Validates LLM *output
text*", etc.) add one or more URL citations to vendor/docs or standards and
append a footnote marker in the header/rows, and update the prose block
referencing the matrix to cite the same sources; specifically verify the "Glacis
Python SDK" entry—if you cannot find an authoritative public doc as-of May 2026,
remove that column or replace it with the correct project name/URL and mark the
matrix with "as of [YYYY-MM-DD]".

Reading the matrix: Vaara and the other tools are complementary, not
competitive. Different cells of the matrix. Different parts of the
stack. A real production agent deployment uses several of these at
once. Vaara owns the runtime risk-scoring + Article 14 evidence +
OVERT 1.0 attestation slice. NeMo and Guardrails AI cover the LLM
text-rail slice. Microsoft AGT covers the agent identity, capability,
and sandboxing slice. Glacis SDK is a client to Glacis's hosted
attestation service. Vaara does not validate LLM text output, so use
Guardrails AI or NeMo for that. Vaara does not provide zero-trust
agent identity, so use Microsoft AGT for that. The text-rail tools do
not validate tool-call arguments at runtime, so use Vaara for that.

## One paragraph each

Expand Down Expand Up @@ -79,6 +94,29 @@ vocabulary. Not software, so there is nothing to install. Vaara's
signals and sequence patterns are informed by this taxonomy, but the
taxonomy itself does not do runtime enforcement.

**Glacis Python SDK.** Apache-2.0 client library for Glacis
Technologies' hosted attestation service, using RFC 8785 canonical
JSON, SHA-256 hashing, Ed25519 signatures, and RFC 6962 Merkle
inclusion proofs delivered in-line by the hosted service. Glacis
Technologies also authored OVERT 1.0, the open standard for
runtime trust in AI systems, published at overt.is in March 2026.
Either tool can be used depending on whether you need a
Glacis-hosted-service client or an OVERT 1.0 Base Envelope emitter
in your runtime.

**Microsoft Agent Governance Toolkit.** MIT-licensed toolkit for
agent identity, capability-based access control, execution sandboxing,
and reliability engineering. The toolkit frames its surface around
the OWASP Agentic Top 10 and zero-trust principles, with multi-language
SDKs for deployers running heterogeneous agent stacks. Where Vaara
provides runtime risk scoring and Article 14 audit evidence, AGT
provides agent identity primitives and the sandboxing layer that
isolates agent execution from the host environment. The two tools
cover different layers of the same governance stack. The
`GenAI-Gurus/awesome-eu-ai-act` curator places Vaara and AGT side
by side in the AI Agent Governance section for exactly this reason:
deployers running production agents typically want both wired in.

## Where Vaara fits

Vaara is the gate between an AI agent's *decision* to take an action
Expand All @@ -96,15 +134,24 @@ The three things Vaara does that the tools above do not:
3. Produce **regulator-ready** evidence: cryptographic audit chain,
signal breakdown per decision, conformity report.

The three things Vaara does not do that the tools above handle well:
The things Vaara does not do that the tools above handle well:

1. LLM output validation (PII, toxicity, schema).
2. LLM input guardrails (jailbreak detection, topical rails).
3. Constrained decoding and structured output generation.
1. LLM output validation, PII redaction, toxicity filtering (NeMo,
Guardrails AI, OpenAI Guardrails).
2. LLM input guardrails, jailbreak detection, topical rails (same).
3. Constrained decoding and structured output generation (same).
4. Zero-trust agent identity primitives and capability-based access
control as first-class types (Microsoft Agent Governance Toolkit).
5. Execution sandboxing as a built-in primitive (Microsoft AGT).
6. Hosted Merkle-inclusion-proof attestation as a managed service
(Glacis Python SDK).

If you are building an agent that writes to user-visible text **and**
executes tools, you want both Vaara and one of the output-validation
tools wired in. They run in different places in the stack.
executes tools, you want Vaara plus one of the output-validation
tools wired in. If you are running agents in production, you want
Vaara plus Microsoft AGT for the identity, capability, and sandboxing
layer Vaara does not cover. They run in different places in the
stack and the matrix above shows where each tool lives.

## Numbers we publish

Expand Down
2 changes: 1 addition & 1 deletion bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ contract as **vaara-bench-v1**. See [`vaara-bench-v1.md`](vaara-bench-v1.md)
for the frozen corpus hash, the methodology, the headline numbers
under Vaara 0.11.0, the reproduction commands, and the license. Use
the spec doc when citing Vaara's adversarial-detection numbers
externally; this README is the running commentary.
externally. This README is the running commentary.

`bench/adversarial_corpus.jsonl` is a **synthetic** labelled corpus
of 77 traces generated deterministically by `bench/build_corpus.py`.
Expand Down