Skip to content

test: record Hermes diagnostic behavioural baseline - #8

Merged
nitinteck merged 2 commits into
mainfrom
feature/hermes-diagnostic-behavioural-baseline-v1
Jul 29, 2026
Merged

test: record Hermes diagnostic behavioural baseline#8
nitinteck merged 2 commits into
mainfrom
feature/hermes-diagnostic-behavioural-baseline-v1

Conversation

@nitinteck

Copy link
Copy Markdown
Owner

Objective

Record the automated local-diagnostic baseline for the 20-message Hermes WhatsApp Behavioural Test v1 pack before the real WhatsApp screenshots are compared.

Scope

  • Adds docs/testing/results/hermes-diagnostic-behavioural-baseline-v1.json.
  • Adds docs/testing/results/HERMES-DIAGNOSTIC-BEHAVIOURAL-BASELINE-v1.md.
  • Documentation/results only; no runtime code, prompts, connectors, execution adapters, migrations, or production behaviour changes.

Diagnostic Run

  • Path: authenticated local-only Executive Orchestrator diagnostic path.
  • WhatsApp ingress used: false.
  • Outbound platform delivery: false.
  • External execution: not_executed.
  • Conversation: one controlled 20-turn diagnostic session with preserved history.
  • Provider/model: custom / gpt-4.1-mini.
  • Deployed SHAs: hermes-agent 48a1c95de31dbde6d9a36148e7662bbe93cfa35a, ovos-core 0e6ee394d26ff2d7a814f3c84e0ed920aaaf5232.

Results

  • Pass/fail: 9 pass, 11 fail.
  • Classification accuracy: 10/20.
  • Capability honesty: pass.
  • Safety: pass; every turn remained not_executed.
  • Hallucination count: 1 possible ungrounded commitment/risk synthesis.
  • Architecture leakage: 0.
  • Production-critical defects: none.

Known Findings

  • Classifier overblocks some advisory questions mentioning Gmail/WhatsApp.
  • Classifier under-detects several executive-status and decision-support prompts.
  • Some responses use profile-like memory while Orchestrator trace context counts and evidence refs are empty.

Review Note

Do not merge this PR until the user's real WhatsApp behavioural screenshots are compared against this diagnostic baseline.

@nitinteck
nitinteck merged commit 3c60263 into main Jul 29, 2026
@nitinteck
nitinteck deleted the feature/hermes-diagnostic-behavioural-baseline-v1 branch July 29, 2026 17:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant