From 040640295e3e1112d954b83c917180bd73bd81ec Mon Sep 17 00:00:00 2001 From: mnajafian-nv Date: Thu, 4 Jun 2026 02:20:47 -0700 Subject: [PATCH 1/4] fix: tighten Hermes wrapped observability contract Signed-off-by: mnajafian-nv --- ATTRIBUTIONS-Rust.md | 202 +++--------------- crates/cli/src/launcher.rs | 24 ++- crates/cli/tests/cli_tests.rs | 89 ++++++++ crates/cli/tests/coverage/launcher_tests.rs | 46 ++++ crates/cli/tests/coverage/server_tests.rs | 200 +++++++++++++++++ .../unit/observability/openinference_tests.rs | 99 +++++++++ docs/nemo-relay-cli/hermes.mdx | 58 +++-- patches/hermes-agent/notes.md | 20 +- third_party/README-hermes-agent.md | 7 + 9 files changed, 545 insertions(+), 200 deletions(-) diff --git a/ATTRIBUTIONS-Rust.md b/ATTRIBUTIONS-Rust.md index 38dd9222b..30290863b 100644 --- a/ATTRIBUTIONS-Rust.md +++ b/ATTRIBUTIONS-Rust.md @@ -3123,9 +3123,8 @@ limitations under the License. ## block-buffer - 0.10.4 **Repository URL**: https://github.com/RustCrypto/utils -**License Type(s)**: MIT OR Apache-2.0 -### License: https://spdx.org/licenses/ -### License File: LICENSE-APACHE +**License Type(s)**: Apache-2.0 +### License: https://spdx.org/licenses/Apache-2.0.html ``` Apache License Version 2.0, January 2004 @@ -3328,35 +3327,7 @@ distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -``` - -### License File: LICENSE-MIT -``` -Copyright (c) 2018-2019 The RustCrypto Project Developers - -Permission is hereby granted, free of charge, to any -person obtaining a copy of this software and associated -documentation files (the "Software"), to deal in the -Software without restriction, including without -limitation the rights to use, copy, modify, merge, -publish, distribute, sublicense, and/or sell copies of -the Software, and to permit persons to whom the Software -is furnished to do so, subject to the following -conditions: - -The above copyright notice and this permission notice -shall be included in all copies or substantial portions -of the Software. -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF -ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED -TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A -PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT -SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY -CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION -OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR -IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER -DEALINGS IN THE SOFTWARE. ``` ## block-buffer - 0.12.0 @@ -7927,9 +7898,8 @@ limitations under the License. ## crypto-common - 0.1.7 **Repository URL**: https://github.com/RustCrypto/traits -**License Type(s)**: MIT OR Apache-2.0 -### License: https://spdx.org/licenses/ -### License File: LICENSE-APACHE +**License Type(s)**: Apache-2.0 +### License: https://spdx.org/licenses/Apache-2.0.html ``` Apache License Version 2.0, January 2004 @@ -8132,35 +8102,7 @@ distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -``` -### License File: LICENSE-MIT -``` -Copyright (c) 2021 RustCrypto Developers - -Permission is hereby granted, free of charge, to any -person obtaining a copy of this software and associated -documentation files (the "Software"), to deal in the -Software without restriction, including without -limitation the rights to use, copy, modify, merge, -publish, distribute, sublicense, and/or sell copies of -the Software, and to permit persons to whom the Software -is furnished to do so, subject to the following -conditions: - -The above copyright notice and this permission notice -shall be included in all copies or substantial portions -of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF -ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED -TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A -PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT -SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY -CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION -OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR -IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER -DEALINGS IN THE SOFTWARE. ``` ## crypto-common - 0.2.1 @@ -8613,9 +8555,8 @@ SOFTWARE. ## digest - 0.10.7 **Repository URL**: https://github.com/RustCrypto/traits -**License Type(s)**: MIT OR Apache-2.0 -### License: https://spdx.org/licenses/ -### License File: LICENSE-APACHE +**License Type(s)**: Apache-2.0 +### License: https://spdx.org/licenses/Apache-2.0.html ``` Apache License Version 2.0, January 2004 @@ -8818,35 +8759,7 @@ distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -``` - -### License File: LICENSE-MIT -``` -Copyright (c) 2017 Artyom Pavlov - -Permission is hereby granted, free of charge, to any -person obtaining a copy of this software and associated -documentation files (the "Software"), to deal in the -Software without restriction, including without -limitation the rights to use, copy, modify, merge, -publish, distribute, sublicense, and/or sell copies of -the Software, and to permit persons to whom the Software -is furnished to do so, subject to the following -conditions: -The above copyright notice and this permission notice -shall be included in all copies or substantial portions -of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF -ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED -TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A -PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT -SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY -CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION -OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR -IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER -DEALINGS IN THE SOFTWARE. ``` ## digest - 0.11.2 @@ -13360,28 +13273,27 @@ limitations under the License. **Repository URL**: https://github.com/fizyk20/generic-array.git **License Type(s)**: MIT ### License: https://spdx.org/licenses/MIT.html -### License File: LICENSE ``` -The MIT License (MIT) - -Copyright (c) 2015 Bartłomiej Kamiński - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to deal -in the Software without restriction, including without limitation the rights -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in all -copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +The MIT License (MIT) + +Copyright (c) 2015 Bartłomiej Kamiński + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ``` @@ -20430,9 +20342,8 @@ SOFTWARE. ## md-5 - 0.10.6 **Repository URL**: https://github.com/RustCrypto/hashes -**License Type(s)**: MIT OR Apache-2.0 -### License: https://spdx.org/licenses/ -### License File: LICENSE-APACHE +**License Type(s)**: Apache-2.0 +### License: https://spdx.org/licenses/Apache-2.0.html ``` Apache License Version 2.0, January 2004 @@ -20635,37 +20546,7 @@ distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -``` -### License File: LICENSE-MIT -``` -Copyright (c) 2006-2009 Graydon Hoare -Copyright (c) 2009-2013 Mozilla Foundation -Copyright (c) 2016 Artyom Pavlov - -Permission is hereby granted, free of charge, to any -person obtaining a copy of this software and associated -documentation files (the "Software"), to deal in the -Software without restriction, including without -limitation the rights to use, copy, modify, merge, -publish, distribute, sublicense, and/or sell copies of -the Software, and to permit persons to whom the Software -is furnished to do so, subject to the following -conditions: - -The above copyright notice and this permission notice -shall be included in all copies or substantial portions -of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF -ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED -TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A -PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT -SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY -CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION -OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR -IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER -DEALINGS IN THE SOFTWARE. ``` ## memchr - 2.8.0 @@ -38917,9 +38798,8 @@ limitations under the License. ## version_check - 0.9.5 **Repository URL**: https://github.com/SergioBenitez/version_check -**License Type(s)**: MIT/Apache-2.0 -### License: https://spdx.org/licenses/ -### License File: LICENSE-APACHE +**License Type(s)**: Apache-2.0 +### License: https://spdx.org/licenses/Apache-2.0.html ``` Apache License Version 2.0, January 2004 @@ -39122,29 +39002,7 @@ distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -``` - -### License File: LICENSE-MIT -``` -The MIT License (MIT) -Copyright (c) 2017-2018 Sergio Benitez -Permission is hereby granted, free of charge, to any person obtaining a copy of -this software and associated documentation files (the "Software"), to deal in -the Software without restriction, including without limitation the rights to -use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of -the Software, and to permit persons to whom the Software is furnished to do so, -subject to the following conditions: - -The above copyright notice and this permission notice shall be included in all -copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS -FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR -COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER -IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN -CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ``` ## walkdir - 2.5.0 diff --git a/crates/cli/src/launcher.rs b/crates/cli/src/launcher.rs index acf5d268a..1e8d4115a 100644 --- a/crates/cli/src/launcher.rs +++ b/crates/cli/src/launcher.rs @@ -87,24 +87,29 @@ struct TransparentRun { agent: CodingAgent, prepared: PreparedRun, resolved: ResolvedConfig, - listener: TcpListener, + listener: Option, gateway_url: String, dry_run: bool, print: bool, } impl TransparentRun { - // Resolves configuration, binds the ephemeral listener, and builds agent-specific launch wiring - // without starting the gateway or spawning the child command. + // Resolves configuration and builds agent-specific launch wiring. Live runs also bind the + // ephemeral listener here so health checks and agent preparation share one resolved gateway + // address; dry runs stay side-effect free and use a placeholder dynamic URL for inspection. async fn new(command: RunCommand, inherited: Option<&ServerArgs>) -> Result { let dry_run = command.dry_run; let print = command.print; let mut resolved = resolve_run_config(&command, inherited)?; let (agent, argv) = resolve_agent_and_argv(&command, &resolved.agents)?; - let listener = TcpListener::bind("127.0.0.1:0").await?; - let address = listener.local_addr()?; - let gateway_url = format!("http://{address}"); - resolved.gateway.bind = address; + let (listener, gateway_url) = if dry_run { + (None, "http://127.0.0.1:".to_string()) + } else { + let listener = TcpListener::bind("127.0.0.1:0").await?; + let address = listener.local_addr()?; + resolved.gateway.bind = address; + (Some(listener), format!("http://{address}")) + }; let prepared = PreparedRun::new(agent, argv, &gateway_url, &resolved, dry_run)?; Ok(Self { @@ -132,10 +137,13 @@ impl TransparentRun { if self.dry_run { return Ok(ExitCode::SUCCESS); } + let listener = self + .listener + .expect("live transparent runs must allocate a gateway listener"); self.prepared .print_live_status(self.agent, &self.gateway_url, &self.resolved); execute_live_run( - self.listener, + listener, self.resolved.gateway, &self.gateway_url, self.prepared, diff --git a/crates/cli/tests/cli_tests.rs b/crates/cli/tests/cli_tests.rs index a3533022e..3fd74cbec 100644 --- a/crates/cli/tests/cli_tests.rs +++ b/crates/cli/tests/cli_tests.rs @@ -141,6 +141,33 @@ fn cli_easy_path_invokes_setup_when_no_config_found() { ); } +#[test] +fn cli_hermes_easy_path_invokes_setup_when_no_config_found() { + let temp = tempfile::tempdir().unwrap(); + let xdg = temp.path().join("xdg"); + std::fs::create_dir_all(&xdg).unwrap(); + let cwd = temp.path().join("workdir"); + std::fs::create_dir_all(&cwd).unwrap(); + + let output = Command::new(gateway_bin()) + .current_dir(&cwd) + .env("XDG_CONFIG_HOME", &xdg) + .env("HOME", temp.path()) + .arg("hermes") + .output() + .unwrap(); + + assert!( + !output.status.success(), + "Hermes easy path should exit non-zero when no config + no TTY for setup" + ); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!( + stderr.contains("setup requires a TTY"), + "expected non-TTY setup error in stderr, got:\n{stderr}" + ); +} + #[test] fn cli_bare_invocation_invokes_setup_when_no_config_found() { let temp = tempfile::tempdir().unwrap(); @@ -257,6 +284,35 @@ command = "hermes --yolo chat" assert!(stdout.contains("argv = hermes --yolo chat")); } +#[test] +fn cli_run_dry_run_infers_hermes_without_config() { + let temp = tempfile::tempdir().unwrap(); + let xdg = temp.path().join("xdg"); + let home = temp.path().join("home"); + std::fs::create_dir_all(&xdg).unwrap(); + std::fs::create_dir_all(&home).unwrap(); + + let output = Command::new(gateway_bin()) + .current_dir(temp.path()) + .env("XDG_CONFIG_HOME", &xdg) + .env("HOME", &home) + .args(["run", "--dry-run", "--print", "--", "hermes"]) + .output() + .unwrap(); + + assert!( + output.status.success(), + "dry-run should stay inspection-only without config: stderr={}", + String::from_utf8_lossy(&output.stderr) + ); + let stdout = String::from_utf8_lossy(&output.stdout); + assert!(stdout.contains("agent = hermes")); + assert!(stdout.contains("gateway_url = http://127.0.0.1:")); + assert!(stdout.contains("argv = hermes")); + assert!(stdout.contains("env.HERMES_ACCEPT_HOOKS = 1")); + assert!(stdout.contains("would temporarily merge NeMo Relay hooks")); +} + #[test] fn cli_run_dry_run_uses_project_user_and_env_config_layers() { let temp = tempfile::tempdir().unwrap(); @@ -383,6 +439,39 @@ fn cli_hook_forward_posts_payload_headers_and_prints_response() { assert!(request.contains(r#"{"hook_event_name":"sessionStart"}"#)); } +#[test] +fn cli_hook_forward_hermes_shell_hook_returns_empty_object() { + let (server_url, received) = spawn_single_request_server(200, r#"{}"#); + let mut child = Command::new(gateway_bin()) + .args([ + "hook-forward", + "hermes", + "--gateway-url", + &server_url, + "--fail-closed", + ]) + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .spawn() + .unwrap(); + child + .stdin + .take() + .unwrap() + .write_all(br#"{"session_id":"smoke-hermes","hook_event_name":"on_session_start"}"#) + .unwrap(); + let output = child.wait_with_output().unwrap(); + let request = received.recv().unwrap(); + + assert!(output.status.success()); + assert_eq!(String::from_utf8_lossy(&output.stdout).trim(), r#"{}"#); + assert!(request.contains("POST /hooks/hermes HTTP/1.1")); + assert!( + request.contains(r#"{"session_id":"smoke-hermes","hook_event_name":"on_session_start"}"#) + ); +} + #[test] fn cli_hook_forward_reports_http_failure_when_fail_closed() { let (server_url, received) = spawn_single_request_server(503, "unavailable"); diff --git a/crates/cli/tests/coverage/launcher_tests.rs b/crates/cli/tests/coverage/launcher_tests.rs index c7fc25f9e..84485f7db 100644 --- a/crates/cli/tests/coverage/launcher_tests.rs +++ b/crates/cli/tests/coverage/launcher_tests.rs @@ -762,6 +762,52 @@ async fn dry_run_does_not_spawn_agent() { assert_eq!(code, ExitCode::SUCCESS); } +#[tokio::test] +async fn dry_run_does_not_allocate_gateway_listener() { + let temp = tempfile::tempdir().unwrap(); + let config = temp.path().join("config.toml"); + std::fs::write( + &config, + r#" +[upstream] +openai_base_url = "http://file-openai" +anthropic_base_url = "http://file-anthropic" + +[agents.hermes] +command = "hermes chat" +"#, + ) + .unwrap(); + let command = RunCommand { + agent: Some(CodingAgent::Hermes), + config: Some(config), + openai_base_url: None, + anthropic_base_url: None, + session_metadata: None, + plugin_config: None, + dry_run: true, + print: true, + command: vec![], + }; + + let run = TransparentRun::new(command, None).await.unwrap(); + + assert!(run.listener.is_none()); + assert_eq!(run.gateway_url, "http://127.0.0.1:"); + assert_eq!(run.resolved.gateway.bind, "127.0.0.1:0".parse().unwrap()); + assert!( + run.prepared + .env + .contains(&("NEMO_RELAY_GATEWAY_URL".into(), run.gateway_url.clone())) + ); + assert!( + run.prepared + .notes + .iter() + .any(|note| note.contains("would temporarily merge NeMo Relay hooks")) + ); +} + #[tokio::test] async fn wait_for_health_reports_unready_gateway() { let error = wait_for_health("http://127.0.0.1:1") diff --git a/crates/cli/tests/coverage/server_tests.rs b/crates/cli/tests/coverage/server_tests.rs index 1d6ac2ad8..677a38e79 100644 --- a/crates/cli/tests/coverage/server_tests.rs +++ b/crates/cli/tests/coverage/server_tests.rs @@ -321,6 +321,206 @@ async fn serve_listener_observability_plugin_records_non_hermes_hooks() { assert!(!agent_starts.contains(&"claude-code".to_string())); } +#[tokio::test] +async fn serve_listener_hermes_api_hooks_write_atof_category_profile_and_fidelity() { + let _guard = PLUGIN_TEST_LOCK.lock().await; + let _ = nemo_relay::plugin::clear_plugin_configuration(); + + let temp = tempfile::tempdir().unwrap(); + let atof_dir = temp.path().join("atof"); + std::fs::create_dir_all(&atof_dir).unwrap(); + let mut config = test_config(); + config.plugin_config = Some(json!({ + "version": 1, + "components": [ + { + "kind": "observability", + "enabled": true, + "config": { + "version": 1, + "atof": { + "enabled": true, + "output_directory": atof_dir, + "filename": "events.jsonl", + "mode": "overwrite" + } + } + } + ] + })); + + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let address = listener.local_addr().unwrap(); + let url = format!("http://{address}"); + let (shutdown_tx, shutdown_rx) = oneshot::channel(); + let handle = + tokio::spawn(async move { serve_listener(listener, config, Some(shutdown_rx)).await }); + + wait_for_gateway(&url).await; + let client = test_http_client(); + + let response = client + .post(format!("{url}/hooks/hermes")) + .json(&json!({ + "hook_event_name": "pre_api_request", + "session_id": "hermes-atof-exact", + "extra": { + "task_id": "task-1", + "api_request_id": "turn-1:api:2", + "api_call_count": 2, + "model": "qwen", + "provider": "custom", + "request": { + "method": "POST", + "body": { + "model": "qwen", + "messages": [ + { "role": "user", "content": "hello" } + ], + "tools": [ + { "type": "function", "function": { "name": "search_files" } } + ] + } + } + } + })) + .send() + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::OK); + + let response = client + .post(format!("{url}/hooks/hermes")) + .json(&json!({ + "hook_event_name": "post_api_request", + "session_id": "hermes-atof-exact", + "extra": { + "task_id": "task-1", + "api_request_id": "turn-1:api:2", + "api_call_count": 2, + "model": "qwen", + "response": { + "model": "qwen", + "finish_reason": "tool_calls", + "assistant_message": { + "role": "assistant", + "content": "", + "tool_calls": [ + { + "id": "call-1", + "type": "function", + "function": { + "name": "search_files", + "arguments": "{\"query\":\"needle\"}" + } + } + ] + }, + "usage": { + "prompt_tokens": 10, + "completion_tokens": 5, + "cost": { "total": 0.0042 } + } + } + } + })) + .send() + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::OK); + + let response = client + .post(format!("{url}/hooks/hermes")) + .json(&json!({ + "hook_event_name": "pre_api_request", + "session_id": "hermes-atof-lossy", + "extra": { + "task_id": "task-2", + "api_call_count": 4, + "model": "qwen", + "provider": "custom", + "request": null, + "message_count": 2 + } + })) + .send() + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::OK); + + shutdown_tx.send(()).unwrap(); + handle.await.unwrap().unwrap(); + + let events = std::fs::read_to_string(temp.path().join("atof/events.jsonl")).unwrap(); + let llm_events = events + .lines() + .map(|line| serde_json::from_str::(line).unwrap()) + .filter(|event| event["category"] == "llm") + .collect::>(); + assert_eq!( + llm_events.len(), + 4, + "expected Hermes LLM exports, got {llm_events:?}" + ); + + let start = llm_events + .iter() + .find(|event| { + event["scope_category"] == "start" + && event["metadata"]["api_call_id"] == json!("turn-1:api:2") + }) + .unwrap(); + assert_eq!(start["category_profile"]["model_name"], json!("qwen")); + assert_eq!(start["metadata"]["provider_payload_exact"], json!(true)); + assert_eq!( + start["metadata"]["fidelity_source"], + json!("hermes_api_hooks_sanitized") + ); + assert_eq!( + start["data"]["content"]["messages"][0]["content"], + json!("hello") + ); + assert_eq!( + start["data"]["content"]["tools"][0]["function"]["name"], + json!("search_files") + ); + + let end = llm_events + .iter() + .find(|event| { + event["scope_category"] == "end" + && event["metadata"]["api_call_id"] == json!("turn-1:api:2") + }) + .unwrap(); + assert_eq!(end["category_profile"]["model_name"], json!("qwen")); + assert_eq!(end["metadata"]["provider_payload_exact"], json!(true)); + assert_eq!(end["data"]["tool_calls"][0]["id"], json!("call-1")); + assert_eq!( + end["data"]["tool_calls"][0]["function"]["name"], + json!("search_files") + ); + assert_eq!(end["data"]["usage"]["prompt_tokens"], json!(10)); + assert_eq!(end["data"]["usage"]["completion_tokens"], json!(5)); + + let lossy_start = llm_events + .iter() + .find(|event| { + event["scope_category"] == "start" + && event["metadata"]["api_call_id"] == json!("hermes-atof-lossy:task-2:4") + }) + .unwrap(); + assert_eq!(lossy_start["category_profile"]["model_name"], json!("qwen")); + assert_eq!( + lossy_start["metadata"]["provider_payload_exact"], + json!(false) + ); + assert_eq!( + lossy_start["data"]["content"]["fidelity"]["provider_payload_exact"], + json!(false) + ); + assert_eq!(lossy_start["data"]["content"]["message_count"], json!(2)); +} + #[tokio::test] async fn serve_listener_activates_any_registered_plugin_kind() { let _guard = PLUGIN_TEST_LOCK.lock().await; diff --git a/crates/core/tests/unit/observability/openinference_tests.rs b/crates/core/tests/unit/observability/openinference_tests.rs index fc8d11f6f..a8910b736 100644 --- a/crates/core/tests/unit/observability/openinference_tests.rs +++ b/crates/core/tests/unit/observability/openinference_tests.rs @@ -2136,6 +2136,105 @@ fn annotated_llm_payloads_emit_flattened_openinference_message_and_tool_attribut assert_attr(&attributes, "llm.finish_reason", "tool_use"); } +#[test] +fn hermes_exact_api_payloads_emit_openinference_text_usage_and_metadata() { + let (provider, exporter) = make_provider(); + let mut processor = + OpenInferenceEventProcessor::new(provider.clone(), "test-scope".to_string()); + let uuid = Uuid::now_v7(); + let metadata = json!({ + "provider_payload_exact": true, + "fidelity_source": "hermes_api_hooks_sanitized" + }); + + processor.process(&Event::Scope(ScopeEvent::new( + BaseEvent::builder() + .uuid(uuid) + .name("custom") + .data(json!({ + "model": "qwen", + "messages": [{ "role": "user", "content": "hello" }], + "tools": [ + { "type": "function", "function": { "name": "search_files" } } + ] + })) + .metadata(metadata.clone()) + .build(), + ScopeCategory::Start, + Vec::new(), + EventCategory::llm(), + Some(CategoryProfile::builder().model_name("qwen").build()), + ))); + processor.process(&Event::Scope(ScopeEvent::new( + BaseEvent::builder() + .uuid(uuid) + .name("custom") + .data(json!({ + "content": "", + "tool_calls": [ + { + "id": "call-1", + "type": "function", + "function": { + "name": "search_files", + "arguments": "{\"query\":\"needle\"}" + } + } + ], + "usage": { + "prompt_tokens": 10, + "completion_tokens": 5, + "cost": { "total": 0.0042 } + }, + "model": "qwen", + "finish_reason": "tool_calls" + })) + .metadata(metadata) + .build(), + ScopeCategory::End, + Vec::new(), + EventCategory::llm(), + Some(CategoryProfile::builder().model_name("qwen").build()), + ))); + + processor.force_flush().unwrap(); + + let spans = exporter.get_finished_spans().unwrap(); + assert_eq!(spans.len(), 1); + let attributes = attr_map(&spans[0].attributes); + assert_eq!( + attributes.get("openinference.span.kind"), + Some(&"LLM".to_string()) + ); + assert_eq!(attributes.get("llm.model_name"), Some(&"qwen".to_string())); + assert_eq!( + attributes.get("input.value"), + Some(&"user: hello".to_string()) + ); + assert_eq!( + attributes.get("output.value"), + Some(&"Requested tools: search_files".to_string()) + ); + assert_eq!( + attributes.get("llm.token_count.prompt"), + Some(&"10".to_string()) + ); + assert_eq!( + attributes.get("llm.token_count.completion"), + Some(&"5".to_string()) + ); + assert_eq!( + attributes.get("llm.cost.total"), + Some(&"0.0042".to_string()) + ); + assert_attr_contains(&attributes, "metadata", "\"provider_payload_exact\":true"); + assert_attr_contains( + &attributes, + "metadata", + "\"fidelity_source\":\"hermes_api_hooks_sanitized\"", + ); +} + #[test] fn llm_end_with_inconsistent_manual_usage_omits_invalid_total_tokens() { let (provider, exporter) = make_provider(); diff --git a/docs/nemo-relay-cli/hermes.mdx b/docs/nemo-relay-cli/hermes.mdx index 9656d0a15..e2b7f3e49 100644 --- a/docs/nemo-relay-cli/hermes.mdx +++ b/docs/nemo-relay-cli/hermes.mdx @@ -8,14 +8,18 @@ SPDX-License-Identifier: Apache-2.0 */} Use this guide to observe local Hermes Agent sessions with NeMo Relay through -Hermes shell hooks and the `nemo-relay` gateway. This gateway path is -separate from the Hermes third-party patch set under `patches/hermes-agent/`; -use the gateway when you want hook forwarding without rebuilding a patched -Hermes checkout. - -Hermes shell hooks provide session, subagent, tool, and LLM hint lifecycle -events. Complete LLM request and response observability still requires model -traffic to route through the gateway. +Hermes shell hooks and the `nemo-relay` gateway. This wrapped gateway path is +the release-safe Hermes integration path today. It is separate from the Hermes +third-party patch set under `patches/hermes-agent/`, which remains useful for +comparison and future plugin compatibility work but is not the priority path +for current Hermes validation. + +Hermes shell hooks provide session, subagent, tool, and LLM lifecycle events. +For NeMo Relay contract work, `pre_api_request`, `post_api_request`, and +`api_request_error` are the authoritative LLM lifecycle hooks. The legacy +`pre_llm_call` and `post_llm_call` hooks still exist, but only as private +hint-style signals. Complete LLM request and response observability still +requires model traffic to route through the gateway. ## Transparent Run @@ -32,11 +36,18 @@ Pass Hermes arguments after `--`: nemo-relay hermes -- chat --provider custom ``` -This shortcut is equivalent to `nemo-relay run -- hermes`. The wrapper starts a -gateway on a dynamic `127.0.0.1` port and exports `NEMO_RELAY_GATEWAY_URL` for -the launched process. Hermes hook configuration is not temporary in this mode. -Install hooks first, or configure equivalent Hermes shell hooks, so approved -hook commands can discover the dynamic gateway URL. +Once NeMo Relay config exists, this shortcut is equivalent to +`nemo-relay run -- hermes`. The wrapper starts a gateway on a dynamic +`127.0.0.1` port and exports `NEMO_RELAY_GATEWAY_URL` for the launched +process. After initial NeMo Relay setup exists, Hermes hook configuration is +temporary in this mode: the launcher merges the NeMo Relay hook-forward +commands into the configured Hermes hook file for the run and restores the +original file afterward. The wrapper also sets `HERMES_ACCEPT_HOOKS=1` so +Hermes can use the injected hook commands without extra manual approval +prompts. + +If no NeMo Relay config exists yet, `nemo-relay hermes` triggers the setup flow +first and then launches Hermes through the same wrapped pipeline. Inspect what would be launched without starting Hermes: @@ -47,6 +58,9 @@ nemo-relay run \ -- hermes ``` +Dry-run stays inspection-only in this mode: it prints a placeholder ephemeral +gateway URL and does not bind a listener or launch Hermes. + ## Shared Config Create `.nemo-relay/config.toml` for project defaults or @@ -89,7 +103,13 @@ records the path under `[agents.hermes].hooks_path` in `.nemo-relay/config.toml` The generated Hermes hooks cover `on_session_start`, `on_session_end`, `on_session_finalize`, `on_session_reset`, `pre_llm_call`, `post_llm_call`, -`pre_tool_call`, `post_tool_call`, `subagent_start`, and `subagent_stop`. +`pre_api_request`, `post_api_request`, `api_request_error`, `pre_tool_call`, +`post_tool_call`, `subagent_start`, and `subagent_stop`. + +The API hooks are the main Hermes LLM lifecycle path for NeMo Relay. The legacy +LLM hooks remain installed because they can still provide useful private hints, +but they are not treated as equal peers to the API hooks in the observability +contract. Hermes hook forwarding prefers `NEMO_RELAY_GATEWAY_URL` when set (this is what `nemo-relay hermes` injects on every run). When launched outside the wrapper — @@ -104,6 +124,16 @@ nemo-relay --bind 127.0.0.1:4040 Then point Hermes provider traffic at `http://127.0.0.1:4040` for any provider mode that exposes a local OpenAI-compatible or Anthropic-compatible base URL. +This is the practical routed-provider validation path today. + +Important distinction: + +- Wrapped execution is the authoritative path for Hermes hook-path validation. +- Wrapped execution does not automatically rewrite Hermes provider `base_url`, + `custom_providers`, or `api_mode`. +- Routed `/v1/messages`, `/v1/chat/completions`, or `/v1/responses` validation + therefore requires explicit Hermes provider configuration in addition to the + wrapped or standalone gateway. ## Smoke Test diff --git a/patches/hermes-agent/notes.md b/patches/hermes-agent/notes.md index a9e03aae3..3cdacff26 100644 --- a/patches/hermes-agent/notes.md +++ b/patches/hermes-agent/notes.md @@ -6,12 +6,20 @@ SPDX-License-Identifier: Apache-2.0 # NeMo-Relay Hermes Integration — Operator Notes These notes are the operator runbook for installing the tracked Hermes + -NeMo-Relay integration from a fresh NeMo-Relay checkout. The maintained patch is -runtime-only: it wires Hermes to the NeMo-Relay plugin entry point, hooks, and -ACG override seam, but it does not carry Hermes-side tests or smoke harnesses. -At runtime, the plugin emits a NeMo-Relay session scope plus manual LLM/tool -lifecycle spans and uses `AtifExporter` to materialize trajectory JSON on -session finalization. +NeMo-Relay integration from a fresh NeMo-Relay checkout. + +Priority note: + +- The wrapped gateway path in `docs/nemo-relay-cli/hermes.mdx` is the P0 + release-safe Hermes validation path. +- This patch runbook is the P1 comparison and compatibility path until Hermes + ships a tagged release that includes the built-in NeMo Relay plugin. + +The maintained patch is runtime-only: it wires Hermes to the NeMo-Relay plugin +entry point, hooks, and ACG override seam, but it does not carry Hermes-side +tests or smoke harnesses. At runtime, the plugin emits a NeMo-Relay session +scope plus manual LLM/tool lifecycle spans and uses `AtifExporter` to +materialize trajectory JSON on session finalization. All commands assume your working directory is the NeMo-Relay repo root unless a step says otherwise. diff --git a/third_party/README-hermes-agent.md b/third_party/README-hermes-agent.md index 5b7c1d35b..80c3247b3 100644 --- a/third_party/README-hermes-agent.md +++ b/third_party/README-hermes-agent.md @@ -13,6 +13,13 @@ detailed operator runbook. It covers the pinned checkout, editable install with the `nemo-relay` extra, environment variables, ATIF output, OpenInference export, and smoke validation. +This patch path is no longer the primary Hermes rollout path for NeMo Relay. +For current release-safe Hermes validation, start with the wrapped gateway path +documented in [docs/nemo-relay-cli/hermes.mdx](../docs/nemo-relay-cli/hermes.mdx). +Use the patch path when you need to compare behavior against the tracked Hermes +plugin integration or prepare for future Hermes releases that carry the built-in +NeMo Relay plugin. + ## Quick Path From the NeMo Relay repository root: From c4c324476731cd9521614f4602d71939c0344b07 Mon Sep 17 00:00:00 2001 From: mnajafian-nv Date: Thu, 4 Jun 2026 02:52:57 -0700 Subject: [PATCH 2/4] fix: restore Rust attributions for CI Signed-off-by: mnajafian-nv --- ATTRIBUTIONS-Rust.md | 202 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 172 insertions(+), 30 deletions(-) diff --git a/ATTRIBUTIONS-Rust.md b/ATTRIBUTIONS-Rust.md index 30290863b..38dd9222b 100644 --- a/ATTRIBUTIONS-Rust.md +++ b/ATTRIBUTIONS-Rust.md @@ -3123,8 +3123,9 @@ limitations under the License. ## block-buffer - 0.10.4 **Repository URL**: https://github.com/RustCrypto/utils -**License Type(s)**: Apache-2.0 -### License: https://spdx.org/licenses/Apache-2.0.html +**License Type(s)**: MIT OR Apache-2.0 +### License: https://spdx.org/licenses/ +### License File: LICENSE-APACHE ``` Apache License Version 2.0, January 2004 @@ -3327,7 +3328,35 @@ distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. +``` + +### License File: LICENSE-MIT +``` +Copyright (c) 2018-2019 The RustCrypto Project Developers + +Permission is hereby granted, free of charge, to any +person obtaining a copy of this software and associated +documentation files (the "Software"), to deal in the +Software without restriction, including without +limitation the rights to use, copy, modify, merge, +publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software +is furnished to do so, subject to the following +conditions: + +The above copyright notice and this permission notice +shall be included in all copies or substantial portions +of the Software. +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF +ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED +TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A +PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT +SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR +IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +DEALINGS IN THE SOFTWARE. ``` ## block-buffer - 0.12.0 @@ -7898,8 +7927,9 @@ limitations under the License. ## crypto-common - 0.1.7 **Repository URL**: https://github.com/RustCrypto/traits -**License Type(s)**: Apache-2.0 -### License: https://spdx.org/licenses/Apache-2.0.html +**License Type(s)**: MIT OR Apache-2.0 +### License: https://spdx.org/licenses/ +### License File: LICENSE-APACHE ``` Apache License Version 2.0, January 2004 @@ -8102,7 +8132,35 @@ distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. +``` +### License File: LICENSE-MIT +``` +Copyright (c) 2021 RustCrypto Developers + +Permission is hereby granted, free of charge, to any +person obtaining a copy of this software and associated +documentation files (the "Software"), to deal in the +Software without restriction, including without +limitation the rights to use, copy, modify, merge, +publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software +is furnished to do so, subject to the following +conditions: + +The above copyright notice and this permission notice +shall be included in all copies or substantial portions +of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF +ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED +TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A +PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT +SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR +IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +DEALINGS IN THE SOFTWARE. ``` ## crypto-common - 0.2.1 @@ -8555,8 +8613,9 @@ SOFTWARE. ## digest - 0.10.7 **Repository URL**: https://github.com/RustCrypto/traits -**License Type(s)**: Apache-2.0 -### License: https://spdx.org/licenses/Apache-2.0.html +**License Type(s)**: MIT OR Apache-2.0 +### License: https://spdx.org/licenses/ +### License File: LICENSE-APACHE ``` Apache License Version 2.0, January 2004 @@ -8759,7 +8818,35 @@ distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. +``` + +### License File: LICENSE-MIT +``` +Copyright (c) 2017 Artyom Pavlov + +Permission is hereby granted, free of charge, to any +person obtaining a copy of this software and associated +documentation files (the "Software"), to deal in the +Software without restriction, including without +limitation the rights to use, copy, modify, merge, +publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software +is furnished to do so, subject to the following +conditions: +The above copyright notice and this permission notice +shall be included in all copies or substantial portions +of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF +ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED +TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A +PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT +SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR +IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +DEALINGS IN THE SOFTWARE. ``` ## digest - 0.11.2 @@ -13273,27 +13360,28 @@ limitations under the License. **Repository URL**: https://github.com/fizyk20/generic-array.git **License Type(s)**: MIT ### License: https://spdx.org/licenses/MIT.html +### License File: LICENSE ``` -The MIT License (MIT) - -Copyright (c) 2015 Bartłomiej Kamiński - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to deal -in the Software without restriction, including without limitation the rights -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in all -copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +The MIT License (MIT) + +Copyright (c) 2015 Bartłomiej Kamiński + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ``` @@ -20342,8 +20430,9 @@ SOFTWARE. ## md-5 - 0.10.6 **Repository URL**: https://github.com/RustCrypto/hashes -**License Type(s)**: Apache-2.0 -### License: https://spdx.org/licenses/Apache-2.0.html +**License Type(s)**: MIT OR Apache-2.0 +### License: https://spdx.org/licenses/ +### License File: LICENSE-APACHE ``` Apache License Version 2.0, January 2004 @@ -20546,7 +20635,37 @@ distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. +``` +### License File: LICENSE-MIT +``` +Copyright (c) 2006-2009 Graydon Hoare +Copyright (c) 2009-2013 Mozilla Foundation +Copyright (c) 2016 Artyom Pavlov + +Permission is hereby granted, free of charge, to any +person obtaining a copy of this software and associated +documentation files (the "Software"), to deal in the +Software without restriction, including without +limitation the rights to use, copy, modify, merge, +publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software +is furnished to do so, subject to the following +conditions: + +The above copyright notice and this permission notice +shall be included in all copies or substantial portions +of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF +ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED +TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A +PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT +SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR +IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +DEALINGS IN THE SOFTWARE. ``` ## memchr - 2.8.0 @@ -38798,8 +38917,9 @@ limitations under the License. ## version_check - 0.9.5 **Repository URL**: https://github.com/SergioBenitez/version_check -**License Type(s)**: Apache-2.0 -### License: https://spdx.org/licenses/Apache-2.0.html +**License Type(s)**: MIT/Apache-2.0 +### License: https://spdx.org/licenses/ +### License File: LICENSE-APACHE ``` Apache License Version 2.0, January 2004 @@ -39002,7 +39122,29 @@ distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. +``` + +### License File: LICENSE-MIT +``` +The MIT License (MIT) +Copyright (c) 2017-2018 Sergio Benitez +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS +FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR +COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ``` ## walkdir - 2.5.0 From 3d07e4ada64df57052df9c8880f405d48f90d716 Mon Sep 17 00:00:00 2001 From: mnajafian-nv Date: Thu, 4 Jun 2026 03:13:16 -0700 Subject: [PATCH 3/4] docs: correct Hermes wrapped run equivalence Signed-off-by: mnajafian-nv --- docs/nemo-relay-cli/hermes.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/nemo-relay-cli/hermes.mdx b/docs/nemo-relay-cli/hermes.mdx index e2b7f3e49..2f8e5a645 100644 --- a/docs/nemo-relay-cli/hermes.mdx +++ b/docs/nemo-relay-cli/hermes.mdx @@ -37,7 +37,7 @@ nemo-relay hermes -- chat --provider custom ``` Once NeMo Relay config exists, this shortcut is equivalent to -`nemo-relay run -- hermes`. The wrapper starts a gateway on a dynamic +`nemo-relay run --agent hermes`. The wrapper starts a gateway on a dynamic `127.0.0.1` port and exports `NEMO_RELAY_GATEWAY_URL` for the launched process. After initial NeMo Relay setup exists, Hermes hook configuration is temporary in this mode: the launcher merges the NeMo Relay hook-forward From 27fafe1ca71b71be013a663ed030f864357554de Mon Sep 17 00:00:00 2001 From: mnajafian-nv Date: Thu, 4 Jun 2026 08:31:53 -0700 Subject: [PATCH 4/4] test: narrow Hermes wrapped observability scope Signed-off-by: mnajafian-nv --- crates/cli/src/launcher.rs | 24 ++++------- crates/cli/tests/cli_tests.rs | 29 ------------- crates/cli/tests/coverage/launcher_tests.rs | 46 --------------------- docs/nemo-relay-cli/hermes.mdx | 9 +--- patches/hermes-agent/notes.md | 20 +++------ third_party/README-hermes-agent.md | 7 ---- 6 files changed, 15 insertions(+), 120 deletions(-) diff --git a/crates/cli/src/launcher.rs b/crates/cli/src/launcher.rs index 1e8d4115a..acf5d268a 100644 --- a/crates/cli/src/launcher.rs +++ b/crates/cli/src/launcher.rs @@ -87,29 +87,24 @@ struct TransparentRun { agent: CodingAgent, prepared: PreparedRun, resolved: ResolvedConfig, - listener: Option, + listener: TcpListener, gateway_url: String, dry_run: bool, print: bool, } impl TransparentRun { - // Resolves configuration and builds agent-specific launch wiring. Live runs also bind the - // ephemeral listener here so health checks and agent preparation share one resolved gateway - // address; dry runs stay side-effect free and use a placeholder dynamic URL for inspection. + // Resolves configuration, binds the ephemeral listener, and builds agent-specific launch wiring + // without starting the gateway or spawning the child command. async fn new(command: RunCommand, inherited: Option<&ServerArgs>) -> Result { let dry_run = command.dry_run; let print = command.print; let mut resolved = resolve_run_config(&command, inherited)?; let (agent, argv) = resolve_agent_and_argv(&command, &resolved.agents)?; - let (listener, gateway_url) = if dry_run { - (None, "http://127.0.0.1:".to_string()) - } else { - let listener = TcpListener::bind("127.0.0.1:0").await?; - let address = listener.local_addr()?; - resolved.gateway.bind = address; - (Some(listener), format!("http://{address}")) - }; + let listener = TcpListener::bind("127.0.0.1:0").await?; + let address = listener.local_addr()?; + let gateway_url = format!("http://{address}"); + resolved.gateway.bind = address; let prepared = PreparedRun::new(agent, argv, &gateway_url, &resolved, dry_run)?; Ok(Self { @@ -137,13 +132,10 @@ impl TransparentRun { if self.dry_run { return Ok(ExitCode::SUCCESS); } - let listener = self - .listener - .expect("live transparent runs must allocate a gateway listener"); self.prepared .print_live_status(self.agent, &self.gateway_url, &self.resolved); execute_live_run( - listener, + self.listener, self.resolved.gateway, &self.gateway_url, self.prepared, diff --git a/crates/cli/tests/cli_tests.rs b/crates/cli/tests/cli_tests.rs index 3fd74cbec..3a67b9d44 100644 --- a/crates/cli/tests/cli_tests.rs +++ b/crates/cli/tests/cli_tests.rs @@ -284,35 +284,6 @@ command = "hermes --yolo chat" assert!(stdout.contains("argv = hermes --yolo chat")); } -#[test] -fn cli_run_dry_run_infers_hermes_without_config() { - let temp = tempfile::tempdir().unwrap(); - let xdg = temp.path().join("xdg"); - let home = temp.path().join("home"); - std::fs::create_dir_all(&xdg).unwrap(); - std::fs::create_dir_all(&home).unwrap(); - - let output = Command::new(gateway_bin()) - .current_dir(temp.path()) - .env("XDG_CONFIG_HOME", &xdg) - .env("HOME", &home) - .args(["run", "--dry-run", "--print", "--", "hermes"]) - .output() - .unwrap(); - - assert!( - output.status.success(), - "dry-run should stay inspection-only without config: stderr={}", - String::from_utf8_lossy(&output.stderr) - ); - let stdout = String::from_utf8_lossy(&output.stdout); - assert!(stdout.contains("agent = hermes")); - assert!(stdout.contains("gateway_url = http://127.0.0.1:")); - assert!(stdout.contains("argv = hermes")); - assert!(stdout.contains("env.HERMES_ACCEPT_HOOKS = 1")); - assert!(stdout.contains("would temporarily merge NeMo Relay hooks")); -} - #[test] fn cli_run_dry_run_uses_project_user_and_env_config_layers() { let temp = tempfile::tempdir().unwrap(); diff --git a/crates/cli/tests/coverage/launcher_tests.rs b/crates/cli/tests/coverage/launcher_tests.rs index 84485f7db..c7fc25f9e 100644 --- a/crates/cli/tests/coverage/launcher_tests.rs +++ b/crates/cli/tests/coverage/launcher_tests.rs @@ -762,52 +762,6 @@ async fn dry_run_does_not_spawn_agent() { assert_eq!(code, ExitCode::SUCCESS); } -#[tokio::test] -async fn dry_run_does_not_allocate_gateway_listener() { - let temp = tempfile::tempdir().unwrap(); - let config = temp.path().join("config.toml"); - std::fs::write( - &config, - r#" -[upstream] -openai_base_url = "http://file-openai" -anthropic_base_url = "http://file-anthropic" - -[agents.hermes] -command = "hermes chat" -"#, - ) - .unwrap(); - let command = RunCommand { - agent: Some(CodingAgent::Hermes), - config: Some(config), - openai_base_url: None, - anthropic_base_url: None, - session_metadata: None, - plugin_config: None, - dry_run: true, - print: true, - command: vec![], - }; - - let run = TransparentRun::new(command, None).await.unwrap(); - - assert!(run.listener.is_none()); - assert_eq!(run.gateway_url, "http://127.0.0.1:"); - assert_eq!(run.resolved.gateway.bind, "127.0.0.1:0".parse().unwrap()); - assert!( - run.prepared - .env - .contains(&("NEMO_RELAY_GATEWAY_URL".into(), run.gateway_url.clone())) - ); - assert!( - run.prepared - .notes - .iter() - .any(|note| note.contains("would temporarily merge NeMo Relay hooks")) - ); -} - #[tokio::test] async fn wait_for_health_reports_unready_gateway() { let error = wait_for_health("http://127.0.0.1:1") diff --git a/docs/nemo-relay-cli/hermes.mdx b/docs/nemo-relay-cli/hermes.mdx index 2f8e5a645..ad4e4c433 100644 --- a/docs/nemo-relay-cli/hermes.mdx +++ b/docs/nemo-relay-cli/hermes.mdx @@ -8,11 +8,7 @@ SPDX-License-Identifier: Apache-2.0 */} Use this guide to observe local Hermes Agent sessions with NeMo Relay through -Hermes shell hooks and the `nemo-relay` gateway. This wrapped gateway path is -the release-safe Hermes integration path today. It is separate from the Hermes -third-party patch set under `patches/hermes-agent/`, which remains useful for -comparison and future plugin compatibility work but is not the priority path -for current Hermes validation. +Hermes shell hooks and the `nemo-relay` gateway. Hermes shell hooks provide session, subagent, tool, and LLM lifecycle events. For NeMo Relay contract work, `pre_api_request`, `post_api_request`, and @@ -58,9 +54,6 @@ nemo-relay run \ -- hermes ``` -Dry-run stays inspection-only in this mode: it prints a placeholder ephemeral -gateway URL and does not bind a listener or launch Hermes. - ## Shared Config Create `.nemo-relay/config.toml` for project defaults or diff --git a/patches/hermes-agent/notes.md b/patches/hermes-agent/notes.md index 3cdacff26..a9e03aae3 100644 --- a/patches/hermes-agent/notes.md +++ b/patches/hermes-agent/notes.md @@ -6,20 +6,12 @@ SPDX-License-Identifier: Apache-2.0 # NeMo-Relay Hermes Integration — Operator Notes These notes are the operator runbook for installing the tracked Hermes + -NeMo-Relay integration from a fresh NeMo-Relay checkout. - -Priority note: - -- The wrapped gateway path in `docs/nemo-relay-cli/hermes.mdx` is the P0 - release-safe Hermes validation path. -- This patch runbook is the P1 comparison and compatibility path until Hermes - ships a tagged release that includes the built-in NeMo Relay plugin. - -The maintained patch is runtime-only: it wires Hermes to the NeMo-Relay plugin -entry point, hooks, and ACG override seam, but it does not carry Hermes-side -tests or smoke harnesses. At runtime, the plugin emits a NeMo-Relay session -scope plus manual LLM/tool lifecycle spans and uses `AtifExporter` to -materialize trajectory JSON on session finalization. +NeMo-Relay integration from a fresh NeMo-Relay checkout. The maintained patch is +runtime-only: it wires Hermes to the NeMo-Relay plugin entry point, hooks, and +ACG override seam, but it does not carry Hermes-side tests or smoke harnesses. +At runtime, the plugin emits a NeMo-Relay session scope plus manual LLM/tool +lifecycle spans and uses `AtifExporter` to materialize trajectory JSON on +session finalization. All commands assume your working directory is the NeMo-Relay repo root unless a step says otherwise. diff --git a/third_party/README-hermes-agent.md b/third_party/README-hermes-agent.md index 80c3247b3..5b7c1d35b 100644 --- a/third_party/README-hermes-agent.md +++ b/third_party/README-hermes-agent.md @@ -13,13 +13,6 @@ detailed operator runbook. It covers the pinned checkout, editable install with the `nemo-relay` extra, environment variables, ATIF output, OpenInference export, and smoke validation. -This patch path is no longer the primary Hermes rollout path for NeMo Relay. -For current release-safe Hermes validation, start with the wrapped gateway path -documented in [docs/nemo-relay-cli/hermes.mdx](../docs/nemo-relay-cli/hermes.mdx). -Use the patch path when you need to compare behavior against the tracked Hermes -plugin integration or prepare for future Hermes releases that carry the built-in -NeMo Relay plugin. - ## Quick Path From the NeMo Relay repository root: