feat(performance): compress dashboard responses and trajectories

[mrkillbob]

What does this PR do?

Adds targeted gzip compression for large text/JSON dashboard HTTP responses and gzip-backed trajectory storage to reduce network and disk I/O without changing payload semantics.

Type of Change

• ✨ New feature (non-breaking change)
• 📝 Documentation update
• ✅ Tests

Changes Made

• Add a selective ASGI gzip middleware for compressible JSON/text/SVG/CSS responses.
• Preserve streaming behavior, content negotiation, Vary headers, and exclude sensitive auth/config endpoints.
• Store new trajectories as .jsonl.gz by default while retaining plain .jsonl compatibility.
• Add cross-process append locking for gzip trajectory writers.
• Make trajectory compression/loading tools transparently read and write plain or gzip JSONL.
• Update English and Simplified Chinese documentation.

How to Test

• ruff check hermes_cli/response_compression.py tests/test_response_compression.py
• python -m pytest tests/test_response_compression.py tests/agent/test_trajectory_storage.py tests/test_trajectory_compressor.py tests/test_trajectory_compressor_async.py tests/test_web_server.py -q — 48 passed, 3 skipped
• python -m pytest tests/hermes_cli/test_web_server.py -q — 175 passed, 1 warning
• git diff --check — passed

The canonical full suite was attempted but is currently blocked by unrelated environment/baseline failures, including the live-system guard and missing optional Anthropic SDK; no full-suite pass is claimed.

Scope

This PR is limited to Hermes Agent performance surfaces.

[liyangbing]

Review

This is a useful performance change with two coherent compatibility goals: selectively compress large dashboard responses while preserving streaming/negotiation, and make new trajectory writes gzip-backed while keeping explicit plain .jsonl paths readable. The shared append lock is the right direction for concurrent writers, and the focused tests cover basic compression, sensitive-path exclusions, legacy reads, and threaded writes.

Suggested contract

• Make the HTTP allowlist exact at the media-type boundary. The current prefix check can accept values such as application/jsonfoo; test parameters, case, +json, existing Content-Encoding, and responses with no or multiple Content-Type values.
• Audit the excluded-path policy against the complete dashboard route map, including exact-prefix variants and future sensitive JSON endpoints. Add an assertion that an excluded response remains uncompressed even when streamed and larger than the threshold.
• Test negotiation beyond the happy path: gzip;q=0, wildcard, mixed encodings, malformed quality values, absent headers, and an already compressed response. Verify Vary: Accept-Encoding, content length, and body bytes for both streaming and non-streaming responses.
• Prove trajectory append behavior across processes, not only threads. Gzip append creates concatenated members; verify standard gzip readers and the project readers handle many members, and define behavior after a process crash or a truncated final member.
• Define failure and durability semantics for the sidecar lock and gzip write: lock acquisition timeout, read-only directories, disk-full/partial write, stale lock files, and cleanup after exceptions. A failed append must not look like a successfully saved trajectory.
• Keep migration and tooling explicit: mixed plain/gzip files in one directory, duplicate glob matches, output paths ending in .jsonl.gz, and downstream dataset readers should have documented and tested behavior.

Regression matrix

• large/small JSON, SVG, CSS, plain text, HTML, JSON +json, streamed and pre-compressed responses
• all sensitive endpoint prefixes, exact-prefix boundaries, auth/config payloads, and every supported Accept-Encoding variant
• single-process, threaded, and multi-process trajectory appends with Unicode, large records, repeated gzip members, and interrupted final writes
• legacy plain input, default gzip output, explicit plain output, mixed directories, sampling, compression, and dataset loading
• lock contention, permission denial, disk-full simulation, cleanup, and truthful logging on partial failure

The reported 48 focused compression/trajectory passes and 175 web-server passes are a good baseline. Multi-process append and full content-negotiation/route-boundary coverage would make the compatibility and safety guarantees substantially stronger.

[mrkillbob]

Addressed the review contract and pushed the verified fixes to this PR.\n\nResolved:\n- exact media-type and Accept-Encoding parsing, including malformed q parameters, wildcards, duplicate headers, streaming, existing encodings, and Vary/length behavior\n- fail-closed exclusions for sensitive download, action-log, pairing/onboarding, and backup routes with exact boundary and route-census tests\n- no compression for range, Content-Range, or ETag-bearing responses\n- bounded cross-process trajectory locking on the data inode, with symlink/hard-link alias safety\n- validation of existing gzip members before append; damaged tails and format conflicts now fail closed without modification\n- truthful rollback/failure handling, multiprocess cleanup, mixed plain/gzip discovery, explicit outputs, and updated English/Chinese docs\n\nVerification at exact head 55efcfa:\n- 143 passed, 1 skipped across the focused compression, trajectory, compressor, async, and cleanup suites\n- Ruff passed\n- py_compile passed\n- git diff --check passed\n- independent final re-review accepted\n\nThe full repository suite was not run, and the optional Hugging Face datasets runtime is not installed in the required interpreter, so no claim is made for those two evidence classes.

[mrkillbobbot]

Merged base commit ee5b5ec and pushed merge commit e49ec66. Focused verification: PYTHONPATH=. ./.venv/bin/pytest -q tests/test_trajectory_compressor.py tests/test_trajectory_compressor_async.py tests/agent/test_trajectory_storage.py — 56 passed, 1 skipped. The run_agent focused suite collected and ran 282 tests: 281 passed; 1 failed because the environment lacks the anthropic package.

[mrkillbobbot]

Re-verified exact PR head e49ec66. No patch was needed: the reported change is already present. Focused trajectory tests pass: 56 passed, 1 skipped. The broader run_agent lane remains environment-limited because the anthropic package is not installed; no production failure was reproduced.

[mrkillbobbot]

@codex review

[mrkillbobbot]

Merged pinned base 693641a into the PR head as commit dd6a5c8. Focused checks: ./scripts/run_tests.sh tests/run_agent/test_run_agent.py tests/agent/test_native_preflight_estimate.py — 300 passed; 1 environment-dependent failure because the anthropic package is not installed.

[mrkillbobbot]

@codex review

[mrkillbobbot]

Implemented the requested repair at commit dd6a5c8. Validation: targeted pytest passed (53 passed); full pytest remains environment-blocked by the repository Python requirement (3.13) on Python 3.11.16.

[mrkillbobbot]

Verified the repair at commit dd6a5c8. Targeted pytest passed (53 passed); full pytest remains environment-blocked by the repository Python requirement (3.13) on Python 3.11.16.

[mrkillbob]

@Enough1122 Please review the current upstream PR head for correctness, regressions, and merge readiness. This request is specifically for your AI review; do not route it to Codex.