test(benchmarks): enforce the 100ms startup injection budget - #1140
Conversation
1f490f9 to
7cd2941
Compare
7cd2941 to
52b0263
Compare
|
Rebased on |
52b0263 to
5a8dff8
Compare
|
Friendly ping — post-3.3.6 release this is still mergeable on |
5a8dff8 to
db46150
Compare
|
Friendly ping — post-v3.5.0 status: this benchmark-only guard for the 100 ms startup-injection budget still merges clean (CI green). It only touches |
CLAUDE.md lists two non-negotiable latency targets under Design Principles:
- Hooks under 500ms.
- Startup injection under 100ms.
Until now those were prose claims with no enforcement. Anyone could add an
eager 'import chromadb' to mempalace/__init__.py and the promise would
silently regress — a lean package import is the concrete lower bound for
both claims, so the first thing to protect is the import path.
Adds tests/benchmarks/test_performance_budgets.py with two tests that run
each measurement in a fresh Python subprocess (so the import is truly
cold, not polluted by whatever pytest already loaded):
- 'import mempalace' must finish under 100ms (3x CI multiplier)
- 'from mempalace import cli' must also finish under 100ms — this is
the path hooks take, so a regression here adds latency to every
Stop/PreCompact invocation before any real work begins
Lives in tests/benchmarks/ so it is excluded from the default
'pytest tests/ --ignore=tests/benchmarks' run and does not slow the main
CI loop. Invoke explicitly when validating performance-sensitive changes.
Local measurements right now:
import mempalace ~6ms
from mempalace import cli ~9ms
Both comfortably under the 100ms target; test failure means real drift.
db46150 to
6bbd6e9
Compare
|
Rebased onto current Re-checked whether this is still a gap before re-surfacing it: Touches only |
2 participants
What and Why
CLAUDE.md lists two non-negotiable latency targets under Design Principles:
Until now those were prose claims with no enforcement. Any contributor could add an eager
import chromadbtomempalace/__init__.pyand the promise would silently regress. A lean package import is the concrete lower bound for both claims, so the first thing to protect is the import path.Change Summary
Adds
tests/benchmarks/test_performance_budgets.pywith two regression tests that each run in a fresh Python subprocess — so the import is truly cold, not polluted by whatever pytest already loaded:test_package_import_under_startup_budget—import mempalacemust finish under 100mstest_cli_import_under_startup_budget—from mempalace import climust also finish under 100ms; this is the path hooks take on invocation, so a regression adds latency to every Stop/PreCompact call before any real work beginsBudget is scaled 3× on CI (
CI=trueenv var) to absorb shared-runner jitter without flaking.Why
tests/benchmarks/Lives under
tests/benchmarks/so it is excluded from the defaultrun — it does not slow the main CI loop. Invoke explicitly when validating performance-sensitive changes:
If the maintainers prefer this to run in main CI, happy to move it and add a
pytest -m perfmarker instead.Local Measurements
On macOS Python 3.13, three cold runs each:
import mempalacefrom mempalace import cliBoth comfortably under the 100ms target. The test fails when real drift appears, with an error message that points the reader at the most common cause (a new eager import in
__init__.py).Test Plan
pytest tests/benchmarks/test_performance_budgets.py -v— 2/2 passruff check/ruff format --check— cleanFollow-up (out of scope)
CLAUDE.md also claims Hooks under 500ms. Enforcing that requires a palace fixture and full hook invocation — a larger PR. If this approach is welcome I'll send a second PR for the hook budget using
mempal_save_hook.shagainst a tmp palace.