feat(autotuner): Autotuner v2 (autotune_v2) — managed persistence, deployment-matched measurement, runner contract

[YangXu1990uiuc]

📌 Description

Autotuner v2 — an evolution of the autotuner exposed through a standalone autotune_v2(), additive over autotune() (which stays byte-identical). It started as a managed persistent cache and has grown, across review, into five composing pillars:

1. Managed persistence — FlashInfer owns cache keys, schema, environment isolation, and invalidation; frameworks stop hashing filenames, parsing the JSON schema, broadcasting raw cache files, and managing staleness. Per-entry atomic store, safe for concurrent all-ranks tuning.
2. Deployment-matched measurement (MeasurementPolicy) — tune the way you serve (eager vs CUDA-graph host-cost semantics); part of the store's environment identity.
3. Runner contract — self-describing tactics + runtime revalidation (validate_tactic, covering in-memory and on-disk hits) so a stale/invalid tactic is a loud fallback, never a silent wrong kernel or a dead server.
4. Distributed consistency — a shared store plus autotune_v2_reload() finalize step so homogeneous ranks converge on byte-identical tactics (composes with Add set_autotune_process_group to synchronize tactic choice across ranks #3187).
5. Selection quality — a default-candidate coverage guard (a tuned result cannot lose to not-tuning at the probe inputs) and an accuracy harness that quantifies regret against a deployment-faithful oracle.

RFC (design + evidence, for public/customer review): #3920. Detailed rationale, the review-round fixes, framework-migration drafts, and the production-B200 validation are in the thread below.

Summary

FlashInfer owns the persistence and compatibility rules for autotuning results, so frameworks stop constructing cache filenames, parsing FlashInfer's JSON schema, broadcasting raw cache files, and managing invalidation themselves — and, via MeasurementPolicy, tuning measures the way the deployment actually runs.

Exposed as a standalone autotune_v2() context manager, deliberately disjoint from autotune(cache=<json path>) (v1) so the design can iterate without inheriting v1 semantics — flashinfer/autotuner/ changes are purely additive hooks and autotune() itself is untouched.

API

with flashinfer.autotune_v2(measure=MeasurementPolicy(execution_mode="cuda_graph")):
    model(dummy_inputs)          # startup warmup: tune + publish atomically
model(inputs)                    # serving: reuses entries, no context needed

# fresh process, same environment: hydrate + serve
with flashinfer.autotune_v2(mode="replay", measure=policy):
    pass

mode	persistent_cache	meaning
"tune"	True (default)	tune misses, publish to disk
"tune"	False	tune in-memory only (no disk I/O)
"replay"	True	serve from the on-disk cache (no profiling)
"replay"	False	no-op unless already hydrated

Attach semantics: persistent_cache=True attaches the store for the remainder of the process (like v1's load_configs) — a survey of vLLM/sglang showed both serve outside any context, so a context-scoped store would silently regress serving to heuristics after warmup exits. The context scopes only when profiling cost may be paid. cache_root (or FLASHINFER_AUTOTUNE_CACHE_DIR) is a placement-only directory; the schema/environment namespaces live below it.

MeasurementPolicy — tune the way you deploy

The decisive question is whether per-call host cost counts: under CUDA-graph serving it is paid once at capture (excluding it is correct); under eager serving it is paid every call. Measured on SM100 (bmm_fp8 M=8, kernel ~8µs): the same cuDNN candidate reads 8.0µs host-excluded vs ~330µs host-included — the eager ranking flips to cublas < cutlass ≪ cuDNN, matching real framework behavior.

MeasurementPolicy(execution_mode="cuda_graph")  # profile under real capture+replay
MeasurementPolicy(execution_mode="eager")       # host cost included (no delay kernel)
# default "auto" = today's legacy behavior; cold_l2 composes. The timer is derived
# from execution_mode and is always CUDA events: "auto"/"cuda_graph" use the
# events+delay-kernel window, "eager" uses events without the delay kernel.
# CUPTI activity-span timing is NOT selected by any public mode; it exists only
# as a private diagnostic override (_timer="cupti") used by the accuracy harness.

The policy is part of the store's environment identity (manifest → env hash): entries tuned under different policies never overwrite each other. A CUPTI activity-span timer is implemented as a diagnostic-only route (per-iteration GPU spans, median; no delay kernel; cuptiIsTracingSessionRunning pre-check + graceful events fallback around the legacy single-subscriber limitation) and is reachable only via the private _timer="cupti" override, not through execution_mode. It is not used for any shipped policy; see the validation report's CUPTI appendix for why (it cannot rank eagerly-launched multi-kernel ops such as moe_cute_dsl).

Store design

<root>/v2/<environment_hash>/          # sha256 of canonical manifest (flashinfer/CUDA/
  manifest.json                        #   cuBLAS/cuDNN backend+frontend/GPU [+ policy])
  entries/<operation_hash>.json        # one atomic file per tuned operation

• One entry per tuned op, published via tempfile + atomic os.replace as soon as it is tuned — no exit-time read/merge/write, no locks; concurrent all-ranks tuning is safe (redundant work, last valid write wins), so per-rank cache files become unnecessary.
• Missing/malformed/key-mismatched entries are cache misses, never errors; each entry embeds its canonical key.
• Hit/miss memoized (decoded once); the serving hot path re-reads neither the filesystem nor JSON.
• Incompatible environments hash to different directories: nothing to invalidate manually, downgrade-safe.
• v1 and v2 never share in-memory state — a v1 save_configs structurally cannot observe v2 entries (regression-tested).

Also included

• Runner contract: validate_tactic(inputs, tactic) revalidation on in-memory and on-disk hits (a rejected tactic is a loud fallback, not a blind replay); default-candidate coverage guard.
• Distributed: autotune_v2_reload() finalize step for rank convergence.
• benchmarks/bench_autotuner_accuracy.py: accuracy harness (oracle sweep → regret / top-1 / winner-flip-rate under each policy, scored against both oracles), with an interleaved drift-protected oracle and per-round SM-clock recording. Validated on a production B200 across bmm_fp8 / mm_fp4 / cutlass MoE (thread).
• Observability: loud INFO at store attach, once-per-op hit provenance logs.

Explicitly out of scope (this PR)

• opaque export/install artifact for multi-node distribution (only needed for leader-tunes-then-broadcast without a shared FS; under discussion)
• flipping execution_mode="auto" to "cuda_graph" (pending forced-capture validation across the op suite)
• per-runner adoption of validate_tactic (cuDNN first, via its Yanqinz/autotuner tactic #3707 structured tactics — the hook is here; runners opt in separately)
• representative-probe construction for MoE (EP-deflated shapes / skewed-routing dummies — the [Perf] cutlass-MoE autotuner tactic selection regresses high-concurrency throughput (unrepresentative profiling: routing distribution / effective per-expert M) #3622/[Perf] trtllm_fp4_block_scale_moe autotuner can pick slower-than-default tactics and is not EP/DP-aware (MXFP4, SM100) #3537 op-level track; sibling PRs)
• serialized backend artifacts (e.g. cuDNN plan sidecar, after Yanqinz/autotuner tactic #3707); GC/pruning; cross-job single-flight; comm-op tuning; any change to v1 behavior

Relation to existing work

Orthogonal to #3707 (structured (engine_id, knobs) cuDNN tactics): entries store whatever tactic representation runners produce — compound-tuple round-trip is covered by tests; the environment hash contains index-tactic drift meanwhile.

Testing

238 GPU-free tests pass in tests/autotuner/ (the v2 file test_autotune_cache_v2.py covering: both real consumer patterns — serve-after-context-exit, hydrate-only; corruption tolerance; env-hash and measurement-policy isolation; bare-serving-uses-ambient-identity and cache_root-switch repopulation; atomicity; v1 coexistence; runner-reorder / extras-collision key completeness; guard win-persist-replay; reload convergence; same-process revalidation fallback — plus the upstream suites incl. the #3187 AST guards). Pre-commit clean.

On-GPU validation on SM100 and a production B200: tune → atomic publish → fresh-process serve; a 4-process concurrent tuning storm into one shared store (SGLang all-ranks pattern) with store integrity intact; and the accuracy harness across three ops (bmm_fp8 / mm_fp4 / cutlass MoE) — details, matrices, and framework-migration drafts in the thread. Branch went through several external review rounds plus a four-lens cleanup pass.

🤖 Generated with Claude Code

🔍 Related Issues

RFC #3920 (design) · #3707 (structured tactics, composed) · #3187 (cross-rank sync, composed) · addresses classes from #3186, #3537, #3566, #3622, #3648, #3719 (see the RFC's issue-bucket table)

🚀 Pull Request Checklist

Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete.

✅ Pre-commit Checks

• I have installed pre-commit by running pip install pre-commit (or used your preferred method).
• I have installed the hooks with pre-commit install.
• I have run the hooks manually with pre-commit run --all-files and fixed any reported issues.

🧪 Tests

• Tests have been added or updated as needed (tests/autotuner/test_autotune_cache_v2.py, 28 GPU-free tests; full tests/autotuner/ suite green).
• All tests are passing.

Reviewer Notes

Opt-in and additive: autotune() and the v1 cache path are byte-identical; v2 state lives behind autotune_v2(). Suggested review order: flashinfer/autotune_cache.py (store + API), then the four hook sites in flashinfer/autotuner/autotuner.py (search_cache 2.5, publish, measurement routing, store stack), then the tests.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

This PR adds a managed v2 autotune cache backend, wires it into autotune_v2() and AutoTuner, and adds tests covering persistence, reuse, isolation, compatibility, and lifecycle behavior.

Changes

Managed autotune cache

Layer / File(s)	Summary
Cache root and atomic write helpers flashinfer/autotune_cache.py	Defines the v2 schema version, cache root selection, canonical JSON serialization, and atomic JSON file writing helpers.
ManagedAutotuneCache implementation flashinfer/autotune_cache.py	Implements environment-hashed entry storage, manifest initialization, per-key lookup and publish operations, and per-process memoization for hits and misses.
autotune_v2 context manager flashinfer/autotune_cache.py, flashinfer/__init__.py	Adds the managed autotune context manager and exposes it from the package top level.
AutoTuner managed cache wiring flashinfer/autotuner.py	Adds managed backend state, managed-cache lookup in search_cache(), eager publish during profiling, and memo clearing on cache reset.
Persistence and lookup tests tests/autotuner/test_autotune_cache_v2.py	Adds scaffolding and coverage for publishing entries, lookup reuse, corruption handling, environment isolation, tuple round-tripping, and publish failure tolerance.
Compatibility and lifecycle tests tests/autotuner/test_autotune_cache_v2.py	Adds coverage for v1 and PathLike compatibility, root relocation, persistent-cache disablement, disk separation, attach idempotence, hydrate-only reuse, and context-entry failure cleanup.

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related PRs

• flashinfer-ai/flashinfer#2554: Both PRs modify flashinfer/autotuner.py cache lookup and tactic serialization behavior.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly identifies Autotuner v2 and managed persistence, which are central changes. Its additional references to measurement and runner contracts remain related to the stated feature scope.
Description check	✅ Passed	The description includes the required description, related issues, testing, checklist, and reviewer notes sections. It provides detailed scope, API, implementation, testing, and out-of-scope informati…

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[gemini-code-assist]

Code Review

This pull request introduces an experimental managed on-disk autotune cache (v2) that stores tuning results per-entry under environment-hashed directories, allowing for automatic, atomic persistence without manual invalidation. The reviewer feedback focuses on two main improvements: first, ensuring that _file_configs and _logged_file_hits are properly saved and restored upon entering and exiting the autotune context to prevent configuration leakage or loss in nested contexts; second, explicitly specifying encoding="utf-8" when reading and writing cache files to guarantee cross-platform consistency.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

To prevent configuration leaks and loss of previously loaded configs in nested or sequential autotune contexts, we should save the previous state of _file_configs and _logged_file_hits on entry so they can be restored on exit.

    use_managed_cache = cache is True
    managed_prev = None
    file_configs_prev = None
    logged_file_hits_prev = None
    if use_managed_cache:
        # Managed v2 backend: per-entry files under an environment-hashed
        # directory.  Nothing is loaded up front (entries are read lazily
        # on cache miss) and there is no exit-time save (winners are
        # published atomically as soon as they are tuned).
        from .autotune_cache import ManagedAutotuneCache

        with tuner._lock:
            file_configs_prev = dict(tuner._file_configs)
            logged_file_hits_prev = set(tuner._logged_file_hits)
            tuner._file_configs.clear()
            tuner._logged_file_hits.clear()
            managed_prev = tuner._managed_cache
            tuner._managed_cache = ManagedAutotuneCache(manifest=_collect_metadata())
    elif isinstance(cache, str):
        with tuner._lock:
            file_configs_prev = dict(tuner._file_configs)
            logged_file_hits_prev = set(tuner._logged_file_hits)
            tuner._file_configs.clear()
            tuner._logged_file_hits.clear()

[YangXu1990uiuc]

This comment targeted an early iteration where v2 rode on autotune(cache=True) and mutated _file_configs. The API has since been forked into a standalone autotune_v2(): autotune() is byte-identical to main, _file_configs/_logged_file_hits are never touched by v2, and the managed store deliberately uses attach semantics (process-lifetime, like load_configs) rather than save/restore scoping — both vLLM and SGLang serve outside any context, so a context-scoped store would silently regress serving to heuristics after warmup. Nested/sequential contexts therefore need no state restoration by design.

[gemini-code-assist]

Restore the saved _file_configs and _logged_file_hits in the exception handler to ensure the state is correctly rolled back if context setup fails.

Suggested change

	if use_managed_cache:
	with tuner._lock:
	tuner._managed_cache = managed_prev
	if use_managed_cache or isinstance(cache, str):
	with tuner._lock:
	if use_managed_cache:
	tuner._managed_cache = managed_prev
	tuner._file_configs = file_configs_prev
	tuner._logged_file_hits = logged_file_hits_prev

[YangXu1990uiuc]

This comment targeted an early iteration where v2 rode on autotune(cache=True) and mutated _file_configs. The API has since been forked into a standalone autotune_v2(): autotune() is byte-identical to main, _file_configs/_logged_file_hits are never touched by v2, and the managed store deliberately uses attach semantics (process-lifetime, like load_configs) rather than save/restore scoping — both vLLM and SGLang serve outside any context, so a context-scoped store would silently regress serving to heuristics after warmup. Nested/sequential contexts therefore need no state restoration by design.

[gemini-code-assist]

Restore the saved _file_configs and _logged_file_hits on normal exit of the context manager to prevent configuration leakage into outer or subsequent contexts.

Suggested change

	if use_managed_cache:
	# Managed entries were already published during tuning; just
	# restore the previously active backend (if any).
	with tuner._lock:
	tuner._managed_cache = managed_prev
	# Save configs on exit when tuning with a legacy cache path,
	# but only if new profiling results were added this session
	# and the cache file was valid (no environment mismatch).
	if cache is not None and cache_valid and tune_mode and tuner._dirty:
	elif isinstance(cache, str) and cache_valid and tune_mode and tuner._dirty:
	tuner.save_configs(cache)
	if use_managed_cache:
	# Managed entries were already published during tuning; just
	# restore the previously active backend (if any).
	with tuner._lock:
	tuner._managed_cache = managed_prev
	tuner._file_configs = file_configs_prev
	tuner._logged_file_hits = logged_file_hits_prev
	# Save configs on exit when tuning with a legacy cache path,
	# but only if new profiling results were added this session
	# and the cache file was valid (no environment mismatch).
	elif isinstance(cache, str) and cache_valid and tune_mode and tuner._dirty:
	tuner.save_configs(cache)
	with tuner._lock:
	tuner._file_configs = file_configs_prev
	tuner._logged_file_hits = logged_file_hits_prev
	elif isinstance(cache, str):
	with tuner._lock:
	tuner._file_configs = file_configs_prev
	tuner._logged_file_hits = logged_file_hits_prev

[YangXu1990uiuc]

This comment targeted an early iteration where v2 rode on autotune(cache=True) and mutated _file_configs. The API has since been forked into a standalone autotune_v2(): autotune() is byte-identical to main, _file_configs/_logged_file_hits are never touched by v2, and the managed store deliberately uses attach semantics (process-lifetime, like load_configs) rather than save/restore scoping — both vLLM and SGLang serve outside any context, so a context-scoped store would silently regress serving to heuristics after warmup. Nested/sequential contexts therefore need no state restoration by design.

[gemini-code-assist]

Specify encoding="utf-8" in os.fdopen to ensure that the JSON file is written consistently across different platforms (especially on Windows where the default encoding is often not UTF-8).

Suggested change

	with os.fdopen(fd, "w") as f:
	with os.fdopen(fd, "w", encoding="utf-8") as f:

[YangXu1990uiuc]

Fixed in ab85c2c — encoding="utf-8" added to the store's write path.

[gemini-code-assist]

Specify encoding="utf-8" when opening the cache file to ensure consistent reading across different platforms, matching the encoding used when writing.

Suggested change

	with open(path, "r") as f:
	with open(path, "r", encoding="utf-8") as f:

[YangXu1990uiuc]

Fixed in ab85c2c — encoding="utf-8" added to the store's read path, matching the writer.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@flashinfer/autotuner.py`:
- Around line 625-639: Managed cache activation is not context-safe because the
shared _managed_cache field can be reused or restored out of order across nested
or overlapping cache contexts, causing legacy cache=str paths to mix with the
managed backend. Update the cache context handling in autotuner.py around the
cache=True and cache=str branches (and the corresponding restore/cleanup sites)
to snapshot and restore the full backend state per context/thread, or maintain a
backend stack so each context gets its own active cache backend. Ensure the
legacy cache path explicitly disables managed cache while it is active, and that
exiting one context cannot resurrect or disable another active context’s
backend.

In `@tests/autotuner/test_autotune_cache_v2.py`:
- Around line 109-114: The test in AutoTuner.get().choose_one assigns an unused
runner variable when unpacking the return value, which should be changed to a
throwaway placeholder to satisfy Ruff. Update the unpacking in the
autotune(False, cache=True) block so only the used tactic value is named,
keeping the assertion focused and avoiding the unused local.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 27c6c5a7-ddd8-4f79-ba0f-cba0f7e5393a

📥 Commits

Reviewing files that changed from the base of the PR and between 5b8da12 and 26abedb6d444f7d809a2d62fba58be4f3f3f081f.

📒 Files selected for processing (3)

• flashinfer/autotune_cache.py
• flashinfer/autotuner.py
• tests/autotuner/test_autotune_cache_v2.py

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

flashinfer/autotuner.py (1)

1098-1101: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Stale API reference in comment.

The comment says the managed cache is "set by autotune(cache=True)", but the activation API is now persist=True; cache=True raises TypeError (verified by test_invalid_persist_and_cache_types_raise). Update to avoid misleading maintainers.

📝 Proposed fix

-                # 2.5 Managed v2 per-entry cache (experimental, set by
-                #     autotune(cache=True)).  One JSON file per entry, read
+                # 2.5 Managed v2 per-entry cache (experimental, set by
+                #     autotune(persist=True)).  One JSON file per entry, read

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@flashinfer/autotuner.py` around lines 1098 - 1101, The managed v2 per-entry
cache comment is referring to the old autotune(cache=True) API, which is now
obsolete and misleading. Update the wording in the nearby cache
documentation/comments around the managed cache path to reference
autotune(persist=True) instead, keeping the description of lazy JSON loading and
promotion into _file_configs accurate and aligned with the current API.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@flashinfer/autotuner.py`:
- Around line 1098-1101: The managed v2 per-entry cache comment is referring to
the old autotune(cache=True) API, which is now obsolete and misleading. Update
the wording in the nearby cache documentation/comments around the managed cache
path to reference autotune(persist=True) instead, keeping the description of
lazy JSON loading and promotion into _file_configs accurate and aligned with the
current API.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 0a6dc4ca-6261-4295-888c-bf1978407bfe

📥 Commits

Reviewing files that changed from the base of the PR and between 26abedb6d444f7d809a2d62fba58be4f3f3f081f and 31625846f45a4db712215be892e70143761e1b29.

📒 Files selected for processing (3)

• flashinfer/autotune_cache.py
• flashinfer/autotuner.py
• tests/autotuner/test_autotune_cache_v2.py

🚧 Files skipped from review as they are similar to previous changes (1)

• flashinfer/autotune_cache.py

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

flashinfer/autotuner.py (1)

1009-1057: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Do not let managed v2 contexts read legacy _file_configs.

Inside autotune_v2(), search_cache() still checks _file_configs before _managed_cache, so a previously loaded v1 JSON config can override or satisfy a v2 managed lookup. Gate the legacy block behind managed_cache is None, and consult only the managed backend while it is active.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@flashinfer/autotuner.py` around lines 1009 - 1057, The search_cache logic in
autotuner.py still lets legacy _file_configs satisfy lookups before the managed
v2 backend, so autotune_v2() can be overridden by v1 configs. Update the cache
resolution in search_cache() to skip the _file_configs block whenever
self._managed_cache is active, and only consult self._managed_cache for v2
contexts while preserving the existing logging and runner_id/tactic return flow.

♻️ Duplicate comments (1)

flashinfer/autotuner.py (1)

834-887: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

Make the active managed backend thread/context-local.

_managed_stack is process-global and _managed_cache returns the top entry, so overlapping autotune_v2(root=...) contexts on different threads can read/write whichever backend was entered most recently. Tokenized removal fixes teardown, but not per-thread active-backend routing.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@flashinfer/autotuner.py` around lines 834 - 887, The active managed backend
selection is still process-global because `_managed_stack` and the
`_managed_cache` property always use the top entry, so concurrent
`autotune_v2(root=...)` contexts can route reads/writes to the wrong backend.
Make the active backend thread/context-local in `flashinfer.autotuner` by
storing the current managed backend per thread or context (similar to
`_override_local` / `_skip_ops_local`) while keeping the existing tokenized
stack for safe teardown. Update `_managed_cache` and any push/pop logic in the
autotune_v2-managed path so each thread sees its own active backend instead of
the last-entered one.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@flashinfer/autotune_cache.py`:
- Around line 186-189: The persistence handling in autotune_v2 treats any truthy
persist value as enabled, so path-like inputs can silently use root=None and
write to the default cache location. Update autotune_v2 to either normalize
non-bool persist values into root when they are path-like, or explicitly reject
them with TypeError before persistence is enabled. Apply the same fix to the
related persistence flow in the later autotune_v2 logic so both paths use
consistent validation.

---

Outside diff comments:
In `@flashinfer/autotuner.py`:
- Around line 1009-1057: The search_cache logic in autotuner.py still lets
legacy _file_configs satisfy lookups before the managed v2 backend, so
autotune_v2() can be overridden by v1 configs. Update the cache resolution in
search_cache() to skip the _file_configs block whenever self._managed_cache is
active, and only consult self._managed_cache for v2 contexts while preserving
the existing logging and runner_id/tactic return flow.

---

Duplicate comments:
In `@flashinfer/autotuner.py`:
- Around line 834-887: The active managed backend selection is still
process-global because `_managed_stack` and the `_managed_cache` property always
use the top entry, so concurrent `autotune_v2(root=...)` contexts can route
reads/writes to the wrong backend. Make the active backend thread/context-local
in `flashinfer.autotuner` by storing the current managed backend per thread or
context (similar to `_override_local` / `_skip_ops_local`) while keeping the
existing tokenized stack for safe teardown. Update `_managed_cache` and any
push/pop logic in the autotune_v2-managed path so each thread sees its own
active backend instead of the last-entered one.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 6ad32f8b-af5b-499e-a4c3-10ff8df4c2e5

📥 Commits

Reviewing files that changed from the base of the PR and between 177e7909c3df30728ab68a7ea15fb43eaa9f19bf and 58b11d1e054830c6ca262ff625d482a2ce900f27.

📒 Files selected for processing (4)

• flashinfer/__init__.py
• flashinfer/autotune_cache.py
• flashinfer/autotuner.py
• tests/autotuner/test_autotune_cache_v2.py

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

flashinfer/autotuner.py (1)

1031-1057: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Include managed-cache hits in the tuned-range warning. choose_one() only checks profiling_cache and _file_configs, so a v2-only process can fall back outside the tuned bucket range without the perf-cliff warning. A small public accessor on ManagedAutotuneCache would let has_tune_data include backend hits without reaching into _hits directly.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@flashinfer/autotuner.py` around lines 1031 - 1057, The tuned-range warning in
choose_one() is missing managed-cache coverage, so v2-only hits can bypass the
perf-cliff check. Add a small public accessor on ManagedAutotuneCache to expose
whether tune data exists for a key, then update has_tune_data in autotuner.py to
include managed cache hits alongside profiling_cache and _file_configs without
reading _hits directly.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@flashinfer/autotune_cache.py`:
- Around line 303-318: The managed cache attachment in autotune_v2 should be
rolled back if entering the delegated autotune context fails, because
_attach_managed_cache can leave the process-wide cache in a modified state.
Update the autotune_v2 context manager so that the call to autotune(...) is
wrapped with cleanup logic that restores the prior AutoTuner managed cache when
__enter__ raises, using the existing tuner = AutoTuner.get() and
_attach_managed_cache path as the rollback point.

---

Outside diff comments:
In `@flashinfer/autotuner.py`:
- Around line 1031-1057: The tuned-range warning in choose_one() is missing
managed-cache coverage, so v2-only hits can bypass the perf-cliff check. Add a
small public accessor on ManagedAutotuneCache to expose whether tune data exists
for a key, then update has_tune_data in autotuner.py to include managed cache
hits alongside profiling_cache and _file_configs without reading _hits directly.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 11fae79a-1d86-4671-9b35-cbe9487f9246

📥 Commits

Reviewing files that changed from the base of the PR and between 58b11d1e054830c6ca262ff625d482a2ce900f27 and 38e2bb530ad2f7fb70fcf358864bfae62e070d2c.

📒 Files selected for processing (3)

• flashinfer/autotune_cache.py
• flashinfer/autotuner.py
• tests/autotuner/test_autotune_cache_v2.py

[YangXu1990uiuc]

Update (2 new commits):

• 1a247605 — observability for attach semantics: loud INFO log at store attach (repr: env_dir + memo counts), complementing the once-per-op source=managed cache hit log.
• e83f0201 — MeasurePolicy (first half of the measurement-policy milestone): autotune_v2(measure=MeasurePolicy(cuda_graph=..., cold_l2=...)) forces profiling to match the deployment (CUDA-graph replay vs eager; cold vs hot L2), promoting the previously op-author-only TuningConfig knobs to a user-selectable per-context policy (None fields inherit). Winners depend on how they were measured, so the policy is part of the store's environment identity (manifest → env_hash): entries tuned under different policies never overwrite each other, and consumers attach with the same policy. Inert when unused; v1 untouched. 22 GPU-free tests now.

Not yet included: CUPTI timing core (next), and the tuner-accuracy quantification harness (regret / top-1 / winner-flip-rate / replay-fidelity vs an oracle sweep — methodology in the proposal doc).

[YangXu1990uiuc]

Measurement-policy milestone complete (e83f0201 + b77ccea8):

MeasurePolicy(cuda_graph=..., cold_l2=..., timer="cupti") — profile the way the deployment runs. The CUPTI timer measures per-iteration GPU kernel spans from activity records (mirrors bench_gpu_time_with_cupti): no delay kernel needed — GPU-side timestamps structurally exclude host launch latency. On-GPU validation: identical winners and identical 2x tactic ratios under all five policy combinations, while absolute magnitudes legitimately diverge (67.7µs events vs 82.1µs cupti for the same matmul: steady-state throughput vs isolated-call latency semantics) — which is exactly why the timer is part of the policy and of the store's environment identity. cupti-python>=13 detected once, graceful events fallback. 24 GPU-free tests; 147 total in the autotuner suites.

[YangXu1990uiuc]

@flashinfer-bot run

[Vinnie6167]

/bot run tests/autotuner

[flashinfer-bot]

GitLab MR !1421 has been created, and the CI pipeline #66310023 is currently running. I'll report back once the pipeline job completes.