Skip to content

fix(agentx): default to uncapped traces with explicit 256k exceptions / 默认使用完整轨迹并显式列出 256k 例外 #4287

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
functionstackx merged 2 commits into from
Oct 18, 2026
Merged

fix(agentx): default to uncapped traces with explicit 256k exceptions / 默认使用完整轨迹并显式列出 256k 例外 #4287

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c5294e5
fix(agentx): default to the uncapped trace corpus
functionstackx Oct 18, 2026
f863d1a
fix(agentx): use the full trace corpus for Muse Spark
functionstackx Oct 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions inferencex-e2e/docs/agentx-standalone.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,14 @@ environment variables override preset defaults. Model, tokenizer, dataset,
concurrency, and output remain caller inputs; `--tokenizer-trust-remote-code` is
needed only for tokenizers with custom code.

InferenceX's `python3 -m infx.bench agentic` wrapper defaults to the full,
uncapped corpus (`semianalysis_cc_traces_weka_062126`). Model prefixes beginning
with `dsr1`, `qwen3.5`, `qwen3.8next`, or `musespark` explicitly select the
256k-capped corpus (`semianalysis_cc_traces_weka_062126_256k`). New model families
use the full corpus unless added to `CAPPED_FAMILIES` in
`infx/bench/agentic/traces.py`. `WEKA_LOADER_OVERRIDE` can pin either loader
regardless of model family; it does not configure the standalone `agentx` CLI.

The default random seed is **42**. Keep this seed when reproducing AgentX results,
or pass `--random-seed 123` to use a different seed.

Expand Down
4 changes: 2 additions & 2 deletions inferencex-e2e/infx/bench/agentic/traces.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,12 @@
UNCAPPED: "semianalysisai/cc-traces-weka-062126",
CAPPED: "semianalysisai/cc-traces-weka-062126-256k",
}
UNCAPPED_FAMILIES = ("dsv4", "glm5.2", "glm5.3", "minimaxm3", "kimik3")
CAPPED_FAMILIES = ("dsr1", "qwen3.5", "qwen3.8next", "musespark")


def resolve(model_prefix: str, override: str | None) -> tuple[str, str]:
"""``(loader, hf_dataset)``; ``override`` (``WEKA_LOADER_OVERRIDE``) pins either corpus."""
loader = override or (UNCAPPED if model_prefix.startswith(UNCAPPED_FAMILIES) else CAPPED)
loader = override or (CAPPED if model_prefix.startswith(CAPPED_FAMILIES) else UNCAPPED)
if loader not in LOADERS:
raise InputError(f"unknown WEKA_LOADER_OVERRIDE={loader!r}; allowed: {', '.join(LOADERS)}")
return loader, LOADERS[loader]
2 changes: 1 addition & 1 deletion inferencex-e2e/infx/tests/bench/test_agentic_command.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -426,7 +426,7 @@ def frontend(method: str, path: str) -> tuple[int, object]:

assert result.returncode == 0, result.stderr
assert _events(tmp_path) == [
f"hf download --repo-type dataset {traces.LOADERS[traces.CAPPED]}",
f"hf download --repo-type dataset {traces.LOADERS[traces.UNCAPPED]}",

@claude claude Bot Oct 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 nit (optional): Two updated expectations still build the expected corpus from traces constants, so they pass even if those constants are wrong. sweep:traces\.LOADERS\[traces\.|"--public-dataset", traces\. Example: test_agentic_command.py:429 expects traces.LOADERS[traces.UNCAPPED]. test_agentic_replay.py:43 expects traces.UNCAPPED. Fix: hard-code semianalysisai/cc-traces-weka-062126 and semianalysis_cc_traces_weka_062126, as the new parametrized test already does.

Why this was flagged

AGENTS.md:94 and AGENTS.md:103 say expected values must be worked out by hand and never taken from the implementation or its helpers. The diff rewrites test_agentic_command.py:429 to expect hf download --repo-type dataset {traces.LOADERS[traces.UNCAPPED]}. It rewrites test_agentic_replay.py:43 to expect "--public-dataset", traces.UNCAPPED. If someone swaps or mistypes the UNCAPPED or LOADERS values in traces.py, the shim downloads the wrong dataset and replay gets the wrong --public-dataset, but both tests still pass. The new test_corpus_is_uncapped_unless_explicitly_capped_or_pinned in the same diff already uses hand-written literals, so it shows the intended pattern.

Verification: Trigger: someone changes or mistypes the UNCAPPED/LOADERS constants in traces.py. test_agentic_command.py:429 and test_agentic_replay.py:43 still build their expected value from the module under test. AGENTS.md:103 requires "expected values worked out independently by hand, never derived by calling the implementation or its helpers". The pattern predates this PR.

"GET /v1/models",
"GET /v1/chat/completions",
"replay",
Expand Down
41 changes: 31 additions & 10 deletions inferencex-e2e/infx/tests/bench/test_agentic_replay.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ def test_default_point_leaves_the_replay_policy_to_the_scenario():
"--model", "deepseek-ai/DeepSeek-V4-Pro", "--tokenizer", "deepseek-ai/DeepSeek-V4-Pro",
"--concurrency", "8", "--benchmark-duration", "3600", "--tokenizer-trust-remote-code",
"--output-artifact-dir", "/results/aiperf_artifacts",
"--public-dataset", traces.CAPPED,
"--public-dataset", traces.UNCAPPED,
] # fmt: skip


Expand Down Expand Up @@ -168,16 +168,37 @@ def test_malformed_replay_inputs_are_rejected(overrides, message):


@pytest.mark.parametrize(
("prefix", "override", "loader"),
("prefix", "override", "capped"),
[
("big41flash", None, traces.UNCAPPED),
("other", None, traces.CAPPED),
("other", traces.UNCAPPED, traces.UNCAPPED),
("dsr1", None, True),
("qwen3.5", None, True),
("qwen3.8next", None, True),
("musespark", None, True),
("qwen3.5-variant", None, True),
("dsv4", None, False),
("dsv41flash", None, False),
("glm5.2", None, False),
("glm5.3", None, False),
("minimaxm3", None, False),
("kimik3", None, False),
("new-model", None, False),
("qwen3.9", None, False),
("new-model", "", False),
("qwen3.5", traces.UNCAPPED, False),
("new-model", traces.CAPPED, True),
],
)
def test_corpus_follows_the_model_family_context_unless_pinned(
monkeypatch, prefix, override, loader
):
monkeypatch.setattr(traces, "UNCAPPED_FAMILIES", ("big4",))
def test_corpus_is_uncapped_unless_explicitly_capped_or_pinned(prefix, override, capped):
loader, dataset = (
("semianalysis_cc_traces_weka_062126_256k", "semianalysisai/cc-traces-weka-062126-256k")
if capped
else ("semianalysis_cc_traces_weka_062126", "semianalysisai/cc-traces-weka-062126")
)
env = {**BASE_ENV, "MODEL_PREFIX": prefix}
if override is not None:
env["WEKA_LOADER_OVERRIDE"] = override

config = ReplayConfig.from_env(env, Path("/results"))

assert traces.resolve(prefix, override) == (loader, traces.LOADERS[loader])
assert (config.loader, config.dataset) == (loader, dataset)
assert _value(replay_argv(config, "/venv/bin/agentx"), "--public-dataset") == loader
Loading