Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion MODELS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

English | [中文](MODELS_zh.md)

This document tracks every model benchmarked by InferenceX: when it was added, which benchmark scenarios are currently active for it, and which scenarios are deprecated. Results for active scenarios are published to <https://inferencex.com/>.
This document tracks every model benchmarked by InferenceX-e2e: when it was added, which benchmark scenarios are currently active for it, and which scenarios are deprecated. Results for active scenarios are published to <https://inferencex.com/>.

Check warning on line 5 in MODELS.md

View check run for this annotation

Claude / Claude Code Review

MODELS.md now refers to undefined "InferenceX-e2e", inconsistent with rest of repo

MODELS.md and MODELS_zh.md now say the doc tracks models benchmarked by "InferenceX-e2e" instead of "InferenceX", but this term appears nowhere else in the repo (README.md, AGENTS.md, and 28+ other docs all use the unsuffixed "InferenceX") and is not defined anywhere. Worth confirming this rename is intentional (and should propagate elsewhere) before merging, since it could just as easily be an automation artifact.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 MODELS.md and MODELS_zh.md now say the doc tracks models benchmarked by "InferenceX-e2e" instead of "InferenceX", but this term appears nowhere else in the repo (README.md, AGENTS.md, and 28+ other docs all use the unsuffixed "InferenceX") and is not defined anywhere. Worth confirming this rename is intentional (and should propagate elsewhere) before merging, since it could just as easily be an automation artifact.

Extended reasoning...

What changed

The diff swaps a single word in the intro sentence of MODELS.md/MODELS_zh.md: "benchmarked by InferenceX" → "benchmarked by InferenceX-e2e". Nothing else in either file, or anywhere else in the repo, changes.

Why this is worth flagging despite being the PRs stated intent

The PR title and description do say this substitution is the entire point of the change ("MODELS doc now says ... InferenceX-e2e instead of InferenceX"), so on its face this is not an accidental typo the author overlooked mid-edit — it is the deliberate diff. I take that objection seriously and it is the reason I am not marking this normal.

But "the PR description matches the diff" is a different question from "the term is correct." A grep across the entire repository for InferenceX-e2e returns exactly the two lines this PR touches — zero other hits. Meanwhile the canonical name InferenceX appears 225+ times across README.md (# InferenceX™, Open Source Continuous Inference Standard), AGENTS.md ("InferenceX is an open-source automated benchmarking system..."), CONTRIBUTING.md, and every other doc in the repo. Results are published to inferencex.com, and the repo itself is SemiAnalysisAI/InferenceX. No workflow, script, or doc anywhere defines an entity called "InferenceX-e2e" or explains how it would differ from InferenceX (e.g., a distinct end-to-end test harness vs. the umbrella benchmarking project).

This PR is tagged [Klaud Cold], i.e., produced by an automated docs-maintenance agent, not a human author making a considered product-naming decision. For that class of change, "the diff matches the PR description" only tells you the agent executed its own plan faithfully — it does not tell you the plan itself was sound. A one-word rename to a brand/product name that (a) is unique to two lines in the whole repo, (b) has no definition anywhere, and (c) contradicts the same name used in README.md and AGENTS.md is a classic shape for an LLM-introduced hallucination/drift that happens to be internally consistent with its own PR description, since the agent wrote both.

Step-by-step proof

  1. grep -rn "InferenceX-e2e" . → only MODELS.md:5 and MODELS_zh.md:5 (this PRs two changed lines).
  2. grep -rn "InferenceX" README.md | head -1# InferenceX™, Open Source Continuous Inference Standard — confirms the canonical, trademarked name has no -e2e suffix.
  3. grep -n "InferenceX is" AGENTS.mdInferenceX is an open-source automated benchmarking system... — the repos own architecture doc, unmodified by this PR, still uses the unsuffixed name.
  4. No file/workflow defines InferenceX-e2e, so a reader of MODELS.md landing on this sentence has no way to resolve what makes it different from the InferenceX defined two clicks away in README.md.

Suggested resolution

Before merging, confirm whether "InferenceX-e2e" is an intentional new term that should be introduced (with a definition) and propagated consistently, or whether it should simply read "InferenceX" to match the rest of the repo. Given the total absence of the term elsewhere, reverting to "InferenceX" is the safer default absent an explicit definition.


## Scenarios

Expand Down
2 changes: 1 addition & 1 deletion MODELS_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

[English](MODELS.md) | 中文

本文档记录 InferenceX 基准测试覆盖的所有模型:加入日期、当前启用的基准测试场景,以及已弃用的场景。启用场景的结果发布于 <https://inferencex.com/>。
本文档记录 InferenceX-e2e 基准测试覆盖的所有模型:加入日期、当前启用的基准测试场景,以及已弃用的场景。启用场景的结果发布于 <https://inferencex.com/>。

## 场景

Expand Down