fix(benchmark): refresh NVIDIA cost evidence - #1073
Merged
Merged
Conversation
Signed-off-by: Seongho Bae <me@seonghobae.me>
seonghobae
enabled auto-merge (squash)
September 5, 2026 05:58
|
Warning Review limit reachedNext included review available in 27 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (5)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Signed-off-by: Seongho Bae <me@seonghobae.me>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 510ad9915b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| separate evidence classes. | ||
|
|
||
| As reviewed on 2026-08-05, NVIDIA's NIM General FAQ states that NVIDIA Developer | ||
| As reviewed on 2026-09-05, NVIDIA's NIM General FAQ states that NVIDIA Developer |
There was a problem hiding this comment.
The refreshed evidence now claims the NVIDIA FAQ was reviewed on September 5 and uses that claim to extend live-run validity, but the full citation at line 241 still says it was retrieved on August 5. This leaves the evidence-grade document internally contradictory and provides no bibliographic record of the retrieval underlying the new validity window; update the citation to the actual September review date.
Useful? React with 👍 / 👎.
This was referenced Sep 5, 2026
seonghobae
added a commit
that referenced
this pull request
Sep 5, 2026
Resolve the NVIDIA hosted-access evidence conflict by combining both sides: this branch's cited source and main's validity date. main's #1073 refreshed only the dates (2026-08-05 -> 2026-09-05, valid_until 2026-09-04 -> 2026-10-05) and never touched the source document, so switching the citation to "Run NIM Anywhere" is this branch's own delta -- taking main's side would leave the PR a no-op. The dates go the other way: the established convention is reviewed_at + 30 days (base: 08-05 -> 09-04), which main's 2026-10-05 follows and this branch's 2026-10-04 misses by one. Two things the conflict markers did not show: - nim_benchmark.py:2538 validates source_url against a hard-coded literal and auto-merged cleanly to this branch's "run-anywhere" while line 116 was still conflicted. Picking main's "/product" there would have left the module rejecting its own default evidence. - CHANGELOG.md still read 2026-10-04 after the resolution; it auto-merged without conflict because only this branch had touched it. The conflicting test mutation is the same test on both sides, each adjusted to its own dates. Under valid_until=2026-10-05 only main's mutation still triggers "validity precedes", so it is the one kept. Verified: 3396 passed, 1 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Sep 5, 2026
seonghobae
pushed a commit
that referenced
this pull request
Sep 5, 2026
test_live_run_without_evidence_fixture_still_fails_closed_on_expired_evidence asserted that a test not requesting the current_actual_cost_evidence fixture observes real-world ACTUAL_COST_EVIDENCE as expired -- true only because today happened to be past the literal valid_until_date at authoring time. Once #1073 refreshed that production evidence (reviewed_at_date=2026-09-05, valid_until_date=2026-10-05), this PR's own required "Tests and package quality" check failed for real: the evidence gate no longer raised, so the test fell through to a genuine network probe against NVIDIA's live API and got HTTP 403 -- the exact same class of ticking-time-bomb bug this PR is otherwise fixing, just introduced by this PR's own new regression test. Fix: the test now deliberately monkeypatches ACTUAL_COST_EVIDENCE to a fixed past window (2020) itself, proving the same opt-in-fixture-vs-file-wide mechanism without depending on real wall-clock time relative to whatever the production valid_until_date currently is. Verified locally against both the unmodified branch and a merge of this branch with current origin/main (post-#1073 evidence). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX
seonghobae
pushed a commit
that referenced
this pull request
Sep 5, 2026
test_live_run_without_evidence_fixture_still_fails_closed_on_expired_evidence asserted that a test not requesting the current_actual_cost_evidence fixture observes real-world ACTUAL_COST_EVIDENCE as expired -- true only because today happened to be past the literal valid_until_date at authoring time. Once #1073 refreshed that production evidence (reviewed_at_date=2026-09-05, valid_until_date=2026-10-05), this PR's own required "Tests and package quality" check failed for real: the evidence gate no longer raised, so the test fell through to a genuine network probe against NVIDIA's live API and got HTTP 403 -- the exact same class of ticking-time-bomb bug this PR is otherwise fixing, just introduced by this PR's own new regression test. Fix: the test now deliberately monkeypatches ACTUAL_COST_EVIDENCE to a fixed past window (2020) itself, proving the same opt-in-fixture-vs-file-wide mechanism without depending on real wall-clock time relative to whatever the production valid_until_date currently is. Verified locally against both the unmodified branch and a merge of this branch with current origin/main (post-#1073 evidence). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX
This was referenced Sep 5, 2026
seonghobae
added a commit
that referenced
this pull request
Sep 10, 2026
Explicit user-authorized bypass to reduce the stalled infrastructure queue. Reviewed both changed files at 59a8f4e, including the resolved closed-PR admission regression. Security and Quality 34073908489, Scan 34073908487 and SAST 34073908484 succeeded; later Draft-event Quality 34085323868 is skipped, not a success, and compatibility CodeQL 34073908462 remains failed. The expired NIM evidence foundation was already corrected through #1073, not open #1069. Remove the pull_request base filter, retain main-only push and trusted Ready/open admission, and coalesce obsolete PR verification runs by workflow/repository/PR. No formal approval, fresh integrated-head GREEN, scanner bypass or fake status is claimed. Preserve #1060 until complete effective-delta equivalence is independently verified. Normal merge retains all later main runtime repairs and requests new integrated push verification.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
문제
NVIDIA hosted-endpoint 비용 근거의
valid_until_date가 2026-09-04로 만료돼 2026-09-05부터 live benchmark 경로와 전체 테스트 5개가 선행 실패했습니다.근거 갱신
2026-09-05에 NVIDIA NIM General FAQ를 다시 확인했습니다. NVIDIA Developer Program의 hosted API endpoint는 prototyping·research·development·testing 용도로 무료이며, production 사용은 NVIDIA AI Enterprise license가 필요하다는 기존 경계가 그대로 유지됩니다.
공식 출처: https://docs.api.nvidia.com/nim/docs/product
수정
검증
uv run pytest -q tests/test_nim_benchmark.py tests/test_nim_benchmark_release_acceptance.py126 passed
git diff --check통과