[TRTLLM-12838][infra] CBTS: coverage-based test selection - #16776
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughCBTS adds optional coverage-database test narrowing, maps changed Python lines to qualified names, preserves impacted and untrusted tests, removes safe YAML entries, exposes structured diagnostics, and propagates coverage settings through Jenkins and dry-run execution. ChangesCoverage-tier CBTS selection
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant JenkinsPipeline
participant main_py
participant CoverageSelector
participant TouchDB
participant coverage_tier_py
participant YAMLTestDB
JenkinsPipeline->>main_py: pass coverage database path
main_py->>CoverageSelector: run initial selection
CoverageSelector-->>main_py: fallback result and selector state
main_py->>TouchDB: open coverage database
main_py->>coverage_tier_py: apply coverage tier
coverage_tier_py->>TouchDB: query impacted and skippable tests
coverage_tier_py-->>main_py: narrowed stages and diagnostics
main_py->>YAMLTestDB: write narrowed test database
Suggested reviewers: 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
jenkins/L0_MergeRequest.groovy (2)
809-815: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
reasons.join('; ')will now print raw MaptoString()output.
result.reasonsused to be a list of strings;main.pyin this PR turns every entry into a structuredMap({source: ..., reason: ..., ...}). This log line still does a plain.join('; '), so the "CBTS: deferring" console message will render as GroovyMapliterals (e.g.[source:fallback, reason:unhandled_files, files:[...]]) instead of readable text.🩹 Suggested fix: format each structured reason before joining
- pipeline.echo("CBTS: deferring — Python returned scope=null. " + - "Reasons: ${result.reasons.join('; ')}") + def reasonStr = result.reasons.collect { r -> + if (r instanceof Map) { + def src = r.source ?: "?" + def rest = r.findAll { k, v -> k != "source" }.collect { k, v -> "${k}=${v}" }.join(", ") + return rest ? "[${src}] ${rest}" : "[${src}]" + } + return r.toString() + }.join('; ') + pipeline.echo("CBTS: deferring — Python returned scope=null. Reasons: ${reasonStr}")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@jenkins/L0_MergeRequest.groovy` around lines 809 - 815, Update the scope-null handling in the pipeline flow around _cbtsParseSelectionResult and _cbtsReportDecision so result.reasons maps are converted to readable reason text before joining. Reuse the structured fields, including source and reason and any relevant details, rather than calling join directly on the maps; preserve the existing deferring message and fallback decision behavior.
954-968: 🗄️ Data Integrity & Integration | 🔴 Critical | ⚡ Quick winCoverage-tier multi-GPU re-enable is broken:
_cbtsParseSelectionResultnever forwardsenable_multi_gpu/coverage_dropped_stagesto the Groovy consumer.main.py'sSelectionResult.to_json()emits these two new fields, but the Groovy-side JSON parser explicitly whitelists a fixed set of keys that doesn't include them, so they never reachtestFilter[(CBTS_RESULT)].
jenkins/L0_MergeRequest.groovy#L954-L968: addenable_multi_gpu: data.enable_multi_gpu ?: falseandcoverage_dropped_stages: data.coverage_dropped_stages ?: []to the map literal returned by_cbtsParseSelectionResult.jenkins/L0_Test.groovy#L5714-L5718: no code change needed here — this consumer is correct and will start working oncecbts.enable_multi_gpuis actually populated by the fix above; currently it always evaluates falsy, so multi-GPU stages are never re-added under coverage-tier selection even whenMULTI_GPU_FILE_CHANGEDis true.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@jenkins/L0_MergeRequest.groovy` around lines 954 - 968, The Groovy parser _cbtsParseSelectionResult in jenkins/L0_MergeRequest.groovy:954-968 must forward enable_multi_gpu with a false default and coverage_dropped_stages with an empty-list default in its returned map. Make no code change to jenkins/L0_Test.groovy:5714-5718; its consumer is already correct and will work once the parser populates cbts.enable_multi_gpu.
🧹 Nitpick comments (2)
jenkins/scripts/cbts/coverage_selection/qualname_map.py (1)
79-86: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueComplete the public API docstrings.
jenkins/scripts/cbts/coverage_selection/qualname_map.py#L79-L86: documentsource,lines, and theok=Falseparse-failure result using Google-styleArgsandReturns.jenkins/scripts/cbts/coverage_selection/selector.py#L108-L112: documentresidual_files,diffs, and allCoverageResultoutcomes using Google-style sections.As per coding guidelines, “Prefer docstrings for external interfaces, use Google-style docstrings, document public function arguments.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@jenkins/scripts/cbts/coverage_selection/qualname_map.py` around lines 79 - 86, The public API docstrings are missing parameter and result documentation. In jenkins/scripts/cbts/coverage_selection/qualname_map.py lines 79-86, update qualnames_for_lines with Google-style Args for source and lines and Returns describing the qualname set and the ok=False result when parsing fails. In jenkins/scripts/cbts/coverage_selection/selector.py lines 108-112, update the affected public function’s docstring with Google-style Args for residual_files and diffs and Returns covering every possible CoverageResult outcome.Source: Coding guidelines
jenkins/scripts/cbts/coverage_tier.py (1)
175-251: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
must_run_reasonstally is computed but never surfaced.
_build_narrowingreturns aCounterof why each entry was kept (rule_kept/impacted/untrusted/no_data/coarse), andapply_coverage_tierstores it onCoverageTierResult.must_run_reasons(Line 239). Nothing downstream reads it —main.py's coverage-tier block only consumestier.detail,tier.affected_stages,tier.dropped,tier.removed. This diagnostic is silently discarded even though the PR explicitly aims to "emit structured selection reasons."♻️ Suggested fix: fold the tally into `detail`
detail={ "source": "coverage", "files": len(residual), "impacted": n_impacted, "untrusted": cov.n_untrusted, "removed_cases": n_removed, "dropped_stages": len(dropped), "outcome": "narrowed" if narrowed else "nothing_removable", + **({"must_run_reasons": dict(must_run_reasons)} if must_run_reasons else {}), **({"no_data_funcs": list(cov.no_data_funcs)} if cov.no_data_funcs else {}), },🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@jenkins/scripts/cbts/coverage_tier.py` around lines 175 - 251, Expose the must-run reason tally in the structured coverage-tier output by adding the existing must_run_reasons value returned from _build_narrowing to CoverageTierResult.detail in apply_coverage_tier. Preserve the existing must_run_reasons field and include the tally under a clear stable detail key so downstream main.py consumers receive it.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@jenkins/scripts/cbts/coverage_selection/selector.py`:
- Around line 18-32: The Ruff auto-fixes must be applied and staged in both
affected files: rerun the configured pre-commit hook for
jenkins/scripts/cbts/coverage_selection/selector.py lines 18-32 and
jenkins/scripts/cbts/tools/coverage_explain.py lines 30-43, then stage the
resulting edits so the release check passes.
In `@jenkins/scripts/cbts/coverage_tier.py`:
- Around line 18-42: Re-run the ruff pre-commit hook for coverage_tier.py,
retain all four auto-fixes it applies, then stage the modified file and create a
new commit containing those changes so the hook completes successfully.
In `@jenkins/scripts/cbts/tools/coverage_explain.py`:
- Around line 86-103: The coverage explanation must mirror
CoverageSelector.decide() safety gates. In
jenkins/scripts/cbts/tools/coverage_explain.py lines 86-103, check
db.file_has_touch_rows(cf) before reporting file or function impact; when false,
stop and report the coverage decision as unsupported. In lines 124-143, remove
the selector’s untrusted tests from skip_s and report those tests as
forced-kept, preserving the selector’s actual behavior.
---
Outside diff comments:
In `@jenkins/L0_MergeRequest.groovy`:
- Around line 809-815: Update the scope-null handling in the pipeline flow
around _cbtsParseSelectionResult and _cbtsReportDecision so result.reasons maps
are converted to readable reason text before joining. Reuse the structured
fields, including source and reason and any relevant details, rather than
calling join directly on the maps; preserve the existing deferring message and
fallback decision behavior.
- Around line 954-968: The Groovy parser _cbtsParseSelectionResult in
jenkins/L0_MergeRequest.groovy:954-968 must forward enable_multi_gpu with a
false default and coverage_dropped_stages with an empty-list default in its
returned map. Make no code change to jenkins/L0_Test.groovy:5714-5718; its
consumer is already correct and will work once the parser populates
cbts.enable_multi_gpu.
---
Nitpick comments:
In `@jenkins/scripts/cbts/coverage_selection/qualname_map.py`:
- Around line 79-86: The public API docstrings are missing parameter and result
documentation. In jenkins/scripts/cbts/coverage_selection/qualname_map.py lines
79-86, update qualnames_for_lines with Google-style Args for source and lines
and Returns describing the qualname set and the ok=False result when parsing
fails. In jenkins/scripts/cbts/coverage_selection/selector.py lines 108-112,
update the affected public function’s docstring with Google-style Args for
residual_files and diffs and Returns covering every possible CoverageResult
outcome.
In `@jenkins/scripts/cbts/coverage_tier.py`:
- Around line 175-251: Expose the must-run reason tally in the structured
coverage-tier output by adding the existing must_run_reasons value returned from
_build_narrowing to CoverageTierResult.detail in apply_coverage_tier. Preserve
the existing must_run_reasons field and include the tally under a clear stable
detail key so downstream main.py consumers receive it.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: a4e8fe46-939f-4256-acec-6e03d0ed6157
📒 Files selected for processing (11)
jenkins/L0_MergeRequest.groovyjenkins/L0_Test.groovyjenkins/scripts/cbts/coverage_selection/qualname_map.pyjenkins/scripts/cbts/coverage_selection/selector.pyjenkins/scripts/cbts/coverage_tier.pyjenkins/scripts/cbts/main.pyjenkins/scripts/cbts/rules/base.pyjenkins/scripts/cbts/rules/tests_def_rule.pyjenkins/scripts/cbts/rules/waives_rule.pyjenkins/scripts/cbts/tools/coverage_explain.pyjenkins/scripts/cbts/tools/dryrun.py
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@jenkins/scripts/cbts/rules/out_of_scope_rule.py`:
- Line 59: Remove the broad "jenkins/" no-op prefix from the OutOfScopeRule
configuration before merging. If it is required for temporary validation, gate
it behind explicit test-only configuration so normal CBTS runs do not classify
every Jenkins change as out of scope.
In `@tensorrt_llm/_torch/pyexecutor/py_executor.py`:
- Line 106: Remove the temporary “[cbts-test] no-op touch to validate
coverage-based selection” comment from the production code, leaving the
surrounding implementation unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 23a719fe-5256-435e-8601-e5531343b7d7
📒 Files selected for processing (2)
jenkins/scripts/cbts/rules/out_of_scope_rule.pytensorrt_llm/_torch/pyexecutor/py_executor.py
8e99945 to
8ce112a
Compare
|
/bot run |
8ce112a to
34ba032
Compare
|
/bot run |
There was a problem hiding this comment.
🧹 Nitpick comments (1)
jenkins/scripts/cbts/main.py (1)
390-392: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAvoid swallowing every coverage-tier failure.
This converts programming defects in Tier 2 into an ordinary fallback, making coverage regressions hard to detect. Catch expected operational failures at this boundary (or a dedicated coverage-tier exception) and preserve unexpected-error diagnostics.
As per coding guidelines, “Avoid broad exception handling; catch specific exceptions instead of using bare
except:.”🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@jenkins/scripts/cbts/main.py` around lines 390 - 392, Update the exception handling around the coverage-tier invocation in the Tier 2 flow to catch only expected operational failures or the dedicated coverage-tier exception, while preserving the existing fallback note and tier=None behavior for those cases. Do not convert unexpected programming errors into ordinary fallback; allow them to propagate with their diagnostics, and update the handler near the existing note and tier assignments rather than broadening error handling elsewhere.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@jenkins/scripts/cbts/main.py`:
- Around line 390-392: Update the exception handling around the coverage-tier
invocation in the Tier 2 flow to catch only expected operational failures or the
dedicated coverage-tier exception, while preserving the existing fallback note
and tier=None behavior for those cases. Do not convert unexpected programming
errors into ordinary fallback; allow them to propagate with their diagnostics,
and update the handler near the existing note and tier assignments rather than
broadening error handling elsewhere.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 39c1ca16-ae60-4c99-b8e9-b0e33d4122af
📒 Files selected for processing (13)
jenkins/L0_MergeRequest.groovyjenkins/L0_Test.groovyjenkins/scripts/cbts/coverage_selection/qualname_map.pyjenkins/scripts/cbts/coverage_selection/selector.pyjenkins/scripts/cbts/coverage_tier.pyjenkins/scripts/cbts/main.pyjenkins/scripts/cbts/rules/base.pyjenkins/scripts/cbts/rules/out_of_scope_rule.pyjenkins/scripts/cbts/rules/tests_def_rule.pyjenkins/scripts/cbts/rules/waives_rule.pyjenkins/scripts/cbts/tools/coverage_explain.pyjenkins/scripts/cbts/tools/dryrun.pytensorrt_llm/_torch/pyexecutor/py_executor.py
🚧 Files skipped from review as they are similar to previous changes (10)
- jenkins/scripts/cbts/rules/tests_def_rule.py
- jenkins/scripts/cbts/rules/base.py
- jenkins/scripts/cbts/rules/out_of_scope_rule.py
- tensorrt_llm/_torch/pyexecutor/py_executor.py
- jenkins/scripts/cbts/rules/waives_rule.py
- jenkins/L0_MergeRequest.groovy
- jenkins/L0_Test.groovy
- jenkins/scripts/cbts/coverage_tier.py
- jenkins/scripts/cbts/coverage_selection/selector.py
- jenkins/scripts/cbts/coverage_selection/qualname_map.py
|
PR_Github #61223 [ run ] triggered by Bot. Commit: |
|
/bot kill |
|
PR_Github #61226 [ kill ] triggered by Bot. Commit: |
34ba032 to
28d14b0
Compare
|
PR_Github #61223 [ run ] completed with state |
|
PR_Github #61226 [ kill ] completed with state |
|
/bot run |
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
jenkins/scripts/cbts/main.py (1)
197-214: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
self.pairs/self.handledset only insiderun(), never initialized in__init__.They're read externally as
selector.pairs/selector.handledinmain()(lines 388), which makes them externally-visible class members. As per coding guidelines,"initialize externally visible class members in the constructor". CurrentlySelector.__init__(lines 198-199) only setsself.stages, sopairs/handledare absent from the object untilrun()executes, and any future caller that inspects them before callingrun()gets anAttributeError.♻️ Proposed fix
class Selector: def __init__(self, stages: dict[str, Stage]) -> None: self.stages = stages + self.pairs: list[tuple[Rule, RuleResult]] = [] + self.handled: set[str] = set()🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@jenkins/scripts/cbts/main.py` around lines 197 - 214, Initialize the externally visible Selector state in Selector.__init__ by setting pairs to an empty list of rule/result tuples and handled to an empty set of file names. Keep run() updating these members with the computed values before they are consumed by main().Source: Coding guidelines
🧹 Nitpick comments (1)
jenkins/scripts/cbts/tools/dryrun.py (1)
214-221: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDuplicates
_fmt_reasonfrommain.pyinstead of reusing it.This block re-implements the same dict-vs-non-dict reason formatting that
main.pyalready defines as_fmt_reason(r).report_cbts_decision.pyapparently imports/reuses that helper rather than reimplementing it;dryrun.pyshould do the same to avoid the two renderings drifting apart.♻️ Proposed fix
+from main import _fmt_reason # noqa: E402 + ... - lines.append("reasons:") - for r in result.get("reasons", []): - if isinstance(r, dict): - src = r.get("source", "?") - rest = ", ".join(f"{k}={v}" for k, v in r.items() if k != "source") - lines.append(f" - [{src}] {rest}" if rest else f" - [{src}]") - else: - lines.append(f" - {r}") + lines.append("reasons:") + lines.extend(f" - {_fmt_reason(r)}" for r in result.get("reasons", []))🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@jenkins/scripts/cbts/tools/dryrun.py` around lines 214 - 221, Replace the inline reason-formatting logic in the report-generation block with the existing _fmt_reason helper from main.py. Import or reuse that helper in dryrun.py and pass each reason to it, preserving the current “reasons:” output structure while ensuring formatting remains consistent.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@jenkins/scripts/cbts/main.py`:
- Around line 197-214: Initialize the externally visible Selector state in
Selector.__init__ by setting pairs to an empty list of rule/result tuples and
handled to an empty set of file names. Keep run() updating these members with
the computed values before they are consumed by main().
---
Nitpick comments:
In `@jenkins/scripts/cbts/tools/dryrun.py`:
- Around line 214-221: Replace the inline reason-formatting logic in the
report-generation block with the existing _fmt_reason helper from main.py.
Import or reuse that helper in dryrun.py and pass each reason to it, preserving
the current “reasons:” output structure while ensuring formatting remains
consistent.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: c9b5ee70-2616-4357-9ad6-bde91f02cfce
📒 Files selected for processing (14)
jenkins/L0_MergeRequest.groovyjenkins/L0_Test.groovyjenkins/scripts/cbts/coverage_selection/qualname_map.pyjenkins/scripts/cbts/coverage_selection/selector.pyjenkins/scripts/cbts/coverage_tier.pyjenkins/scripts/cbts/main.pyjenkins/scripts/cbts/rules/base.pyjenkins/scripts/cbts/rules/out_of_scope_rule.pyjenkins/scripts/cbts/rules/tests_def_rule.pyjenkins/scripts/cbts/rules/waives_rule.pyjenkins/scripts/cbts/tools/coverage_explain.pyjenkins/scripts/cbts/tools/dryrun.pyjenkins/scripts/cbts/tools/report_cbts_decision.pytensorrt_llm/_torch/pyexecutor/py_executor.py
🚧 Files skipped from review as they are similar to previous changes (9)
- jenkins/scripts/cbts/rules/base.py
- tensorrt_llm/_torch/pyexecutor/py_executor.py
- jenkins/scripts/cbts/rules/tests_def_rule.py
- jenkins/L0_Test.groovy
- jenkins/L0_MergeRequest.groovy
- jenkins/scripts/cbts/rules/waives_rule.py
- jenkins/scripts/cbts/coverage_selection/qualname_map.py
- jenkins/scripts/cbts/coverage_selection/selector.py
- jenkins/scripts/cbts/coverage_tier.py
28d14b0 to
c43b960
Compare
|
PR_Github #61233 [ run ] triggered by Bot. Commit: |
|
PR_Github #61233 [ run ] completed with state
|
|
/bot run |
…DB lag commit_distance could not fire in the pipeline: the checkout is a depth-1 clone with a single-SHA refspec, so no candidate revision is ever in the object store and the git call failed on every candidate before the compare API answered. Keeping it also kept a second, differently-defined measurement -- git answers against whatever ref it is handed, and a merely stale ref returns a smaller number rather than an error -- plus the --repo-root plumbing that existed only to feed it. Query the compare API and nothing else. Without a token the lag is null and the ranking degrades to its build-number tie-break, which is what happened anyway whenever git could not answer. Also collapse the credential binding to the file's existing one-line form and correct the comment above it: the reference point is the tip of main, not HEAD. Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
…points --dest and --print-url were never wired: _cbtsCoverageAudit calls the script once with --print-selection and then does its own wget and tar, so fetch_latest_touch_db, extract_touch_db and latest_tarball_url had no callers in the pipeline or anywhere else in the tree. Remove them along with the two flags, leaving the module to answer the one question it is asked -- which post-merge build's DB to use. Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
…it comment The module docstring still described the lag as coming from local git, a path removed two commits ago -- two earlier edits to that paragraph had silently matched nothing because they were written with different line wrapping. Restate it once, and fold the three-line comment above the credential binding into one. Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
The echo still read "behind HEAD" from when git measured the distance against the workspace. The compare API measures it against the tip of main, which is a different number in a PR build: HEAD carries the PR's own commits, main does not. Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
…freshness The DB's lag behind main was recorded but never acted on. Decline the tier past --coverage-max-lag: a DB that far behind no longer describes who touches what in the code under test, and narrowing on it risks dropping a case the change actually reaches. A lag that could not be measured is treated the same way -- freshness that cannot be shown is not assumed -- so a GitHub compare outage turns the tier off rather than letting it run unverified. The default of 100 comes from what the producer currently manages: main gains roughly 24-36 commits a day, and of the last twelve post-merge builds only four uploaded a coverage tarball, the newest of them 58 commits behind. A tighter bound would decline nearly every PR today; 100 admits the DBs that actually exist while still rejecting one several days stale. The verdict is recorded as coverage_freshness (ok / stale / unknown, empty when no DB was consulted) and posted as s_coverage_freshness, so the decline rate is queryable per cause instead of only readable inside s_reason -- which is what should drive the threshold from here. Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
Widening an import-executed qualname to its file's row set recovers tests but does not bound the impact: a test that imports the file and never enters any function in it records nothing there, so it is missing from the file set too. On llmapi/llm_args.py the widening goes 509 -> 735 holders against roughly 746 importers, and the gap is exactly the population the bound was supposed to cover. Decline instead, for module bodies, class bodies and signature / decorator lines, until the producer records the pool workers' import phase. A missing patch (binary / rename / oversized) and unparsable source decline for the same reason -- the changed scope cannot be established at all. Function-body changes are untouched and keep precise narrowing. This is expensive: over the last 200 main commits, 92 touched tensorrt_llm/**/*.py and 76 of those (82.6%) changed at least one import-executed line, mostly by adding a method, whose def lands on the class body. Tier 2 will therefore stand down on most core-Python PRs until import- phase capture lands, which is the correct trade while the narrowing cannot be justified from the data. Closure changes keep the widening for now. It carries the same incompleteness and is not sound either -- a decorator's wrapper is created at import time, so the enclosing scope it attributes to can itself be import-only -- but it covers 4.3% of commits rather than 82.6%. Recorded in SELECTION.md 4.1 as the next candidate. Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
A residual file whose whole diff survives strip_noop_diff_lines as empty changed no executable line, so it was widening the impacted set to everything touching that file for no reason. Contribute nothing instead. This is not the deletion case: iter_diff_post_line_numbers anchors `-` lines at the following post-image line, so removing code still yields line numbers and still resolves to a qualname. The branch fires only when every `+` and `-` line was blank or comment. The one way it could misread is a `#`-leading line inside a multi-line string, which the regex cannot tell from a comment. Over the last 200 main commits, 9 of 375 core-Python diffs were comment-only and none was string content. Tier 1's rules already read every diff through the same stripping, so this adds no assumption the pipeline was not already making. Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
…plays A replay has no lag to measure -- the DB is whatever the operator passed and the commits are historical -- so main.py read it as unmeasurable and declined every PR, making every dry run report the gate rather than the selection logic it exists to exercise. Pass 0. Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
…p now costs Both cross-references described the consumer as widening to file level, which is no longer what happens for import-executed changes -- those decline outright. State that under 5.2, where the blind spot is described, so the reason to record the workers' import phase is visible from the producer side; and move 5.1's pointer to the closure section that still does widen. Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
… not main's tip The freshness gate measured the DB against the tip of main, which answers "is this DB recent" rather than "does this DB still describe the code under test". CI checks out the PR head (env.gitlabMergeRequestLastCommit), so the revision the DB has to match is the PR's merge base: a DB one commit off the tip scores as fresh for a PR branched three hundred commits back, and its function-to-test edges no longer describe that code. Ranking and gating now use different numbers. The lag against main's tip still ranks candidates and reports overall freshness; a new drift measures the ranked winner against the PR's merge base and is what the gate decides on. Drift sums both sides of the compare rather than picking one. The dangerous failure is an edge the code under test has and the DB never recorded, and both directions produce it -- an older DB misses callers added since, a newer DB reflects a call path deleted since. The fail-closed bound is symmetric too and catches only whole-function absence, never a row set that is merely too narrow. Summing is also the only form that handles a diverged base: a PR targeting a release branch, which a main-collected DB does not describe at all, scores as the large number it is instead of slipping through on one small term. Direction rides along as drift_status, recorded but never weighted. Any step that cannot be answered leaves the drift null, which the gate reads as unknown and declines. The limit is --coverage-max-drift, default 30. Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
…ough, not six flags Groovy destructured artifact.py's --print-selection JSON into six CLI flags, which main.py reassembled into the decision JSON -- the same data serialized three times, with the SHAs shell-quoted along the way and a new field costing an edit in both layers. Only drift is ever read: the freshness gate decides on it. The other five are record-only passthrough. So the whole blob now travels as one file, written verbatim to cbts_coverage_db.json and read via --coverage-db-meta. Six conditional appends in the Groovy collapse to one unconditional line, the audit helper's return map goes from seven keys to two, and a missing, empty or unparsable meta leaves drift null, which the gate already declines on. Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
…he pipeline
_cbtsCoverageAudit had grown to string-building a log line out of JSON fields,
mkdir, wget, tar, and writeFile -- all of it shell and Groovy around a Python
tool that already had the selection in hand.
artifact.py --prepare DIR now does the whole fetch and prints {path, meta}.
Groovy keeps only what it alone can do: bind the credential and run
coverage_audit.py over the result. The helper goes from roughly forty lines to
twenty, and its return map no longer has to be kept in step with the JSON.
The tarball is streamed rather than buffered -- it is past 200 MB, so reading
it into memory was not an option -- and gets its own socket timeout: the 15s
tuned for the small metadata calls expires mid-transfer on an artifact that
size. Retries replace trtllm_utils.llmExecStepWithRetry, which cannot wrap a
step that no longer exists as a shell command.
Verified end to end against build 2895: 226 MB fetched, the 2.2 GB sqlite
unpacked, meta written, and main.py declining it as stale at drift 235.
Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
…repacking them
_cbtsCoverageAudit unpacked prepare's {path, meta} and built a fresh map of the
same two keys, so the failure branches had to name them too -- and meta was
dead there, the caller guarding on path alone.
It now returns that map verbatim, or null. The caller guards on the map itself,
which also covers artifact.py printing an empty object.
Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
…al dir provoked A scratch `rules/` in the working tree resolves as a first-party module, so ruff's isort splits `from rules._helpers` into its own section and inserts a blank line. CI checks out clean, sees the same name as third-party, and takes the line back out. Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>
d5448cd to
03a7302
Compare
|
/bot run |
|
PR_Github #67340 [ run ] triggered by Bot. Commit: |
|
PR_Github #67340 [ run ] completed with state
|
|
/bot skip --commit "skip ci as the failure is not related to this pr" |
GitHub Bot Help
Provide a user friendly way for developers to interact with a Jenkins server. Run See details below for each supported subcommand. Details
Launch build/test pipelines. All previously running jobs will be killed.
kill
Kill all running builds associated with pull request. skip
Skip testing for latest commit on pull request. reuse-pipeline
Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break. |
|
/bot skip --comment "skip ci as the failure is not related to this pr" |
|
PR_Github #67428 [ skip ] triggered by Bot. Commit: |
|
PR_Github #67428 [ skip ] completed with state |
4 participants
Summary
Adds Tier-2 coverage-based test selection (CBTS) for residual core-Python changes. The Jenkins pipeline now runs
_cbtsCoverageAudit(pipeline)to discover an optional sqlite coverage DB path and, when present, threads it intojenkins/scripts/cbts/main.pyas--coverage-db(omitted when empty or when audit failures are non-fatal), preserving existing CBTS behavior when coverage is unavailable.Key new/updated coverage-tier behavior:
co_qualnames (jenkins/scripts/cbts/coverage_selection/qualname_map.py).jenkins/scripts/cbts/coverage_selection/selector.py).jenkins/scripts/cbts/coverage_tier.py).reasons(list of dicts) plus coverage-tier fields likeenable_multi_gpuandcoverage_dropped_stages(jenkins/scripts/cbts/main.py), with consistent rendering indryrun.pyandreport_cbts_decision.py.coverage_explain.pyto explain kept/removed coverage cases for a specific commit SHA, and extendsdryrun.pywith opt-in--coverage-dbreplay/formatting support.Supporting infra tweaks:
_cbtsCoverageAuditis converted from best-effort “shadow audit” into a return-value helper that yields the sqlite path on success and""otherwise; logs updated to reflect skipping Tier 2 when coverage artifacts can’t be found/processed (jenkins/L0_MergeRequest.groovy).parallelJobsFilteredonly whencbts.enable_multi_gpuis true andMULTI_GPU_FILE_CHANGEDis set (jenkins/L0_Test.groovy).jenkins/changes (jenkins/scripts/cbts/rules/out_of_scope_rule.py).tensorrt_llm/_torch/pyexecutor/py_executor.pyfor coverage-based validation.Dev Engineer Review
Correctness / behavior
jenkins/L0_MergeRequest.groovy:_cbtsCoverageAudit(pipeline)now returns the sqlite coverage artifact path on success and""on non-fatal failures;--coverage-dbis only appended to the Python invocation when the returned path is non-empty. Updated logging aligns with Tier 2 being skipped when the artifact is unavailable/unusable.jenkins/scripts/cbts/main.py: implements optional coverage-tier execution behind--coverage-db, updatesSelectionResultJSON withenable_multi_gpuandcoverage_dropped_stages, and migratesreasonsfromlist[str]tolist[dict]with structured rendering/printing.jenkins/scripts/cbts/coverage_selection/qualname_map.py: AST-based line→qualname attribution with scope selection rules; returns(set(), False)onSyntaxErrorandqualnames_for_lines(...)->(set[str], bool)to drive conservative downstream behavior.jenkins/scripts/cbts/coverage_selection/selector.py: validates residual residual inputs, derives post-diff line numbers, maps them to qualnames, queries TouchDB for impacted tests, caches “untrusted” tests, and avoids marking untrusted tests skippable unless impacted; uses conservative “all touching-file tests impacted” fallback when mapping/reads fail.jenkins/scripts/cbts/coverage_tier.py: narrows the test-db by removing only SAFE entries and accounting for must-run reasons (impacted/untrusted/no-data). Includes stage-dropping logic with explicit exclusions for multi-GPU and post-merge variants.jenkins/scripts/cbts/tools/coverage_explain.py: diff+qualname mapping and TouchDB-based classification of KEPT vs REMOVED nodeids for a specific SHA; includes optional kept-case printing and safe fallbacks.jenkins/scripts/cbts/tools/dryrun.py+jenkins/scripts/cbts/tools/report_cbts_decision.py: consistent formatting for structuredreasonsobjects (dict-based reasons rendered as[source] k=v, ...).Jenkins pipeline / scoping
jenkins/L0_Test.groovy: conditional re-add of multi-GPU stages intoparallelJobsFilteredgated bycbts.enable_multi_gpuandMULTI_GPU_FILE_CHANGED.jenkins/scripts/cbts/rules/out_of_scope_rule.py: treatsjenkins/as out-of-scope in addition to the existing more specific CBTS script prefix.Schema / API consistency
jenkins/scripts/cbts/rules/base.py: addsRuleResult.detail: dict[str, object]defaulting to empty dict for structured rule payloads.jenkins/scripts/cbts/rules/tests_def_rule.pyandwaives_rule.py: populatesdetailwith narrowed-path counts and waived added/removed counts respectively.jenkins/scripts/cbts/main.py+ tooling: updates reason formatting/serialization to match the structured schema end-to-end.Risk / areas to double-check
coverageDbPathis non-empty, and that empty/failed audit does not alter non-coverage CBTS semantics.QA Engineer Review
No test changes.
Description
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.