Repository navigation
docs(bazel): propose consolidating first-party code onto one Bazel module #445
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from 5 commits
Commits
Show all changes
19 commits
Select commit
Hold shift + click to select a range
3748a4c
docs(bazel): propose consolidating onto a single Bazel module
balajinvda 537f2f5
docs(bazel): revise consolidation plan after review
balajinvda 5d3f4bc
docs(bazel): scope nvsnap, fix lane claim, harden the inventory tool
balajinvda 39aa7e5
fix(bazel): reject dirty trees, fix tempdir leak, scope the exception…
balajinvda 194824f
fix(bazel): stop the inventory false-greening, re-decide nvsnap on cu…
balajinvda 211f8da
fix(bazel): check git exit status, split closure paths and guard mech…
balajinvda 52331dc
fix(bazel): check cleanliness before creating scratch file, align clo…
balajinvda 4374b6d
fix(bazel): propagate selector grep failures instead of counting zero
balajinvda 2c904d4
fix(bazel): report the post-consolidation state instead of exiting si…
balajinvda a762035
fix(bazel): distinguish absent inputs from unparseable ones
balajinvda 5799b38
refactor(bazel): rewrite the inventory tool in Python to remove a def…
balajinvda 878f188
refactor(bazel): reimplement the inventory tool in Go with a real parser
balajinvda a9f3ea2
fix(bazel): reject Starlark forms the inventory parser silently misha…
balajinvda 9766891
refactor(bazel): parse MODULE.bazel with a real Starlark AST parser
balajinvda 5378f55
ci(bazel): point setup-go at this module's go.sum
balajinvda a48d39a
fix(bazel): resolve constants, chained receivers, and reject included…
balajinvda 74d5c9d
docs(bazel): drop the inventory tool, keep the plan
balajinvda 2f9a33f
docs(bazel): add derivations, a non-waivable invariant, and nvsnap's …
balajinvda 8c966c3
docs(bazel): pilot on nvsnap then image-credential-helper, not the wo…
balajinvda File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,310 @@ | ||
| # Bazel consolidation: one root module for first-party code | ||
|
|
||
| Status: proposed. Not started. | ||
| Owner: TBD. | ||
|
|
||
| Measurements below were taken on main `1c29b4b7` (2026-07-25). They move. | ||
| Regenerate them with `tools/ci/bazel-consolidation-inventory` rather than | ||
| trusting this prose, and update the stamp above when they change. The first | ||
| draft of this document was invalidated within hours by an in-flight change that | ||
| pinned the worker Bazel versions. | ||
|
|
||
| ## Summary | ||
|
|
||
| First-party code builds as 19 nested service modules plus the root. That layout | ||
| is inherited from the pre-monorepo history, when each service was its own | ||
| repository. The proposal is that all 19 first-party service modules converge on | ||
| one root Bazel module, with services as packages, while releases and ownership | ||
| stay per component. | ||
|
|
||
| An earlier draft carried `nvsnap` as a standing exception on the grounds that its | ||
| base image is not publicly pullable. That rationale is stale. `.github/workflows/bazel.yml` | ||
| still records it, but the code disagrees: `nvsnap` pins public `gcr.io/distroless/static`, | ||
| `docker.io/library/ubuntu`, and `docker.io/library/alpine` bases by digest, builds | ||
| its images on `@distroless_static`, and documents a public-input-only source build. | ||
| Private NGC is its publish destination, not its base. Phase 1 therefore has to | ||
| resolve `nvsnap` rather than assume it: migrate it with the rest, or record a real | ||
| reason under the exception contract. Its absence from the public CI matrix is a | ||
| row decision, not an architectural one. | ||
|
|
||
| Scope boundary, and the most common misreading of this proposal: one Bazel | ||
| module does not mean one Go module, one Cargo workspace, or one lockfile per | ||
| language. It means one Bazel dependency graph. Component `go.mod`, `go.sum`, and | ||
| Cargo workspaces stay where Go and Cargo tooling, publishing, or ownership need | ||
| them. The target is one root `MODULE.bazel.lock` and one Bazel-resolved graph. | ||
|
|
||
| ## Why this is a correctness change, not tidiness | ||
|
|
||
| Several services consume libraries that already live in this repository through | ||
| published Go pseudoversions mapped in their nested modules. A library and its | ||
| consumer can therefore change atomically in one commit while Bazel still tests | ||
| the consumer against the older published copy of the library. A single root | ||
| graph removes that class of false green. | ||
|
|
||
| The duplication is the secondary argument: | ||
|
|
||
| | Item | Count | | ||
| |---|---| | ||
| | Nested service modules | 19 | | ||
| | `MODULE.bazel.lock` files | 20, 6.3 MB total | | ||
| | Subtree `.bazelrc` | 19, of which 17 distinct | | ||
| | `workspace_status.sh` copies | 20, 7 variants | | ||
| | Copied `rules/oci` files | 120 files, 6944 lines | | ||
| | `oci.pull` declarations | 23 (18 from nvcr.io), 8 images, 7 digests | | ||
|
|
||
| The isolation that separate modules provide is largely unused: `rules_oci`, | ||
| `rules_pkg`, and `aspect_bazel_lib` are on identical versions in all 19. | ||
|
|
||
| Two important qualifications. The `rules/oci` and stamping copies contain real | ||
| behavioral differences, so they need a shared API with per-component | ||
| configuration, not mechanical deletion. And the 23 base-image pulls resolve to | ||
| 8 distinct images; each collapses to one repository definition and one digest | ||
| pin, but not necessarily to one fetch or one compilation. That depends on how | ||
| many Bazel invocations CI runs and whether they share repository and action | ||
| caches with compatible action keys. | ||
|
|
||
| ## Java is the proof, not the exception | ||
|
|
||
| `nv-boot-parent` and `cloud-tasks` already build in the root module while | ||
| retaining Maven and POM workflows, a root Maven dependency hub and lock, | ||
| Java-specific rules, component CI metadata, Testcontainers and Docker-host test | ||
| lanes, externally consumed labels, and independent releases. | ||
|
|
||
| The lesson is not to make Java, Go, and Rust identical. It is that one root | ||
| graph can host specialized rules, dependency hubs, test lanes, and release | ||
| metadata per component. Splitting Java into standalone modules is an explicit | ||
| non-goal. | ||
|
|
||
| ## Primary risk: label rebasing and visibility | ||
|
|
||
| This is the highest-risk part of the migration and it is structural, not | ||
| mechanical. Removing a workspace root changes what its labels mean. | ||
|
|
||
| - Nested root-relative labels such as `//internal/...`, `//rules/oci`, | ||
| `//platforms`, and `//:Cargo.lock` resolve differently once the module root | ||
| moves. | ||
| - `//:__subpackages__` currently scopes to one service. Under the repository | ||
| root it expands to the entire repository. There are 156 such declarations | ||
| outside vendored code. | ||
| - There are 644 `//visibility:public` occurrences outside vendored code and | ||
| zero `package_group` definitions. The expressed API surface is therefore very | ||
| broad and uniform rather than absent: almost everything is public, and nothing | ||
| declares a narrow boundary. Consolidation must inventory that surface before | ||
| restricting it, because today's breadth is load-bearing for consumers. | ||
|
|
||
| Before any boundary is removed, define: default-private service packages, | ||
| explicit exported APIs, allowed cross-layer dependency groups, a query or aspect | ||
| check for forbidden cross-service edges, and root-run Gazelle regeneration with | ||
| an idempotence check. | ||
|
|
||
| ## CI: extend the existing hybrid, do not rebuild it | ||
|
|
||
| An earlier draft of this document claimed CI scheduling is a hand-maintained | ||
| approximation of the dependency graph and proposed replacing it with | ||
| `bazel query rdeps`. That was wrong, and the correction matters because it | ||
| changes what work is left. | ||
|
|
||
| There are two layers: | ||
|
|
||
| 1. Target selection inside a job already uses `rdeps`. `.github/workflows/bazel.yml` | ||
| maps changed files to labels, queries reverse dependencies, and falls back to | ||
| a full build for non-modify changes, global files, and file types it does not | ||
| model. That is a deliberate hybrid and it should be kept. | ||
| 2. Job selection across the matrix is static: which subtree lanes to spawn comes | ||
| from a list in the workflow plus path-prefix matching, because 19 modules | ||
| cannot be one query. | ||
|
|
||
| Consolidation does not add `rdeps`. It retires the static module-root list, but | ||
| it does not eliminate job or lane selection. Affected targets still have to be | ||
| partitioned by component metadata, target tags, and execution requirements: | ||
| Docker-host and Testcontainers lanes, Java component lanes, Cargo parity tests, | ||
| BYOO generation, and release and artifact staging. | ||
| Root-scoped Java already demonstrates this, routing on the `component_kind` and | ||
| `ci_lane` fields of its component descriptors rather than on a module root. | ||
|
|
||
| The hybrid must be extended during every phase, not deferred to the end. `rdeps` | ||
| alone cannot safely handle added, deleted, renamed, generated, configuration, | ||
| container, or release files, nor anything not modeled in the graph. The | ||
| conservative fallback is a feature. | ||
|
|
||
| ## Costs and risks | ||
|
|
||
| 1. Merging language dependency graphs is the real work. Do not assume conflicts | ||
| resolve by taking the newer version: Kubernetes dependency families span | ||
| several versions with conflicting `replace` directives. Toolchains also | ||
| differ, with first-party Go SDK versions from 1.25.0 through 1.25.11 and | ||
| protobuf and Rust toolchains spanning major versions. The older 1.23.0 pin | ||
| reported by a naive scan comes only from a vendored third-party module and is | ||
| not in scope. Each decision needs an owner review and service tests. | ||
| 2. A dependency bump becomes repository wide. Breakage surfaces immediately | ||
| instead of never, which is the goal, but it changes how changes are staged. | ||
| 3. The analysis graph per invocation grows. Query-driven target selection is the | ||
| mitigation; remote caching is not. Caching avoids re-executing actions, but | ||
| the analysis phase still loads and analyzes whatever the invocation asks for, | ||
| so the fix is asking for less rather than caching more. | ||
| 4. This is incremental work measured in months and must not be attempted as one | ||
| change. | ||
|
|
||
| ## Constraint: the parallel migration window | ||
|
|
||
| An external project builds against this repository by Bazel label during the | ||
| parallel migration period: | ||
|
|
||
| ``` | ||
| @nvcf//src/control-plane-services/cloud-tasks/nvct-core:nvct_core | ||
| @nvcf//src/control-plane-services/cloud-tasks/nvct-core:nvct_core_test_fixtures | ||
| ``` | ||
|
coderabbitai[bot] marked this conversation as resolved.
Resolved
|
||
|
|
||
| Those labels are a public interface until it cuts over. Do not rename them or | ||
| restrict their visibility before then. Nothing else here is blocked by it, | ||
| because those targets already build in the root module. | ||
|
|
||
| ## Per-component exit criteria | ||
|
|
||
| A component is done when all of the following hold. These are the migration's | ||
| definition of done; a phase is not complete until every component it touched | ||
| satisfies them. | ||
|
|
||
| - In-repository Go dependencies resolve to root-local `//src/...` labels, not to | ||
| published `@com_github_nvidia_nvcf...` copies. This is the correctness goal, so | ||
| it is the criterion that cannot be waived. | ||
|
coderabbitai[bot] marked this conversation as resolved.
Resolved
|
||
| - The component's public labels are inventoried before visibility is restricted, | ||
| and every external consumer of a removed label has been identified. | ||
| - Build, tests, OCI image equivalence, version stamping, and staged release | ||
| behavior all pass. | ||
| - Tag prefixes, artifact names, and registry destinations are unchanged. A | ||
| consolidation that silently renames a published artifact has failed, however | ||
| green the build is. | ||
|
|
||
| The root-local-label criterion is scoped to components that have migrated. It is | ||
| not waived for first-party NVCF code that a component vendors. NVCA currently has | ||
| 80 BUILD files referring to vendored copies of NVCF Go libraries, and that is | ||
| precisely the false-green condition this proposal exists to remove. A component | ||
| may be excepted from moving; it may not be excepted from the correctness goal | ||
| while still claiming to be done. | ||
|
|
||
| ## Exception contract | ||
|
|
||
| An exception is a decision, not a deferral, and it has to be written down as one. | ||
| Every nested module retained past Phase 8 records: | ||
|
|
||
| - Owner, and the rationale for staying nested. | ||
| - The retained module and its CI entry point. | ||
| - How the component sources first-party dependencies, and whether it can still | ||
| test against a stale published or vendored copy of an in-repository library. | ||
| - Residual correctness risk, stated plainly. | ||
| - A dated revisit or removal trigger. | ||
|
|
||
| An entry missing any of these is not an exception; it is unfinished work. | ||
|
|
||
| The 22 tracked `MODULE.bazel` files are three different things, and conflating | ||
| them obscures the end state: | ||
|
|
||
| | Category | Count | End state | | ||
| |---|---|---| | ||
| | First-party service modules | 19 | Converge on the root | | ||
| | Root module | 1 | The destination | | ||
| | Migration scaffolding (`rules/oci-destinations`) | 1 | Removable once its five consumers migrate | | ||
| | Vendored third-party (`cel.dev/expr`) | 1 | Never in scope; excluded from the guard | | ||
|
|
||
| Only the first category can hold an architectural exception. `rules/oci-destinations` | ||
| is scaffolding that five nested modules currently depend on, so it is retired by | ||
| Phase 8 rather than excepted. A vendored third-party module is a guard exclusion, | ||
| not a decision anyone has to justify. | ||
|
|
||
| So: migrated services satisfy the root-label criteria; approved service exceptions | ||
| satisfy the exception contract above; vendored third-party modules are excluded | ||
| from the guard entirely. NVCA, ESS, or BYOO join the second group only if their | ||
| subphase concludes so, and the summary is updated when that happens. | ||
|
|
||
| ## Phases | ||
|
|
||
| Each phase lands independently and must leave every declared CI lane green, not | ||
| only `bazel build //...` and `bazel test //...` at the repository root. Lanes | ||
| that route by component metadata or execution requirements, such as the | ||
| Docker-host and Java component lanes, count toward that invariant. | ||
|
|
||
| 1. Correct the inventory, remove stale Bazel documentation, and settle the | ||
| exception question. Two of the 22 tracked modules are not first-party services | ||
| and are handled by category rather than by exception: `rules/oci-destinations` | ||
| is migration scaffolding retired once its five consumers migrate, and the | ||
| vendored `cel.dev/expr` module is excluded from the guard outright. `nvsnap` | ||
| is the only open question, and its recorded rationale contradicts its code, so | ||
| this phase decides it on current evidence. The Phase 2 guard still needs an | ||
| allowlist rather than a blanket ban, but the allowlist should be short and | ||
| every entry should carry the exception contract. | ||
|
coderabbitai[bot] marked this conversation as resolved.
Resolved
|
||
| 2. Establish visibility policy, root-run Gazelle enforcement, the nested-module | ||
| guard, and the extended hybrid affected-target CI. | ||
| 3. Converge Bazel and shared toolchain versions. This is real work: sampling two | ||
| subtrees on the root Bazel version produced two different failures, one from | ||
| `sh_test` no longer being a built-in global and one from a protoc and | ||
| protobuf mismatch. | ||
| 4. Centralize the OCI, platform, and stamping contracts behind one API covering | ||
| Go, Rust, multi-binary images, extra layers, environment, entrypoints, | ||
| platforms, and registry outputs. Gate it on equivalence: compare image | ||
| manifests and configuration before and after. Define how per-service versions | ||
| are stamped, given that one workspace-status command runs once per Bazel | ||
| invocation while releases stay per service. | ||
| 5. Migrate the four worker services. They are the most uniform and prove the Go | ||
| dependency merge and the release path end to end. | ||
| 6. Migrate the remaining ordinary Go services. The three special cases are | ||
| scheduled explicitly rather than deferred, because Phase 8 cannot complete | ||
| while any of them is unresolved: | ||
| - 6a. NVCA. 1261 vendored BUILD files and direct `//vendor/...` dependencies | ||
| whose labels will not survive a root move. Decide between rebasing the | ||
| vendored labels and dropping the vendored tree in favor of module | ||
| resolution before any code moves. | ||
| - 6b. ESS. Retains its Go workspace while the root Bazel graph ingests its | ||
| modules, consistent with the scope boundary above: one Bazel module does not | ||
| mean one Go workspace. The work is teaching the root graph to build those | ||
| modules, not collapsing the workspace. | ||
| - 6c. BYOO. The collector already builds through a declared Bazel genrule. | ||
| The problem is that the action uses host Go with `local` and `no-sandbox`, | ||
| so it is neither hermetic nor remotely cacheable. Make the action hermetic, | ||
| retain its dedicated local lane, or record an exception. | ||
| Each subphase carries its own owner and exit criteria. If one is judged not | ||
| worth doing, it moves to the exception allowlist under the contract below | ||
| rather than staying open. Moving a component there changes the end state: | ||
| the summary's count of converging modules drops, and that component keeps its | ||
| own lockfile and module-root CI lane. | ||
| 7. Move the Rust trees into the root Bazel module while initially retaining | ||
| their Cargo workspaces, lockfiles, and separate crate hub names. One Bazel | ||
| module can host several crate hubs. Merging them into one Cargo graph is a | ||
| later optimization needing its own justification. Add and retain | ||
| `cargo test --workspace --all-targets` until Bazel declares every Rust | ||
| integration test that Cargo currently discovers. GitHub CI has no Cargo lane | ||
| today, so this is new work rather than something already in place. | ||
| 8. Retire nested configuration and the static module-root list once graph | ||
| coverage is complete, leaving the allowlisted exceptions in place. Lane | ||
| routing by component metadata and execution requirements stays. | ||
|
|
||
| ## Adjacent cleanup, independent of consolidation | ||
|
|
||
| 1. Publish the Bazel CI image from this repository. It is currently built | ||
| elsewhere and mirrored by hand, and that indirection already caused a release | ||
| path to run a toolchain several versions behind what CI validates with. | ||
| 2. Add a base image freshness check. Pinned digests are correct and | ||
| reproducible, but nothing reports when a newer base is published, so a | ||
| patched base can go unnoticed. This is detection, not caching. | ||
| 3. Require a digest on every `oci.pull`. One pull uses a mutable tag, which is | ||
| neither reproducible nor safe to cache. | ||
|
|
||
| ## Explicit non-goals | ||
|
|
||
| - Splitting the Java tree into standalone modules. It is already in the target | ||
| shape. Open tasks proposing that split should be closed rather than executed. | ||
| - One Go module, one Cargo workspace, or one lockfile per language. | ||
| - Merging the Rust crate hubs as part of this work. | ||
| - A single large migration. | ||
| - Removing per-service release granularity. | ||
| - Keeping `nvsnap` permanently out of scope on the current stated rationale. | ||
| That rationale no longer matches the code and Phase 1 must re-decide it. | ||
|
|
||
| ## Open questions | ||
|
|
||
| 1. Does any service genuinely need an independent dependency version today? The | ||
| data says no, but confirm with each owning team before removing the ability. | ||
| 2. What is the correct single Bazel version, given the subtrees and the root | ||
| disagree and converging is a behavior change for the subtrees. | ||
| 3. Should the workers be the pilot? They are the most uniform, which is why they | ||
| are proposed, but they also release frequently. | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.