fix(autoscaler): build a multi-arch image by moving TLS to rustls - #529
Closed
balajinvda wants to merge 1 commit into
Closed
fix(autoscaler): build a multi-arch image by moving TLS to rustls#529balajinvda wants to merge 1 commit into
balajinvda wants to merge 1 commit into
Conversation
Closes the arm64 gap reported in #527. The control observability profile installs this service, and no published tag resolves for linux/arm64 because none has ever been built. Three defects, each masking the next. The oci.pull requested "linux/arm64" while nvcr.io publishes its aarch64 manifest under the arm64/v8 variant, and rules_oci matches the platform string exactly, so the pull failed. http-invocation pins the same image at the same digest with arm64/v8 and has always built both architectures. This is an independent bug that would have blocked the build even after the rest was fixed. openssl-sys then could not cross-compile. Its build script probes AARCH64_UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR, finds it unset, and falls back to OPENSSL_LIB_DIR, which points at the host's x86_64 libraries. Vendored OpenSSL was tried first and fails under crate_universe exactly as the existing comment described, so the dependency had to go. The service now uses rustls, matching http-invocation, which runs entirely on rustls with no openssl and is why it builds both architectures. scylla moves to rustls-023, reqwest to rustls-tls, and the direct OpenSSL certificate handling is replaced. openssl-sys is no longer anywhere in the image dependency graph; the remaining OpenSSL use is test-only certificate generation, moved to dev-dependencies. TWO BEHAVIOUR NOTES FOR REVIEW: 1. Cassandra server certificate verification remains DISABLED. The OpenSSL implementation called set_verify(SslVerifyMode::NONE) after loading the CA, so the CA never validated the peer. rustls has no equivalent switch, so an explicit NoServerVerification verifier reproduces it. This preserves current behaviour rather than tightening it: enabling verification could drop every existing Cassandra connection and belongs in its own reviewed change. 2. The explicit certificate/private-key match check in the TimeseriesDb client is removed. rustls performs the equivalent when the identity is installed, so the coverage remains, but a mismatched pair now fails later and with a less specific message. Verified: the image index carries both amd64 and arm64, and a query over the image dependency graph returns zero openssl-sys targets. NOT verified: that mTLS to a real Cassandra cluster still works. A green build proves the types line up and nothing more, which is why this is a draft. Co-authored-by: Balaji Ganesan <bganesan@nvidia.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
Contributor
🛡️ CodeQL Analysis🚨 Found 2 issue(s) Severity Breakdown:
📋 Top Issues🔗 View full details in Security tab 🕐 Last updated: 2026-07-29 18:00:21 UTC | Commit: 4417989 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Closes #527. The control observability profile installs the function autoscaler, and no published
nvcf-function-autoscalertag resolves forlinux/arm64— not because a version was mis-selected, but because an arm64 image has never been built.What was actually wrong
Three defects, each hiding the next. Only the second matches the reason recorded in the code.
1. Wrong platform string.
oci.pullrequestedlinux/arm64; nvcr.io publishes its aarch64 manifest under thearm64/v8variant, andrules_ocimatches exactly:http-invocationpins the same image at the same digest withlinux/arm64/v8and has always built both architectures. This is an independent bug that would have blocked the build even after everything else was fixed.2.
openssl-syscannot cross-compile. Its build script probes for a target-prefixed variable, doesn't find one, and falls back to the host's:Vendored OpenSSL was tried first and fails under
crate_universeexactly as the existing comment predicted, so the dependency had to go.3. arm64 was absent from
DEFAULT_PLATFORMSin this service's forked copy ofrules/oci— the only one of 19 modules missing it.What changed
The service moves to rustls, matching
http-invocation, which runs entirely on rustls with zero openssl and is precisely why it builds both architectures.scylla:openssl-010→rustls-023reqwest:native-tls→rustls-tlsx509-parser[dev-dependencies], keepingopenssl-sysout of the image graphWhat is not verified
mTLS against a real Cassandra cluster. A green build proves the types line up. It proves nothing about whether the handshake succeeds with real certificates. This needs someone with cluster access to confirm before it ships.
Two deliberate behaviour notes for that reviewer:
Server certificate verification remains disabled. The OpenSSL implementation called
set_verify(SslVerifyMode::NONE)after loading the CA, so the CA never validated the peer. rustls has no equivalent switch, so an explicitNoServerVerificationverifier reproduces it. Current behaviour is preserved rather than tightened — turning verification on could drop every existing Cassandra connection, and that belongs in its own reviewed change. It is now visible in a documented verifier rather than buried in a builder chain.The explicit certificate/private-key match check is removed from the TimeseriesDb client. rustls performs the equivalent when installing the identity, so coverage remains, but a mismatched pair now fails later and less specifically.
Customer Release Notes
The function autoscaler image now supports linux/arm64 in addition to linux/amd64, allowing control observability profile installs on arm64 Kubernetes nodes.
Plan Summary
Not applicable.
Usage
Not applicable.
Testing
Not tested: mTLS handshake against a live Cassandra cluster.
Notes
The
DEFAULT_PLATFORMScomment claiming this was solely an openssl problem is corrected. It was accurate about openssl but omitted the platform-string bug, and sent me down the wrong path twice before I built it and read the actual error.Worth a follow-up: nothing enforces that a service's forked
rules/ocilists both platforms. This one dropped arm64 in its private copy and nothing noticed until an arm64 install failed. A guard comparing each copy'sDEFAULT_PLATFORMSagainst the root would have caught it at the introducing commit.References
Closes #527. Relates to #522.
Related Merge Requests/Pull Requests
None.
Dependencies
Adds
rustls0.23,rustls-pemfile2,x509-parser0.16,pem3. Removesopensslfrom runtime dependencies (retained as a dev-dependency for test fixtures). All are widely used, actively maintained, Apache-2.0/MIT licensed.