feat: (BREAKING) addition of launch timeout for nodeclaim lifecycle #2349

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

k8s-ci-robot merged 6 commits into kubernetes-sigs:main from rschalo:launch-ttl-fix

Jul 15, 2025

Contributor

rschalo commented Jul 3, 2025 •

edited

Loading

Fixes #N/A

Description
If Karpenter encounters issues launching instances then we should retry for a shorter amount of time than the full registration timeout. This unblocks provisioning decisions for pods that may be stuck waiting for compute that will not come up due to launch failures. This also allows Karpenter to make new decisions about what compute should be provisioned.

BREAKING

This PR changes an alpha metric for nodeclaim disruptions. The metric reason liveness will change to registration_timeout.

How was this change tested?
Unit tests.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.


          feat: addition of launch TTL for nodeclaim lifecycle

979741a

k8s-ci-robot added the cncf-cla: yes label

k8s-ci-robot requested review from engedaam and tallaxes

July 3, 2025 20:08

k8s-ci-robot added the size/M label

coveralls commented Jul 3, 2025 •

edited

Loading

Pull Request Test Coverage Report for Build 16301866352

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

21 of 33 (63.64%) changed or added relevant lines in 1 file are covered.
211 unchanged lines in 17 files lost coverage.
Overall coverage increased (+0.08%) to 81.972%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/controllers/nodeclaim/lifecycle/liveness.go	21	33	63.64%

Files with Coverage Reduction	New Missed Lines	%
pkg/apis/v1/labels.go	1	78.95%
pkg/test/options.go	1	96.43%
pkg/controllers/nodeclaim/lifecycle/liveness.go	2	64.18%
pkg/controllers/disruption/emptiness.go	4	89.29%
pkg/controllers/provisioning/scheduling/topologydomaingroup.go	4	86.21%
pkg/controllers/disruption/singlenodeconsolidation.go	6	93.68%
pkg/controllers/disruption/helpers.go	7	87.5%
pkg/controllers/provisioning/scheduling/preferences.go	7	88.76%
pkg/cloudprovider/types.go	8	95.63%
pkg/controllers/disruption/multinodeconsolidation.go	12	86.86%

Totals
Change from base Build 16014134817:	0.08%
Covered Lines:	10317
Relevant Lines:	12586

💛 - Coveralls

DerekFrank reviewed

View reviewed changes

pkg/controllers/nodeclaim/lifecycle/liveness.go Show resolved Hide resolved

pkg/controllers/nodeclaim/lifecycle/liveness.go Outdated Show resolved Hide resolved

pkg/controllers/nodeclaim/lifecycle/liveness.go Outdated Show resolved Hide resolved

pkg/controllers/nodeclaim/lifecycle/liveness.go Outdated Show resolved Hide resolved

jonathan-innis reviewed

View reviewed changes

pkg/controllers/nodeclaim/lifecycle/liveness.go Outdated Show resolved Hide resolved

pkg/controllers/nodeclaim/lifecycle/liveness.go Outdated Show resolved Hide resolved

pkg/controllers/nodeclaim/lifecycle/liveness.go Outdated Show resolved Hide resolved

pkg/controllers/nodeclaim/lifecycle/liveness_test.go Show resolved Hide resolved


          pr responses

77c4939

k8s-ci-robot added size/L and removed size/M labels

rschalo changed the title ~~feat: addition of launch TTL for nodeclaim lifecycle~~ feat: (BREAKING) addition of launch timeout for nodeclaim lifecycle

jonathan-innis reviewed

View reviewed changes

pkg/controllers/nodeclaim/lifecycle/liveness.go Outdated Show resolved Hide resolved

pkg/controllers/nodeclaim/lifecycle/liveness_test.go Show resolved Hide resolved

pkg/controllers/nodeclaim/lifecycle/liveness.go Outdated Show resolved Hide resolved


          update test

40558f7

jonathan-innis reviewed

View reviewed changes

pkg/controllers/nodeclaim/lifecycle/liveness.go Outdated Show resolved Hide resolved

pkg/controllers/nodeclaim/lifecycle/liveness_test.go Outdated

    
              		// expect that the nodeclaim was not deleted

              		ExpectExists(ctx, env.Client, nodeClaim)

              	})

              	It("should use the status condition transition times for timeouts, not the creation timestamp", func() {

Member

jonathan-innis Jul 15, 2025

We may want to have a test here that validates the RequeueAfter that we are getting both for the Registered and Launched timeouts

pkg/controllers/nodeclaim/lifecycle/liveness_test.go Show resolved Hide resolved

pkg/controllers/nodeclaim/lifecycle/liveness_test.go Show resolved Hide resolved

pkg/controllers/nodeclaim/lifecycle/liveness.go Outdated Show resolved Hide resolved

rschalo added 2 commits

July 15, 2025 10:05


          requeue for launch timeout

c861aa8


          test expansion

1ed07dd

rschalo commented

View reviewed changes

pkg/controllers/nodeclaim/lifecycle/liveness_test.go

    
              		fakeClock.Step(16 * time.Minute)

              		result := ExpectObjectReconciled(ctx, env.Client, nodeClaimController, nodeClaim)

              		Expect(result.RequeueAfter).To(Not(Equal(0 * time.Second)))

              		Expect(result.RequeueAfter > 0*time.Second && result.RequeueAfter < 15*time.Minute).To(BeTrue())

Contributor Author

rschalo Jul 15, 2025 •

edited

Loading

Expect that the requeueAfter exists and is within the registration timeout window


          clean up

ae0d3dc

jonathan-innis approved these changes

View reviewed changes

Member

jonathan-innis left a comment

/lgtm
/approve

k8s-ci-robot assigned jonathan-innis

k8s-ci-robot added the lgtm label

Contributor

k8s-ci-robot commented Jul 15, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jonathan-innis, rschalo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [jonathan-innis]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added the approved label

k8s-ci-robot merged commit 280d3fb into kubernetes-sigs:main

16 checks passed

rschalo deleted the launch-ttl-fix branch

July 15, 2025 19:17

DerekFrank mentioned this pull request

REQUEST: New membership for DerekFrank kubernetes/org#5703

Closed

11 tasks

jmdeal mentioned this pull request

docs: add additional breaking changes to the 1.7 upgrade guide aws/karpenter-provider-aws#8492

Merged

3 tasks

jigisha620 pushed a commit to jigisha620/karpenter that referenced this pull request


          feat: (BREAKING) addition of launch timeout for nodeclaim lifecycle (k…

4dc5299

…ubernetes-sigs#2349)

harshad3339 added a commit to acquia/karpenter that referenced this pull request

* chore: bump go version to 1.24.4 (kubernetes-sigs#2298)

* chore: Only log that the command succeeded when it actually did (kubernetes-sigs#2302)

* fix: Fix bug with MarkForDeletion before creating replacements (kubernetes-sigs#2300)

* perf: Refactor the eviction queue to be multithreaded (kubernetes-sigs#2252)

* docs: Add Bizfly Cloud provider (kubernetes-sigs#2303)

* chore: Bump lifecycle cache expiration to one hour (kubernetes-sigs#2307)

* chore: Use cluster state to check replacement NodeClaim existence (kubernetes-sigs#2308)

* chore(deps): bump github.com/samber/lo from 1.50.0 to 1.51.0 in the go-deps group (kubernetes-sigs#2315)

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore: bump operatorpkg (kubernetes-sigs#2314)

* chore(deps): bump the k8s-go-deps group across 1 directory with 4 updates (kubernetes-sigs#2317)

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore: Refactor Orchestration Queue and Handle Mark/Unmark Deletion in Queue (kubernetes-sigs#2305)

* chore(deps): bump the k8s-go-deps group with 7 updates (kubernetes-sigs#2326)

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* perf: multithreaded orchestration queue (kubernetes-sigs#2293)

* test: Add nodeclaim name when you have garbage collection (kubernetes-sigs#2333)

* perf: Reduce multiple patch calls in instance termination (kubernetes-sigs#2324)

* fix: add helm rbac for kwok-provider to update finalizers (kubernetes-sigs#2336)

Signed-off-by: Max Cao <[email protected]>

* feat: configure CRD status operator with larger histogram buckets (kubernetes-sigs#2328)

* chore(deps): bump sigs.k8s.io/yaml from 1.4.0 to 1.5.0 in the k8s-go-deps group (kubernetes-sigs#2339)

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump github.com/docker/docker from 28.2.2+incompatible to 28.3.0+incompatible in the go-deps group (kubernetes-sigs#2340)

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix: Fix re-retrieving object on retry (kubernetes-sigs#2337)

* fix: Fix overriding error with patch call (kubernetes-sigs#2338)

* fix: add missing rlock to disruption queue (kubernetes-sigs#2348)

* test: allow e2e tests to output junit report (kubernetes-sigs#2334)

Signed-off-by: Max Cao <[email protected]>

* docs: Add Oracle Cloud Infrastructure (OCI) provider  (kubernetes-sigs#2342)

* fix: no longer allow the same hostname to take multiple capacity (kubernetes-sigs#2356)

* feat: support auto relaxing min values (kubernetes-sigs#2299)

* fix: update provider ID to ensure that Cloud Provider tests pass (kubernetes-sigs#2363)

* fix: remove unsupported capacity_type label from karpenter_nodeclaims… (kubernetes-sigs#2364)

* fix: update deletionTimestamp on terminating pods when after nodeDeletionTimestamp (kubernetes-sigs#2316)

Co-authored-by: Amanuel Engeda <[email protected]>

* chore: promote ReservedCapacity feature gate to beta (kubernetes-sigs#2365)

* fix: flakiness in expiration tests (kubernetes-sigs#2366)

* test: Bump the termination time for the deletion timestamp (kubernetes-sigs#2367)

* chore(deps): bump github.com/docker/docker from 28.3.0+incompatible to 28.3.1+incompatible in the go-deps group (kubernetes-sigs#2355)

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix: pod errors when nodepool requirements filter all instance types (kubernetes-sigs#2341)

* refactor: Create a NopValidator for the disruption testing (kubernetes-sigs#2369)

* chore(deps): bump the go-deps group with 2 updates (kubernetes-sigs#2373)

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* refactor: Update disruption testing from PR comments (kubernetes-sigs#2372)

* feat: (BREAKING) addition of launch timeout for nodeclaim lifecycle (kubernetes-sigs#2349)

* chore: Consider node.kubernetes.io/not-ready:NoExecute as ephemeral (kubernetes-sigs#2265)

* perf: Optimistically delete from the cache after launch (kubernetes-sigs#2380)

* docs: Node Overlay RFC (kubernetes-sigs#2166)

* fix: handle multiple PDBs for the same pod more gracefully (kubernetes-sigs#2379)

* docs: Add IBM Cloud provider (kubernetes-sigs#2396)

Signed-off-by: Josephine Pfeiffer <[email protected]>

* fix: rate limit eviction when PDBs are blocking (kubernetes-sigs#2399)

* feat: Add the Node Overlay CRD (kubernetes-sigs#2296)

* chore: ignore pods that use unsupported provisioner in the storageClass (kubernetes-sigs#2400)

* feat: Add a feature flag for Node Overlay (kubernetes-sigs#2404)

* feat: Add StaticCapacity feature flag (kubernetes-sigs#2405)

* fix(BREAKING): update naming of karpenter_pods_drained_total (kubernetes-sigs#2421)

* fix: pod metrics when pod is terminal (kubernetes-sigs#2417)

* chore: ignore pods that have unbound pvc with volumeBindingMode immediate (kubernetes-sigs#2415)

* docs: static capacity RFC (kubernetes-sigs#2309)

* chore: bump go version to 1.24.6 (kubernetes-sigs#2432)

* feat: Create optional operator arguments to leverage leader lease functionality (kubernetes-sigs#2433)

* chore(deps): bump the go-deps group with 5 updates (kubernetes-sigs#2442)

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump actions/checkout from 4.2.2 to 5.0.0 in /.github/actions/install-pyroscope in the action-deps group (kubernetes-sigs#2428)

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump the actions-deps group across 1 directory with 2 updates (kubernetes-sigs#2443)

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump actions/cache from 4.2.3 to 4.2.4 in /.github/actions/install-deps in the action-deps group (kubernetes-sigs#2425)

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix: do not block drifted nodes from being terminated if consolidation is disabled (kubernetes-sigs#2423)

* chore: Pin GH action SHAs for run-bench-test (kubernetes-sigs#2448)

* chore: update operatorpkg (kubernetes-sigs#2455)

* chore: Track NodeClaims in NodePoolState (kubernetes-sigs#2449)

* chore(deps): bump the k8s-go-deps group across 1 directory with 7 updates (kubernetes-sigs#2456)

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* perf: Add flag to disable costly metrics controllers (kubernetes-sigs#2354)

* perf: concurrent reconciles CPU-based scaling (kubernetes-sigs#2406)

* perf: Disruption Queue Retry Duration Scaling (kubernetes-sigs#2411)

* perf: Typed Bucket Scaling (kubernetes-sigs#2420)

* ci: Include K8s version 1.33 and 1.34 in testing (kubernetes-sigs#2465)

* chore: increase MaxInstanceTypes to give cloud-providers more control over instance type truncation (kubernetes-sigs#2430)

* chore(deps): bump the go-deps group with 2 updates (kubernetes-sigs#2461)

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump amannn/action-semantic-pull-request from 6.0.1 to 6.1.1 in the actions-deps group (kubernetes-sigs#2462)

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* ci: revert k8s 1.34 addition (kubernetes-sigs#2475)

* fix: Don't schedule a pod with DRA requirements (kubernetes-sigs#2384)

* fix: support arbitrary reserved capacity labels for drift (kubernetes-sigs#2476)

* chore(deps): bump actions/checkout from 4.2.2 to 5.0.0 in /.github/actions/install-prometheus in the action-deps group (kubernetes-sigs#2426)

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix: Fix nil pointer exception for multiNodeConsolidation (kubernetes-sigs#2472)

* fix: avoid hash collisions with duplicate match expressions (kubernetes-sigs#2479)

* ci: enable k8s 1.34 tests (kubernetes-sigs#2481)

* fix: Validate unsupported provisioners on bound PVs (kubernetes-sigs#2480)

* refactor: use iterator for iterating state nodes (kubernetes-sigs#2483)

* fix: make toolchain failing due to deletion of asciicheck (kubernetes-sigs#2485)

* fix: Handle PVC edge cases handled by kube-scheduler (kubernetes-sigs#2488)

* chore: Change appName from const to var (kubernetes-sigs#2489)

* fix: Handle unbound volumes with volumeName defined (kubernetes-sigs#2487)

* chore(deps): bump actions/setup-go from 5.5.0 to 6.0.0 in /.github/actions/install-deps in the action-deps group (kubernetes-sigs#2494)

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump actions/setup-python from 5.6.0 to 6.0.0 in the actions-deps group (kubernetes-sigs#2493)

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump the go-deps group with 6 updates (kubernetes-sigs#2491)

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump the k8s-go-deps group with 4 updates (kubernetes-sigs#2492)

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore: remove duplicate reconcile logging (kubernetes-sigs#2496)

* chore: bump operatorpkg version (kubernetes-sigs#2500)

* perf: Update the Node Repair Controller for requeue time  (kubernetes-sigs#2286)

* feat: Add NodeOverlay Controller Support (kubernetes-sigs#2306)

* chore(deps): bump the k8s-go-deps group with 3 updates (kubernetes-sigs#2504)

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore: rolling back to 1.34 (kubernetes-sigs#2512)

* fix: handle nil selector when hashing in topology (kubernetes-sigs#2511)

* feat: Support Pod Level Resources (kubernetes-sigs#2383)

Signed-off-by: Tsubasa Nagasawa <[email protected]>

* fix: merge limits into requests when constructing ds pods (kubernetes-sigs#2514)

* fix: default CPU_REQUESTS when non-positive value is provided (kubernetes-sigs#2516)

* fix(node): prevent empty providerID causing false NodeClaim matches (kubernetes-sigs#2507)

* feat: Support Static Capacity (kubernetes-sigs#2521)

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Jason Deal <[email protected]>
Co-authored-by: Jonathan Innis <[email protected]>
Co-authored-by: Andrew Mitchell <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Ryan Mistretta <[email protected]>

* fix: over provisioning static nodeclaims during controller crashes (kubernetes-sigs#2534)

* chore: drop consistency error to info log (kubernetes-sigs#2542)

* fix: flaky static provisioning unit test (kubernetes-sigs#2546)

* fix: nodepool crd definition should explicitly say replicas field as alpha (kubernetes-sigs#2554)

* chore: Update NodeRegistrationHealthy SC to use a buffer mechanism (kubernetes-sigs#2520)

---------

Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Max Cao <[email protected]>
Signed-off-by: Josephine Pfeiffer <[email protected]>
Signed-off-by: Tsubasa Nagasawa <[email protected]>
Co-authored-by: Derek Frank <[email protected]>
Co-authored-by: Jonathan Innis <[email protected]>
Co-authored-by: Lê Minh Quân <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jigisha Patil <[email protected]>
Co-authored-by: Amanuel Engeda <[email protected]>
Co-authored-by: Max Cao <[email protected]>
Co-authored-by: Aidan Rowe <[email protected]>
Co-authored-by: Daniel Lopes <[email protected]>
Co-authored-by: Saurav Agarwalla <[email protected]>
Co-authored-by: cosimomeli <[email protected]>
Co-authored-by: Jason Deal <[email protected]>
Co-authored-by: Reed Schalo <[email protected]>
Co-authored-by: Josephine Pfeiffer <[email protected]>
Co-authored-by: Sumukha Radhakrishna <[email protected]>
Co-authored-by: Andy Townsend <[email protected]>
Co-authored-by: Sumukha Radhakrishna <[email protected]>
Co-authored-by: ryan-mist <[email protected]>
Co-authored-by: Brandon Wagner <[email protected]>
Co-authored-by: Alima Azamat <[email protected]>
Co-authored-by: Andrew Mitchell <[email protected]>
Co-authored-by: Tsubasa Nagasawa <[email protected]>
Co-authored-by: Neil <[email protected]>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved cncf-cla: yes lgtm size/L