🐛 awsmachine: only register machine to LB when it's running #5040

r4f4 · 2024-06-28T13:25:11Z

What type of PR is this?

/kind bug

What this PR does / why we need it:

The AWS docs [1] state that to register an instance with the Load Balancer target groups, the instance must be running. Currently, CAPA tries to register it while it's still pending, causing the following error:

0314 22:24:18.512176  701419 awsmachine_controller.go:605] "failed to reconcile LB attachment" err=<
    [could not register machine to load balancer: could not register control plane instance "i-039bb22b1e8df7c99" with load balancer: failed to register instance with target group 'mrb-capa-67-d2pfx-int-22623': InvalidTarget: The following targets are not in a running state and cannot be registered: 'i-039bb22b1e8df7c99'
        status code: 400, request id: 17514354-77ac-42b1-a882-489760563bbd, could not register machine to load balancer: could not register control plane instance "i-039bb22b1e8df7c99" with load balancer: failed to register instance with target group 'mrb-capa-67-d2pfx-ext-6443': InvalidTarget: The following targets are not in a running state and cannot be registered: 'i-039bb22b1e8df7c99'
        status code: 400, request id: 84e1849c-abb7-4af9-9220-8791fdc1a3fb]
 >
I0314 22:24:18.512325  701419 recorder.go:104] "events: Failed to register control plane instance \"i-039bb22b1e8df7c99\" with load balancer: failed to register instance with target group 'mrb-capa-67-d2pfx-ext-6443': InvalidTarget: The following targets are not in a running state and cannot be registered: 'i-039bb22b1e8df7c99'\n\tstatus code: 400, request id: 84e1849c-abb7-4af9-9220-8791fdc1a3fb" type="Warning" object={"kind":"AWSMachine","namespace":"openshift-cluster-api-guests","name":"mrb-capa-67-d2pfx-bootstrap","uid":"58af1162-380b-4f3e-93fe-c0e81401070e","apiVersion":"infrastructure.cluster.x-k8s.io/v1beta2","resourceVersion":"562"} reason="FailedAttachControlPlaneELB"

Even though this doesn't stop the install from succeeding, let's wait for the instance state to be "running" and with that avoid unnecessary AWS API calls.

[1] https://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-register-targets.html#register-instances

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #5037

Special notes for your reviewer:

Checklist:

Release note:

Fix registering instances to Load Balancers when instances are still pending.

k8s-ci-robot · 2024-06-28T13:25:20Z

Hi @r4f4. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

mtulio

/lgtm

k8s-ci-robot · 2024-06-28T13:49:27Z

@mtulio: changing LGTM is restricted to collaborators

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

r4f4 · 2024-06-28T17:01:45Z

No regressions found in Openshift e2e CI tests: openshift/installer#8678

richardcase · 2024-07-01T13:16:11Z

/ok-to-test

r4f4 · 2024-07-01T14:09:11Z

/hold
I'll look into the failing tests.

The AWS docs [1] state that to register an instance with the Load Balancer target groups, the instance must be running. Currently, CAPA tries to register it while it's still pending, causing the following error: ``` 0314 22:24:18.512176 701419 awsmachine_controller.go:605] "failed to reconcile LB attachment" err=< [could not register machine to load balancer: could not register control plane instance "i-039bb22b1e8df7c99" with load balancer: failed to register instance with target group 'mrb-capa-67-d2pfx-int-22623': InvalidTarget: The following targets are not in a running state and cannot be registered: 'i-039bb22b1e8df7c99' status code: 400, request id: 17514354-77ac-42b1-a882-489760563bbd, could not register machine to load balancer: could not register control plane instance "i-039bb22b1e8df7c99" with load balancer: failed to register instance with target group 'mrb-capa-67-d2pfx-ext-6443': InvalidTarget: The following targets are not in a running state and cannot be registered: 'i-039bb22b1e8df7c99' status code: 400, request id: 84e1849c-abb7-4af9-9220-8791fdc1a3fb] > I0314 22:24:18.512325 701419 recorder.go:104] "events: Failed to register control plane instance \"i-039bb22b1e8df7c99\" with load balancer: failed to register instance with target group 'mrb-capa-67-d2pfx-ext-6443': InvalidTarget: The following targets are not in a running state and cannot be registered: 'i-039bb22b1e8df7c99'\n\tstatus code: 400, request id: 84e1849c-abb7-4af9-9220-8791fdc1a3fb" type="Warning" object={"kind":"AWSMachine","namespace":"openshift-cluster-api-guests","name":"mrb-capa-67-d2pfx-bootstrap","uid":"58af1162-380b-4f3e-93fe-c0e81401070e","apiVersion":"infrastructure.cluster.x-k8s.io/v1beta2","resourceVersion":"562"} reason="FailedAttachControlPlaneELB" ``` Even though this doesn't stop the install from succeeding, let's wait for the instance state to be "running" and with that avoid unnecessary AWS API calls. [1] https://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-register-targets.html#register-instances

r4f4 · 2024-07-04T16:49:07Z

With the updated version:

DEBUG I0704 16:33:28.785101     109 awsmachine_controller.go:968] "registering to v2 load balancer"
DEBUG I0704 16:33:29.293878     109 recorder.go:104] "Cannot register control plane instance \"i-03b7a9be210c6c6d4\" with load balancer: instance is not running" logger="events" type="Warning" object={"kind":"AWSMachine","namespace":"openshift-cluster-api-guests","name":"rdossant-installer-07-vctwp-master-2","uid":"f07dd126-1edf-4c9a-9047-064adcad7e91","apiVersion":"infrastructure.cluster.x-k8s.io/v1beta2","resourceVersion":"551"} reason="FailedAttachControlPlaneELB"
DEBUG I0704 16:33:29.670727     109 awsmachine_controller.go:641] "done reconciling instance" instance={"id":"i-03b7a9be210c6c6d4","instanceState":"pending","type":"m6i.xlarge","subnetId":"subnet-0f01b76318ad9cb2d","imageId":"ami-01d43c90109a44820","securityGroupIds":["sg-0a695045b014e15a9","sg-0e01b50f92a13f13c","sg-073e69c91a0db8276"],"iamProfile":"rdossant-installer-07-vctwp-master-profile","addresses":[{"type":"InternalDNS","address":"ip-10-0-83-84.us-east-2.compute.internal"},{"type":"InternalIP","address":"10.0.83.84"}],"privateIp":"10.0.83.84","enaSupport":true,"ebsOptimized":false,"tags":{"MachineName":"openshift-cluster-api-guests/rdossant-installer-07-vctwp-master-2","Name":"rdossant-installer-07-vctwp-master-2","kubernetes.io/cluster/rdossant-installer-07-vctwp":"owned","sigs.k8s.io/cluster-api-provider-aws/cluster/rdossant-installer-07-vctwp":"owned","sigs.k8s.io/cluster-api-provider-aws/role":"control-plane"},"availabilityZone":"us-east-2c","volumeIDs":["vol-0d1ddd57774d66010"],"instanceMetadataOptions":{"httpEndpoint":"enabled","httpPutResponseHopLimit":1,"httpTokens":"optional","instanceMetadataTags":"disabled"},"privateDnsName":{"enableResourceNameDnsAAAARecord":false,"enableResourceNameDnsARecord":false,"hostnameType":"ip-name"}}
DEBUG I0704 16:33:29.670765     109 awsmachine_controller.go:643] "but find the instance is pending, requeue" instance="i-03b7a9be210c6c6d4"
[...]
DEBUG I0704 16:33:51.738230     109 awsmachine_controller.go:569] "EC2 instance state changed" state="running" instance-id="i-03b7a9be210c6c6d4"
[...]
DEBUG I0704 16:33:55.973309     109 awsmachine_controller.go:968] "registering to v2 load balancer"
DEBUG I0704 16:33:55.973454     109 recorder.go:104] "Control plane instance \"i-03b7a9be210c6c6d4\" is registered with load balancer" logger="events" type="Normal" object={"kind":"AWSMachine","namespace":"openshift-cluster-api-guests","name":"rdossant-installer-07-vctwp-master-2","uid":"f07dd126-1edf-4c9a-9047-064adcad7e91","apiVersion":"infrastructure.cluster.x-k8s.io/v1beta2","resourceVersion":"551"} reason="SuccessfulAttachControlPlaneELB"
DEBUG I0704 16:33:59.098666     109 awsmachine_controller.go:641] "done reconciling instance" instance={"id":"i-03b7a9be210c6c6d4","instanceState":"running","type":"m6i.xlarge","subnetId":"subnet-0f01b76318ad9cb2d","imageId":"ami-01d43c90109a44820","securityGroupIds":["sg-0a695045b014e15a9","sg-0e01b50f92a13f13c","sg-073e69c91a0db8276"],"iamProfile":"rdossant-installer-07-vctwp-master-profile","addresses":[{"type":"InternalDNS","address":"ip-10-0-83-84.us-east-2.compute.internal"},{"type":"InternalIP","address":"10.0.83.84"}],"privateIp":"10.0.83.84","enaSupport":true,"ebsOptimized":false,"tags":{"MachineName":"openshift-cluster-api-guests/rdossant-installer-07-vctwp-master-2","Name":"rdossant-installer-07-vctwp-master-2","kubernetes.io/cluster/rdossant-installer-07-vctwp":"owned","sigs.k8s.io/cluster-api-provider-aws/cluster/rdossant-installer-07-vctwp":"owned","sigs.k8s.io/cluster-api-provider-aws/role":"control-plane"},"availabilityZone":"us-east-2c","volumeIDs":["vol-0d1ddd57774d66010"],"instanceMetadataOptions":{"httpEndpoint":"enabled","httpPutResponseHopLimit":1,"httpTokens":"optional","instanceMetadataTags":"disabled"},"privateDnsName":{"enableResourceNameDnsAAAARecord":false,"enableResourceNameDnsARecord":false,"hostnameType":"ip-name"}}

/test pull-cluster-api-provider-aws-e2e-blocking

r4f4 · 2024-07-04T17:42:06Z

/test pull-cluster-api-provider-aws-e2e

r4f4 · 2024-07-04T20:45:11Z

/hold cancel

I don't see any regressions on Openshift e2e tests.

alexander-demicev

/lgtm

richardcase · 2024-07-17T10:36:24Z

/test pull-cluster-api-provider-aws-e2e
/test pull-cluster-api-provider-aws-e2e-eks

richardcase · 2024-07-23T06:45:43Z

/milestone v2.6.0

cnmcavoy · 2024-07-23T21:07:29Z

/lgtm also

damdo

/lgtm

/assign @richardcase

for approval

AndiDog · 2024-07-29T08:04:20Z

/approve

richardcase · 2024-07-29T08:43:26Z

Its already approved but also:

/approve

k8s-ci-robot · 2024-07-29T08:43:32Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: AndiDog, richardcase

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [AndiDog,richardcase]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

damdo · 2024-07-29T08:59:42Z

@r4f4 does this need backporting?

r4f4 · 2024-07-29T09:25:06Z

@damdo It's not a serious issue but I think it'd be nice to backport.

damdo · 2024-07-29T09:33:12Z

@richardcase wdyt?

This bump includes the following fix to only register instances to the LB when they are in a running state: * kubernetes-sigs/cluster-api-provider-aws#5040 This should avoid unnecessary AWS API calls and red herring error messages in the log output.

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. labels Jun 28, 2024

k8s-ci-robot requested review from damdo and vincepri June 28, 2024 13:25

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-priority labels Jun 28, 2024

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jun 28, 2024

r4f4 mentioned this pull request Jun 28, 2024

OCPBUGS-36327: bump capa for instance register fix openshift/installer#8678

Merged

mtulio reviewed Jun 28, 2024

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 1, 2024

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 1, 2024

r4f4 force-pushed the fix-5037 branch from d967ea4 to 172038a Compare July 4, 2024 16:16

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jul 4, 2024

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 4, 2024

alexander-demicev reviewed Jul 10, 2024

View reviewed changes

k8s-ci-robot assigned alexander-demicev Jul 10, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 10, 2024

k8s-ci-robot added this to the v2.6.0 milestone Jul 23, 2024

damdo reviewed Jul 24, 2024

View reviewed changes

k8s-ci-robot assigned richardcase and damdo Jul 24, 2024

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 29, 2024

k8s-ci-robot merged commit 0159435 into kubernetes-sigs:main Jul 29, 2024
23 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 awsmachine: only register machine to LB when it's running #5040

🐛 awsmachine: only register machine to LB when it's running #5040

r4f4 commented Jun 28, 2024

k8s-ci-robot commented Jun 28, 2024

mtulio left a comment

k8s-ci-robot commented Jun 28, 2024

r4f4 commented Jun 28, 2024

richardcase commented Jul 1, 2024

r4f4 commented Jul 1, 2024

r4f4 commented Jul 4, 2024

r4f4 commented Jul 4, 2024

r4f4 commented Jul 4, 2024

alexander-demicev left a comment

richardcase commented Jul 17, 2024

richardcase commented Jul 23, 2024

cnmcavoy commented Jul 23, 2024

damdo left a comment

AndiDog commented Jul 29, 2024

richardcase commented Jul 29, 2024

k8s-ci-robot commented Jul 29, 2024

damdo commented Jul 29, 2024

r4f4 commented Jul 29, 2024

damdo commented Jul 29, 2024

🐛 awsmachine: only register machine to LB when it's running #5040

🐛 awsmachine: only register machine to LB when it's running #5040

Conversation

r4f4 commented Jun 28, 2024

k8s-ci-robot commented Jun 28, 2024

mtulio left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Jun 28, 2024

r4f4 commented Jun 28, 2024

richardcase commented Jul 1, 2024

r4f4 commented Jul 1, 2024

r4f4 commented Jul 4, 2024

r4f4 commented Jul 4, 2024

r4f4 commented Jul 4, 2024

alexander-demicev left a comment

Choose a reason for hiding this comment

richardcase commented Jul 17, 2024

richardcase commented Jul 23, 2024

cnmcavoy commented Jul 23, 2024

damdo left a comment

Choose a reason for hiding this comment

AndiDog commented Jul 29, 2024

richardcase commented Jul 29, 2024

k8s-ci-robot commented Jul 29, 2024

damdo commented Jul 29, 2024

r4f4 commented Jul 29, 2024

damdo commented Jul 29, 2024