Skip to content

[v12] Log troubleshooting information when InvalidInstanceID errors are found during EC2 discovery #25641

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lxea merged 5 commits into from
May 8, 2023
Merged

[v12] Log troubleshooting information when InvalidInstanceID errors are found during EC2 discovery #25641

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
195fce2
Log troubleshooting information when InvalidInstanceID errors are found
Apr 27, 2023
4406219
resolve comments
Apr 27, 2023
780c97c
resolve comments
May 2, 2023
39be365
resolve comments
May 4, 2023
d086039
fix typo
May 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions docs/pages/server-access/guides/ec2-discovery.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -399,6 +399,19 @@ error json: cannot unmarshal object into Go struct field DownloadContentPlugin.s

It is likely that you're running an older SSM agent version. Upgrade to SSM agent version 3.1 or greater to resolve.

### `InvalidInstanceId: Instances [[i-123]] not in a valid state for account 456`

The following problems can cause this error:
- The Discovery Service doesn't have permission to access the managed node.
- AWS Systems Manager Agent (SSM Agent) isn't running. Verify that SSM Agent is running.
- SSM Agent isn't registered with the SSM endpoint. Try reinstalling SSM Agent.
- The discovered instance does not have permission to receive SSM
commands, verify the instance includes the AmazonSSMManagedInstanceCore IAM policy.

See SSM RunCommand error codes and troubleshooting information in AWS documentation for more details:
- https://docs.aws.amazon.com/systems-manager/latest/userguide/troubleshooting-managed-instances.html
- https://docs.aws.amazon.com/systems-manager/latest/APIReference/API_SendCommand.html#API_SendCommand_Errors

## Next steps

- Read [Joining Nodes via AWS IAM
Expand Down
7 changes: 6 additions & 1 deletion lib/srv/discovery/discovery.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,9 @@ import (

"github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v3"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/awserr"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/aws/aws-sdk-go/service/ssm"
"github.com/gravitational/trace"
"github.com/sirupsen/logrus"
"golang.org/x/exp/slices"
Expand Down Expand Up @@ -404,7 +406,10 @@ func (s *Server) handleEC2Discovery() {
instances.AccountID, genEC2InstancesLogStr(ec2Instances.Instances))

if err := s.handleEC2Instances(ec2Instances); err != nil {
if trace.IsNotFound(err) {
var aErr awserr.Error
if errors.As(err, &aErr) && aErr.Code() == ssm.ErrCodeInvalidInstanceId {
s.Log.WithError(err).Error("SSM SendCommand failed with ErrCodeInvalidInstanceId. Make sure that the instances have AmazonSSMManagedInstanceCore policy assigned. Also check that SSM agent is running and registered with the SSM endpoint on that instance and try restarting or reinstalling it in case of issues. See https://docs.aws.amazon.com/systems-manager/latest/APIReference/API_SendCommand.html#API_SendCommand_Errors for more details.")
} else if trace.IsNotFound(err) {
s.Log.Debug("All discovered EC2 instances are already part of the cluster.")
} else {
s.Log.WithError(err).Error("Failed to enroll discovered EC2 instances.")
Expand Down