Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking 73 Kubernetes AWS Account Migrations from CNCF #4626

Closed
73 tasks done
hh opened this issue Jan 13, 2023 · 33 comments
Closed
73 tasks done

Tracking 73 Kubernetes AWS Account Migrations from CNCF #4626

hh opened this issue Jan 13, 2023 · 33 comments
Assignees
Labels
area/dns DNS records for k8s.io, kubernetes.io, k8s.dev, etc., code in dns/ area/infra/aws Issues or PRs related to Kubernetes AWS infrastructure sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra.
Milestone

Comments

@hh
Copy link
Member

hh commented Jan 13, 2023

There are 73 total Kubernetes accounts in the CNCF AWS Account. We will need to track these migrations.

arn:aws:servicequotas:::organizations/L-29A0C5DF is a request to raise the default max accounts from 10 to 200.

The process for moving accounts between organisations is outlined here: https://aws.amazon.com/premiumsupport/knowledge-center/organizations-move-accounts/

Kubernetes OU (3)

registry.k8s.io OU (1)

aws-ebs-csi-driver OU (1)

kops OU (4)

sig-k8s-infra OU (12)

capa OU (52)

@hh hh added area/dns DNS records for k8s.io, kubernetes.io, k8s.dev, etc., code in dns/ sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra. labels Jan 13, 2023
@hh hh changed the title Tracking Kubernetes AWS Account Migrations from CNCF Tracking 73 Kubernetes AWS Account Migrations from CNCF Jan 13, 2023
@hh
Copy link
Member Author

hh commented Jan 16, 2023

When planning our migration, we need to consider the following:

  • You have a valid payment method on the member account to address any charges that are incurred while the accounts are migrating.
    I've opened Case ID: 11777279581 with AWS to research, but I'm trying to avoid doing this by hand. Suggestions and workarounds welcome here.
  • Before migrating, consider the number of accounts in your organization. If an increase is needed, see Quotas for AWS Organizations for more information.
    I've submitted this, waiting on resolution. No update to arn:aws:servicequotas:::organizations/L-29A0C5DF yet.
  • You backed up any reports from the member accounts that you need to keep. The member accounts can't access these reports after leaving the Organization.
    @dims @ameukam Let me know of any to migrate here.

@hh
Copy link
Member Author

hh commented Jan 16, 2023

From https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_accounts_remove.html#orgs_manage_accounts_remove-from-master

Effects of removing an account from an organization

When you remove an account from an organization, no direct changes are made to the account. However, the following indirect effects occur:

  • The account is now responsible for paying its own charges and must have a valid payment method attached to the account.
    I'm working on this now, trying to find some automation.
  • Integration with other services might be disabled. If you remove an account from an organization that has integration with an AWS service enabled, the users in that account can no longer use that service.
    I'm not aware of any, but there just wanted this highlighted.

@hh
Copy link
Member Author

hh commented Jan 17, 2023

It looks like we will need to detach each of the 73 accounts from the CNCF org, and at the moment I disconnect them we will need to provide credit card details (which can only be done via the Web Console / UI).

While I can do that, an account can only be part of a single org. So once detached, each of the accounts will need to be authenticated to to accept the invitation to join the new Kubernetes org.

It's unclear to me the best way to handle authenticating to these accounts as the passwords are likely handled in a distributed way by the Kubernetes community.

@hh
Copy link
Member Author

hh commented Jan 17, 2023

I'm looking at options for authenticating to and accepting the handshake to join the new org once they are detached from the CNCF org.

We must authenticate as each account, and AssumeRole doesn't look like it will work once the accounts are standalone. Also the current passwords are unknown to the CNCF at an administrative level.

We'll need to decide on a process. We could reset the password on each account as we go and share each of them with #sig-k8s-infra, but I'm open to options.

I've thought about trying it for one of the top level accounts that is less used, like [email protected] to test the process, but wanted to get buy-in for the reset password approach OR possibly escalating a ticket with Premium Support with AWS.

/cc @dims @ameukam

@dims
Copy link
Member

dims commented Jan 17, 2023

@hh will let @ameukam reply on the "try with top level account". If we can escalate a ticket, we should just start that without delay.

@hh
Copy link
Member Author

hh commented Jan 17, 2023

@dims can you help to escalate this Case ID : 11777279581 ?

Our official support plan is Basic, so I'm not sure I can do much on my end:
https://support.console.aws.amazon.com/support/home?region=us-west-2#/case/?displayId=11777279581&language=en

I'd also like to ensure that this service quota request to increase the default number of accounts was lifted from 10 to 200. We'll hit a roadblock pretty quick otherwise. It says Status: "CASE_CLOSED" but it's unclear if the quota was actually lifted. It's within this ticket, but I still can't verify. It was discussed in the body of this Case ID as well.

 aws service-quotas list-requested-service-quota-change-history                                                     
{
    "RequestedQuotas": [
        {
            "Id": "99e343d85b6c4879bf57682d792e24ddKWJ1tFl9",
            "CaseId": "11753963261",
            "ServiceCode": "organizations",
            "ServiceName": "AWS Organizations",
            "QuotaCode": "L-29A0C5DF",
            "QuotaName": "Default maximum number of accounts",
            "DesiredValue": 200.0,
            "Status": "CASE_CLOSED",
            "Created": "2023-01-13T23:58:00.061000+13:00",
            "LastUpdated": "2023-01-14T20:39:57.106000+13:00",
            "Requester": "{\"accountId\":\"348685125169\",\"callerArn\":\"arn:aws:iam::348685125169:root\"}",
            "QuotaArn": "arn:aws:servicequotas::348685125169:organizations/L-29A0C5DF",
            "GlobalQuota": true,
            "Unit": "None"
        }
    ]
}

@hh
Copy link
Member Author

hh commented Jan 17, 2023

I'll work with @jeefy and @taylorwaggoner to get billing/card details to use during the migration. Once I have those we can try to move [email protected] manually.

Steps:

@upodroid
Copy link
Member

It is worth finding out if we need to retain most of these accounts. Are these accounts actually holding live infra or is it part of CI/CD usage like boskos projects? If they are part of a CI process and we can supply new accounts, we can provision fresh ones, reconfigure IAM, and use that opportunity to start on a clean slate.

@hh
Copy link
Member Author

hh commented Jan 18, 2023

It is worth finding out if we need to retain most of these accounts.
@upodroid I agree

Are these accounts actually holding live infra or is it part of CI/CD usage like boskos projects?
Is there anyone who could identify these easily given the list? I have some automation ready to go through them, but we need a plan, it's a lot of accounts.

are they part of a CI process?
This migration is an ideal time to identify and consolidate.

I would suggest we add someone (or a role) from sig-k8s-infra to each of these accounts / mailing lists, so password resets can occur.

The Kubernetes OU is limited to accounts that contain k8s in the email:

aws organizations list-accounts --query 'sort_by(Accounts,&Email)[?contains(Email,k8s)==true].[Email]' | grep \" | wc -l

73

All but four of these are email lists maintained by kubernetes.io (google workspace groups / email lists?):
aws organizations list-accounts --query 'sort_by(Accounts,&Email)[?contains(Email,k8s)==true`].[Email]' | grep " | grep -v kubernetes.io;

@upodroid
Copy link
Member

Getting access to the *@kubernetes.io emails is easy.

Add your email to the groups in this file. Ignore everything between the + and @ symbols to get the email.

https://github.com/kubernetes/k8s.io/blob/main/groups/sig-k8s-infra/groups.yaml#L270

@hh
Copy link
Member Author

hh commented Jan 18, 2023

Here are the emails associated with kubernetes.io:

aws organizations list-accounts --query 'sort_by(Accounts,&Email)[?contains(Email,`kubernetes.io`)==`true`].[Email]' | grep \"  | sed 's/+.*@/@/' | sort -u
        "[email protected]"
        "[email protected]"
        "[email protected]"
        "[email protected]"
        "[email protected]"

I'll put in a PR to add [email protected] to each of these @upodroid.

hh added a commit to hh/k8s.io that referenced this issue Jan 18, 2023
hh added a commit to hh/k8s.io that referenced this issue Jan 18, 2023
This list is associated with an AWS account that is part of a the migration to a new Kubernetes AWS Org in kubernetes#4626
This is to enable a password reset for that account and ensure someone within #sig-k8s-infra has password access to all Kubernetes AWS Org member accounts.
@hh
Copy link
Member Author

hh commented Jan 18, 2023

Created #4645 for:

Created #4646 for:

I couldn't find these two lists in sig-k8s-infra/groups.yaml:

I'll use stsAssumeRole to see if there in any infrastructure contained within those last two. We will need to decide on a place to store the changed passwords (and ideally an AWS account password rotation policy).

@ameukam
Copy link
Member

ameukam commented Jan 18, 2023

We need to move the AWS accounts under those emails:

we can revisit migration of the accounts used by Kops and CAPA.

@ameukam
Copy link
Member

ameukam commented Jan 18, 2023

It is worth finding out if we need to retain most of these accounts. Are these accounts actually holding live infra or is it part of CI/CD usage like boskos projects? If they are part of a CI process and we can supply new accounts, we can provision fresh ones, reconfigure IAM, and use that opportunity to start on a clean slate.

I'm in favor of this approach. we can have a mix of accounts created and accounts migrated.

@hh
Copy link
Member Author

hh commented Jan 18, 2023

Regarding account limit quota increase request, there was an update:

Service: AWS Organizations
Region: US East (Northern Virginia)
Limit name: Default maximum number of accounts
New limit value: 200

For a quota increase of this type, I will need to collaborate with our service team to get approval.

The team can take anything between 24-48 hours to provide feedback to the request, this is to ensure that we can meet your needs while keeping existing infrastructure safe.

I have marked the request as a priority in order to get feedback as soon as possible.

@ameukam
Copy link
Member

ameukam commented Jan 18, 2023

@hh Sent an invite for [email protected] account. Can you talk a look in the org handshakes in the CNCF org:

$ aws organizations invite-account-to-organization --target Id=REDACTED,Type=ACCOUNT
{
    "Handshake": {
        "Id": "h-8201bc74bc5846f480ea2e63bed0b7dc",
        "Arn": "arn:aws:organizations::REDACTED:handshake/o-kz4vlkihvy/invite/h-8201bc74bc5846f480ea2e63bed0b7dc",
        "Parties": [
            {
                "Id": "kz4vlkihvy",
                "Type": "ORGANIZATION"
            },
            {
                "Id": "REDACTED",
                "Type": "ACCOUNT"
            }
        ],
        "State": "OPEN",
        "RequestedTimestamp": "2023-01-18T20:43:46.778000+00:00",
        "ExpirationTimestamp": "2023-02-02T20:43:46.778000+00:00",
        "Action": "INVITE",
        "Resources": [
            {
                "Value": "o-kz4vlkihvy",
                "Type": "ORGANIZATION",
                "Resources": [
                    {
                        "Value": "[email protected]",
                        "Type": "MASTER_EMAIL"
                    },
                    {
                        "Value": "Kubernetes AWS Admins",
                        "Type": "MASTER_NAME"
                    },
                    {
                        "Value": "ALL",
                        "Type": "ORGANIZATION_FEATURE_SET"
                    }
                ]
            },
            {
                "Value": "REDACTED",
                "Type": "ACCOUNT"
            }
        ]
    }
}

@hh
Copy link
Member Author

hh commented Jan 30, 2023

[email protected] is now migrated

@hh
Copy link
Member Author

hh commented Jan 30, 2023

[email protected] (previously: [email protected]) is now migrated

@hh
Copy link
Member Author

hh commented Jan 30, 2023

[email protected] (previously [email protected]) has an updated email and is ready to migrate: @ameukam I'd like to do this one synchronously

I'd like to get @Riaankl to migrate the remaining accounts *@lists.cncf.io and k8s-infra-aws-admins+*kubernetes.io. The migration process is a bit more streamlined when we have access to everything now.

@hh
Copy link
Member Author

hh commented Jan 30, 2023

/assign @Riaankl

@hh
Copy link
Member Author

hh commented Jan 30, 2023

@mrbobbytables: We will be generating a lot of passwords that need to be saved and likely accessed by #sig-k8s-infra and the sigs these are assigned to. @dims mentioned there is a community 1Password account we should be using. Do you have details?

@hh
Copy link
Member Author

hh commented Feb 3, 2023

All but three accounts migrated. Will work to put passwords into 1Password next week.

@ameukam
Copy link
Member

ameukam commented Feb 6, 2023

/area infra/aws
/milestone v1.27

@k8s-ci-robot k8s-ci-robot added the area/infra/aws Issues or PRs related to Kubernetes AWS infrastructure label Feb 6, 2023
@k8s-ci-robot k8s-ci-robot added this to the v1.27 milestone Feb 6, 2023
@riaankleinhans
Copy link
Contributor

Only account remaining to be migrated is [email protected]
We do not get the password reset email & link.
@ameukam @dims do you have access to that?

@dims
Copy link
Member

dims commented Feb 7, 2023

@Riaankl check with CNCF folks like Amye? if we can figure out who is on that list, we can coordinate with them

@riaankleinhans
Copy link
Contributor

@Riaankl check with CNCF folks like Amye? if we can figure out who is on that list, we can coordinate with them

Will do.

@riaankleinhans
Copy link
Contributor

@Riaankl check with CNCF folks like Amye? if we can figure out who is on that list, we can coordinate with them

Got that sorted, last account migrated!

@ameukam
Copy link
Member

ameukam commented Feb 7, 2023

@Riaankl check with CNCF folks like Amye? if we can figure out who is on that list, we can coordinate with them

Got that sorted, last account migrated!

@Riaankl if everything is done here, do you mind close this issue ? cc @jeefy @hh

@riaankleinhans
Copy link
Contributor

@ameukam I still need to add the passwords for all the accounts in 1password
Think we can close after that. Thanks

@riaankleinhans
Copy link
Contributor

All passowrds for the accounts that I migrated was added to the 1password vault AWS CI Accouts
@hh @jeefy @ameukam

/close

@k8s-ci-robot
Copy link
Contributor

@Riaankl: Closing this issue.

In response to this:

All passowrds for the accounts that I migrated was added to the 1password vault AWS CI Accouts
@hh @jeefy @ameukam

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@github-project-automation github-project-automation bot moved this from 🏗 In progress to ✅ Done in AWS Infrastructure (SIG K8s Infra) Feb 9, 2023
@hh
Copy link
Member Author

hh commented Feb 10, 2023

aws organizations describe-organization
{
    "Organization": {
        "Id": "o-kz4vlkihvy",
        "Arn": "arn:aws:organizations::348685125169:organization/o-kz4vlkihvy",
        "FeatureSet": "ALL",
        "MasterAccountArn": "arn:aws:organizations::348685125169:account/o-kz4vlkihvy/348685125169",
        "MasterAccountId": "348685125169",
        "MasterAccountEmail": "[email protected]",
        "AvailablePolicyTypes": [
            {
                "Type": "SERVICE_CONTROL_POLICY",
                "Status": "ENABLED"
            }
        ]
    }
}
aws organizations list-accounts | jq -r '.Accounts[] | select(.Status="ACTIVE") | .Name + " : " + .Email'  | sort
Kubernetes AWS Admins : [email protected]
aws-ebs-csi-driver Infra : [email protected]
capa-2020-07-12-01 : [email protected]
capa-2020-07-12-02 : [email protected]
capa-2020-07-13-00 : [email protected]
capa-2020-07-13-01 : [email protected]
capa-2020-07-13-02 : [email protected]
capa-2020-07-13-03 : [email protected]
capa-2020-07-13-04 : [email protected]
capa-2020-07-13-05 : [email protected]
capa-2020-07-13-06 : [email protected]
capa-2020-07-13-07 : [email protected]
capa-2020-07-13-08 : [email protected]
capa-2020-07-13-09 : [email protected]
capa-2020-07-13-10 : [email protected]
capa-2020-07-13-11 : [email protected]
capa-2020-07-13-12 : [email protected]
capa-2020-07-13-13 : [email protected]
capa-2020-11-19-00 : [email protected]
capa-2020-11-19-01 : [email protected]
capa-2020-11-19-02 : [email protected]
capa-2020-11-19-03 : [email protected]
capa-2020-11-19-04 : [email protected]
capa-2020-11-19-05 : [email protected]
capa-2020-11-19-06 : [email protected]
capa-2020-11-19-07 : [email protected]
capa-2020-11-19-08 : [email protected]
capa-2020-11-19-09 : [email protected]
capa-2020-11-19-10 : [email protected]
capa-2020-11-19-11 : [email protected]
capa-2020-11-19-12 : [email protected]
capa-2020-11-19-13 : [email protected]
capa-account-00 : [email protected]
capa-account-01 : [email protected]
capa-account-02 : [email protected]
capa-account-03 : [email protected]
capa-account-04 : [email protected]
capa-account-05 : [email protected]
capa-account-06 : [email protected]
capa-account-07 : [email protected]
capa-account-08 : [email protected]
capa-account-09 : [email protected]
capa-account-2020-02-18-a : [email protected]
capa-account-2020-02-18-b : [email protected]
capa-account-2020-02-18-c : [email protected]
capa-account-2020-02-18-d : [email protected]
capa-account-2020-02-18-e : [email protected]
capa-account-2020-02-18-f : [email protected]
capa-account-2020-02-18-g : [email protected]
capa-account-2020-02-18-h : [email protected]
capa-account-2020-02-18-i : [email protected]
capa-account-2020-02-18-j : [email protected]
cncf-k8s-infra-aws-capa-ami : [email protected]
cncf-k8s-infra-aws-capa-demo : [email protected]
e2e-kops-2020-03-15-a : [email protected]
e2e-kops-2020-03-15-b : [email protected]
e2e-kops-2020-03-15-c : [email protected]
e2e-kops-2020-03-15-d : [email protected]
image-builder-aws-2020-11-19-00 : [email protected]
image-builder-aws-2020-11-19-02 : [email protected]
image-builder-aws-2020-11-19-03 : [email protected]
image-builder-aws-2020-11-19-04 : [email protected]
image-builder-aws-2020-11-19-05 : [email protected]
image-builder-aws-2020-11-19-06 : [email protected]
image-builder-aws-2020-11-19-07 : [email protected]
image-builder-aws-2020-11-19-08 : [email protected]
image-builder-aws-2020-11-19-09 : [email protected]
image-builder-aws-2020-11-19-10 : [email protected]
image-builder-aws-2021-03-28-01 : [email protected]
image-builder-aws-2021-25-03/28/21-01 : k8s-infra-aws-admins+image20212503/28/[email protected]
k8s-infra-artifacts-k8s-io-prod : [email protected]
k8s-infra-aws-admins+accounts : [email protected]
k8s-infra-aws-admins+registry-k8s-io : [email protected]
k8s-infra-aws-root-account : [email protected]
k8s-infra-networking : [email protected]
k8s-infra-organisation-incident-response : [email protected]
k8s-infra-organisation-logging : [email protected]
k8s-infra-organisation-security : [email protected]
k8s-infra-registry-k8s-io-prod : [email protected]
k8s-infra-security-audit : [email protected]
k8s-infra-security-engineering : [email protected]
k8s-infra-security-incident-response : [email protected]
k8s-infra-security-logs : [email protected]
k8s-infra-security-logs : [email protected]
k8s-infra-shared-services : [email protected]
k8s-infra-shared-services : [email protected]
sig-release-leads : [email protected]

@hh hh closed this as completed Feb 10, 2023
@jeefy
Copy link
Member

jeefy commented Apr 7, 2023

Looks like some were missed, filed #5111 to capture that effort, but linking here as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/dns DNS records for k8s.io, kubernetes.io, k8s.dev, etc., code in dns/ area/infra/aws Issues or PRs related to Kubernetes AWS infrastructure sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra.
Projects
Status: Done
Development

No branches or pull requests

7 participants