Skip to content

capg/nightly: remove check credentials#625

Merged
k8s-ci-robot merged 1 commit intokubernetes-sigs:masterfrom
cpanato:update-gcp-night-build
Jun 4, 2021
Merged

capg/nightly: remove check credentials#625
k8s-ci-robot merged 1 commit intokubernetes-sigs:masterfrom
cpanato:update-gcp-night-build

Conversation

@cpanato
Copy link
Copy Markdown
Member

@cpanato cpanato commented Jun 3, 2021

What this PR does / why we need it:

Trying to setup the job to run the image builds for CAPG.

The nightly job: https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/wg-k8s-infra/trusted/image-builder/image-builder-periodics.yaml uses a service account so in theory, we don't need to expected the GOOGLE_APPLICATION_CREDENTIALS will be injected.
Per this document: https://github.com/kubernetes/test-infra/tree/master/workload-identity#pods we can just use it

but not sure if require more changes here or in the job definition. This is a debug PR to check.

I will try to set up something similar using my personal account and see if I can simulate a similar thing.

/assign @dims @ameukam @codenrhoden @spiffxp

Which issue(s) this PR fixes (optional, in fixes #(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged): Fixes #

Additional context
Add any other context for the reviewers

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jun 3, 2021
@dims
Copy link
Copy Markdown
Member

dims commented Jun 3, 2021

/approve
/lgtm
/hold

@k8s-ci-robot k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. lgtm "Looks good to me", indicates that a PR is ready to be merged. labels Jun 3, 2021
@cpanato
Copy link
Copy Markdown
Member Author

cpanato commented Jun 3, 2021

Steps I did to simulate the prow job using this PR change

  • Created a service account in GCP with the permission roles/compute.instanceAdmin.v1
  • Created a VM and attached the service account created in the above step
  • SSH into the VM
  • git clone the image-builder repo
  • run the ci-gce-nightly.sh script

It starts to run but then failed because

 Ask a project owner to grant you the iam.serviceAccountUser role on the service account
  • So added this extra role
  • Run again the script
  • Now it creates the packer VM and it is running

some snippet of the logs

root@test-image:/home/prow/go/src/sigs.k8s.io/image-builder# ./images/capi/scripts/ci-gce-nightly.sh                                                                                                                                                                                                                                                                                                                                [20/421]                        Credentialed Accounts                                                                                                                                                                                                                                                                                                                                                                                               ACTIVE  ACCOUNT                                                                                                                                                                                                                                                                                                                                                                                                                             *       testing-image-builder@cpanato-general.iam.gserviceaccount.com                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   To set the active account, run:                                                                                                                                                                                                                                                                                                                                                                                                                 $ gcloud config set account `ACCOUNT`

groupadd: group 'packer' already exists
hack/ensure-ansible.sh
hack/ensure-packer.sh
packer build -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/config/kubernetes.json"  -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/config/cni.json"  -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/config/containerd.json"  -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/config/ansible-args.json"  -var-file="/home/prow/go
/src/sigs.k8s.io/image-builder/images/capi/packer/config/goss-args.json"  -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/config/common.json"  -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/config/additional_components.json"  -force -color=true -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/gce/ubuntu-1804.json" -var-file="/home/prow/go/src/sigs.
k8s.io/image-builder/images/capi/packer/gce/ci/nightly/overwrite-1-18.json"  packer/gce/packer.json
ubuntu-1804: output will be in this color.

==> ubuntu-1804: Checking image does not exist...
==> ubuntu-1804: Creating temporary rsa SSH key for instance...
==> ubuntu-1804: Using image: ubuntu-1804-bionic-v20210514
==> ubuntu-1804: Creating instance...
    ubuntu-1804: Loading zone: us-central1-a
    ubuntu-1804: Loading machine type: n1-standard-1
    ubuntu-1804: Requesting instance creation...
    ubuntu-1804: Waiting for creation operation to complete...
    ubuntu-1804: Instance has been created!
==> ubuntu-1804: Waiting for the instance to become running...
    ubuntu-1804: IP: 35.193.44.47
==> ubuntu-1804: Using ssh communicator to connect: 35.193.44.47
==> ubuntu-1804: Waiting for SSH to become available...
==> ubuntu-1804: Connected to SSH!
==> ubuntu-1804: Provisioning with shell script: /tmp/packer-shell474386350
==> ubuntu-1804: Provisioning with Ansible...
    ubuntu-1804: Setting up proxy adapter for Ansible....
 ....

    ubuntu-1804:
    ubuntu-1804: PLAY [all] *********************************************************************
    ubuntu-1804:
    ubuntu-1804: TASK [Gathering Facts] *********************************************************
    ubuntu-1804: ok: [default]
    ubuntu-1804:
    ubuntu-1804: TASK [setup : Put templated sources.list in place] *****************************
    ubuntu-1804: changed: [default]
    ubuntu-1804:
    ubuntu-1804: TASK [setup : Put templated apt.conf.d/90proxy in place when defined] **********
    ubuntu-1804: changed: [default]
    ubuntu-1804:
    ubuntu-1804: TASK [setup : perform a dist-upgrade] ******************************************
    ubuntu-1804: changed: [default]
....

    ubuntu-1804:
    ubuntu-1804: TASK [sysprep : Truncate shell history] ****************************************
    ubuntu-1804: ok: [default] => (item={'path': '/root/.bash_history'})
    ubuntu-1804: ok: [default] => (item={'path': '/home/ubuntu/.bash_history'})
    ubuntu-1804:
    ubuntu-1804: PLAY RECAP *********************************************************************
    ubuntu-1804: default                    : ok=78   changed=59   unreachable=0    failed=0    skipped=93   rescued=0    ignored=0
    ubuntu-1804:
==> ubuntu-1804: Deleting instance...
    ubuntu-1804: Instance has been deleted!
==> ubuntu-1804: Deleting previous image...
==> ubuntu-1804: Creating image...
==> ubuntu-1804: Deleting disk...
    ubuntu-1804: Disk has been deleted!
Build 'ubuntu-1804' finished after 7 minutes 47 seconds.

==> Wait completed after 7 minutes 47 seconds

==> Builds finished. The artifacts of successful builds are:
--> ubuntu-1804: A disk image was created: cluster-api-ubuntu-1804-v1-18-18-nightly

We need this PR to get in + a new role for the service account in k/k8s.io which I can add next

here the screenshot of one image created

Screenshot 2021-06-03 at 13 47 26

cc @ameukam @dims @spiffxp

@cpanato
Copy link
Copy Markdown
Member Author

cpanato commented Jun 3, 2021

/test pull-azure-vhds

@cpanato
Copy link
Copy Markdown
Member Author

cpanato commented Jun 3, 2021

updated screenshot both three images

Screenshot 2021-06-03 at 14 09 52

@cpanato
Copy link
Copy Markdown
Member Author

cpanato commented Jun 3, 2021

/retest

1 similar comment
@cpanato
Copy link
Copy Markdown
Member Author

cpanato commented Jun 3, 2021

/retest

@codenrhoden
Copy link
Copy Markdown
Contributor

/retest
The merge of #624 should take care of the Azure flakes

@cpanato
Copy link
Copy Markdown
Member Author

cpanato commented Jun 4, 2021

all green now @codenrhoden

/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 4, 2021
@codenrhoden
Copy link
Copy Markdown
Contributor

/approve

@k8s-ci-robot
Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: codenrhoden, cpanato, dims

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 4, 2021
@k8s-ci-robot k8s-ci-robot merged commit b1d9fd8 into kubernetes-sigs:master Jun 4, 2021
@ameukam
Copy link
Copy Markdown
Member

ameukam commented Jun 4, 2021

@cpanato Images are successfully built:

gcloud compute images list --filter=\"family=capi-ubuntu-1804-k8s-\" --filter="name~^cluster-api" --format="value(name)" --project=k8s-staging-cluster-api-gcp
cluster-api-ubuntu-1804-v1-18-18-nightly
cluster-api-ubuntu-1804-v1-19-10-nightly
cluster-api-ubuntu-1804-v1-20-6-nightly

@dims
Copy link
Copy Markdown
Member

dims commented Jun 4, 2021

Nice @cpanato @ameukam !!!! this is a big win 👍🏾

@spiffxp
Copy link
Copy Markdown
Contributor

spiffxp commented Jun 5, 2021

I have a security concern with the way the k8s-infra role was assigned, ref: kubernetes/k8s.io#2124 (comment)

Which specific service account needs to be used? We should be binding the role against that. Not the whole project

@cpanato cpanato deleted the update-gcp-night-build branch June 7, 2021 11:09
@cpanato
Copy link
Copy Markdown
Member Author

cpanato commented Jun 7, 2021

lets address the issues in this discussion here: kubernetes/k8s.io#2124 (comment) and get that fixed.

there are other things I would like to do after this, like make the images public, so we can use in the CAPG jobs and avoid to recreate that everytime

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants