Migrate slack-infra services to aaa cluster

[spiffxp]

• origin gcp-project: kubernetes-tools
• origin cluster-name: tools-cluster
• apps:
  • slackin (slack.k8s.io)
  • slack-event-log
  • slack-moderator
  • slack-welcomer
• repo: https://github.com/kubernetes-sigs/slack-infra

/wg k8s-infra
/area cluster-infra
/assign @bartsmykla @ameukam

[spiffxp]

/sig contributor-experience

[nikhita]

fyi @Katharine @BenTheElder @jeefy @mrbobbytables

[BenTheElder]

there's also the tempellis image hosting, we should move that to a staging project probably.

[bartsmykla]

My step by step plan to move slack infra to the aaa cluster:

State	Step	WIP	Done	Blocked by	Ref	Details
🟢	Update deploy instructions		#784			Creating PR with updated instructions about how to deploy the slack-infra and two missing resource manifests (ingress and certificate)
🟢	Create static IP address		#793			Address: slack-infra-ingress-prod / 34.107.195.71
🔴	Deploy ingress resource to receive efemeral IP address				#793 (comment)	Done, but the temporary address assigned was not used as the assigned name was impossible to change
🟢	Update the DNSes with temporary subdomain		#793, #795, #796, #797			Updating DNSes with temporary subdomain (slack-staging.k8s.io) pointing to the address from the step above
🟢	Update the Ingress with static IP annotation		#784, #793			Updating the ingress resource created in step no. 1 with annotation: kubernetes.io/ingress.global-static-ip-name: slack-infra-ingress-prod
🟢	Deploy secrets		#issuecomment-618052639			Asking one of the people who have access to encrypted secrets (@dims, @spiffxp, @thockin) to deploy them to the slack-infra namespace¹
🟢	Figure out if new services can coexist with the old one		Confirmation from @Katharine at slack			Figuring out if deploying slack-event-log, slack-moderator and slack-welcomer without first turning off the already existing instances is possible
🟢	Deploy the rest of resources					²
🟢	Update the real subdomains		#814, #815, #817			If everything will work fine, updating the DNSes and certificate with proper subdomains (slack.k8s.io and slack.kubernetes.io) pointing to the new ingress (slack-infra-ingress-prod IP address)
🟢	Remove temporary dns records		#816, #818			After confirmation everything works as expected removing dns records for subdomains: slack-staging.k8s.io and slack-staging.kubernetes.io
⚪️	Get consensus if we can remove duplicated manifests					Getting consensus if we can remove resource manifests (listed below) from our repository and update deploy instructions to use the ones existing at k-sigs/slack-infra/cluster³
⚪️	If agreed, remove duplicated recources
🟢	Add missing bad-domains ConfigMap		#784		slack discussion, #751 (comment)	Add missing bad-domains ConfigMap manifest to our repository
🟠	Create slack-tools staging container registry	#802			k/k8s.io/k8s.gcr.io#creating-staging-repos
⚪️	Remove ingress and certificate manifests from the slack infra repo
⚪️	Enable automate builds for slack-event-logs				k/k8s.io/k8s.gcr.io#enabling-automatic-builds
⚪️	Enable automate builds for slack-moderator				k/k8s.io/k8s.gcr.io#enabling-automatic-builds
⚪️	Enable automate builds for slack-welcomer				k/k8s.io/k8s.gcr.io#enabling-automatic-builds
⚪️	Create slackin directory in the slack-infra repo				Where to find slackin container image
⚪️	Enable automate builds for slackin				k/k8s.io/k8s.gcr.io#enabling-automatic-builds
⚪️	Release the container images for all of the slack-tools and put them in repository managed by us	#802
⚪️	Update the slack infra repo with the new images
⚪️	Update our repository with the new images

References

¹

git-crypt unlock
kubectl apply -n slack-infra -f secrets/
git-crypt lock

²

kubectl apply -n slack-infra -f resources/

³

slack-infra/resources/slack-event-log/deployment.yaml
slack-infra//resourcesslack-event-log/service.yaml

slack-infra/resources/slack-moderator/deployment.yaml
slack-infra/resources/slack-moderator/service.yaml

slack-infra/resources/slack-welcomer/deployment.yaml
slack-infra/resources/slack-welcomer/message.yaml
slack-infra/resources/slack-welcomer/service.yaml

slack-infra/resources/slackin/deployment.yaml
slack-infra/resources/slackin/service.yaml

[bartsmykla]

@dims, @spiffxp, @thockin can one of you deploy the secrets (step 6) as it doesn't conflicts with other steps?

[spiffxp]

/assign
taking a look

[spiffxp]

spiffxp@cloudshell:~/k8s.io (kubernetes-public)$ for f in slack-infra/*/*-secret.yaml; do kubectl apply -n slack-infra -f $f; done
secret/slack-event-log-config created
secret/recaptcha created
secret/slackin-token created
secret/slack-moderator-config created
secret/slack-welcomer-config created

spiffxp@cloudshell:~/k8s.io (kubernetes-public)$ kubectl get secrets -n slack-infra
NAME                     TYPE                                  DATA   AGE
default-token-dtsxg      kubernetes.io/service-account-token   3      17h
recaptcha                Opaque                                2      18s
slack-event-log-config   Opaque                                1      19s
slack-moderator-config   Opaque                                1      17s
slack-welcomer-config    Opaque                                1      16s
slackin-token            Opaque                                1      18s

[spiffxp]

/unassign
I would ask #slack-admins about 7

I'm... marginally ok with 11, but I don't want us to get to a point where answering "what's the source of what is deployed in our cluster" becomes a byzantine task of manually tracking everything down. In an ideal world it's a machine-answerable question.

How can we re-use those externally defined resources + patch in / overlay our own needs? kustomize? kpt?

[ameukam]

Sounds like a good plan @bartsmykla.:)

[ameukam]

@spiffxp can you deploy the ConfigMap bad-domains in slack-infra : https://github.com/kubernetes-sigs/slack-infra/blob/master/cluster/slackin/deployment.yaml#L51-L54 ?

[bartsmykla]

Yes, it looks like we are missing this to run the slackin

[bartsmykla]

So @spiffxp applied the ConfigMap and I just have to update the repository with the missing manfest

[bartsmykla]

When I today checked what's the status of slack-infra I realized we can't use existing images which are hosted at internal-accesible only GCR registry so we have to start deploying them to registries managed by us.

Warning  Failed     16s (x2 over 30s)  kubelet, gke-aaa-pool2-20200316195138785800000-fafa121a-zhw5  Failed to pull image "gcr.io/kubernetes-tools/slackin-kubernetes@sha256:d1a9b02239e690d5cbd78c76475a112aedf279dbd8ef401de1b8394f817a23b3": rpc error: code = Unknown desc = Error response from daemon: pull access denied for gcr.io/kubernetes-tools/slackin-kubernetes, repository does not exist or may require 'docker login'

ref. #802

[bartsmykla]

As suggested and discussed at slack, we proceed with @munnerz with DNS update (ref. #814, #815 and #817), all four tools are right now working at the aaa cluster.

We have tested inviting the new members via slack.k8s.io (slackin), receiving welcome messages (slack-welcomer) and reporting messages (slack-moderator). We are also in the process od checking if slack-event-log works as expected (I'll need confirmation from @munnerz as access to that channel is restricted to slack admins).

I suggest to leave the old infrastructure still running for the next two weeks to be sure everything is fine, and then we can get rid of another cluster.

I will followup with the PR updarting the certificate.yaml file and one removing the temporary slack-staging.k8s.io subdomain.

[bartsmykla]

Following up PR for removal of slack-staging.k8s.ip dns record and also with update of the certificate.yaml #818

[bartsmykla]

There are two things I would like to still do, even if the whole slack-infra already works in out new infra.

1. Get consensus if we can remove manifests of basic resources (like deplyoments and services) from our repository and instead use the ones from k-sigs/slack-infra/cluster and do it if we'll decide it's a good idea.
2. Currently every image which slack tools are using are still pointing to the registry not managed by us, so I would like to create the staging registry and then configure the automate builds for the images and use them in our cluster

[spiffxp]

/close
slack-infra has been migrated over, there is followup to be done re:images and certs

[k8s-ci-robot]

@spiffxp: Closing this issue.

In response to this:

/close
slack-infra has been migrated over, there is followup to be done re:images and certs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.