Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

api: make pluto and moondog IMDSv2 compliant #705

Merged
merged 2 commits into from
Feb 4, 2020
Merged

api: make pluto and moondog IMDSv2 compliant #705

merged 2 commits into from
Feb 4, 2020

Conversation

etungsten
Copy link
Contributor

@etungsten etungsten commented Feb 3, 2020 •
edited
Loading

Issue #, if available: Partially addresses #685

Description of changes:

  • moondog now fetches an IMDS session token that's valid for 60 seconds before retrieving data from IMDS
  • pluto now fetches an IMDS session token that's valid for 60 seconds before querying IMDS for dynamic settings.

Testing:
Built new Thar image with changes.

Launched instance without requiring http token (IMDSv1 setup).
moondog runs fine, sundog runs fine, kubelet starts up fine, instance connects to my cluster fine.

bash-5.0# systemctl status moondog
● moondog.service - Thar userdata configuration system
     Loaded: loaded (/x86_64-thar-linux-gnu/sys-root/usr/lib/systemd/system/moondog.service; enabled; vendor preset: enabled)
     Active: active (exited) since Mon 2020-02-03 20:05:44 UTC; 15min ago
   Main PID: 2343 (code=exited, status=0/SUCCESS)
      Tasks: 0 (limit: 4430)
     Memory: 0B
     CGroup: /system.slice/moondog.service

Feb 03 20:05:43 ip-192-168-52-249.us-west-2.compute.internal systemd[1]: Starting Thar userdata configuration system...
Feb 03 20:05:44 ip-192-168-52-249.us-west-2.compute.internal systemd[1]: Started Thar userdata configuration system.
bash-5.0# systemctl status sundog
● sundog.service - User-specified setting generators
     Loaded: loaded (/x86_64-thar-linux-gnu/sys-root/usr/lib/systemd/system/sundog.service; enabled; vendor preset: enabled)
     Active: active (exited) since Mon 2020-02-03 20:05:44 UTC; 16min ago
   Main PID: 2433 (code=exited, status=0/SUCCESS)
      Tasks: 0 (limit: 4430)
     Memory: 0B
     CGroup: /system.slice/sundog.service

Feb 03 20:05:44 ip-192-168-52-249.us-west-2.compute.internal systemd[1]: Starting User-specified setting generators...
Feb 03 20:05:44 ip-192-168-52-249.us-west-2.compute.internal systemd[1]: Started User-specified setting generators.
bash-5.0# systemctl status kubelet
● kubelet.service - Kubelet
     Loaded: loaded (/x86_64-thar-linux-gnu/sys-root/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2020-02-03 20:05:46 UTC; 17min ago
       Docs: https://github.com/kubernetes/kubernetes
    Process: 2720 ExecStartPre=/sbin/iptables -P FORWARD ACCEPT (code=exited, status=0/SUCCESS)
    Process: 2764 ExecStartPre=/usr/bin/host-ctr -source ${POD_INFRA_CONTAINER_IMAGE} -pull-image-only -containerd-socket /run/containerd/containerd.sock -nam
espace k8s.io (code=exited, status=0/SUCCESS)
   Main PID: 2855 (kubelet)
      Tasks: 16 (limit: 4430)
     Memory: 255.0M
        CPU: 1min 15.116s
     CGroup: /system.slice/kubelet.service
             └─2855 /usr/bin/kubelet --cloud-provider aws --config /etc/kubernetes/kubelet/config --kubeconfig /etc/kubernetes/kubelet/kubeconfig --container-ru
....

Now with http tokens required (enforced IMDSv2).
moondog runs fine. sundog runs fine.

bash-5.0# systemctl status sundog
● sundog.service - User-specified setting generators
     Loaded: loaded (/x86_64-thar-linux-gnu/sys-root/usr/lib/systemd/system/sundog.service; enabled; vendor preset: enabled)
     Active: active (exited) since Sat 2020-02-01 00:04:36 UTC; 2 days ago
   Main PID: 2342 (code=exited, status=0/SUCCESS)
      Tasks: 0 (limit: 4430)
     Memory: 0B
     CGroup: /system.slice/sundog.service

Feb 01 00:04:36 ip-192-168-43-151.us-west-2.compute.internal systemd[1]: Starting User-specified setting generators...
Feb 01 00:04:36 ip-192-168-43-151.us-west-2.compute.internal systemd[1]: Started User-specified setting generators.
bash-5.0# systemctl status moondog
● moondog.service - Thar userdata configuration system
     Loaded: loaded (/x86_64-thar-linux-gnu/sys-root/usr/lib/systemd/system/moondog.service; enabled; vendor preset: enabled)
     Active: active (exited) since Sat 2020-02-01 00:04:36 UTC; 2 days ago
   Main PID: 2283 (code=exited, status=0/SUCCESS)
      Tasks: 0 (limit: 4430)
     Memory: 0B
     CGroup: /system.slice/moondog.service

Feb 01 00:04:36 ip-192-168-43-151.us-west-2.compute.internal systemd[1]: Starting Thar userdata configuration system...
Feb 01 00:04:36 ip-192-168-43-151.us-west-2.compute.internal systemd[1]: Started Thar userdata configuration system.

kubelet is having trouble starting the cni-plugin because the cni image doesn't support IMDSv2 yet. This will be addressed outside of this PR. Other than that kubelet has the right configuration from settings generated by sundog and moondog

bash-5.0# systemctl status kubelet
● kubelet.service - Kubelet
     Loaded: loaded (/x86_64-thar-linux-gnu/sys-root/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2020-02-01 00:04:39 UTC; 2 days ago
       Docs: https://github.com/kubernetes/kubernetes
    Process: 2648 ExecStartPre=/sbin/iptables -P FORWARD ACCEPT (code=exited, status=0/SUCCESS)
    Process: 2733 ExecStartPre=/usr/bin/host-ctr -source ${POD_INFRA_CONTAINER_IMAGE} -pull-image-only -containerd-socket /run/containerd/containerd.sock -nam
espace k8s.io (code=exited, status=0/SUCCESS)
   Main PID: 2835 (kubelet)
      Tasks: 15 (limit: 4430)
     Memory: 273.8M
        CPU: 42min 8.291s
     CGroup: /system.slice/kubelet.service
             └─2835 /usr/bin/kubelet --cloud-provider aws --config /etc/kubernetes/kubelet/config --kubeconfig /etc/kubernetes/kubelet/kubeconfig --container-ru
ntime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --network-plugin cni --root-dir /var/lib/kubelet --cert-dir /var/lib/kubelet/
pki --volume-plugin-dir /var/lib/kubelet/plugins/volume/exec --node-ip 192.168.43.151 --node-labels  --register-with-taints  --pod-infra-container-image 60240
1143452.dkr.ecr.us-west-2.amazonaws.com/eks/pause-amd64:3.1

Feb 03 20:28:12 ip-192-168-43-151.us-west-2.compute.internal kubelet[2835]: E0203 20:28:12.201922    2835 kubelet.go:2172] Container runtime network not ready
: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Feb 03 20:28:13 ip-192-168-43-151.us-west-2.compute.internal kubelet[2835]: E0203 20:28:13.801289    2835 pod_workers.go:190] Error syncing pod 716b5165-4486-
11ea-a850-02fe0b2a3c5c ("aws-node-wfnw5_kube-system(716b5165-4486-11ea-a850-02fe0b2a3c5c)"), skipping: failed to "StartContainer" for "aws-node" with CrashLoo
pBackOff: "Back-off 5m0s restarting failed container=aws-node pod=aws-node-wfnw5_kube-system(716b5165-4486-11ea-a850-02fe0b2a3c5c)"
Feb 03 20:28:17 ip-192-168-43-151.us-west-2.compute.internal kubelet[2835]: E0203 20:28:17.202824    2835 kubelet.go:2172] Container runtime network not ready
: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

tjkirch
tjkirch reviewed Feb 3, 2020
View reviewed changes
workspaces/api/pluto/src/main.rs Outdated Show resolved Hide resolved
workspaces/api/moondog/src/main.rs Show resolved Hide resolved
workspaces/api/pluto/src/main.rs Outdated Show resolved Hide resolved
workspaces/api/moondog/src/main.rs Outdated Show resolved Hide resolved
workspaces/api/moondog/src/main.rs Outdated Show resolved Hide resolved
@iliana iliana mentioned this pull request Feb 3, 2020
Closed
10 tasks
@etungsten
Copy link
Contributor Author

Addresses @tjkirch 's comments.

Building a new image to test.

tjkirch
tjkirch reviewed Feb 3, 2020
View reviewed changes
workspaces/api/moondog/src/main.rs Outdated Show resolved Hide resolved
workspaces/api/pluto/src/main.rs Outdated Show resolved Hide resolved
workspaces/api/pluto/src/main.rs Show resolved Hide resolved
workspaces/api/moondog/src/main.rs Show resolved Hide resolved
@etungsten
moondog: IMDSv2 compliance
586c804 
moondog now retrieves a imds session token before requesting data from
IMDS.
@etungsten
pluto: use IMDSv2 to retrieve instance metadata
3c11c39 
Pluto now creates a imds sessions token and uses it to retrieve instance
metadata
@etungsten
Copy link
Contributor Author

Addresses @tjkirch 's comments.

zmrow
zmrow approved these changes Feb 4, 2020
View reviewed changes
Copy link
Contributor

@zmrow zmrow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sweet - thanks for doing this!

🐎

@etungsten etungsten merged commit 94e9aed into develop Feb 4, 2020
@etungsten etungsten deleted the api-imdsv2 branch February 4, 2020 17:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zmrow zmrow zmrow approved these changes

@tjkirch tjkirch tjkirch approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@etungsten @tjkirch @zmrow