runtimes: enable log export feature

[harshanarayana]

What

This feature adds an ability to export Cluster logs using the k3d binaries.

Why

Tools like kind has been supporting the ability to export logs from the cluster using their native tooling for a while. This ability to export the logs using k3d can be super helpful when doing local development and test experiments. Instead of having to individually collect the logs, this command can be one stop shop for all standard commands to be aggregated.

Implications

This changes the Runtime interface and hence this is a backward incompatible change.

If this is an acceptable change to be included, I will be more than happy to take care of including a few more things to the supported export of logs.

❯ bin/k3d debug export-logs export-log-test
❯ tree debug-logs-export-log-test/
debug-logs-export-log-test/
├── k3d-export-log-test-server-0
│   ├── cluster-info.log
│   ├── containerd.log
│   ├── crictl-images.log
│   ├── crictl-ps.log
│   ├── kubectl-get-node.yaml
│   ├── kubectl-version.log
│   ├── log
│   │   ├── containers
│   │   │   ├── coredns-7448499f4d-62f57_kube-system_coredns-b66d5470354351c562ee3f3f2edbff20d117e294549e0e92836a81b61ab71978.log
│   │   │   ├── helm-install-traefik-44sht_kube-system_helm-14c069fc9a9bb06878083cbff48a68138852eaa283d605b1d2d51dad3ef94814.log
│   │   │   ├── helm-install-traefik-crd-fh29n_kube-system_helm-63fbaed67883b443593f803fea47c2d60fe42311ebfd7fbe2f9a2a3dd4ec4b91.log
│   │   │   ├── local-path-provisioner-5ff76fc89d-m5wz6_kube-system_local-path-provisioner-ebde80834fa702282e3c0f1ee871172a3690e27c55c39c884b1e23cb09b94d15.log
│   │   │   ├── metrics-server-86cbb8457f-lg7tq_kube-system_metrics-server-b16a150d61faa07a257db24758b95807ec131f08d4006eb1dbe7970e1145020c.log
│   │   │   ├── svclb-traefik-hst27_kube-system_lb-port-443-bdb21890b8d32bcffb1cfacd12a157ea0d73cde672dbea703279166dafa49ebf.log
│   │   │   ├── svclb-traefik-hst27_kube-system_lb-port-80-6d1cfb81abf4a5573998d49a00621a03d9fe05c7f18033bf59f4e412905a8be6.log
│   │   │   └── traefik-6b84f7cbc-l574b_kube-system_traefik-bd236ceb3ef60e87e3495afc039ad2f2dacc628c1f9c056d11dd9a137ac7642a.log
│   │   ├── k3d-entrypoints_250117074528.log
│   │   └── pods
│   │       ├── kube-system_coredns-7448499f4d-62f57_0a9c8c5a-594f-468a-9c6a-b8b9c8db6875
│   │       │   └── coredns
│   │       │       └── 0.log
│   │       ├── kube-system_helm-install-traefik-44sht_59ef59f1-a663-46f9-9e7b-166bc0091e5b
│   │       │   └── helm
│   │       │       └── 2.log
│   │       ├── kube-system_helm-install-traefik-crd-fh29n_10a82388-d43f-474c-b8c2-867c510fd1cb
│   │       │   └── helm
│   │       │       └── 0.log
│   │       ├── kube-system_local-path-provisioner-5ff76fc89d-m5wz6_bc459eed-fdc3-4c9d-85b6-d26703ac4881
│   │       │   └── local-path-provisioner
│   │       │       └── 0.log
│   │       ├── kube-system_metrics-server-86cbb8457f-lg7tq_01ee3bec-bf2a-405b-9232-96ccd203f3af
│   │       │   └── metrics-server
│   │       │       └── 0.log
│   │       ├── kube-system_svclb-traefik-hst27_ae354af9-c674-49c7-96af-9d17634b90fd
│   │       │   ├── lb-port-443
│   │       │   │   └── 0.log
│   │       │   └── lb-port-80
│   │       │       └── 0.log
│   │       └── kube-system_traefik-6b84f7cbc-l574b_1aa62a25-4d5f-439c-9465-09dfb1a825f9
│   │           └── traefik
│   │               └── 0.log
│   └── ps-aef.log
├── k3d-export-log-test-server-0.log
├── k3d-export-log-test-serverlb
└── k3d-export-log-test-serverlb.log

21 directories, 26 files

❯ bin/k3d debug export-logs export-log-test --node k3d-export-log-test-server-0
❯ tree debug-logs-export-log-test/
debug-logs-export-log-test/
├── k3d-export-log-test-server-0
│   ├── cluster-info.log
│   ├── containerd.log
│   ├── crictl-images.log
│   ├── crictl-ps.log
│   ├── kubectl-get-node.yaml
│   ├── kubectl-version.log
│   ├── log
│   │   ├── containers
│   │   │   ├── coredns-7448499f4d-62f57_kube-system_coredns-b66d5470354351c562ee3f3f2edbff20d117e294549e0e92836a81b61ab71978.log
│   │   │   ├── helm-install-traefik-44sht_kube-system_helm-14c069fc9a9bb06878083cbff48a68138852eaa283d605b1d2d51dad3ef94814.log
│   │   │   ├── helm-install-traefik-crd-fh29n_kube-system_helm-63fbaed67883b443593f803fea47c2d60fe42311ebfd7fbe2f9a2a3dd4ec4b91.log
│   │   │   ├── local-path-provisioner-5ff76fc89d-m5wz6_kube-system_local-path-provisioner-ebde80834fa702282e3c0f1ee871172a3690e27c55c39c884b1e23cb09b94d15.log
│   │   │   ├── metrics-server-86cbb8457f-lg7tq_kube-system_metrics-server-b16a150d61faa07a257db24758b95807ec131f08d4006eb1dbe7970e1145020c.log
│   │   │   ├── svclb-traefik-hst27_kube-system_lb-port-443-bdb21890b8d32bcffb1cfacd12a157ea0d73cde672dbea703279166dafa49ebf.log
│   │   │   ├── svclb-traefik-hst27_kube-system_lb-port-80-6d1cfb81abf4a5573998d49a00621a03d9fe05c7f18033bf59f4e412905a8be6.log
│   │   │   └── traefik-6b84f7cbc-l574b_kube-system_traefik-bd236ceb3ef60e87e3495afc039ad2f2dacc628c1f9c056d11dd9a137ac7642a.log
│   │   ├── k3d-entrypoints_250117074528.log
│   │   └── pods
│   │       ├── kube-system_coredns-7448499f4d-62f57_0a9c8c5a-594f-468a-9c6a-b8b9c8db6875
│   │       │   └── coredns
│   │       │       └── 0.log
│   │       ├── kube-system_helm-install-traefik-44sht_59ef59f1-a663-46f9-9e7b-166bc0091e5b
│   │       │   └── helm
│   │       │       └── 2.log
│   │       ├── kube-system_helm-install-traefik-crd-fh29n_10a82388-d43f-474c-b8c2-867c510fd1cb
│   │       │   └── helm
│   │       │       └── 0.log
│   │       ├── kube-system_local-path-provisioner-5ff76fc89d-m5wz6_bc459eed-fdc3-4c9d-85b6-d26703ac4881
│   │       │   └── local-path-provisioner
│   │       │       └── 0.log
│   │       ├── kube-system_metrics-server-86cbb8457f-lg7tq_01ee3bec-bf2a-405b-9232-96ccd203f3af
│   │       │   └── metrics-server
│   │       │       └── 0.log
│   │       ├── kube-system_svclb-traefik-hst27_ae354af9-c674-49c7-96af-9d17634b90fd
│   │       │   ├── lb-port-443
│   │       │   │   └── 0.log
│   │       │   └── lb-port-80
│   │       │       └── 0.log
│   │       └── kube-system_traefik-6b84f7cbc-l574b_1aa62a25-4d5f-439c-9465-09dfb1a825f9
│   │           └── traefik
│   │               └── 0.log
│   └── ps-aef.log
└── k3d-export-log-test-server-0.log

20 directories, 25 files

[harshanarayana]

@iwilltry42 PTAL when you can

[harshanarayana]

v1
├── k3d-test-server-0
│   ├── cluster-info.log
│   ├── containerd.log
│   ├── crictl-images.log
│   ├── crictl-ps.log
│   ├── kubectl-get-node.yaml
│   ├── kubectl-version.log
│   ├── log
│   │   ├── containers
│   │   │   ├── coredns-7448499f4d-8dhmx_kube-system_coredns-6dd51f354f8f27ff014ef38849e3ffcfaff1508c7042cd1425f4c7730a6eaaa7.log
│   │   │   ├── helm-install-traefik-crd-gj8kc_kube-system_helm-d769ed944dda3c7e9b130aa01da5094c093dd1e324d2446a4f10445d5f1c8542.log
│   │   │   ├── helm-install-traefik-h964r_kube-system_helm-8d911021b0aae5e12b0273185512a66342308fd73eee32562c213675afa34c13.log
│   │   │   ├── local-path-provisioner-5ff76fc89d-bjf4n_kube-system_local-path-provisioner-0b8f980454508a9f9bd2b9480b94736aa8c5fc0edf7652e9649cfa8b4ad36014.log
│   │   │   ├── metrics-server-86cbb8457f-zwrn7_kube-system_metrics-server-6c05dcf0fa7336f67ac8f2efe2c25f2dc97ee62a0cf3632fe70fbc05264c46a2.log
│   │   │   ├── svclb-traefik-rxrwz_kube-system_lb-port-443-4293306ac37d02a902620275e186abf243a0f84d948224d600dfc5e3626bbf6f.log
│   │   │   ├── svclb-traefik-rxrwz_kube-system_lb-port-80-dbec7113890731d8963a83205812fefbd53fdd907ab3240fa3733f39aaaab574.log
│   │   │   └── traefik-6b84f7cbc-hh6mh_kube-system_traefik-13e5435613b7e5f3c7026a403ae79128fea4c6ceb38a0305e3e9718277ca8a50.log
│   │   ├── k3d-entrypoints_240717122814.log
│   │   └── pods
│   │       ├── kube-system_coredns-7448499f4d-8dhmx_7078afe2-1e8f-443b-a4eb-d2fee1cd6956
│   │       │   └── coredns
│   │       │       └── 0.log
│   │       ├── kube-system_helm-install-traefik-crd-gj8kc_fb93c486-4511-4301-b53f-6794d447eea4
│   │       │   └── helm
│   │       │       └── 0.log
│   │       ├── kube-system_helm-install-traefik-h964r_8562b2fb-7182-462c-b44e-4ee578d8104f
│   │       │   └── helm
│   │       │       └── 0.log
│   │       ├── kube-system_local-path-provisioner-5ff76fc89d-bjf4n_00d2a204-7623-4595-a136-33038c3742a0
│   │       │   └── local-path-provisioner
│   │       │       └── 0.log
│   │       ├── kube-system_metrics-server-86cbb8457f-zwrn7_0be65e2c-bb1f-423f-9b6c-da41927a648a
│   │       │   └── metrics-server
│   │       │       └── 0.log
│   │       ├── kube-system_svclb-traefik-rxrwz_567fab59-c009-4639-83f4-c6e81f75a309
│   │       │   ├── lb-port-443
│   │       │   │   └── 0.log
│   │       │   └── lb-port-80
│   │       │       └── 0.log
│   │       └── kube-system_traefik-6b84f7cbc-hh6mh_9e65392d-4d3a-4290-aa28-7d1e3bab87b4
│   │           └── traefik
│   │               └── 0.log
│   └── ps-aef.log
├── k3d-test-server-0.log
├── k3d-test-serverlb
└── k3d-test-serverlb.log

It will generate a struct like this.

[harshanarayana]

@iwilltry42 I was hoping if you could spare some time and take a look at this and see if this is acceptable addition to the tool.

This can be really useful with kubernetes-sigs/e2e-framework#441

[iwilltry42]

Hi @harshanarayana , sorry for letting you wait this long.
This is pretty nice and the implementation looks good, thank you!
I'm just thinking that we may want to put this below a debug subcommand, so we don't pollute the "main CLI", i.e. k3d debug export-logs. WDYT?

Also, in the future it might be cool to add filters to choose which logs should/shouldn't be exported.

[harshanarayana]

@iwilltry42 Thanks for taking a look and getting back to me. And no worries about the delay in response. I can understand.

I have moved the export sub command to the debug chain and now k3d debug export-logs is how you can use this tooling.

I have added a provision to pass the components as an arg in the Interface definition. But I will keep the implementation for that in a different PR. I would appreciate if you can share a few example of what a filter might look like. I will be happy to open a PR with the changes required for the same.

PTAL at the current state and let me know if we can work towards getting this merged.

[iwilltry42]

I'm fine with how it is now. We can always add filters in the future as needed. I was thinking of stripping down the export e.g. to just k3s logs (docker container logs), K3s component logs (containerd et al) or kube component logs (kube-system pods).

Thanks for your efforts!

[harshanarayana]

I'm fine with how it is now. We can always add filters in the future as needed. I was thinking of stripping down the export e.g. to just k3s logs (docker container logs), K3s component logs (containerd et al) or kube component logs (kube-system pods).

Thanks for your efforts!

Understood. Let me get a Draft PR open with some of these changes and we can take this discussion there.

Thanks for the merge @iwilltry42