Log output for container in 'ct install' does not terminate

[pathob]

Is this a request for help?: No

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Report

Version of Helm and Kubernetes:

$ helm version
version.BuildInfo{Version:"v3.5.2", GitCommit:"167aac70832d3a384f65f9745335e9fb40169dc2", GitTreeState:"dirty", GoVersion:"go1.15.7"}
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"archive", BuildDate:"2020-07-28T00:00:00Z", GoVersion:"go1.15rc1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.17", GitCommit:"f00bd5fc5d1b9001e612847aa77f4e9c66758075", GitTreeState:"clean", BuildDate:"2021-03-22T23:03:44Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
$ ct version
Version:	 v3.3.1
Git commit:	 71d0e1e82c5c8b66ce4d9704426dc13b7075829d
Date:		 2020-11-30T20:10:04Z
License:	 Apache 2.0

What happened:

When running ct install ... the installation and tests do not terminate. The execution of ct gets stuck after:

<== Logs of init container crowd-uinddlbjpw-0
------------------------------------------------------------------------------------------------------------------------
>>> kubectl get pods crowd-uinddlbjpw-0 --no-headers --namespace crowd-uinddlbjpw --output jsonpath={.spec.containers[*].name}
------------------------------------------------------------------------------------------------------------------------
==> Logs of container crowd-uinddlbjpw-0
------------------------------------------------------------------------------------------------------------------------
>>> kubectl logs crowd-uinddlbjpw-0 --namespace crowd-uinddlbjpw --container crowd

The pod reaches running state and the displayed log is okay. But it just gets stuck there.

What you expected to happen:

As the installation (there are no actual tests yet) was successful, ct should just terminate successfully too.

If I copy the command for printing the logs from the ct output (kubectl logs crowd-uinddlbjpw-0 --namespace crowd-uinddlbjpw --container crowd) and execute it manually, the log output also terminates as expected.

How to reproduce it (as minimally and precisely as possible):

Unfortunately, the repo is currently private. If required, I can try to build a minimal version of that case.

Anything else we need to know:

I noticed this behavior in automated tests in Kind first. Manual executions in Azure (AKS) show the same behavior.

[pathob]

Problem still exists with chart-testing v3.4.0.

[ctron]

I am running into the same issue. With helm/[email protected] and v3.4.0 of chart testing.

[ctron]

Is there any workaround for this issue?

[pathob]

Not that I know and this project here seems quite abandoned... Does not even feels like worth the effort to debug the chart-testing code if PR's are not accepted anymore...

[ctron]

Not that I know and this project here seems quite abandoned... Does not even feels like worth the effort to debug the chart-testing code if PR's are not accepted anymore...

I noticed that too. Can you recommend any alternative?

[pathob]

Unfortunately no...

[smlx]

If required, I can try to build a minimal version of that case.

Please do this as I cannot reproduce this issue.

[ctron]

I happens in most cases when I run this CI job: https://github.com/eclipse/packages/actions/runs/1328443622 … not sure if you can see the logs. I can export them for you if that helps. Or tweak some settings to get different output.

[smlx]

Sorry, I don't see any problems in that CI job. Could you please explain clearly what you think is broken about the ct install behaviour in the CI job that you linked?

[ctron]

Sure. There are 4 failed matrix jobs in the last step of the overall pipeline. There is a Helm install timeout of 30m. All jobs currently are expected to fail.

One of the jobs properly aborted after ~39 minutes (30min+) collecting the logs, but exiting the job. The other 3 face the same issue. Terminate the test, but fail to exit the process. So the overall CI job got manually cancelled.

[pathob]

@smlx I will try to build another example that demonstrates the issue.

[smlx]

Ah I see. Probably this is because the kubectl command does not have a --request-timeout argument. When the kind cluster is torn down the request hangs, because by default kubectl does not set a request timeout.

I'll send a PR shortly.

[pathob]

I just had another look into the issue. The issue for me happens when a very long line from the kubectl log should be printed. The line generated by the Docker image in my case is almost 80000 characters long. It looks like ct cannot handle that while kubectl does not have any issue with that.