[BUG] Spec tests are unable to find local Helm installation.

[svteb]

Describe the bug
According to INSTALL.md and SOURCE_INSTALL.md, helm is an optional dependency that gets installed locally during the build/setup of testsuite. This applies to running the regular workloads, but not for spec tests, which will fail without a global installation.

To Reproduce

1. Remove any global helm installation in /bin/...
2. Go through source installation steps (clone, shards install, crystal build src/cnf-testsuite.cr).
3. ./cnf-testsuite setup
4. Run crystal spec

ubuntu@spec-test:~/testsuite$ crystal spec
current_branch during compile: "main"
current_tag during compile: "main-2024-02-28-020614-04c37af4"
.....F..--: 1: /home/ubuntu/testsuite/tools/helm/linux-amd64/helm: not found
--: 1: /home/ubuntu/testsuite/tools/helm/linux-amd64/helm: not found
--: 1: /home/ubuntu/testsuite/tools/helm/linux-amd64/helm: not found

...

Expected behavior
Spec tests should be capable of finding the local helm installation.

Solution
This is speculation on my part but I think that the path is incorrect for spec tests as upon running find, this is the output:

ubuntu@spec-test:~$ find -name helm
./.config/helm
./.cache/helm
./testsuite/lib/tar/utils/helm
./testsuite/lib/helm
./.cnf-testsuite/tools/helm
./.cnf-testsuite/tools/helm/linux-amd64/helm

where /.cnf-testsuite/tools/helm/linux-amd64/helm leads to an actual helm binary, which I presume is the path that is being used in the main workload (didn't check).

[macaktom]

I can look into this one.

[macaktom]

After some troubleshooting, I found out, that for spec tests variable ENV["CUSTOM_HELM_PATH"] is empty and because of that, it gives you binary path, which uses your current directory (local_helm_full_path function in helm project) instead of helm_local_install_dir, which is declared in helmenv_setup.cr. For workload and platform tests it works as expected, but not for spec tests.

Possible workarounds:

1. export CUSTOM_HELM_PATH variable

or

2. add this import to helmenv_setup.cr in observability_spec.cr. Probably better would be to add it to spec_helper (which is imported by all tests, path will be little bit different)
   require "./../../src/tasks/helmenv_setup.cr"

I also tried to add CUSTOM_HELM_PATH to helm_local_install task, which is used by setup task, but it also didn't work for spec tests.

Any ideas for better solution, or why this happens only for spec tests?

[ptacemic]

I am checking this now

[ptacemic]

can we close it now ?

[horecoli]

Hello, I don't think this is fixed. I found a spec test that fails when Helm is not installed globally. Here are the results:

crystal spec spec/utils/utils_spec.cr:151
current_branch during compile: "main"
current_tag during compile:
✖️ FAILED: Found 0 privileged containers: []
✖️ FAILED: Found 0 privileged containers: []
F

Failures:

1) Utils 'all_cnfs_task_runner' should run a test against all cnfs in the cnfs directory if there is not cnf-config argument passed to it
Failure/Error: (task_response).should eq(["✔️ PASSED: No privileged containers",

Expected: ["✔️ PASSED: No privileged containers", "✖️ FAILED: Found 1 privileged containers: [\"privileged-coredns\"]"]
got: ["✖️ FAILED: Found 0 privileged containers: []", "✖️ FAILED: Found 0 privileged containers: []"]

# spec/utils/utils_spec.cr:186

Finished in 9:47 minutes
1 examples, 1 failures, 0 errors, 0 pending

Failed examples:

crystal spec spec/utils/utils_spec.cr:151 # Utils 'all_cnfs_task_runner' should run a test against all cnfs in the cnfs directory if there is not cnf-config argument passed to it

Spec test failed and did not perform the necessary cleanup at the end. However, when I executed the test within this spec test, it passed without any issues. In the info logs, I observed the following:

INFO - Helm.uninstall stderr: --: 1: /home/ubuntu/test_main/testsuite/tools/helm/linux-amd64/helm: not found

When I installed Helm globally, it started working correctly without any issues.:

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh
helm version
version.BuildInfo{Version:"v3.14.4", GitCommit:"81c902a123462fd4052bc5e9aa9c513c4c8fc142", GitTreeState:"clean", `GoVersion:"go1.21.9"}``

crystal spec spec/utils/utils_spec.cr:151
current_branch during compile: "main"
current_tag during compile:

✔️ PASSED: No privileged containers
✖️ FAILED: Found 1 privileged containers: ["privileged-coredns"]
Successfully cleaned up coredns-1609263557
Successfully cleaned up privileged-coredns
.

Finished in 9:32 minutes
1 examples, 0 failures, 0 errors, 0 pending

[agentpoyo]

No helm errors when running crystal spec:

pair@cnfdev4:~/workspace/drew/testsuite-ptac$ crystal spec
current_branch during compile: "heads/origin/issue_1909"
current_tag during compile: 
........................................Error: no repo named "badrepo" found
.F........F...Error: uninstall: Release not loaded: prometheus: release: not found
....F....