new-cnf-installation: Fully transition to new CNF installation method

Remove old installation method, use the new one instead.
Change specs accordingly.
Remove unnecessary config parameters.
Fully enable multiple deployments installation.

Refs: #2169
Signed-off-by: Konstantin Yarovoy <[email protected]>

.gitignore

@@ -8,6 +8,7 @@
 *.swp
 *.swo
 /cnfs
+/installed_cnf_files
 /results
 /tools/chaos_mesh
 /tools/helm

CNF_TESTSUITE_YML_USAGE.md

@@ -118,22 +118,6 @@ image_registry_fqdns:
 
 Described below: [link](#5G-parameters)
 
-#### Dynamic parameters
-
-Dynamic parameters are not meant to be set up manually. They exist due to limitations of testsuite design.
-
-##### source_cnf_dir
-
-Directory where config is located.
-
-##### destintaion_cnf_dir
-
-Internal directory where CNF installation files are copied, currently in `cnfs` folder
-
-##### installation_method
-
-Installation method of the CNF and additional info according to it.
-
 #### Deployments
 
 Deployments are defined as three arrays, each for different installation method. Each array element represents one deployment, and they are meant to represent a single CNF together (Not implemented yet).

INSTALL.md

@@ -235,9 +235,9 @@ For more details on points, see our [POINTS.md](./POINTS.md) documentation.
 
 #### Cleaning Up
 
-Run the following to cleanup the specific cnf-testsuite test (this is assuming you installed the cnf-testsuite.yml in your present working directory):
+Run the following to uninstall the CNF (this is assuming you installed the cnf-testsuite.yml in your present working directory):
 ```
-cnf-testsuite cnf_cleanup cnf-config=./cnf-testsuite.yml
+cnf-testsuite cnf_cleanup
 ```
 You can also run `cleanall` and cnf-testsuite will attempt to cleanup everything.
 

SOURCE_INSTALL.md

@@ -151,7 +151,7 @@ crystal spec
 
 ### Setup
 
-Now that we have a `cnf-testsuite` binary, we can run `setup` to ensure it has all the pre-requisites needed in order to successfully run tests and setup required cnfs/ directory and other files required for cnf-testsuite.
+Now that we have a `cnf-testsuite` binary, we can run `setup` to ensure it has all the pre-requisites needed in order to successfully run tests and setup required installed_cnf_files/ directory and other files required for cnf-testsuite.
 
 - Run the following to setup cnf-testsuite:
   ```
@@ -255,10 +255,10 @@ For more details on points, see our [POINTS.md](./POINTS.md) documentation.
 
 #### Cleaning Up
 
-Run the following to cleanup the specific cnf-testsuite test:
+Run the following to uninstall the CNF:
 
 ```
-./cnf-testsuite cnf_cleanup cnf-config=./cnf-testsuite.yml
+./cnf-testsuite cnf_cleanup
 ```
 
 You can also run `cleanall` and cnf-testsuite will attempt to cleanup everything.

example-cnfs/coredns/README.md

@@ -35,8 +35,8 @@ Run the all the tests
 
 Check the results file
 
-Cleanup the cnf test setup (including undeployment of CoreDNS)
+Uninstall the CNF (including undeployment of CoreDNS)
 
 ```
-./cnf-testsuite cnf_cleanup cnf-path=example-cnfs/coredns
+./cnf-testsuite cnf_cleanup
 ```

example-cnfs/envoy/README.md

@@ -30,8 +30,8 @@ Run the test suite:
 ./cnf-testsuite all
 ```
 
-Envoy cleanup
+Envoy uninstallation
 
 ```
-./cnf-testsuite cnf_cleanup cnf-path=example-cnfs/envoy
+./cnf-testsuite cnf_cleanup
 ```

example-cnfs/ip-forwarder/README.md

@@ -31,19 +31,10 @@ export KUBECONFIG=$(pwd)/admin.conf ; ./cnf-testsuite cnf_setup cnf-path=example
 ### Testing
 Run the test suite: `export KUBECONFIG=$(pwd)/admin.conf ; ./cnf-testsuite all`
 
-### Automated cleanup
+### Automated uninstallation
 ```
-export KUBECONFIG=$(pwd)/admin.conf ; ./cnf-testsuite cnf_cleanup cnf-path=example-cnfs/ip-forwarder/cnf-testsuite.yml
+export KUBECONFIG=$(pwd)/admin.conf ; ./cnf-testsuite cnf_cleanup
 ```
 
-### Manual installation
-1. Install helm version 3
-1. Make the cnfs/ip-forwarder diretory 
-1. If you are testing the cnf source, clone the source into the cnfs/ip-forwarder directory
-1. Copy the cnf-testsuite.yml into the cnfs/ip-forwarder directory
-1. Deploy the CNF using helm: `helm install cnfs/ip-forwarder/vpp`
-1. Wait for the installation to finish (all pods are ready)
-1. Run the test suite: `export KUBECONFIG=$(pwd)/admin.conf ; ./cnf-testsuite all`
-
 
 

example-cnfs/linkerd2/README.md

@@ -33,8 +33,8 @@ Run the test suite:
 ./cnf-testsuite all
 ```
 
-linkerd cleanup
+linkerd uninstallation
 
 ```
-./cnf-testsuite cnf_cleanup cnf-path=example-cnfs/linkerd2/cnf-testsuite.yml
+./cnf-testsuite cnf_cleanup
 ```

example-cnfs/nsm/README.md

@@ -31,8 +31,8 @@ Run the all the tests
 
 Check the results file
 
-Cleanup the cnf test setup (including undeployment of NSM)
+Uninstall the CNF (including undeployment of NSM)
 
 ```
-./cnf-testsuite cnf_cleanup cnf-config=./example-cnfs/nsm/cnf-testsuite.yml
+./cnf-testsuite cnf_cleanup
 ```

example-cnfs/pantheon-nsm-nat/README.md

@@ -41,9 +41,8 @@ Run the all the tests
 
 Check the results file
 
-Cleanup the cnf test setup (including undeployment of nsm-nat)
+Uninstall the CNF (including undeployment of nsm-nat)
 
 ```
-./cnf-testsuite cnf_cleanup cnf-config=./example-cnfs/pantheon-nsm-nat/cnf-testsuite.yml
-./cnf-testsuite cnf_cleanup cnf-config=./example-cnfs/nsm/cnf-testsuite.yml
+./cnf-testsuite cnf_cleanup
 ```

example-cnfs/pantheon-nsm-nat/nat-cnf/README.md

@@ -31,8 +31,8 @@ crystal src/cnf-testsuite.cr all
 
 Check the results file
 
-Cleanup the cnf test setup (including undeployment of nsm-nat)
+Uninstall the CNF (including undeployment of nsm-nat)
 
 ```
-crystal src/cnf-testsuite.cr cnf_cleanup cnf-config=./example-cnfs/pantheon-nsm-nat/cnf-testsuite.yml
+crystal src/cnf-testsuite.cr cnf_cleanup
 ```

example-cnfs/vpp-3c2n-csp-use-case/README.md

@@ -23,23 +23,14 @@ export KUBECONFIG=$(pwd)/<YourKubeConf> ; ./cnf-testsuite setup
 
 Setup and deploy  service chain
 ```
-export KUBECONFIG=$(pwd)/admin.conf ; ./cnf-testsuite example_cnf_setup example-cnf-path=example-cnfs/vpp-3c2n-csp-use-case/cnf-testsuite.yml
+export KUBECONFIG=$(pwd)/admin.conf ; ./cnf-testsuite cnf_setup example-cnf-path=example-cnfs/vpp-3c2n-csp-use-case/cnf-testsuite.yml
 ```
 
 ### Testing
 Run the test suite: `export KUBECONFIG=$(pwd)/admin.conf ; ./cnf-testsuite all`
 
-### Automated cleanup
+### Automated uninstallation
 ```
-export KUBECONFIG=$(pwd)/admin.conf ; ./cnf-testsuite example_cnf_cleanup example-cnf-path=example-cnfs/vpp-3c2n-csp-use-case/cnf-testsuite.yml
+export KUBECONFIG=$(pwd)/admin.conf ; ./cnf-testsuite cnf_cleanup
 ```
-
-### Manual installation
-1. Install helm version 3
-1. Make the cnfs/vpp-3c2n-csp-use-case diretory 
-1. If you are testing the cnf source, clone the source into the cnfs/vpp-3c2n-csp-use-case directory
-1. Copy the cnf-testsuite.yml into the cnfs/vpp-3c2n-csp-use-case directory
-1. Deploy the service chain using helm: `helm install vpp-3c2n-csp-use-case cnfs/csp`
-1. Wait for the installation to finish (all pods are ready)
-1. Run the test suite: `export KUBECONFIG=$(pwd)/admin.conf ; ./cnf-testsuite all`
 

spec/5g/core_spec.cr

@@ -18,7 +18,7 @@ describe "Core" do
       result = ShellCmd.run_testsuite("smf_upf_heartbeat verbose")
       (/(PASSED).*(Chaos service degradation is less than 50%)/ =~ result[:output]).should_not be_nil
     ensure
-      result = ShellCmd.run_testsuite("cnf_cleanup cnf-config=sample-cnfs/sample_open5gs/cnf-testsuite.yml") 
+      result = ShellCmd.cnf_cleanup() 
       result[:status].success?.should be_true
     end
   end
@@ -29,7 +29,7 @@ describe "Core" do
       result = ShellCmd.run_testsuite("smf_upf_heartbeat verbose baseline_count=300")
       (/(FAILED).*(Chaos service degradation is more than 50%)/ =~ result[:output]).should_not be_nil
     ensure
-      result = ShellCmd.run_testsuite("cnf_cleanup cnf-config=sample-cnfs/sample_open5gs/cnf-testsuite.yml") 
+      result = ShellCmd.cnf_cleanup() 
       result[:status].success?.should be_true
     end
   end
@@ -40,7 +40,7 @@ describe "Core" do
       result = ShellCmd.run_testsuite("suci_enabled verbose")
       (/(PASSED).*(Core uses SUCI 5g authentication)/ =~ result[:output]).should_not be_nil
     ensure
-      result = ShellCmd.run_testsuite("cnf_cleanup cnf-config=sample-cnfs/sample_open5gs/cnf-testsuite.yml") 
+      result = ShellCmd.cnf_cleanup() 
       result[:status].success?.should be_true
     end
   end
@@ -51,7 +51,7 @@ describe "Core" do
       result = ShellCmd.run_testsuite("suci_enabled verbose")
       (/(FAILED).*(Core does not use SUCI 5g authentication)/ =~ result[:output]).should_not be_nil
     ensure
-      result = ShellCmd.run_testsuite("cnf_cleanup cnf-config=sample-cnfs/sample_open5gs_no_auth/cnf-testsuite.yml") 
+      result = ShellCmd.cnf_cleanup() 
       result[:status].success?.should be_true
     end
   end

spec/cluster_setup_spec.cr

@@ -6,8 +6,7 @@ require "cluster_tools"
 
 describe "Cluster Setup" do
   before_each do
-    result = ShellCmd.run_testsuite("cleanup")
-    result[:status].success?.should be_true
+    result = ShellCmd.environment_cleanup()
   end
 
   it "'install_cluster_tools' should give a message if namespace does not exist", tags: ["cluster_setup"]  do

spec/cnf_testsuite_all/cnf_testsuite_spec.cr

@@ -9,8 +9,7 @@ describe CnfTestSuite do
   end
 
   after_all do
-    result = ShellCmd.run_testsuite("samples_cleanup")
-    result[:status].success?.should be_true
+    result = ShellCmd.environment_cleanup()
   end
 
   it "a task should fail with an exit code of 2 when there is an exception", tags: ["security"] do

spec/fixtures/cnf-testsuite-unmapped-keys-and-subkeys.yml

@@ -1,5 +1,5 @@
 ---
-helm_directory: cnfs/coredns/helm_chart/coredns
+helm_directory: installed_cnf_files/coredns/helm_chart/coredns
 release_name: coredns
 helm_repository:
   name: stable 

spec/platform/hardware_and_scheduler_spec.cr

@@ -4,8 +4,7 @@ require "./../../src/tasks/utils/utils.cr"
 
 describe "Platform" do
   before_all do
-    result = ShellCmd.run_testsuite("samples_cleanup")
-    result[:status].success?.should be_true
+    result = ShellCmd.environment_cleanup()
     result = ShellCmd.run_testsuite("setup")
     result[:status].success?.should be_true
   end

spec/platform/platform_spec.cr

@@ -4,19 +4,18 @@ require "../../src/tasks/utils/utils.cr"
 
 describe "Platform" do
   before_all do
-    result = ShellCmd.run_testsuite("samples_cleanup")
-    result[:status].success?.should be_true
+    result = ShellCmd.environment_cleanup()
     result = ShellCmd.run_testsuite("setup")
     result[:status].success?.should be_true
   end
   it "'platform:*' should not error out when no cnf is installed", tags: ["platform"] do
-    result = ShellCmd.run_testsuite("cleanup")
+    result = ShellCmd.environment_cleanup()
     result = ShellCmd.run_testsuite("platform:oci_compliant")
     puts result[:output]
     (/No cnf_testsuite.yml found/ =~ result[:output]).should be_nil
   end
   it "'platform' should not run prerequisites that are prefixed with a ~", tags: ["platform"] do
-    result = ShellCmd.run_testsuite("cleanup")
+    result = ShellCmd.environment_cleanup()
     result = ShellCmd.run_testsuite("platform ~k8s_conformance")
     (/kind=namespace namespace=sonobuoy/ =~ (result[:output] + result[:error])).should be_nil
   end

spec/setup_spec.cr

@@ -9,8 +9,7 @@ require "sam"
 describe "Setup" do
 
   after_each do
-    result = ShellCmd.run_testsuite("cleanup")
-    result[:status].success?.should be_true
+    result = ShellCmd.environment_cleanup()
   end
 
   it "'setup' should completely setup the cnf testsuite environment before installing cnfs", tags: ["setup"]  do
@@ -24,7 +23,7 @@ describe "Setup" do
       result = ShellCmd.cnf_setup("cnf-path=example-cnfs/coredns/cnf-testsuite.yml")
       (/Successfully setup coredns/ =~ result[:output]).should_not be_nil
     ensure
-      result = ShellCmd.run_testsuite("cnf_cleanup cnf-path=example-cnfs/coredns/cnf-testsuite.yml")
+      result = ShellCmd.cnf_cleanup()
       result[:status].success?.should be_true
       (/Successfully cleaned up/ =~ result[:output]).should_not be_nil
     end
@@ -35,7 +34,7 @@ describe "Setup" do
       result = ShellCmd.cnf_setup("cnf-config=example-cnfs/coredns/cnf-testsuite.yml")
       (/Successfully setup coredns/ =~ result[:output]).should_not be_nil
     ensure
-      result = ShellCmd.run_testsuite("cnf_cleanup cnf-config=example-cnfs/coredns/cnf-testsuite.yml")
+      result = ShellCmd.cnf_cleanup()
       result[:status].success?.should be_true
       (/Successfully cleaned up/ =~ result[:output]).should_not be_nil
     end
@@ -48,7 +47,7 @@ describe "Setup" do
       (/Successfully setup coredns/ =~ result[:output]).should_not be_nil
     ensure
 
-      result = ShellCmd.run_testsuite("cnf_cleanup cnf-path=spec/fixtures/cnf-testsuite.yml verbose")
+      result = ShellCmd.cnf_cleanup("verbose")
       result[:status].success?.should be_true
       (/Successfully cleaned up/ =~ result[:output]).should_not be_nil
     end
@@ -59,19 +58,19 @@ describe "Setup" do
       result = ShellCmd.cnf_setup("cnf-path=sample-cnfs/multi_helm_directories/cnf-testsuite.yml")
       (/Successfully setup coredns/ =~ result[:output]).should_not be_nil
     ensure
-      result = ShellCmd.run_testsuite("cnf_cleanup cnf-path=sample-cnfs/multi_helm_directories/cnf-testsuite.yml")
+      result = ShellCmd.cnf_cleanup()
       result[:status].success?.should be_true
       (/Successfully cleaned up/ =~ result[:output]).should_not be_nil
     end
   end
 
-  it "'cnf_setup/cnf_cleanup' should properly install/uninstall old versions of cnf configs", tags: ["setup"] do
+  it "'cnf_setup/cnf_cleanup' should properly install/cleanup old versions of cnf configs", tags: ["setup"] do
     begin
       result = ShellCmd.cnf_setup("cnf-path=spec/fixtures/cnf-testsuite-v1-example.yml")
       result[:status].success?.should be_true
       (/Successfully setup coredns/ =~ result[:output]).should_not be_nil
     ensure
-      result = ShellCmd.run_testsuite("cnf_cleanup cnf-path=spec/fixtures/cnf-testsuite-v1-example.yml")
+      result = ShellCmd.cnf_cleanup()
       result[:status].success?.should be_true
       (/Successfully cleaned up/ =~ result[:output]).should_not be_nil
     end
@@ -84,10 +83,9 @@ describe "Setup" do
       result = ShellCmd.cnf_setup("cnf-path=sample-cnfs/sample-minimal-cnf/cnf-testsuite.yml")
       (/A CNF is already set up. Setting up multiple CNFs is not allowed./ =~ result[:output]).should_not be_nil
     ensure
-      result = ShellCmd.run_testsuite("cnf_cleanup cnf-path=sample-cnfs/sample_coredns/cnf-testsuite.yml")
+      result = ShellCmd.cnf_cleanup()
       result[:status].success?.should be_true
       (/Successfully cleaned up/ =~ result[:output]).should_not be_nil
-      result = ShellCmd.run_testsuite("cnf_cleanup cnf-path=sample-cnfs/sample-minimal-cnf/cnf-testsuite.yml")
     end
   end
 end

spec/spec_helper.cr

@@ -22,12 +22,29 @@ module ShellCmd
   end
 
   def self.cnf_setup(setup_params, cmd_prefix="", expect_failure=false)
-    result = run_testsuite("cnf_setup #{setup_params} wait_count=300", cmd_prefix)
+    timeout_parameter = setup_params.includes?("timeout") ? "" : "timeout=300"
+    result = run_testsuite("cnf_setup #{setup_params} #{timeout_parameter}", cmd_prefix)
     if !expect_failure
       result[:status].success?.should be_true
     else
       result[:status].success?.should be_false
     end
     result
   end
+
+  def self.cnf_cleanup(cleanup_params="", cmd_prefix="", expect_failure=false)
+    timeout_parameter = cleanup_params.includes?("timeout") ? "" : "timeout=300"
+    result = run_testsuite("cnf_cleanup #{cleanup_params} #{timeout_parameter}", cmd_prefix)
+    if !expect_failure
+      result[:status].success?.should be_true
+    else
+      result[:status].success?.should be_false
+    end
+    result
+  end
+
+  def self.environment_cleanup(cleanup_params="", cmd_prefix="")
+    timeout_parameter = cleanup_params.includes?("timeout") ? "" : "timeout=300"
+    result = run_testsuite("cleanup #{cleanup_params} #{timeout_parameter}", cmd_prefix)
+  end
 end