ci: [CNI] Move Nightly Cilium Pipeline test to ACN

[jpayne3506]

Reason for Change:

Moved pipeline to ACN to enable the reuse of existing code for this pipeline and others in the future. Changed the cilium image value to be parameterized for future use.

Creation of new file directories were required to follow the same pattern in overlay-e2e-step of calling the directory with kubectl to call clusterrole, serviceaccount, and clusterrolebinding. Similarly the dameonset and deployment for cilium-agent and cilium-operator had to be pulled out of the directories as they cannot be given a dynamic value while within those directories and using the current pattern.

Issue Fixed:

Requirements:

• uses conventional commit messages
• includes documentation
• adds unit tests

Notes:

Ensured that other pipelines that use the new file directory are updated and working. See below for pipeline status.
CNI Load Test - https://msazure.visualstudio.com/One/_build/results?buildId=73426710&view=logs&j=3713fd4c-b129-5ac5-2b38-c7386192ca24
Nightly latest - https://msazure.visualstudio.com/One/_build/results?buildId=73423183&view=results
Nightly v1.12.8 - https://msazure.visualstudio.com/One/_build/results?buildId=73426500&view=results
Azure PR - https://msazure.visualstudio.com/One/_build/results?buildId=73501911&view=results

PR runs with image repository parameter
CNI Load Test - https://msazure.visualstudio.com/One/_build/results?buildId=74436193&view=results
Nightly latest - https://msazure.visualstudio.com/One/_build/results?buildId=74435386&view=results
Azure PR - ToDo

Added CILIUM_VERSION_TAG and CILIUM_IMAGE_REPOSITORY env variable to

• Azure Container Networking PR
• Azure Container Networking PR - Submodules
• CNI Load Test

Merged changes from

• ci: fix cilium validate state check for PRs #1962
• https://msazure.visualstudio.com/CloudNativeCompute/_git/aks-rp/pullrequest/8053786?path=/ccp/charts/addon-charts/cilium-addon/templates/cilium-operator/clusterrole.yaml&_a=files

Changed number of operator pods to match TSG - https://eng.ms/docs/cloud-ai-platform/azure-core/azure-management-and-platforms/containers-bburns/azure-kubernetes-service/azure-kubernetes-service-troubleshooting-guide/doc/tsg/cilium

• cilium-operator" pods are running (2 pods if there are >= 2 nodes, otherwise 1).
• Default is 2 Nodes from makefile

 overlay-byocni-up:` rg-up overlay-net-up ## Brings up an Overlay BYO CNI cluster
	$(AZCLI) aks create -n $(CLUSTER) -g $(GROUP) -l $(REGION) \
		--node-count $(NODE_COUNT) \

NODE_COUNT ?= 2

Once Approved I will start up automatic scheduling, rename the new pipeline from its default, and stop the automatic scheduling from the old pipeline.

[rbtr]

see #1963 (comment)