[BUG] Exposition of port 9965 on cilium pods and service label selectors missing

[lukibahr]

Describe the bug

After enabling ACNS according the docs https://learn.microsoft.com/en-us/azure/aks/advanced-network-observability-cli?tabs=cilium#visualization-using-byo-grafana, the goal is to visualize hubble metrics in Grafana. Enabling ACNS successfully installs cilium and its pods - you can fetch metrics from the pod by executing in the pod using kubectl exec -it <cilium-pod> -- /bin/bash and (after installing curl or wget in the container) run curl -X GET localhost:9965/metrics.

However, the hubble metrics server port 9965 is not exposed by the cilium pod. The only port which is exposed by the pod is 9962, which references on Cilium metrics only:

    ports:
    - containerPort: 9962
      hostPort: 9962
      name: prometheus
      protocol: TCP

Additionally, the service in kube-system namespace network-observability does not select pods due to missing endpoints because the do not have label selectors which makes it unable to build a servicemonitor for adding scrape config to the prometheus (like it's describe in the docs above). The service network-observability should have a label selector on k8s-app: cilium - see the following yaml snippet:

# this is a customer generated service that selects the pods by the selector. field

apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: aks-managed-kappie
  labels:
    k8s-app: hubble-workaround
  name: network-observability-workaround
  namespace: kube-system
spec:
  ports:
  - name: hubble
    port: 9965
    protocol: TCP
    targetPort: 9965
  - name: cilium
    port: 9962
    protocol: TCP
    targetPort: 9962
  type: ClusterIP
  selector: # missing selector
    k8s-app: cilium

To Reproduce

For steps to reproduce the behavior, see above.

❯ kubectl port-forward -n kube-system svc/network-observability 9965:9965
                                                                                                                                                                      
error: cannot attach to *v1.Service: invalid service 'network-observability': Service is defined without a selector.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• Kubernetes version: Client Version: v1.31.2, Kustomize Version: v5.4.2, Server Version: v1.31.2