Cert bundle not properly returned on AKS

[schneider-lazar]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.):

No

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.):

certificate, tls, bundle

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

NGINX Ingress controller version:
0.20.0 (git-e8d8103)

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:08:19Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:
Azure AKS

• Cloud provider or hardware configuration: Azure Kubernetes Service
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:
  Azure AKS

What happened:

A full bundle certificate is installed with
kubectl create secret tls test-wildcard --cert=certnew2.cer --key=privateKey.KEY

Bundle contains Cert, Intermediate and Root certs of an internal CA (in that sequence). However this also applies for externally signed certs.

The full bundle (including the key at the end of the PEM) is found in the pod/container at
/etc/ingress-controller/ssl/default-test-wildcard.pem

In openssl we can see that the ingress controller returns only the certificate, but not the full certificate chain, hence the chain cannot be validated in openSSL (see attached, anonymized)
openssl.txt

Parameters for ingress install/upgrade
--set rbac.create=false --set controller.extraArgs.default-ssl-certificate="default/test-wildcard" --set-string controller.service.annotations."service.beta.kubernetes.io/azure-load-balancer-internal"=true --set controller.service.loadBalancerIP=$(ingressIP) --set controller.extraArgs.enable-ssl-chain-completion=0

Please remark instead of false we had to use 0 as indicated workaround - otherwise the switch would not work and we got errors: helm/charts#7291

Sidenote (not sure if it is connected):

An issue that should be fixed but is not in our AKS instance - even though general fix is announced - is described here:
https://azure.microsoft.com/en-us/updates/watch-stream-error-fix/
Azure/AKS#676

we still see in our logs a lot of similar errors
E1113 10:35:18.401677 6 streamwatcher.go:109] Unable to decode an event from the watch stream: stream error: stream ID 443; INTERNAL_ERROR where ID varies

What you expected to happen:

We expect the full cert chain to be returned, minimum the intermediate so that openssl validation will not fail (like for all other apps that we have)

How to reproduce it (as minimally and precisely as possible):

Upload self-signed CA cert bundle to AKS ingress and check certificate chain output

Anything else we need to know:

[schneider-lazar]

Issue was resolved by using correct syntax for the bundle. Please note that recommended Snytax for bundel as stated in kubernetes/kubernetes#24669 is NOT working. Keep the certs in the correct order but ONLY use -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- to mark the blocks.