kubectl cluster-info dump dies on first pod with multiple containers

[chrisegner]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.): No

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): cluster-info, dump, "server rejected our request for an unknown reason"

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:48:38Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:42:39Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release):

NAME="Red Hat Enterprise Linux Server"
VERSION="7.3 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="7.3"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.3 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.3:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.3
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.3"

• Kernel (e.g. uname -a): Linux ip-xxx.ec2.internal 3.10.0-514.el7.x86_64 #1 SMP Wed Oct 19 11:24:13 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: kubeadm
• Others:

What happened:
kubectl cluster-info dump fails to get logs from pods with multiple containers. It exits at the first such pod it encounters (in lexical order). The kube-dns pod invariable triggers this.

$ kubectl cluster-info dump --all-namespaces --output-directory "cluster-info-$(date +%s)"
Error from server: the server rejected our request for an unknown reason (get pods 0921nr28g1e5d49q)

What you expected to happen:

1. cluster-info dump should gracefully handle multi-container pods, fetching logs for all of containers.
2. Should a dump fail, kubectl should continue dumping as much of the information from remaining resources as possible.

How to reproduce it (as minimally and precisely as possible):
Run kubectl cluster-info dump --all-namespaces --output-directory "cluster-info-$(date +%s)" with a kube-dns pod running.

Anything else we need to know:

[xingzhou]

I can reproduce the error by using the latest code, let me take some investigation on this first.