system:heapster ClusterRole lacking nodes/stats get privileges

[r0bj]

The system:heapster ClusterRole appears to be missing the ability to GET to nodes/stats resource.

heapster deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: heapster
  namespace: kube-system
spec:
  replicas: 1
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  template:
    metadata:
      labels:
        task: monitoring
        k8s-app: heapster
        version: v6
    spec:
      serviceAccountName: heapster
      containers:
      - name: heapster
        image: gcr.io/google_containers/heapster:v1.5.0
        imagePullPolicy: Always
        command:
        - /heapster
        - --source=kubernetes.summary_api:https://kubernetes.default.svc?kubeletHttps=true&kubeletPort=10250&insecure=true
        - --sink=influxdb:http://dev.os-metrics-db.service.poz-dev.consul:8086?withfields=true&db=k8s
        - --metric-resolution=30s

error messages from API server:

I0123 12:28:05.087431       1 rbac.go:116] RBAC DENY: user "system:serviceaccount:kube-system:heapster" groups ["system:serviceaccounts" "system:serviceaccounts:kube-system" "system:authenticated"] cannot "get" resource "nodes/stats" named "dev-k8s-lb-p1" cluster-wide

error message from heapster:

E0123 12:28:35.033599       1 manager.go:101] Error in scraping containers from kubelet_summary:10.14.30.116:10250: request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=get, resource=nodes, subresource=stats)"

Issue is fixed after adding those lines to ClusterRole system:heapster:

- apiGroups:
  - ""
  resources:
  - nodes/stats
  verbs:
  - get

kubernetes: 1.9.1
heapster: 1.5.0

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[r0bj]

/remove-lifecycle stale

[OrlinVasilev]

@r0bj - have started the kubelet with authorization-webhook true ? currently I'm facing the same issue which I cannot solve

[r0bj]

Yes, kubelet is started with authorization-webhook true.

As a workaround I just created separate ClusterRole and use it for heapster:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: heapster-custom-fix
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  - namespaces
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - extensions
  resources:
  - deployments
  verbs:
  - get
  - list
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/stats
  verbs:
  - get

[OrlinVasilev]

thanks worked for me just starting the kubelet with --authentication-token-webhook=true and later on adding it to the kops config files.
thanks a lot hope will not hit the same on metrics server.

[ljfranklin]

With K8S v1.12.0 I was able to get the Dashboard + Heapster working with this suggestion plus adding the verb create to the nodes/stats section to avoid a 403 error.

[page-fault-in-nonpaged-area]

This fix should be implemented by default until metrics-server migration is complete. took me too long to figure out why heapster isn't working.