Auth loop when no admin permissions set on default namespace

[overag3]

Hi,

to resolve issue #6223 , we updated Kubeapps from version 12.2.10 to version 12.4.2.

We use ADFS to authenticate on Kubeapps and unfortunately after the update, we were unable to access the portal listing the applications already deployed. We systematically return to the page which asks us to connect via OIDC.

We found a workaround by creating a RoleBinding, we tried to use the cluster role reader but without success

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ns-admin
  namespace: default
subjects:
    - apiGroup: rbac.authorization.k8s.io
      kind: User
      name: <[email protected]>
roleRef:
  kind: ClusterRole
  name: admin
  apiGroup: rbac.authorization.k8s.io

I specify we confine users to prevent them from deploying applications elsewhere than in their namespace. We find that this behavior will generate security issues and lead to misunderstandings on the user side.

Is there a solution to solve this problem appeared with the last versions?

Regards

[absoludity]

Hi there @overag3 . I'm not aware of any changes to the authentication of Kubeapps recently, so I'm hesitant to assume anything without further evidence. I'd recommend following the OIDC debugging docs to gather log info or other info that shows why you're authentication is now failing. It can be tricky to find the actual issue, especially as there were upstream issues with ADFS in oauth2proxy (that MS was working to fix) in the past, from memory.

Let us know if you can find more details from the logs, otherwise we'll have to wait until we have time to try the setup with ADFS ourselves to confirm the issue, but it may not happen for a while.

[overag3]

Hi,
in fact, the problem is not at the authentication level, since the logs of the kubeappsapis pods tell me that the user has successfully authenticated. I think the problem is related to the permissions assigned to the user, with the creation of RoleBinding, everything is back to normal.

I can't find anything in Kubeapps pods or kube-apiserver pods logs.

[JoelTrain]

We have been hitting this same issue.

We first noticed the issue on chart https://charts.bitnami.com/bitnami version 12.4.2,
but then found that it persists to oci://registry-1.docker.io/bitnamicharts kubeapps version 12.4.3

During the infinite log in redirect loop an API call to <hostname>/apis/kubeappsapis.plugins.resources.v1alpha1.ResourcesService/CheckNamespaceExists can be seen in the network debugger with a 200 status code but an error in the response header:
rpc error: code = PermissionDenied desc = Forbidden to get the Namespace 'default' due to 'namespaces "default" is forbidden: User "oidc:<username>" cannot get resource "namespaces" in API group "" in the namespace "default"'

We found two different workarounds:

1. Downgrade to chart oci://registry-1.docker.io/bitnamicharts kubeapps version 12.1.2
2. Create a new ClusterRole and ClusterRoleBinding

kind: ClusterRole
metadata:
  name: get-list-watch-namespaces
rules:
  - verbs:
      - get
      - list
      - watch
    apiGroups:
      - ""
    resources:
      - namespaces

kind: ClusterRoleBinding
metadata:
  name: system-authenticated-get-list-watch-namespaces
subjects:
  - kind: Group
    apiGroup: rbac.authorization.k8s.io
    name: system:authenticated
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: get-list-watch-namespaces

Allowing the user logging into using OIDC to "get" namespaces seems to fix the issue but is not desirable in our case, because then the user can see many namespaces in the context switch UI and is defaulted to the first one alphabetically.

[absoludity]

Thanks @overag3 and @JoelTrain for the extra information and explicit work-around, sorry @overag3, I'd misunderstood your issue at first. So it sounds like the issue is that

• somewhere prior to 12.4.2 the CheckNamespaceExists method returning a PermissionDenied was accepted as "user has successfully logged in, even if they don't have permission to access the default namespace" (since Kubeapps should not require that every user can check that the default namespace exists), whereas,
• post 12.4.2, Kubeapps is interpreting the response either as a reason to logout the user, or that the user does not have sufficient perms (which is incorrect as above).

I'll try to dig further, thanks.

[RGPosadas]

Will add that I am facing the same issue, my source of error could be easily seen from the pinniped-proxy logs as:

[INFO] - GET https://<host>/api/v1/namespaces/default 403 Forbidden

Similar to @JoelTrain , the only fix was to allow non-admins to GET namespaces on the default namespace, which is not ideal.

Some data which may or may not be useful:

config.json

{
    "kubeappsCluster": "default",
    "kubeappsNamespace": "apps",
    "helmGlobalNamespace": "apps",
    "carvelGlobalNamespace": "kapp-controller-packaging-global",
    "appVersion": "v2.7.0",
    "authProxyEnabled": true,
    "oauthLoginURI": "/oauth2/start",
    "oauthLogoutURI": "/oauth2/sign_out",
    "authProxySkipLoginPage": false,
    "featureFlags": {
        "apiOnly": {
            "enabled": false,
            "grpc": {
                "annotations": {
                    "nginx.ingress.kubernetes.io/backend-protocol": "GRPC"
                }
            }
        },
        "operators": false,
        "schemaEditor": {
            "enabled": false
        }
    },
    "clusters": [
        "default"
    ],
    "theme": "",
    "remoteComponentsUrl": "",
    "customAppViews": [],
    "skipAvailablePackageDetails": false,
    "createNamespaceLabels": {}
}

[absoludity]

Investigated and fixed in #6269. Once found, both the problem and solution was obvious (it was introduced when we switched backend gRPC libraries), though I still need to investigate why our CI integration tests did not hit this (the intention is that they should, but we could be giving our un-privileged user more privileges than we think).

I'll ping here again when the fix is landed and a development build of the kubeapps-apis image is ready for people to verify. Thanks for your patience :)

[overag3]

@absoludity Thank you very much for fixing this bug !! 👍
Unfortunately, I also found another problem when deleting charts from Kubeapps, I'm going to open a new issue.

Regards