Cannot access projects using JWT Token when argocd is installed from Helm Chart

[chamarakera]

Getting below error after directly installing the argocd Helm Chart into the cluster

$ /usr/bin/argocd  app sync my-app --insecure
FATA[0001] rpc error: code = Unauthenticated desc = invalid session: JWT token for role 'ccp-argocd-nonprod-deploy' issued at '1613054077' does not exist in project 'apps-nonprod' 

Below is how to produce the error

1. Install Helm Chart
2. Configure Repository and create Project in Argocd
3. Run following

$ PROJ=apps-nonprod
$ APP=my-app
$ argocd proj role create-token $PROJ $ROLE
$ JWT=<JWT_TOKEN>
$ argocd proj role add-policy $PROJ $ROLE --action get --permission allow --object $APP
$ argocd app get $APP --auth-token $JWT
FATA[0001] rpc error: code = Unauthenticated desc = invalid session: JWT token for role 'get-role' issued at '1613054077' does not exist in project 'apps-nonprod'

Below is my project's yaml. All though it says JWT token for role does not exists, I can see it in the project CRD

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"argoproj.io/v1alpha1","kind":"AppProject","metadata":{"annotations":{},"finalizers":["resources-finalizer.argocd.argoproj.io"],"name":"apps-nonprod","namespace":"argocd"},"spec":{"clusterResourceWhitelist":[{"group":"*","kind":"*"}],"description":"Kubernetes Application Space for non-production","destinations":[{"namespace":"dev","server":"https://kubernetes.default.svc"},{"namespace":"qa","server":"https://kubernetes.default.svc"},{"namespace":"uat","server":"https://kubernetes.default.svc"},{"namespace":"sit","server":"https://kubernetes.default.svc"},{"namespace":"ppd","server":"https://kubernetes.default.svc"},{"namespace":"perf","server":"https://kubernetes.default.svc"}],"roles":[{"description":"Role for deploying apps via Jenkins CI","name":"ccp-argocd-nonprod-deploy","policies":["p, proj:apps-nonprod:argocd-nonprod-deploy, applications, *, apps-nonprod/*, allow"]}],"sourceRepos":["[email protected]:myrepo/my-config-repo.git"]}}
  creationTimestamp: "2021-02-11T14:28:21Z"
  finalizers:
  - resources-finalizer.argocd.argoproj.io
  generation: 2
  managedFields:
  - apiVersion: argoproj.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
        f:finalizers:
          .: {}
          v:"resources-finalizer.argocd.argoproj.io": {}
      f:spec:
        .: {}
        f:clusterResourceWhitelist: {}
        f:description: {}
        f:destinations: {}
        f:sourceRepos: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: "2021-02-11T14:28:21Z"
  - apiVersion: argoproj.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:roles: {}
    manager: argocd-server
    operation: Update
    time: "2021-02-11T14:34:37Z"
  name: apps-nonprod
  namespace: argocd
  resourceVersion: "4326483"
  selfLink: /apis/argoproj.io/v1alpha1/namespaces/argocd/appprojects/apps-nonprod
  uid: 3e1e5bd3-a22c-492d-bd0a-ad7683ab4626
spec:
  clusterResourceWhitelist:
  - group: '*'
    kind: '*'
  description: Kubernetes Application Space for non-production
  destinations:
  - namespace: dev
    server: https://kubernetes.default.svc
  roles:
  - description: Role for deploying apps via Jenkins CI
    jwtTokens:
    - iat: 1613054077
      id: abaa07c5-671f-46f0-a871-b7b2ef152a25
    name: argocd-nonprod-deploy
    policies:
    - p, proj:apps-nonprod:argocd-nonprod-deploy, applications, *, apps-nonprod/*,
      allow
  sourceRepos:
  - [email protected]:myrepo/my-config-repo.git

Everything works fine when I just apply argocd using the install.yaml down below

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v1.8.4/manifests/install.yaml

[jcstryker]

@ckeragala I noticed you closed this, and it seems related to #5382 . Could you share how you resolved this?