Jenkins Podtemplate with k8s authentication not working - serviceaccount token not exist

[vanlongme]

Jenkins and plugins versions report

Environment

Jenkins: 2.387.1
OS: Linux - 4.18.0-305.el8.x86_64
Java: 11.0.18 - Red Hat, Inc. (OpenJDK 64-Bit Server VM)
---
ace-editor:1.1
all-changes:1.5
analysis-model-api:10.23.1
ant:481.v7b_09e538fcca
antisamy-markup-formatter:159.v25b_c67cd35fb_
apache-httpcomponents-client-4-api:4.5.14-150.v7a_b_9d17134a_5
audit-trail:333.vb_e1b_b_0f1238c
authentication-tokens:1.4
aws-credentials:191.vcb_f183ce58b_9
aws-java-sdk:1.12.406-370.v8f993c987059
aws-java-sdk-cloudformation:1.12.406-370.v8f993c987059
aws-java-sdk-codebuild:1.12.406-370.v8f993c987059
aws-java-sdk-ec2:1.12.406-370.v8f993c987059
aws-java-sdk-ecr:1.12.406-370.v8f993c987059
aws-java-sdk-ecs:1.12.406-370.v8f993c987059
aws-java-sdk-efs:1.12.406-370.v8f993c987059
aws-java-sdk-elasticbeanstalk:1.12.406-370.v8f993c987059
aws-java-sdk-iam:1.12.406-370.v8f993c987059
aws-java-sdk-logs:1.12.406-370.v8f993c987059
aws-java-sdk-minimal:1.12.406-370.v8f993c987059
aws-java-sdk-sns:1.12.406-370.v8f993c987059
aws-java-sdk-sqs:1.12.406-370.v8f993c987059
aws-java-sdk-ssm:1.12.406-370.v8f993c987059
badge:1.9.1
basic-branch-build-strategies:71.vc1421f89888e
blackduck-detect:8.0.0
blueocean:1.27.2
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.2
blueocean-commons:1.27.2
blueocean-config:1.27.2
blueocean-core-js:1.27.2
blueocean-dashboard:1.27.2
blueocean-display-url:2.4.1
blueocean-events:1.27.2
blueocean-git-pipeline:1.27.2
blueocean-github-pipeline:1.27.2
blueocean-i18n:1.27.2
blueocean-jwt:1.27.2
blueocean-personalization:1.27.2
blueocean-pipeline-api-impl:1.27.2
blueocean-pipeline-editor:1.27.2
blueocean-pipeline-scm-api:1.27.2
blueocean-rest:1.27.2
blueocean-rest-impl:1.27.2
blueocean-web:1.27.2
bootstrap4-api:4.6.0-5
bootstrap5-api:5.2.1-3
bouncycastle-api:2.27
branch-api:2.1071.v1a_188a_562481
build-timeout:1.28
built-on-column:1.3
caffeine-api:2.9.3-65.v6a_47d0f4d1fe
checks-api:1.8.1
cloudbees-bitbucket-branch-source:796.v6cb_1559e1673
cloudbees-folder:6.815.v0dd5a_cb_40e0e
command-launcher:90.v669d7ccb_7c31
commons-httpclient3-api:3.1-3
commons-lang3-api:3.12.0-36.vd97de6465d5b_
commons-text-api:1.10.0-36.vc008c8fcda_7b_
conditional-buildstep:1.4.2
configuration-as-code:1569.vb_72405b_80249
credentials:1214.v1de940103927
credentials-binding:523.vd859a_4b_122e6
cucumber-slack-notifier:0.10.0
dark-theme:302.vf069cb_e01486
dashboard-view:2.472.v9ff2a_e6a_c529
discard-old-build:1.07
display-url-api:2.3.7
durable-task:504.vb10d1ae5ba2f
ec2:2.0.6
echarts-api:5.4.0-1
email-ext:2.94
emailext-template:1.5
envinject:2.901.v0038b_6471582
envinject-api:1.199.v3ce31253ed13
external-monitor-job:203.v683c09d993b_9
extra-columns:1.25
favorite:2.4.1
font-awesome-api:6.2.1-1
git:5.0.0
git-client:4.1.0
git-parameter:0.9.18
git-server:99.va_0826a_b_cdfa_d
github:1.37.0
github-api:1.303-417.ve35d9dd78549
github-branch-source:1701.v00cc8184df93
gitlab-api:5.0.1-78.v47a_45b_9f78b_7
gitlab-branch-source:646.vb_9560d64b_69f
gitlab-logo:1.1.0
gitlab-oauth:1.16
gitlab-plugin:1.7.8
global-slack-notifier:1.5
groovy:453.vcdb_a_c5c99890
groovy-postbuild:2.5
handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953
hashicorp-vault-plugin:360.v0a_1c04cf807d
htmlpublisher:1.31
instance-identity:142.v04572ca_5b_265
ionicons-api:31.v4757b_6987003
jackson2-api:2.14.2-319.v37853346a_229
jakarta-activation-api:2.0.1-3
jakarta-mail-api:2.0.1-3
javadoc:226.v71211feb_e7e9
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.8-1
jdk-tool:63.v62d2fd4b_4793
jenkins-design-language:1.27.2
jenkins-multijob-plugin:623.v03401733c9a_9
jersey2-api:2.38-1
jjwt-api:0.11.5-77.v646c772fddb_0
job-dsl:1.81
jobConfigHistory:1207.vd28a_54732f92
jquery:1.12.4-1
jquery3-api:3.6.1-2
jsch:0.1.55.61.va_e9ee26616e7
junit:1177.v90374a_ef4d09
kubernetes:3893.v73d36f3b_9103
kubernetes-client-api:6.4.1-215.v2ed17097a_8e9
kubernetes-credentials:0.10.0
ldap:659.v8ca_b_a_fe79fa_d
login-theme:1.1
mailer:448.v5b_97805e3767
material-theme:0.5.2-rc100.6121925fe229
matrix-auth:3.1.6
matrix-project:785.v06b_7f47b_c631
maven-plugin:3.21
metrics:4.2.13-420.vea_2f17932dd6
mina-sshd-api-common:2.9.2-50.va_0e1f42659a_a
mina-sshd-api-core:2.9.2-50.va_0e1f42659a_a
mission-control-view:0.9.16
momentjs:1.1.1
multi-branch-project-plugin:0.7
node-iterator-api:49.v58a_8b_35f8363
okhttp-api:4.10.0-125.v3593b_a_f8c97b_
pam-auth:1.1
parameterized-scheduler:1.2
parameterized-trigger:2.45
pipeline-aws:1.43
pipeline-build-step:486.vd08f550cceee
pipeline-config-history:1.6
pipeline-github-lib:42.v0739460cda_c4
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-groovy-lib:629.vb_5627b_ee2104
pipeline-input-step:466.v6d0a_5df34f81
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2118.v31fd5b_9944b_5
pipeline-model-definition:2.2118.v31fd5b_9944b_5
pipeline-model-extensions:2.2118.v31fd5b_9944b_5
pipeline-rest-api:2.31
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2118.v31fd5b_9944b_5
pipeline-stage-view:2.31
pipeline-utility-steps:2.15.1
plain-credentials:143.v1b_df8b_d3b_e48
plugin-util-api:2.20.0
popper-api:1.16.1-3
popper2-api:2.11.6-2
prometheus:2.1.1
pubsub-light:1.17
rebuild:1.34
remote-file:1.23
resource-disposer:0.21
role-strategy:587.v2872c41fa_e51
run-condition:1.5
schedule-build:422.v5340de213c7e
scm-api:631.v9143df5b_e4a_a
scm-filter-branch-pr:0.5.1
script-security:1229.v4880b_b_e905a_6
seleniumhtmlreport:1.1
serenity:1.4
simple-theme-plugin:146.v0e67db_a_9052e
slack:631.v40deea_40323b
snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4
solarized-theme:0.1
sonar:2.15
sonargraph-integration:5.0.1
sse-gateway:1.26
ssh:2.6.1
ssh-agent:327.v230ecd01f86f
ssh-credentials:305.v8f4381501156
ssh-slaves:2.877.v365f5eb_a_b_eec
sshd:3.275.v9e17c10f2571
structs:324.va_f5d6774f3a_d
theme-manager:1.6
timestamper:1.22
token-macro:321.vd7cc1f2a_52c8
trilead-api:2.84.v72119de229b_7
uno-choice:2.6.4
variant:59.vf075fe829ccb
view-job-filters:2.3
windows-slaves:1.8.1
workflow-aggregator:596.v8c21c963d92d
workflow-api:1208.v0cc7c6e0da_9e
workflow-basic-steps:1010.vf7a_b_98e847c1
workflow-cps:3641.vf58904a_b_b_5d8
workflow-cps-global-lib:609.vd95673f149b_b
workflow-cps-global-lib-http:2.20.0
workflow-durable-task-step:1234.v019404b_3832a
workflow-job:1282.ve6d865025906
workflow-multibranch:733.v109046189126
workflow-scm-step:400.v6b_89a_1317c9a_
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:839.v35e2736cfd5c
ws-cleanup:0.44

What Operating System are you using (both controller, and any agents involved in the problem)?

Redhat Enterprise 7.
K8s jnlp-jdk11 agent

Reproduction steps

1. Enable k8s authentication on vault
2. Create vault k8s credential, fill vault's role that create above step to this credential
3. Goto Manage Node => Cloud Node => create k8s pod template
4. Create a pipeline that using k8s pod template:

node('worker-lab') {
    stage('test'){
withVault(configuration: [vaultCredentialId: 'test-pod-template', vaultUrl: 'https://vault.testserver.local'], vaultSecrets: [[path: 'test/k8s', secretValues: [[envVar: 'abc', vaultKey: 'a']]]]) {
    container('busybox') {
        sh '''
          echo $abc
          sleep 300
        '''
      }
    }
    }
}
`

### Expected Results

value get from vault path

### Actual Results

```yaml
Agent [worker-lab-9vdlx] is provisioned from template worker-lab
---
apiVersion: "v1"
kind: "Pod"
metadata:
  labels:
    jenkins: "slave"
    jenkins/label-digest: "f2c78489307afc50700aa24a8d8532cb13d7ff86"
    jenkins/label: "worker-lab"
  name: "worker-lab-9vdlx"
  namespace: "jenkins"
spec:
  containers:
  - args:
    - "********"
    - "worker-lab-9vdlx"
    env:
    - name: "JENKINS_SECRET"
      value: "********"
    - name: "JENKINS_TUNNEL"
      value: "10.0.0.89:30000"
    - name: "JENKINS_AGENT_NAME"
      value: "worker-lab-9vdlx"
    - name: "JENKINS_NAME"
      value: "worker-lab-9vdlx"
    - name: "JENKINS_AGENT_WORKDIR"
      value: "/home/jenkins/agent"
    - name: "JENKINS_URL"
      value: "https://jenkins-cloud.xxx.yyy/"
    image: "docker.xxx.yyy/jenkins/inbound-agent:latest-jdk11"
    imagePullPolicy: "Always"
    name: "jnlp"
    resources:
      limits: {}
      requests: {}
    tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
      readOnly: false
    workingDir: "/home/jenkins/agent"
  - args:
    - "9999999"
    command:
    - "sleep"
    image: "docker.xxx.yyy/busybox"
    imagePullPolicy: "IfNotPresent"
    name: "busybox"
    resources:
      limits: {}
      requests: {}
    securityContext:
      privileged: true
    tty: true
    volumeMounts:
    - mountPath: "/home/"
      name: "workspace-volume"
      readOnly: false
    workingDir: "/home/"
  hostNetwork: false
  imagePullSecrets:
  - name: "registry-creds"
  nodeSelector:
    kubernetes.io/os: "linux"
  restartPolicy: "Never"
  serviceAccountName: "jenkins-sa"
  volumes:
  - emptyDir:
      medium: ""
    name: "workspace-volume"


java.nio.file.NoSuchFileException: /var/run/secrets/kubernetes.io/serviceaccount/token
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
	at java.base/sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:182)
	at java.base/java.nio.channels.FileChannel.open(FileChannel.java:292)
	at java.base/java.nio.channels.FileChannel.open(FileChannel.java:345)
	at java.base/java.nio.file.Files.lines(Files.java:4038)
	at java.base/java.nio.file.Files.lines(Files.java:4130)
	at com.datapipe.jenkins.vault.credentials.VaultKubernetesCredential.getToken(VaultKubernetesCredential.java:57)
Caused: com.datapipe.jenkins.vault.exception.VaultPluginException: could not get JWT from Service Account Token
	at com.datapipe.jenkins.vault.credentials.VaultKubernetesCredential.getToken(VaultKubernetesCredential.java:60)
	at com.datapipe.jenkins.vault.credentials.AbstractAuthenticatingVaultTokenCredential.getToken(AbstractAuthenticatingVaultTokenCredential.java:60)
	at com.datapipe.jenkins.vault.credentials.AbstractVaultTokenCredentialWithExpiration.authorizeWithVault(AbstractVaultTokenCredentialWithExpiration.java:31)
	at com.datapipe.jenkins.vault.VaultAccessor.init(VaultAccessor.java:66)
	at com.datapipe.jenkins.vault.VaultAccessor.retrieveVaultSecrets(VaultAccessor.java:161)
	at com.datapipe.jenkins.vault.VaultBindingStep$Execution.doStart(VaultBindingStep.java:115)
	at org.jenkinsci.plugins.workflow.steps.GeneralNonBlockingStepExecution.lambda$run$0(GeneralNonBlockingStepExecution.java:77)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Finished: FAILURE

Anything else?

If remove "withVault" field, and cat token file that mount from k8s service account, it echo that token file exist:

Running on [worker-lab-9vdlx](https://jenkins-cloud.xxx.yyy/computer/worker-lab-9vdlx/) in /home/jenkins/agent/workspace/lab-pod-template/pipeline-test
[Pipeline] {
[Pipeline] stage
[Pipeline] { (test)
[Pipeline] sh
+ cat /var/run/secrets/kubernetes.io/serviceaccount/token
eyJhbGciOiJSUzI1NiIsImtpZCI6IjY2YTAzOTM1YWQwNzMyNzhlNWYwMmY2YWE1MzU0OTIwZGM1MTA3NjAifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTcxMDkyOTI2NywiaWF0IjoxNjc5MzkzMjY3LCJpc3MiOiJodHRwczovL29pZGMuZWtzLmFwLXNvdXRoZWFzdC0xLmFtYXpvbmF3cy5jb20vaWQvMTY2ODlGOTJFOTQ3RUEzRjdEREYyN0NBRTJDMzBCNUUiLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6ImplbmtpbnMiLCJwb2QiOnsibmFtZSI6Indvcmtlci1sYWItOXZkbHgiLCJ1aWQiOiIzYWRlN2VlNy1iNjkxLTQ3YzktYjU5Zi00Yzk3MzVjYTg5NTkifSwic2VydmljZWFjY291bnQiOnsibmFtZSI6ImplbmtpbnMtc2EiLCJ1aWQiOiJmZTA2NDRjYS1jMWJiLTQ4NDItYjI3NS0wYzRlY2I4YTBmMWYifSwid2FybmFmdGVyIjoxNjc5Mzk2ODc0fSwibmJmIjoxNjc5MzkzMjY3LCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6amVua2luczpqZW5raW5zLXNhIn0.cFK2rPKWwtwiTiArLHYpcTfC4e04iQwORqYpMnAaPdC_q9yFX6s-Q3WvR1mglbBLBV6Bfy1nCnB-hi28BNV2ACjwkAfv8cgkPwy0l0a1zTKaYVOx28YDacjmCdRD2mNLN5N9YcYcKxqDuaBbSQ3H8SmLhdXV7EfSn7iAxYFGUi98NyXWOgW5gHbncZDlyVgFWQPh1UxUYh397iZ-KlB4ccv2iUESvJwjkzzjeMz4r5S-WVSANxoEkUVD-JScMoe7uT42zWPLu9JFXO6c9CVFG-EQ9E8AF7-DJs_9yU2RX4YntLdXC8B7r5-8vmXlR2XouwASzE7w6xk254hnpyTyBQ+ sleep 300
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS