Incorrect secret is injected when there are multiple secrets with same key

[klurpicolo]

Version
Java version 11
Kubernetes version 1.21.7

Version after upgrade [issue found]
Spring Boot 2.6.2
Spring Cloud 2021.0.1

Version before upgrade [works fine]
Spring Boot 2.5.3
Spring Cloud 2020.0.3

Describe the bug
We're developing Spring boot backend project and deploy to Kubernetes. There are 2 pods (I'll call them Microservice A and Microservice B) being deployed in the same namespace. Each Microservice have a K8s secrets corresponding to them(which also are in the same namespace). Before upgrading to Spring Boot 2.6.2 and Spring Cloud 2021.0.1, the secrets properties source works correct but after upgrading both A and B, secrets value of Microservice B is injected to Microservice A instead.

Microservice A

Secret
- name : a-service-oauth
  data:
    oauth2.client_id: aaaaa_id
    oauth2.client_secret: aaaaa_secret
- name : a-mongodb
  data:
    url: aaaaa_url

bootstrap.yaml
spring:
  application:
    name: a
  ...
  cloud:
    kubernetes:
      reload:
        enabled: true
        monitoring-secrets: true
      secrets:
        enable-api: true
        sources:
          - name: a-mongodb
          - name: a-service-oauth
      enabled: true
...

Microservice B

Secret
- name : b-service-oauth
- data:
    oauth2.client_id: bbbbb_id
    oauth2.client_secret: bbbbb_secret
- name : b-mongodb
  data:
    url: bbbbb_url

bootstrap.yaml
spring:
  application:
    name: a
  ...
  cloud:
    kubernetes:
      reload:
        enabled: true
        monitoring-secrets: true
      secrets:
        enable-api: true
        sources:
          - name: b-mongodb
          - name: b-service-oauth
      enabled: true

When i investigated bug. I checked the value by exposing via actuator endpoint(/actuator/configprops).

I notice 2 things that incorrect

• Value of client id is bbbbb_id which suppose to be aaaaa_id
• In origin, it shows that value are from secret name a-mongodb which suppose be a-service-oauth.

After I downgrade and check these values that expose via actuator. They are correct.

If any information further is required, please let me know thanks!

[wind57]

That is kind of hard to believe... If Microservice A has a config declaration for a-service-oauth and it does not have b-service-oauth in it's config - it will never even try to read it, so there is no way it will be present.

I am not saying this is not a potential bug, because I can't be sure, I am just telling you that you might not be showing the entire picture here. A minimal, reproducible example, like a github project we can look at, would help a lot. thank you.

[wind57]

Also, I am sure you are not giving the entire picture here, just by the simple fact that you show us this in Secret of Microservice A:

- name : mongodb
  data:
    url: aaaaa_url

And in Microservice B, in Secret:

- name : mongodb
  data:
    url: bbbbb_url

and you say that these are deployed in the same namespace. This is not possible, of course.

[klurpicolo]

Actually, they are a-mongodb and b-mongodb. Sorry for incorrect information. I'll edit to correct that.

That is kind of hard to believe... I agreed. I'm not sure if there is others pieces of information that I should share with you.

I'll try set up another project that can reproduce and share with you.

[wind57]

you could start by showing the exact contents of your entire bootstrap.yaml and your Secrets - I don't need your company name, etc; I just need the exact structure - may be something jumps into my eyes immediately.

[klurpicolo]

Here you are https://gist.github.com/klurpicolo/f5acb4d37cc3a7c0a34281db0c986cb4.

[wind57]

thx, unfortunately I do not see anything obvious. how about logs from both pods in debug mode, at least at start-up.

I mean 1) enable spring logs in debug mode 2) give the logs for me to look at

[klurpicolo]

Here is debug log on start Spring boot https://gist.github.com/klurpicolo/c238846667040821bba6a1269455d116

[wind57]

so, there is no secret reading in your logs :) only config maps. seems like you configured a-service config map, only; which does not match your description of the bug...

[klurpicolo]

Hi @wind57,

Could you help take a look on this log instead https://gist.github.com/klurpicolo/63df6c702cc8171ef66113a336a0eb52 ?
Previous, I used the older version of spring-cloud-starter-kubernetes-fabric8-config which seems to not have log.info.
Also, I set log to debug level on spring cloud kubernetes package.

[wind57]

I still don't have a very good picture in mind of what is going on, but I think there is a strong hint towards this issue/comment.

It seems that you might have multiple secrets with the same key, and because I have introduced that defect - they are messed up. For the time being, I can only recommend for you to stay on a lower version and watch that defect. We will close it and fix it, for sure, but not entirely sure in which version.

Or, if you can, rename your secret keys to use different names.

[klurpicolo]

I'll stay on lower version. Thanks for helping 🙏

[wind57]

There is a PR to fix this here, but it is for our main branch. What this means is that when next release happens (from this branch), it is going to be available only for jdk-17 and up.