Harbor Version Upgrade from 2 to 2.2 Causing Key Issue

[svujasin]

We ran into a key issue when upgrading from Harbor 2 to 2.2. We were unable to sign / push to Harbor. This was the case for both existing images as well as new images. After reviewing the logs of the notary-signer, we noticed the following error:

kubectl logs -l component=notary-signer -n harbor -f
2021/05/11 15:55:14 Updating database.
2021/05/11 15:55:14 schema_migrations table does not require update, skip.
no change
notarysigner database migrated to latest version
{"level":"info","msg":"Version: 0.6.1, Git commit: d6e1431f","time":"2021-05-11T15:55:14Z"}
{"go.version":"go1.15.6","level":"error","msg":"GetKeyInfo: key fb3da5e64d325bc7088d561b43fcd86f8161cc76830460742d48fb0efb039d70 not found","time":"2021-05-14T17:45:20Z"}

I have been unable to determine what key this is referencing. Furthermore, we had tried to reload all of the signer roles and attempt to sign and push again. This was to no avail. The error when attempting to sign and push also is extremely vague - unable to reach trust server at this time: 500.

This is as if the keys are in a corrupted state. We have to remove all the keys and roles, then reload all the roles / keys for each image then resign all the images from scratch in order to remediate. Is there a different way?

[reasonerjt]

Did you back up the data before you upgrade, could you double-check if the data is lost during upgrade?
That should not happen.

[svujasin]

We did backup the data before hand, however we didn’t try a restore and it’s too late to do so. But there was data loss in the signer DB in the private keys table.

[chaospuppy]

We're also seeing this issue following an upgrade from 2.0.0 to 2.2.2.

% k logs -n harbor -l component=notary-signer -c notary-signer                                                                                                                                                                                                                1 ↵
2021/06/11 02:54:18 Failed to Ping DB, sleep for 1 second.
2021/06/11 02:54:19 Failed to Ping DB, sleep for 1 second.
2021/06/11 02:54:20 schema_migrations table does not require update, skip.
no change
notarysigner database migrated to latest version
{"level":"info","msg":"Version: 0.6.1, Git commit: d6e1431f","time":"2021-06-11T02:54:20Z"}
{"go.version":"go1.13.8","level":"error","msg":"GetKeyInfo: key bc3b7e144c09f724fbec9c2f789a36a5862fdf419010b5c5086ef8af3379cc49 not found","time":"2021-06-11T22:17:57Z"}

[chaospuppy]

Additionally, we rolled back the upgrade of Notary from 2.2.2 back to 2.0.0 having observed some other issues, and we restored the notary-signer and notary-server databases to snapshots we had taken before performing the upgrade. We did NOT restore the clair and registry databases, however, as we left harbor core on 2.2.2. Despite having restored the notary databases, we're still seeing the errors listed above. Is there any reason these issues would occur because the registry database hasn't been reverted?

[dkulchinsky]

We're encountering the same issue after upgrade to v2.2.2 from v2.1.5

[harbor-notary-signer-5c677d7c7d-rwz2h notary-signer] {"go.version":"go1.15.6","level":"error","msg":"GetKeyInfo: key 61670107d8407671f2995d7b21c46c47a99576a4881c7810ef68ccaa8ad05858 not found","time":"2021-06-17T04:33:46Z"}

[harbor-notary-server-7785f8887b-7m8vp notary-server] {"go.version":"go1.15.6","http.request.contenttype":"multipart/form-data; boundary=d87dfd2d5007df5193946803a15100b2309e31702ce947cb45ff363ee9b7","http.request.host":"notary.<registry domain>","http.request.id":"4994d5db-fe27-4436-a615-6c669793faad","http.request.method":"POST","http.request.remoteaddr":"<ip>","http.request.uri":"/v2/<GUN>/_trust/tuf/","http.request.useragent":"Go-http-client/1.1","level":"info","msg":"invalid update: Update sent by the client is invalid.: \u0026{ErrValidation An error occurred during validation: rpc error: code = 5 desc = key 61670107d8407671f2995d7b21c46c47a99576a4881c7810ef68ccaa8ad05858 not found}","time":"2021-06-17T04:33:46Z"}

However, the key exists in the notary signer DB:

notarysignerdb=# select key_id,gun from private_keys where key_id='61670107d8407671f2995d7b21c46c47a99576a4881c7810ef68ccaa8ad05858';
                              key_id                              |                                              gun
------------------------------------------------------------------+-----------------------------------------------------------------------------------------------
 61670107d8407671f2995d7b21c46c47a99576a4881c7810ef68ccaa8ad05858 | <GUN>
(1 row)

We did not see any errors in the database migrations during upgrade.

we did however noticed these warnings in the core containers:

2021-06-17T04:16:12Z [WARNING] [/pkg/signature/notary/helper.go:102]: Failed to clear cached root.json: /tmp/notary-cache/tuf/<GUN>/metadata/root.json, error: remove /tmp/notary-cache/tuf/<GUN>/metadata/root.json: no such file or directory, when repo is removed from notary the signature status maybe incorrect

these warnings do not correspond to attempts to sign the images in this GUN, however they do refer to the same GUN.

We're puzzled why notary signer says the key is not found although it's clearly there in the DB and these Failed to clear cached root.json are nothing we ever seen before.

happy to provide more information if needed.

this also seem to be related to #15131

[reasonerjt]

Hi, I can reproduce this issue.

Considering we used golang 1.15 to recompile notary since v2.2.0, I managed reproduce this issue by re-compiling the notary-signer binary with go1.15 and use that binary in v2.1.5 image, error message:

aes.KeyUnwrap(): integrity check failed.

actually the notary's error handling is misleading, that the error happens here:

https://github.com/theupdateframework/notary/blob/332094e0c1c45a46b4c3be9fe239672d900a2fea/signer/keydbstore/sql_keydbstore.go#L124

And the error is lost here:
https://github.com/theupdateframework/notary/blob/332094e0c1c45a46b4c3be9fe239672d900a2fea/signer/keydbstore/sql_keydbstore.go#L226

The discussion in these issues make me think it has something to do with the change crypto/hmac package:
golang/go#41089
dvsekhvalnov/jose2go#26
I currently don't have bandwidth to dig deeper to find which particular change breaks the feature. Next step I think we need to investigate if the issue can be fixed if we bump up the notary's dependency github.com/dvsekhvalnov/jose2go, but this may take some time. Additionally, notary is not actively maintained, so the communication may take more time before they fix this issue.

I'm not sure if we should release a patch to use go1.14 to compile notary-signer because that may cause another breakage for users using released v2.2.x, we need to discuss it.
In short term or even mid-term, could you pin to the image goharbor/notary-signer-photon:v2.1.5 it would be a valid workaround.

[dkulchinsky]

@reasonerjt thanks a lot for getting to the root cause so quickly! I just verified the workaround (rolled signer back to v2.1.5 image) in our test environment and everything works as expected.

Could you elaborate about the concern of building a 2.2.x image of signer with Golang 1.14? since that would essentially be the same as use the image from v2.1.5, isn't it?

[reasonerjt]

@dkulchinsky
For example if we compile notary with go1.14 in v2.2.3 that will likely break, if user upgrade from v2.2.1 which contains notary compiled with g1.15

That said, we may eventually choose to use go1.14 for notary in future releases after discussion. The data has been persisted in notary's db, so I don't have a quick idea if there's a way that breaks no one.

I'll update in this issue.

[dkulchinsky]

@reasonerjt thanks for the clarification, I think I understand now 👍🏼

So the main issue is that any keys stored with Golang 1.14 will not work with Golang 1.15 and vice versa, indeed quite a pickle 😓

[reasonerjt]

Re-open for continue the discussion.
@dkulchinsky I have done some test it seems the issue is fixed in the latest version of github.com/dvsekhvalnov/jose2go, which means if notary can bump up to use the latest version of jose2go go1.15+ will work.

Next we will use go1.14 to compile notary binaries in Harbor v2.2.3 and v2.3.1, and I'll open an issue to notary to notify this problem to notary's maintainers hopefully they can take action to bump up.

[reasonerjt]

Closing this issue as related code changes have been merged.

[wy65701436]

@dkulchinsky
For example if we compile notary with go1.14 in v2.2.3 that will likely break, if user upgrade from v2.2.1 which contains notary compiled with g1.15

That said, we may eventually choose to use go1.14 for notary in future releases after discussion. The data has been persisted in notary's db, so I don't have a quick idea if there's a way that breaks no one.

I'll update in this issue.

For those who are impacted, use the workaround to resolve. https://github.com/goharbor/harbor/wiki/Harbor-FAQs#notary-key-not-found

[axi92]

I got the same error in my notary-signer-photon #16933