After upgrade 2.2.2 → 2.3.5 → 2.5.1 all signatures gone

[axi92]

If you are reporting a problem, please make sure the following information are provided:

Expected behavior and actual behavior:
I had valid trust signatures with version 2.2.2. After the upgrade to 2.5.1 all signatures are gone in the UI.

Steps to reproduce the problem:
Upgrade from 2.2.2 to 2.3.5 to 2.5.1

Versions:
Please specify the versions of following systems.

• harbor version: 2.5.1
• docker engine version: Docker version 20.10.16, build aa7e414
• docker-compose version: docker-compose version 1.29.2, build 5becea4c

Additional context:

• Harbor config files: You can get them by packaging harbor.yml and files in the same directory, including subdirectory.
• Log files: You can get them by package the /var/log/harbor/ .

harbor.yml

hostname: domain.com
http:
  port: 80
https:
  port: 443
  certificate: /data/cert/server.crt
  private_key: /data/cert/server.key
harbor_admin_password: <password>
database:
  password: <password>
  max_idle_conns: 100
  max_open_conns: 900
data_volume: /data
storage_service:
  ca_bundle: 
trivy:
  ignore_unfixed: false
  timeout: 5m0s
  skip_update: false
  offline_scan: false
  insecure: false
jobservice:
  max_job_workers: 10
notification:
  webhook_job_max_retry: 10
chart:
  absolute_url: disabled
log:
  level: debug
  local:
    rotate_count: 50
    rotate_size: 200M
    location: /var/log/harbor
_version: 2.5.0
proxy:
  http_proxy: 
  https_proxy: 
  no_proxy: 
  components:
    - core
    - jobservice
    - trivy
upload_purging:
  enabled: true
  age: 168h
  interval: 24h
  dryrun: false

[axi92]

Errors in some containers:

notary-signer-photon:

a lot of:
{"go.version":"go1.14.15","level":"error","msg":"GetKeyInfo: key d9fef1ccb8299307b034f251c3daa3703fbf184ff741ed1d62a91fd97872e122 not found","time":"2022-06-02T07:30:15Z"}

notary-server-photon:

{"level":"debug","msg":"entered ValidateRoot with dns: domain.com/product/version-info","time":"2022-06-02T07:30:15Z"}
{"level":"debug","msg":"found the following root keys: [745a07f5b40f47e54e17f76c4887c6e32193ff83ded003632f9a9ec78ec66186]","time":"2022-06-02T07:30:15Z"}
{"level":"debug","msg":"found 1 valid leaf certificates for domain.com/product/version-info: 745a07f5b40f47e54e17f76c4887c6e32193ff83ded003632f9a9ec78ec66186","time":"2022-06-02T07:30:15Z"}
{"level":"debug","msg":"found 1 leaf certs, of which 1 are valid leaf certs for domain.com/product/version-info","time":"2022-06-02T07:30:15Z"}
{"level":"debug","msg":"checking root against trust_pinning config for domain.com/product/version-info","time":"2022-06-02T07:30:15Z"}
{"level":"debug","msg":"checking trust-pinning for cert: 745a07f5b40f47e54e17f76c4887c6e32193ff83ded003632f9a9ec78ec66186","time":"2022-06-02T07:30:15Z"}
{"level":"debug","msg":" role has key IDs: 745a07f5b40f47e54e17f76c4887c6e32193ff83ded003632f9a9ec78ec66186","time":"2022-06-02T07:30:15Z"}
{"level":"debug","msg":"verifying signature for key ID: 745a07f5b40f47e54e17f76c4887c6e32193ff83ded003632f9a9ec78ec66186","time":"2022-06-02T07:30:15Z"}
{"level":"debug","msg":"root validation succeeded for domain.com/product/version-info","time":"2022-06-02T07:30:15Z"}
{"level":"debug","msg":"snapshot role has key IDs: 5c672d1106a9b42ecc2a28700d1eac1f4e7885c0b5addbd239f32526d7eacf91","time":"2022-06-02T07:30:15Z"}
{"level":"debug","msg":"verifying signature for key ID: 5c672d1106a9b42ecc2a28700d1eac1f4e7885c0b5addbd239f32526d7eacf91","time":"2022-06-02T07:30:15Z"}
{"level":"debug","msg":"SignTimestamp","time":"2022-06-02T07:30:15Z"}
{"level":"debug","msg":"sign called with 1/1 required keys","time":"2022-06-02T07:30:15Z"}
{"level":"error","msg":"Failed to create a new timestamp","time":"2022-06-02T07:30:15Z"}
{"go.version":"go1.14.15","http.request.host":"notary-server:4443","http.request.id":"b92045ae-97e9-4339-9d0b-3854c08e4aa7","http.request.method":"GET","http.request.remoteaddr":"172.25.0.6:40460","http.request.uri":"/v2/domain.com/product/version-info/_trust/tuf/timestamp.json","http.request.useragent":"Go-http-client/1.1","level":"info","msg":"404 GET timestamp role","domain.com/product/version-info":"gun","time":"2022-06-02T07:30:15Z"}
{"go.version":"go1.14.15","http.request.host":"notary-server:4443","http.request.id":"b92045ae-97e9-4339-9d0b-3854c08e4aa7","http.request.method":"GET","http.request.remoteaddr":"172.25.0.6:40460","http.request.uri":"/v2/domain.com/product/version-info/_trust/tuf/timestamp.json","http.request.useragent":"Go-http-client/1.1","level":"error","msg":"unknown: unknown error: rpc error: code = 5 desc = key d9fef1ccb8299307b034f251c3daa3703fbf184ff741ed1d62a91fd97872e122 not found","time":"2022-06-02T07:30:15Z"}
{"go.version":"go1.14.15","http.request.host":"notary-server:4443","http.request.id":"b92045ae-97e9-4339-9d0b-3854c08e4aa7","http.request.method":"GET","http.request.remoteaddr":"172.25.0.6:40460","http.request.uri":"/v2/domain.com/product/version-info/_trust/tuf/timestamp.json","http.request.useragent":"Go-http-client/1.1","http.response.contenttype":"application/json; charset=utf-8","http.response.duration":"35.527107ms","http.response.status":500,"http.response.written":70,"level":"info","msg":"response completed","time":"2022-06-02T07:30:15Z"}

[axi92]

I tried the workaround on one repo https://github.com/goharbor/harbor/wiki/Harbor-FAQs#notary-key-not-found
As I tried to push an new signed image, it asked me for new initialization of the repo. As far as I understand that states that the trust is gone on this repo. We decided to setup a new harbor instance and make it all clean from the ground up again.

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[axi92]

As I said we setup a new updated harbor and signed everything new.