harbor v2.9.0 migration error, jinja template typo/bug

[drshawnkwang]

If you are reporting a problem, please make sure the following information are provided:

Expected behavior and actual behavior:
Expected: Updating harbor to version 2.9.0 using prepare container and migrate command works

Actual behavior:
docker run -it --rm -v /:/hostfs goharbor/prepare:v2.9.0 migrate -i ./harbor.yml

harbor.yml contains object storage configuration

# Harbor Storage settings by default is using /data dir on local filesystem
# Uncomment storage_service setting If you want to using external storage
storage_service:
#   # storage backend, default is filesystem, options include filesystem, azure, gcs, s3, swift and oss
#   # for more info about this configuration please refer https://docs.docker.com/registry/configuration/
  swift:
    authurl: https://swift.example/auth/v1
    username: user:foobar
    password: secret-replaced
    container: harbor
#   # set disable to true when you want to disable registry redirect
  redirect:
    disable: False

Error message

migrating to version 2.9.0                                                                                                                                                                                                        
Traceback (most recent call last):                                                                                                                                                                                                
  File "/usr/src/app/main.py", line 15, in <module>                                                                                                                                                                               
    cli()                                                                                                                                                                                                                         
  File "/usr/lib/python3.11/site-packages/click/core.py", line 1137, in __call__                                                                                                                                                  
    return self.main(*args, **kwargs)                                                                                                                                                                                             
           ^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                             
  File "/usr/lib/python3.11/site-packages/click/core.py", line 1062, in main                                                                                                                                                      
    rv = self.invoke(ctx)                                                                                                                                                                                                         
         ^^^^^^^^^^^^^^^^                                                                                                                                                                                                         
  File "/usr/lib/python3.11/site-packages/click/core.py", line 1668, in invoke                                                                                                                                                    
    return _process_result(sub_ctx.command.invoke(sub_ctx))                                                                                                                                                                       
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                        
  File "/usr/lib/python3.11/site-packages/click/core.py", line 1404, in invoke                                                                                                                                                    
    return ctx.invoke(self.callback, **ctx.params)                                                                                                                                                                                
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                
  File "/usr/lib/python3.11/site-packages/click/core.py", line 763, in invoke                                                                                                                                                     
    return __callback(*args, **kwargs)                                                                                                                                                                                            
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                            
  File "/usr/src/app/commands/migrate.py", line 43, in migrate                                                                                                                                                                    
    m.migrate(current_input_path, current_output_path)                                                                                                                                                                            
  File "/usr/src/app/migrations/version_2_9_0/__init__.py", line 21, in migrate                                                                                                                                                   
    f.write(tpl.render(**config_dict))                                                                                                                                                                                            
            ^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                             
  File "/usr/lib/python3.11/site-packages/jinja2/environment.py", line 1301, in render                                                                                                                                            
    self.environment.handle_exception()                                                                                                                                                                                           
  File "/usr/lib/python3.11/site-packages/jinja2/environment.py", line 936, in handle_exception                                                                                                                                   
    raise rewrite_traceback_stack(source=source)                                                                                                                                                                                  
  File "/usr/src/app/migrations/version_2_9_0/harbor.yml.jinja", line 117, in top-level template code                                                                                                                             
    disable: {{ value.disabled }}                                                                                                                                                                                                 
^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                         
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'disabled'  

Looking at the jinja template on GitHub

#   # set disable to true when you want to disable registry redirect
  redirect:
      {% if value.disabled is not none %}
    disable: {{ value.disabled }}
      {% else %}
    disable: {{ value.disable }}
      {% endif %}

Looking at this code, there is both the word disable and disabled in the jinja. Someone probably needs to change this so only one word is used. (probably disable?)

Remediation: Modify harbor.yml.jinja to use the keyword disable or disabled consistently.

Steps to reproduce the problem:
Configure harbor.yml to use object storage, with redirect: disable, run the prepare & migration docker command.

Versions:
Please specify the versions of following systems.

• harbor version: 2.8.2
• docker engine version: 24.0.6
• docker-compose version: 2.21.0

Additional context:

• Harbor config files: Snipped provided above
• Log files: Not relevant as issue is with migration

[wy65701436]

hi @drshawnkwang To clarify, the specific problem encountered during the migration from v2.8.2 to v2.9.0?

[wy65701436]

Thanks @drshawnkwang for catching this.

@MinerYang Please validate the issue on our end. If it is an problem, we should await the release of v2.9.1 to address it.
And I kindly request the manual workaround steps for v2.9.0.

cc @YangJiao0817 it needs test case.

[MinerYang]

introduced by this PR
#14615

Thanks @drshawnkwang , we are now fixing it.

[MinerYang]

As a workaround, when you do the harbor.yml migrate, please refer to the following steps.
step 1 uncomment the redirect block

storage_service:
  # redirect:
  #   disable: false

step2 run migrate script

docker run -it --rm -v /:/hostfs goharbor/prepare:v2.9.0 migrate -i ./harbor.yml

step3 add storage_service.redirect.disable field manually as demand

storage_service:
   redirect:
      disable: true
...

[cparadal]

As a workaround, when you do the harbor.yml migrate, please refer to the following steps. step 1 uncomment the redirect block

storage_service:
  # redirect:
  #   disable: false

step2 run migrate script

docker run -it --rm -v /:/hostfs goharbor/prepare:v2.9.0 migrate -i ./harbor.yml

step3 add storage_service.redirect.disable field manually as demand

storage_service:
   redirect:
      disable: true
...

We're having issues upgrading from 2.8.4 to 2.90 (see [related issue(https://github.com//issues/19398)).
As for the workaround, I guess that migration script is something that is run on non-Helm installations of Harbor? What's the workaround if you're trying to upgrade via Helm?

2023-09-26T17:33:56Z [INFO] [/common/dao/pgsql.go:135]: Upgrading schema for pgsql ...
2023-09-26T17:33:56Z [ERROR] [/common/dao/pgsql.go:140]: Failed to upgrade schema, error: "Dirty database version 120. Fix and force version."
2023-09-26T17:33:56Z [FATAL] [/core/main.go:192]: failed to migrate the database, error: Dirty database version 120. Fix and force version.

[MinerYang]

Hi @cparadal ,
Seems it not related to #19320 which is about migrate offline-installer config harbor.yml.
We will keep tracking your helm issue on the harbor-helm side.