Skip to content
This repository has been archived by the owner on Apr 17, 2023. It is now read-only.

Background worker fails after enabling security scanning #1743

Closed
kkoppel opened this issue Mar 28, 2018 · 3 comments
Closed

Background worker fails after enabling security scanning #1743

kkoppel opened this issue Mar 28, 2018 · 3 comments
Assignees
@mssola
Labels
bug

Comments

@kkoppel
Copy link

kkoppel commented Mar 28, 2018

Description

My Portus background worker started to fail after enabling security scanning with Clair v2.0.2.

Steps to reproduce

I recently set up Portus to work with an existing private registry installation with about 50 repositories. The existing Docker images got imported and everything seemed to work fine.
I then enabled security scanning via Clair by setting the following environment variables:
PORTUS_SECURITY_CLAIR_SERVER
PORTUS_SECURITY_CLAIR_HEALTH_PORT

After restarting both the main Portus instance and the background worker, I get the following from the background worker:

[Initialization] Running: 'Registry events', 'Security scanning'
/srv/Portus/lib/portus/security.rb:49:in `vulnerabilities': undefined method `map' for nil:NilClass (NoMethodError)
        from /srv/Portus/lib/portus/background/security_scanning.rb:47:in `block in execute!'
        from /srv/Portus/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `block (2 levels) in find_each'
        from /srv/Portus/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `each'
        from /srv/Portus/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `block in find_each'
        from /srv/Portus/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:124:in `find_in_batches'
        from /srv/Portus/vendor/bundle/ruby/2.5.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:50:in `find_each'
        from /srv/Portus/lib/portus/background/security_scanning.rb:35:in `execute!'
        from /srv/Portus/bin/background.rb:56:in `block (2 levels) in <top (required)>'
        from /srv/Portus/bin/background.rb:54:in `each'
        from /srv/Portus/bin/background.rb:54:in `each_with_index'
        from /srv/Portus/bin/background.rb:54:in `block in <top (required)>'
        from /srv/Portus/bin/background.rb:53:in `loop'
        from /srv/Portus/bin/background.rb:53:in `<top (required)>'
        from /srv/Portus/vendor/bundle/ruby/2.5.0/gems/railties-4.2.10/lib/rails/commands/runner.rb:60:in `load'
        from /srv/Portus/vendor/bundle/ruby/2.5.0/gems/railties-4.2.10/lib/rails/commands/runner.rb:60:in `<top (required)>'
        from /srv/Portus/vendor/bundle/ruby/2.5.0/gems/railties-4.2.10/lib/rails/commands/commands_tasks.rb:123:in `require'
        from /srv/Portus/vendor/bundle/ruby/2.5.0/gems/railties-4.2.10/lib/rails/commands/commands_tasks.rb:123:in `require_command!'
        from /srv/Portus/vendor/bundle/ruby/2.5.0/gems/railties-4.2.10/lib/rails/commands/commands_tasks.rb:90:in `runner'
        from /srv/Portus/vendor/bundle/ruby/2.5.0/gems/railties-4.2.10/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
        from /srv/Portus/vendor/bundle/ruby/2.5.0/gems/railties-4.2.10/lib/rails/commands.rb:17:in `<top (required)>'
        from bin/rails:12:in `require'
        from bin/rails:12:in `<main>'
bundler: failed to load command: packaging/suse/portusctl/bin/portusctl (packaging/suse/portusctl/bin/portusctl)
RuntimeError: Something went wrong while invoking: /srv/Portus/vendor/bundle/ruby/2.5.0/bin/bundler.ruby2.5 exec rails r /srv/Portus/bin/background.rb
  /srv/Portus/packaging/suse/portusctl/lib/runner.rb:12:in `exec'
  /srv/Portus/packaging/suse/portusctl/lib/runner.rb:34:in `block in bundler_exec'
  /srv/Portus/packaging/suse/portusctl/lib/runner.rb:30:in `chdir'
  /srv/Portus/packaging/suse/portusctl/lib/runner.rb:30:in `bundler_exec'
  /srv/Portus/packaging/suse/portusctl/lib/cli.rb:272:in `exec'
  /srv/Portus/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor/command.rb:27:in `run'
  /srv/Portus/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor/invocation.rb:126:in `invoke_command'
  /srv/Portus/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor.rb:369:in `dispatch'
  /srv/Portus/vendor/bundle/ruby/2.5.0/gems/thor-0.19.4/lib/thor/base.rb:444:in `start'
  packaging/suse/portusctl/bin/portusctl:6:in `<top (required)>'

If I restart the background worker, then the same thing happens again after some seconds. I see a bunch of requests being handled by Clair when the background worker starts and all of them get 200 or 201 responses.

Deployment information

Portus is running in Docker containers and the whole setup is based on the secure Docker compose example.

Configuration:

# portusctl exec rake portus:info                                                                                                                                                     
[Mailer config] Host:     portus.***
[Mailer config] Protocol: https://                                                                                                                                                            
Evaluated configuration:                                                                                                                                                                      
---                                                                                                                                                                                           
email:                                                                                                                                                                                        
  from: [email protected]                                                                                                                                                                    
  name: Portus                                                                                                                                                                                
  reply_to: [email protected]                                                                                                                                                              
  smtp:                                                                                                                                                                                       
    enabled: false                                                                                                                                                                            
    address: smtp.example.com                                                                                                                                                                 
    port: 587                                                                                                                                                                                 
    user_name: [email protected]                                                                                                                                                           
    password: "****"                                                                                                                                                                          
    domain: example.com                                                                                                                                                                       
gravatar:                                                                                                                                                                                     
  enabled: true                                                                                                                                                                               
delete:                                                                                                                                                                                       
  enabled: false
ldap:                                                                                                                                                                                         
  enabled: false                                                                                                                                                                              
  hostname: ldap_hostname                                                                                                                                                                     
  port: 389                                                                                                                                                                                   
  method: plain                                                                                                                                                                               
  base: ''                                                                                                                                                                                    
  filter: ''                                                                                                                                                                                  
  uid: uid
  authentication:
    enabled: false
    bind_dn: ''
    password: "****"
  guess_email:
    enabled: false
    attr: ''
oauth:
  google_oauth2:
    enabled: false
    id: ''
    secret: ''
    domain: ''
    options:
      hd: ''
  open_id:
    enabled: false
    identifier: ''
    domain: ''
  github:
    enabled: false
    client_id: ''
    client_secret: ''
    organization: ''
    team: ''
    domain: ''
  gitlab:
    enabled: false
    application_id: ''
    secret: ''
    group: ''
    domain: ''
    server: ''
  bitbucket:
    enabled: false
    key: ''
    secret: ''
    domain: ''
    options:
      team: ''
first_user_admin:
  enabled: true
signup:
  enabled: true
check_ssl_usage:
  enabled: true
registry:
  jwt_expiration_time:
    value: 5
  catalog_page:
    value: 100
  timeout:
    value: 2
  read_timeout:
    value: 120
machine_fqdn:
  value: portus.***
display_name:
  enabled: false
user_permission:
  change_visibility:
    enabled: true
  create_team:
    enabled: true
  manage_team:
    enabled: true
  create_namespace:
    enabled: true
  manage_namespace:
    enabled: true
security:
  clair:
    server: http://clair:6060
    health_port: 6061
  zypper:
    server: ''
  dummy:
    server: ''
anonymous_browsing:
  enabled: true
background:
  registry:
    enabled: true
  sync:
    enabled: true
    strategy: initial

Portus version: 2.3.1@a4ca664b9c30c7a464296297d1868ba301d791cf
mssola added a commit to mssola/Portus that referenced this issue Apr 6, 2018
@mssola
security: fetch the manifest more safely
640e48c 
In order to fetch vulnerabilities, the security scanner first has to
gather the layers to be inspected. In our case, we fetch this
information from the image manifest as given by the Registry.

This commit adds some code so the fetching of this manifest is safer.
For example, it will not freak out if a timeout was reached when
requesting the manifest from the registry.

Fixes SUSE#1743

Signed-off-by: Miquel Sabaté Solà <[email protected]>
@mssola mssola mentioned this issue Apr 6, 2018
Merged
@mssola
Copy link
Collaborator

mssola commented Apr 6, 2018 •
edited
Loading

From the stack trace I gather that this can happen when there is a problem while fetching a manifest from the registry (e.g. it doesn't exist anymore, a timeout was reached, etc.). I've opened #1768 to address this situation. Once it's merged, I'll update the 2.3 Docker image so you could tell me whether you can still reproduce this problem.

Thanks a lot for reporting 👏 (and sorry for the delay, I came back from vacations on Tuesday and I was still catching up 😁)

@mssola mssola added the bug label Apr 6, 2018
@mssola mssola self-assigned this Apr 6, 2018
mssola added a commit that referenced this issue Apr 6, 2018
@mssola
security: fetch the manifest more safely
fed2818 
In order to fetch vulnerabilities, the security scanner first has to
gather the layers to be inspected. In our case, we fetch this
information from the image manifest as given by the Registry.

This commit adds some code so the fetching of this manifest is safer.
For example, it will not freak out if a timeout was reached when
requesting the manifest from the registry.

Fixes #1743

Signed-off-by: Miquel Sabaté Solà <[email protected]>
@mssola
Copy link
Collaborator

mssola commented Apr 6, 2018

@kkoppel the code is already in the 2.3 Docker image. Could you pull the image again and tell me if you are still facing this issue ? We will release a 2.3.4 version in the following weeks and this fix will also be included.

@kkoppel
Copy link
Author

kkoppel commented Apr 7, 2018

Thanks, @mssola !
I've had the background process running on the new Docker image for 30 hours now and have not seen this issue again.

vitoravelino pushed a commit to vitoravelino/Portus that referenced this issue Apr 12, 2018
@mssola
security: fetch the manifest more safely
58d17f3 
In order to fetch vulnerabilities, the security scanner first has to
gather the layers to be inspected. In our case, we fetch this
information from the image manifest as given by the Registry.

This commit adds some code so the fetching of this manifest is safer.
For example, it will not freak out if a timeout was reached when
requesting the manifest from the registry.

Fixes SUSE#1743

Signed-off-by: Miquel Sabaté Solà <[email protected]>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@mssola mssola

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mssola @kkoppel