Fixes for osc-1.x

.gitignore

@@ -6,3 +6,6 @@
 .docker-tmp/
 .bash_history
 osc
+.venv
+.vscode
+.idea

check_source.py

@@ -585,7 +585,7 @@ def run_source_validator(self, old, directory):
         for script in scripts:
             if os.path.isdir(script):
                 continue
-            res = subprocess.run(['/bin/bash', script, '--batchmode', directory, old], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+            res = subprocess.run([script, '--batchmode', directory, old], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
             if res.returncode:
                 text = "Source validator failed. Try \"osc service runall source_validator\"\n"
                 text += res.stdout.decode('utf-8')

docs/pkglistgen.md

@@ -1,25 +1,25 @@
 # Package List Generator
 
-pkglistgen.py is a self contained script to generate and update OBS products for openSUSE and SLE. 
+pkglistgen.py is a self contained script to generate and update OBS products for openSUSE and SLE.
 It works on the products and its staging projects and ports.
 
 The main input is a package named 000package-groups and it will update the content of other packages
 from that. For that it will read [YAML](https://en.wikipedia.org/wiki/YAML) input from e.g. 000package-groups/groups.yml and generate .group files into 000product. The rest of 000package-groups is copied into 000product as well and it runs the OBS product converter service (See [OBS Documentation](https://en.opensuse.org/openSUSE:Build_Service_product_definition) for details)
-The generated release spec files are split into 000release-packages to avoid needless rebuilds. 
+The generated release spec files are split into 000release-packages to avoid needless rebuilds.
 
 ## Input
 
-The package list generator reads several files. The most important are group*.yml (tradionally only groups.yml) within 000package-groups.
+The package list generator reads several files. The most important are group*.yml (traditionally only groups.yml) within 000package-groups.
 
 ### supportstatus.txt
 The file lists the packages and their support level. It's only necessary to list packages here that have a different level than the default level specificied in the groups. The format is plain text: <package name> <level> - the level is handed over 1:1 to KIWI file. Currently used values are: unsupported, l2 and l3
- 
+
 ### group*.yml
 The file is a list of package lists and the special hash 'OUTPUT'. OUTPUT contains an entry for every group file that needs to be written out. The group name of it needs to exist as package list as well. OUTPUT also contains flags for the groups.
 
 We currently support:
  * default-support
- Sets the support level in case there is no explicitly entry in [supportstatus.txt](#supportstatus.txt), defaults to 'unsupported'
+ Sets the support level in case there is no explicitly entry in [supportstatus.txt](#supportstatustxt), defaults to 'unsupported'
  * recommends
  If the solver should take recommends into account when solving the package list, defaults to false.
  * includes
@@ -31,7 +31,7 @@ We currently support:
 
 Be aware that group names must not contain a '-'.
 
-You can also adapt the solving on a package level by putting a hash into the package list. Normally the package name is a string, in case it's a hash the key needs to be the package name and the value is a list of following modifiers: 
+You can also adapt the solving on a package level by putting a hash into the package list. Normally the package name is a string, in case it's a hash the key needs to be the package name and the value is a list of following modifiers:
 
  * recommended
  Evaluate also 'Recommends' in package to determine dependencies. Otherwise only 'required' are considered. Used mainly for patterns in SLE. It can not be combined with platforms, For architecture specific recommends, use patterns.
@@ -46,7 +46,7 @@ You can also adapt the solving on a package level by putting a hash into the pac
  * required
  If the package is missing or is uninstallable, don't leave a comment but put the error as package entry for OBS to create unresolvable error to avoid building a DVD
 
-Note that you can write yaml lists in 2 ways. You can put the modifier lists as multiple lines starting with -, but it's recommended to put them as [M1,M2] behind the package name. See the difference between pkg4 and pkg5 in the example. 
+Note that you can write yaml lists in 2 ways. You can put the modifier lists as multiple lines starting with -, but it's recommended to put them as [M1,M2] behind the package name. See the difference between pkg4 and pkg5 in the example.
 
 #### Example:
 
@@ -68,32 +68,32 @@ OUTPUT:
   - group3:
     includes:
     - list2
-    
+
 group1:
   - pkg1
-  
+
 group2:
   - pkg2: [locked]
   - pkg3
-  
+
 group3:
   - pkg4: [x86_64]
-  
+
 list1:
   - pkg5:
     - x86_64
-  
+
 list2:
   - pkg6: [recommended]
-``` 
+```
 
-## Overlap calculcation
- TODO 
+## Overlap calculation
+ TODO
 
 ## Handling in staging workflow
-If 000package-groups contains a file named summary-staging.txt, the bot will trigger a diff mode on staging projects. 
+If 000package-groups contains a file named summary-staging.txt, the bot will trigger a diff mode on staging projects.
 It will create an equal summary-staging.txt in 000product and create a comment with a human readable diff in the staging
 project. This comment can be replied to. If the reply starts with 'approve', staging accept will apply the diff to this
 txt file. If the reply starts with 'ignore', the bot will continue with the pipeline and do nothing on staging accept.
 
-This way simple changes to the summary can be accepted without a submit request (of which there can only be one at a time).
+This way simple changes to the summary can be accepted without a submit request (of which there can only be one at a time).

gocd/alp-stagings.gocd.yaml

@@ -0,0 +1,142 @@
+---
+format_version: 3
+pipelines:
+  ALP.Stagings.RelPkgs:
+    environment_variables:
+      OSC_CONFIG: /home/go/config/oscrc-staging-bot
+    group: LEO
+    lock_behavior: unlockWhenFinished
+    timer:
+      spec: 0 0 * ? * *
+      only_on_changes: false
+    materials:
+      scripts:
+        git: https://github.com/openSUSE/openSUSE-release-tools.git
+    stages:
+    - Generate.Release.Package:
+        approval: manual
+        jobs:
+          ALP.Staging.A:
+            resources:
+              - repo-checker
+            tasks:
+              - script: ./pkglistgen.py -A https://api.opensuse.org update_and_solve --staging SUSE:ALP:Staging:A --only-release-packages --force
+          ALP.Staging.B:
+            resources:
+              - repo-checker
+            tasks:
+              - script: ./pkglistgen.py -A https://api.opensuse.org update_and_solve --staging SUSE:ALP:Staging:B --only-release-packages --force
+
+  ALP.Staging.A:
+    environment_variables:
+      STAGING_PROJECT: SUSE:ALP:Staging:A
+      STAGING_API: https://api.opensuse.org
+      OSC_CONFIG: /home/go/config/oscrc-staging-bot
+    group: LEO
+    lock_behavior: unlockWhenFinished
+    materials:
+      scripts:
+        auto_update: true
+        git: https://github.com/openSUSE/openSUSE-release-tools.git
+        whitelist:
+          - DO_NOT_TRIGGER
+        destination: scripts
+    stages:
+    - Checks:
+        jobs:
+          Check.Build.Succeeds:
+            resources:
+              - staging-bot
+            tasks:
+              - script: |-
+                  export PYTHONPATH=$PWD/scripts
+                  cd scripts/gocd
+                  ./report-status.py -A $STAGING_API -p $STAGING_PROJECT -n packagelists -r standard -s pending
+                  ./verify-repo-built-successful.py -A $STAGING_API -p $STAGING_PROJECT -r standard
+          Repo.Checker:
+            environment_variables:
+              OSC_CONFIG: /home/go/config/oscrc-staging-bot
+            resources:
+              - repo-checker
+            tasks:
+              - script: |-
+                  ./scripts/staging-installcheck.py -A $STAGING_API -p SUSE:ALP -s $STAGING_PROJECT
+
+    - Update.000product:
+        resources:
+          - repo-checker
+        tasks:
+          - script: |-
+              export PYTHONPATH=$PWD/scripts
+              cd scripts/gocd
+
+              if ../pkglistgen.py --debug -A $STAGING_API update_and_solve --staging $STAGING_PROJECT --force; then
+                ./report-status.py -A $STAGING_API -p $STAGING_PROJECT -n packagelists -r standard -s success
+              else
+                ./report-status.py -A $STAGING_API -p $STAGING_PROJECT -n packagelists -r standard -s failure
+                exit 1
+              fi
+
+    - Enable.images.repo:
+       resources:
+         - staging-bot
+       tasks:
+         - script: |-
+             osc -A $STAGING_API api -X POST "/source/$STAGING_PROJECT?cmd=remove_flag&repository=images&flag=build"
+
+  ALP.Staging.B:
+    environment_variables:
+      STAGING_PROJECT: SUSE:ALP:Staging:B
+      STAGING_API: https://api.opensuse.org
+      OSC_CONFIG: /home/go/config/oscrc-staging-bot
+    group: LEO
+    lock_behavior: unlockWhenFinished
+    materials:
+      scripts:
+        auto_update: true
+        git: https://github.com/openSUSE/openSUSE-release-tools.git
+        whitelist:
+          - DO_NOT_TRIGGER
+        destination: scripts
+    stages:
+    - Checks:
+        jobs:
+          Check.Build.Succeeds:
+            resources:
+              - staging-bot
+            tasks:
+              - script: |-
+                  export PYTHONPATH=$PWD/scripts
+                  cd scripts/gocd
+                  ./report-status.py -A $STAGING_API -p $STAGING_PROJECT -n packagelists -r standard -s pending
+                  ./verify-repo-built-successful.py -A $STAGING_API -p $STAGING_PROJECT -r standard
+          Repo.Checker:
+            environment_variables:
+              OSC_CONFIG: /home/go/config/oscrc-staging-bot
+            resources:
+              - repo-checker
+            tasks:
+              - script: |-
+                  ./scripts/staging-installcheck.py -A $STAGING_API -p SUSE:ALP -s $STAGING_PROJECT
+
+    - Update.000product:
+        resources:
+          - repo-checker
+        tasks:
+          - script: |-
+              export PYTHONPATH=$PWD/scripts
+              cd scripts/gocd
+
+              if ../pkglistgen.py --debug -A $STAGING_API update_and_solve --staging $STAGING_PROJECT --force; then
+                ./report-status.py -A $STAGING_API -p $STAGING_PROJECT -n packagelists -r standard -s success
+              else
+                ./report-status.py -A $STAGING_API -p $STAGING_PROJECT -n packagelists -r standard -s failure
+                exit 1
+              fi
+
+    - Enable.images.repo:
+       resources:
+         - staging-bot
+       tasks:
+         - script: |-
+             osc -A $STAGING_API api -X POST "/source/$STAGING_PROJECT?cmd=remove_flag&repository=images&flag=build"

gocd/pkglistgen.opensuse.gocd.yaml

@@ -145,7 +145,28 @@ pipelines:
             tasks:
             - script: python3 ./skippkg-finder.py -A https://api.opensuse.org -o openSUSE:Leap:15.3:Update:Respin -t openSUSE:Leap:15.3:Update -s SUSE:SLE-15-SP3:Update
             - script: python3 ./pkglistgen.py -d -A https://api.opensuse.org update_and_solve -p openSUSE:Leap:15.3:Update:Respin -s target
-
+  Pkglistgen.openSUSE_Leap_15.4_CR:
+    group: Leap
+    lock_behavior: unlockWhenFinished
+    environment_variables:
+      OSC_CONFIG: /home/go/config/oscrc-staging-bot
+    timer:
+      spec: 0 40 * ? * *
+      only_on_changes: false
+    materials:
+      git:
+        git: https://github.com/openSUSE/openSUSE-release-tools.git
+    stages:
+    - pkglistgen:
+        approval:
+          type: manual
+        jobs:
+          openSUSE_Leap_15.4_Images_target:
+            resources:
+            - repo-checker
+            tasks:
+            - script: python3 ./skippkg-finder.py -A https://api.opensuse.org -o openSUSE:Leap:15.4:Images -t openSUSE:Leap:15.4:Update -s SUSE:SLE-15-SP4:Update
+            - script: python3 ./pkglistgen.py -d -A https://api.opensuse.org update_and_solve -p openSUSE:Leap:15.4:Images -s target
   Update.Repos.Leap.openSUSE_Leap_15.5:
     group: Leap
     lock_behavior: unlockWhenFinished
@@ -167,6 +188,27 @@ pipelines:
             - repo-checker
             tasks:
              - script: python3 ./pkglistgen.py --apiurl https://api.opensuse.org handle_update_repos openSUSE:Leap:15.5
+  Update.Repos.Leap.openSUSE_Leap_15.4_Images:
+    group: Leap
+    lock_behavior: unlockWhenFinished
+    environment_variables:
+      OSC_CONFIG: /home/go/config/oscrc-staging-bot
+    timer:
+      spec: 0 0 0 */3 * ?
+      only_on_changes: false
+    materials:
+      git:
+        git: https://github.com/openSUSE/openSUSE-release-tools.git
+    stages:
+    - Update:
+        approval:
+          type: manual
+        jobs:
+          openSUSE_Leap_15.4_Images:
+            resources:
+            - repo-checker
+            tasks:
+             - script: python3 ./pkglistgen.py --apiurl https://api.opensuse.org handle_update_repos openSUSE:Leap:15.4:Images
   Update.Repos.Leap.openSUSE_Leap_15.3_Update_Respin:
     group: Leap
     lock_behavior: unlockWhenFinished

gocd/pkglistgen.opensuse.gocd.yaml.erb

@@ -119,8 +119,38 @@ pipelines:
             - script: python3 ./skippkg-finder.py -A https://api.opensuse.org -o openSUSE:Leap:15.3:Update:Respin -t openSUSE:Leap:15.3:Update -s SUSE:SLE-15-SP3:Update
             - script: python3 ./pkglistgen.py -d -A https://api.opensuse.org update_and_solve -p <%= project[0] %><%= options %>
 <% end -%>
-
-<% %w(openSUSE:Leap:15.5 openSUSE:Leap:15.3:Update:Respin).each do |project| -%>
+  Pkglistgen.openSUSE_Leap_15.4_CR:
+    group: Leap
+    lock_behavior: unlockWhenFinished
+    environment_variables:
+      OSC_CONFIG: /home/go/config/oscrc-staging-bot
+    timer:
+      spec: 0 40 * ? * *
+      only_on_changes: false
+    materials:
+      git:
+        git: https://github.com/openSUSE/openSUSE-release-tools.git
+    stages:
+    - pkglistgen:
+        approval:
+          type: manual
+        jobs:
+<% ['openSUSE:Leap:15.4:Images/target'].each do |project|
+  project=project.split('/')
+  name=project[0].gsub(':', '_')
+  if project.size > 1
+    options=" -s #{project[1]}"
+    name = name + "_#{project[1]}"
+  end
+  -%>
+          <%= name %>:
+            resources:
+            - repo-checker
+            tasks:
+            - script: python3 ./skippkg-finder.py -A https://api.opensuse.org -o openSUSE:Leap:15.4:Images -t openSUSE:Leap:15.4:Update -s SUSE:SLE-15-SP4:Update
+            - script: python3 ./pkglistgen.py -d -A https://api.opensuse.org update_and_solve -p <%= project[0] %><%= options %>
+<% end -%>
+<% %w(openSUSE:Leap:15.5 openSUSE:Leap:15.4:Images openSUSE:Leap:15.3:Update:Respin).each do |project| -%>
   Update.Repos.Leap.<%= project.gsub(':', '_') %>:
     group: Leap
     lock_behavior: unlockWhenFinished