Rebuildibility poc #2623

jreidinger · 2021-08-19T15:07:28Z

draft to have a tool to test a various approaches for detecting rebuildability issues.

Still WIP, but feel free to comment approach.

codecov-commenter · 2021-08-19T16:12:26Z

Codecov Report

Merging #2623 (524e70f) into master (b133ea6) will decrease coverage by 0.67%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master    #2623      +/-   ##
==========================================
- Coverage   25.20%   24.52%   -0.68%     
==========================================
  Files          91       96       +5     
  Lines       15156    15574     +418     
==========================================
  Hits         3820     3820              
- Misses      11336    11754     +418

Impacted Files	Coverage Δ
osclib/dependency.py	`0.00% <0.00%> (ø)`
osclib/remote_package.py	`0.00% <0.00%> (ø)`
osclib/remote_project.py	`0.00% <0.00%> (ø)`
rebuildability_check.py	`0.00% <0.00%> (ø)`
skippkg-finder.py	`0.00% <0.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b133ea6...524e70f. Read the comment docs.

coolo · 2021-08-21T06:52:37Z

osclib/remote_project.py

+        cleaning_regex = re.compile(r'\.[0-9]+$')
+        for project in linked_projects:
+            for pkg in project.get_packages(inherited = False):
+                # TODO: is there better way then regex?


Note that ruby2.5 also matches your regexp but is not maintenance specific, so relying on regexp is basically a no go. /source/SUSE:SLE-15-SP2:Update/zypper.15590/_meta will tell you <releasename>zypper</releasename>.

Good catch with that ruby...in general I a bit worry about performance as when I check SLE15-SP4 there was a lot of updates in update channel, but lets not preoptimalize in advance.

coolo · 2021-08-21T06:53:08Z

osclib/remote_package.py

+            return cls.from_xml(osc.core.http_GET(url))
+        except HTTPError as e:
+            if e.code == 404:
+                raise PackageNotFound('Package %s not found' % (package_name))


you need to reraise in else - or you will get very strange results

coolo · 2021-08-21T06:56:51Z

rebuildability_check.py

+class RebuildabilityChecker(object):
+    def __init__(self, project_str):
+        self.logger = logging.getLogger('RebuildibilityChecker')
+        self.project = RemoteProject.find(project_str) # apiurl should be read from osc.conf.config['apiurl'], osclib config class looks like has different goal?


that might be just me but could you limit your line length to a value that makes reading the comments possible without scrolling left and right on github page? That would be awesome

goal of this comments is mainly discussion about code where we cooperate. It should not be there before we merge it.
But thanks for catch

coolo · 2021-08-21T06:59:44Z

osclib/remote_package.py

+
+    def copy(self, target_project_name, expand=False):
+        apiurl = osc.conf.config['apiurl']
+        osc.core.copy_pac(apiurl, self.project_name, self.name, apiurl, target_project_name, self.name, expand=expand)


note that copy_pac is greatly expensive operating. For such use cases a revisioned _link is much clearly preferred.

yeah, I expect it is expensive. I just want to avoid any diff issue. Revisioned _link can be solution as I expect in SLE15 it won't change much. If we want to use it on staging it can be more tricky as staging change more frequently and link can be broken. ( question is how big issue it is as it means that it is removed from staging )

if you plan to rebuild all of SLE15 history against a moving target you want to go back to the drawing board anyway.

coolo · 2021-08-21T07:01:15Z

rebuildability_check.py

+        rebuild_project = self.project.create_subproject("Rebuild", title, description) # how to handle if it exists? Clean it and use? What if we do not have permission to create subproject?
+
+        testing_packages = [package.copy(rebuild_project.name, expand=True) for package in packages]
+        while not all([p.builds.is_finished for p in testing_packages]): # builds is object for handling building and is_finished means that all builds are finished ( or disabled )


I assume this is supposed to read as pseudo code? Because you don't need to query all packages' build status just to find out that one is not finished (taking that is_finished is going to be an external API call).

yeah, we decide to start with pseudo code how the API can look like and adapt it as we implement it. Maybe we can even find some project specific query to find unfinished build.

coolo · 2021-08-21T07:07:11Z

rebuildability_check.py

+            # write some progress about number of pass, failures, what is waiting, etc. using p.builds query ( ensure it is not memoized )
+            print("Working like a crazy monk")
+
+        # TODO: delete rebuild project or keep it for inspection?


in principle: all succeeded packages are normally uninteresting. The failed ones you might need to show around. Not sure the yast team ever accepted a "it failed, fix it - that's all I have!" :)

But depending on how often you plan to run this, you surely do not want to have a lot of these projects around as they fill up several hundered GB easily. So reusing them can be key.

Maybe it can be even parameter specific e.g. for SLE-15-SP4 I expect we want to keep it and just reuse project when we want another rebuild test. For some other usage like maintenance update, different behavior can make sense.

Before you discuss such things, learn how long it takes to create and recompile a single project. So before you start to worry about reports and cleanup, just create one of these beasts

yep, that is a goal of this tool, to start playing with it and as side-effect create a bit of high level API that helps with whatever we decided to do.

coolo · 2021-08-23T15:31:16Z

rebuildability_check.py

@@ -23,7 +23,7 @@ def result(self):
        description = "Temporary project including expanded copy of parent to verify if everything can be rebuild from scratch"
        rebuild_project = self.project.create_subproject("Rebuild", title, description) # how to handle if it exists? Clean it and use? What if we do not have permission to create subproject?

-        testing_packages = [package.copy(rebuild_project.name, expand=True) for package in packages]
+        testing_packages = [package.link(rebuild_project.name) for package in packages]


note that only works for truely static targets. In case of update projects, you really want to have a revision to your link. Like osc linkpac --current does

jreidinger · 2021-08-24T05:06:20Z

OK, so first run finished. It uses code after commit ede0f67
repo is at https://build.suse.de/project/show/home:jreidinger:SP4-rebuild-testing:Rebuild
runtime for creating such repo is

real	231m10.171s
user	11m26.858s
sys	1m36.694s

building of such repo started 23.08. at 22:00

coolo · 2021-08-24T07:15:37Z

On the project page, click on the repo name, ignore the disgusting amount of build cyles and look for "Check the rebuild time for x86_64"

jreidinger · 2021-08-24T07:15:59Z

thanks

imobachgs · 2021-08-24T07:52:11Z

What we did in the past: Query /source/?view=info and then compare <filename> with package name

Thanks! I found out exactly that query last night checking the OBS code 😄 because I felt that things were too complicated.

* Extract the logic to read the packages to a separate class, so it is easy to change the overall approach if needed in the future.

coolo · 2021-08-26T10:54:34Z

So your branch eventually finished, any idea what to do with the result?

Use the /source/:project endpoint to get the list of packages

jreidinger · 2021-08-26T13:34:02Z

@coolo we see it together with a bunch of failures and unresolvable stuff. For multi choice we need to fix the script to copy source Prefer configuration. For rest we need to try to find if it is valid or issue with our approach. And then we probably need to talk with release manages/PMs what to do with it. And meanwhile we are preparing alternative approaches that takes less time ( and probably find less issues as it cover limited set of packages ).

coolo · 2021-08-26T13:49:05Z

What prefers do you want to copy? You build against SP3 and all the project configs are inherited. The only thing you would need to copy is if you want to compile sp4 sources against sp3. But in sp4 project configs are no prefers that would fix your issues.

But your project checks if you can rebuild packages against other packages you rebuild, i.e. bootstrap the merged set. I don't think that's what #2581 asks for. And some of the unresolvables and failures might actually inherit from that.

jreidinger · 2021-08-27T20:44:59Z

@coolo interesting note about bootstrapping, if it is really not goal ( which I am not aware, as I thought that e.g. for embedded stuff it is needed ) then the other two options which script support can be interesting. The first one is just to provide list of packages to try to rebuild ( plan is to provide also possibility to specify file path instead of comma separated list ) or provide package and compute what it can break in build dependencies ( e.g. I test it with yast2-core which cause rebuild of all yast2 packages which is correct ). I hope that the second option with build dependencies can be something that can address #2581

jreidinger · 2021-08-27T20:45:45Z

BTW usage to get what needs to check rebuild when new glibc is introduced should be python3 rebuildability_check.py --project SUSE:SLE-15-SP4:GA --apiurl https://api.suse.de --packages=glibc --dependencies=standard --dry-run but it now fails for me ( need to debug it on monday ).

coolo · 2021-08-28T05:52:52Z

Bootstrap is indeed interesting for embedded but for a fraction of the package. But it's impossible if the package sources aren't rebuildable to begin with - and that's what the issue asks for IMO.

jreidinger · 2021-08-29T19:51:06Z

OK, I fix issue with getting proper source project name and now even more interesting issue appear. Build info for update project is empty. No clue why. Will ask on monday on build service chat - https://api.suse.de/build/SUSE:SLE-15-SP3:Update/standard/x86_64/_builddepinfo?view=revpkgnames

coolo · 2021-08-30T05:09:24Z

https://chat.suse.de/channel/help-obs?msg=MqjFS46pCdKJd7LHm

coolo · 2022-02-15T12:44:05Z

Closing the PR - leaving the branch

jreidinger and others added 4 commits August 19, 2021 12:19

add initial idea for code (non-working)

7707cb3

add initial remote project class

83ab175

fix issues found by flaker

6fbcd32

Add a minimal RemotePackage class

45be3e0

jreidinger added 7 commits August 19, 2021 20:52

add helper to list all linked projects

783d253

fix indentation

d78aa31

implement creation of subprojects

16afa9d

implement listing packages

1b42f93

filter out maintenance specific names

8be5a79

fix flake warnings

f5bbfad

implement copy of package

0cecd6e

coolo reviewed Aug 21, 2021

View reviewed changes

imobachgs added 7 commits August 23, 2021 11:47

Extend RemotePackage with more meta data

ca9f8b9

RemotePackage uses 'metadata' for consistency

a1e00ed

Use RemotePackage release name instead of a regexp

7245fcc

Add a RemotePackage#releasename method

692faff

Use RemotePackage#releasename method

bb8347f

Find remote package metadata on-demand

1a19489

Link instead of copying packages during rebuildability check

9bb9dcf

coolo reviewed Aug 23, 2021

View reviewed changes

imobachgs and others added 3 commits August 23, 2021 16:35

RemotePackage#link_pac links against latest rev

fe3a562

optimize getting list of packages

cb3d8ab

fix release name to never return None

ede0f67

jreidinger added 2 commits August 24, 2021 10:20

fixes from review and linter

a96f2b7

fix typo (TODO: rebase whole branch)

2e74ef9

jreidinger mentioned this pull request Aug 24, 2021

skippkg-finder: overwrites NON_FTP_PACKAGES.group for obsoleted/unneeded package from SLE or openSUSE #2625

Merged

jreidinger and others added 3 commits August 24, 2021 16:15

add option for computing dependencies (WIP)

eeaf04e

Use the /source/:project endpoint to get the list of packages

7779e5f

* Extract the logic to read the packages to a separate class, so it is easy to change the overall approach if needed in the future.

Update from code review

3e56c5f

imobachgs added 5 commits August 26, 2021 13:49

Link the package from the proper project

d105770

Fix incident filtering

1fa580e

Update the list of packages with dots to ignore

7be0ca2

Remove outdated code

a9bd926

Merge pull request #2627 from imobachgs/packages-reader

dc186ff

Use the /source/:project endpoint to get the list of packages

jreidinger added 2 commits August 27, 2021 21:21

Implement dry-run and dependencies for packages

283cce7

fix linters

97c55e1

fix project name

e21c9e9

jreidinger added 2 commits September 6, 2021 23:27

add option to specify packages as list

ab1c8c0

report if packages not found

524e70f

coolo closed this Feb 15, 2022

coolo deleted the rebuildibility_POC branch November 11, 2022 09:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rebuildibility poc #2623

Rebuildibility poc #2623

jreidinger commented Aug 19, 2021

codecov-commenter commented Aug 19, 2021 •

edited

Loading

coolo Aug 21, 2021

jreidinger Aug 23, 2021

coolo Aug 21, 2021

coolo Aug 21, 2021

jreidinger Aug 23, 2021

coolo Aug 21, 2021

jreidinger Aug 23, 2021

coolo Aug 23, 2021

coolo Aug 21, 2021

jreidinger Aug 23, 2021

coolo Aug 21, 2021

jreidinger Aug 23, 2021

coolo Aug 23, 2021

jreidinger Aug 23, 2021

coolo Aug 23, 2021

jreidinger commented Aug 24, 2021

coolo commented Aug 24, 2021

jreidinger commented Aug 24, 2021

imobachgs commented Aug 24, 2021

coolo commented Aug 26, 2021

jreidinger commented Aug 26, 2021

coolo commented Aug 26, 2021

jreidinger commented Aug 27, 2021

jreidinger commented Aug 27, 2021

coolo commented Aug 28, 2021

jreidinger commented Aug 29, 2021

coolo commented Aug 30, 2021

coolo commented Feb 15, 2022

Rebuildibility poc #2623

Rebuildibility poc #2623

Conversation

jreidinger commented Aug 19, 2021

codecov-commenter commented Aug 19, 2021 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreidinger commented Aug 24, 2021

coolo commented Aug 24, 2021

jreidinger commented Aug 24, 2021

imobachgs commented Aug 24, 2021

coolo commented Aug 26, 2021

jreidinger commented Aug 26, 2021

coolo commented Aug 26, 2021

jreidinger commented Aug 27, 2021

jreidinger commented Aug 27, 2021

coolo commented Aug 28, 2021

jreidinger commented Aug 29, 2021

coolo commented Aug 30, 2021

coolo commented Feb 15, 2022

codecov-commenter commented Aug 19, 2021 •

edited

Loading