fix(build): The number of build jobs should depend on RAM size#1939
Merged
fix(build): The number of build jobs should depend on RAM size#1939
Conversation
Sometimes the build might fail in OBS because Rust compilation requires huge amount of RAM. The problem happens when running many parallel jobs on a machine with not enough RAM. The build on S390 failed when running 8 jobs with 8GB RAM, so the compilation definitely needs more than 1GB per job. My experimental run `/usr/bin/time -v cargo build -j1` showed maximum used memory 1.1GB on x86_64. To be on the safe side let's require 1.3GB per job. (Different architectures might require more and we need to leave something for the system.) If some architecture needs quite different amount of RAM we can make the setting arch dependent using the `%ifarch` macro.
imobachgs
reviewed
Jan 23, 2025
Contributor
imobachgs
left a comment
There was a problem hiding this comment.
It looks like black magic to me :-) No, seriously, it looks good. I tried the logic on my own machine.
Thanks!
imobachgs
approved these changes
Jan 23, 2025
jreidinger
approved these changes
Jan 23, 2025
lslezak
added a commit
that referenced
this pull request
Jan 24, 2025
## Problem - Related to #1939 - Unfortunately SLE16 uses old rpm version 4.18 which does not support `%{getncpus:proc}` (added in 4.19). - We need to compute the value manually. :-/ ## Details - I wanted to use some `%if` and use the new option in TW. However the macro exists in both but in SLE16 it complains about extra argument which is not accepted. And rpm evaluates macros in both `%if` branches... 😟 - So I basically reverted to my original implementation
Merged
imobachgs
added a commit
that referenced
this pull request
Feb 26, 2025
Prepare for releasing Agama 12: * #1858 * #1887 * #1890 * #1892 * #1893 * #1894 * #1896 * #1898 * #1899 * #1900 * #1901 * #1906 * #1908 * #1909 * #1910 * #1911 * #1912 * #1914 * #1915 * #1917 * #1919 * #1920 * #1921 * #1922 * #1923 * #1924 * #1926 * #1927 * #1928 * #1929 * #1930 * #1931 * #1932 * #1933 * #1934 * #1935 * #1936 * #1937 * #1938 * #1939 * #1942 * #1943 * #1945 * #1948 * #1949 * #1952 * #1953 * #1954 * #1955 * #1957 * #1958 * #1959 * #1961 * #1963 * #1964 * #1967 * #1969 * #1970 * #1971 * #1972 * #1973 * #1974 * #1975 * #1976 * #1977 * #1979 * #1980 * #1981 * #1982 * #1984 * #1986 * #1987 * #1988 * #1990 * #1991 * #1992 * #1993 * #1995 * #1996 * #1997 * #1999 * #2000 * #2001 * #2002 * #2003 * #2004 * #2005 * #2006 * #2007 * #2008 * #2009 * #2010 * #2011 * #2012 * #2013 * #2014 * #2015 * #2016 * #2017 * #2019 * #2021 * #2022 * #2025 * #2027 * #2029 * #2030 * #2031 * #2033 * #2034 * #2035 * #2036 * #2037 * #2038 * #2039 * #2040 * #2045 * #2046 * #2050 * #2053 * #2054 * #2055 * #2056 * #2058 * #2060 * #2061 * #2062 * #2063 * #2064 * #2066 * #2067 * #2068 * #2069 * #2070 * #2071 * #2072 * #2073 * #2075 * #2076 * #2080 * #2082 * #2083
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Details
Sometimes the build might fail in OBS because Rust compilation requires huge amount of RAM. The problem happens when running many parallel jobs on a machine with not enough RAM. The build on S390 failed when running 8 jobs with 8GB RAM.
Originally I wanted to increase the requirement for RAM in the
_constraintsfile from 8GB to 16GB. But that would decrease the number of available workers significantly, esp. on exotic archs like s390. And in most cases the workers run 4 or 8 jobs so requiring 16GB would be an overkill.Solution
So to decrease the needed amount of RAM decrease the maximum number of parallel jobs.
My experimental run
/usr/bin/time -v cargo build -j1showed maximum used memory 1.1GB on x86_64. The compilation on S390 failed with 1GB per job. To be on the safe side let's require 1.3GB per job, different architectures might require more and we need to also leave something for the system and other services.That means with 8GB RAM it should run at most 6 parallel jobs. That should hopefully avoid triggering the OOM killer.
Notes
%ifarchmacro.