Skip to content

[RDY] buildLinux: Add more overrides #34351

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Ericson2314 merged 1 commit into from
Feb 6, 2018
Merged

[RDY] buildLinux: Add more overrides #34351

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions pkgs/os-specific/linux/kernel/common-config.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@

*/

{ stdenv, version, kernelPlatform, extraConfig, features }:
{ stdenv, version, extraConfig, features }:

with stdenv.lib;

Expand Down Expand Up @@ -682,6 +682,5 @@ with stdenv.lib;
WW_MUTEX_SELFTEST? n
''}

${kernelPlatform.kernelExtraConfig or ""}
${extraConfig}
''
14 changes: 7 additions & 7 deletions pkgs/os-specific/linux/kernel/generate-config.pl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,18 +13,18 @@
use IPC::Open2;
use Cwd;

my $wd = getcwd;

# exported via nix
my $debug = $ENV{'DEBUG'};
my $autoModules = $ENV{'AUTO_MODULES'};
my $preferBuiltin = $ENV{'PREFER_BUILTIN'};

my $ignoreConfigErrors = $ENV{'ignoreConfigErrors'};
my $buildRoot = $ENV{'BUILD_ROOT'};
$SIG{PIPE} = 'IGNORE';

# Read the answers.
my %answers;
my %requiredAnswers;
open ANSWERS, "<$ENV{KERNEL_CONFIG}" or die;
open ANSWERS, "<$ENV{KERNEL_CONFIG}" or die "Could not open answer file";
while (<ANSWERS>) {
chomp;
s/#.*//;
Expand All @@ -40,7 +40,7 @@
sub runConfig {

# Run `make config'.
my $pid = open2(\*IN, \*OUT, "make -C $ENV{SRC} O=$wd config SHELL=bash ARCH=$ENV{ARCH}");
my $pid = open2(\*IN, \*OUT, "make -C $ENV{SRC} O=$buildRoot config SHELL=bash ARCH=$ENV{ARCH}");

# Parse the output, look for questions and then send an
# appropriate answer.
Expand Down Expand Up @@ -122,7 +122,7 @@ sub runConfig {
# there. `make config' often overrides answers if later questions
# cause options to be selected.
my %config;
open CONFIG, "<.config" or die;
open CONFIG, "<$buildRoot/.config" or die "Could not read .config";
while (<CONFIG>) {
chomp;
if (/^CONFIG_([A-Za-z0-9_]+)="(.*)"$/) {
Expand All @@ -137,7 +137,7 @@ sub runConfig {
close CONFIG;

foreach my $name (sort (keys %answers)) {
my $f = $requiredAnswers{$name} && $ENV{'ignoreConfigErrors'} ne "1"
my $f = $requiredAnswers{$name} && $ignoreConfigErrors ne "1"
? sub { die "error: " . $_[0]; } : sub { warn "warning: " . $_[0]; };
&$f("unused option: $name\n") unless defined $config{$name};
&$f("option not set correctly: $name (wanted '$answers{$name}', got '$config{$name}')\n")
Expand Down
34 changes: 23 additions & 11 deletions pkgs/os-specific/linux/kernel/generic.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,11 @@
{ buildPackages, runCommand, nettools, bc, perl, gmp, libmpc, mpfr, openssl
, ncurses
, libelf
, utillinux
, writeTextFile, ubootTools
, callPackage
}:

Copy link
Member

@vcunat vcunat Feb 10, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand the intention here. Most of the parameters are unused, and having buildPackages (and others) twice is a bit confusing...

Copy link
Member

@Ericson2314 Ericson2314 Feb 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good point. Sorry I missed this.

Copy link
Member

@Ericson2314 Ericson2314 Feb 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see this PR as setting up the inlining of manual-config.nix into generic.nix. we can probably kill many callPackages with that.

Copy link
Member Author

@teto teto Feb 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I copy pasted from manual-config.nix and forgot to trim it down. Will do in #34351 (comment) sorry

Copy link
Member

@vcunat vcunat Feb 11, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When doing this, we might also clean up the parameters in <nixpkgs/pkgs/os-specific/linux/kernel/linux-*.nix>.

{ stdenv, buildPackages, perl, buildLinux

, # The kernel source tarball.
Expand Down Expand Up @@ -28,7 +36,7 @@
, extraMeta ? {}
, hostPlatform
, ...
}:
} @ args:

assert stdenv.isLinux;

Expand All @@ -45,8 +53,10 @@ let
} // features) kernelPatches;

config = import ./common-config.nix {
inherit stdenv version extraConfig;
kernelPlatform = hostPlatform;
inherit stdenv version ;
# append extraConfig for backwards compatibility but also means the user can't override the kernelExtraConfig part
extraConfig = extraConfig + lib.optionalString (hostPlatform ? kernelExtraConfig ) hostPlatform.kernelExtraConfig;
Copy link
Member

@Ericson2314 Ericson2314 Feb 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@teto great change, I didn't even notice this before :). kernelExtraConfig isn't used anywhere, so let's definitely get rid of that!

Copy link
Member Author

@teto teto Feb 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean moving kernelExtraConfig from platforms.* to the buildLinux calls ?

Copy link
Member

@Ericson2314 Ericson2314 Feb 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh wait, my grep was bad, nevermind.


features = kernelFeatures; # Ensure we know of all extra patches, etc.
};

Expand All @@ -68,7 +78,9 @@ let
nativeBuildInputs = [ perl ];

platformName = hostPlatform.platform.name;
# e.g. "defconfig"
kernelBaseConfig = hostPlatform.platform.kernelBaseConfig;
# e.g. "bzImage"
kernelTarget = hostPlatform.platform.kernelTarget;
autoModules = hostPlatform.platform.kernelAutoModules;
preferBuiltin = hostPlatform.platform.kernelPreferBuiltin or false;
Expand All @@ -83,25 +95,25 @@ let
inherit (kernel) src patches preUnpack;

buildPhase = ''
cd $buildRoot
export buildRoot="''${buildRoot:-build}"

# Get a basic config file for later refinement with $generateConfig.
make HOSTCC=${buildPackages.stdenv.cc.targetPrefix}gcc -C ../$sourceRoot O=$PWD $kernelBaseConfig ARCH=$arch
make HOSTCC=${buildPackages.stdenv.cc.targetPrefix}gcc -C . O="$buildRoot" $kernelBaseConfig ARCH=$arch

# Create the config file.
echo "generating kernel configuration..."
echo "$kernelConfig" > kernel-config
DEBUG=1 ARCH=$arch KERNEL_CONFIG=kernel-config AUTO_MODULES=$autoModules \
PREFER_BUILTIN=$preferBuiltin SRC=../$sourceRoot perl -w $generateConfig
echo "$kernelConfig" > "$buildRoot/kernel-config"
DEBUG=1 ARCH=$arch KERNEL_CONFIG="$buildRoot/kernel-config" AUTO_MODULES=$autoModules \
PREFER_BUILTIN=$preferBuiltin BUILD_ROOT="$buildRoot" SRC=. perl -w $generateConfig
'';

installPhase = "mv .config $out";
installPhase = "mv $buildRoot/.config $out";

enableParallelBuilding = true;
};

kernel = buildLinux {
inherit version modDirVersion src kernelPatches stdenv extraMeta configfile;
kernel = (callPackage ./manual-config.nix {}) {
inherit version modDirVersion src kernelPatches stdenv extraMeta configfile hostPlatform;

config = { CONFIG_MODULES = "y"; CONFIG_FW_LOADER = "m"; };
};
Expand Down
2 changes: 1 addition & 1 deletion pkgs/os-specific/linux/kernel/linux-4.13.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{ stdenv, buildPackages, hostPlatform, fetchurl, perl, buildLinux, ... } @ args:

import ./generic.nix (args // rec {
buildLinux (args // rec {
version = "4.13.16";
extraMeta.branch = "4.13";

Expand Down
2 changes: 1 addition & 1 deletion pkgs/os-specific/linux/kernel/linux-4.14.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

with stdenv.lib;

import ./generic.nix (args // rec {
buildLinux (args // rec {
version = "4.14.17";

# branchVersion needs to be x.y
Expand Down
2 changes: 1 addition & 1 deletion pkgs/os-specific/linux/kernel/linux-4.15.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

with stdenv.lib;

import ./generic.nix (args // rec {
buildLinux (args // rec {
version = "4.15.1";

# modDirVersion needs to be x.y.z, will automatically add .0 if needed
Expand Down
2 changes: 1 addition & 1 deletion pkgs/os-specific/linux/kernel/linux-4.4.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{ stdenv, buildPackages, hostPlatform, fetchurl, perl, buildLinux, ... } @ args:

import ./generic.nix (args // rec {
buildLinux (args // rec {
version = "4.4.115";
extraMeta.branch = "4.4";

Expand Down
2 changes: 1 addition & 1 deletion pkgs/os-specific/linux/kernel/linux-4.9.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{ stdenv, buildPackages, hostPlatform, fetchurl, perl, buildLinux, ... } @ args:

import ./generic.nix (args // rec {
buildLinux (args // rec {
version = "4.9.80";
extraMeta.branch = "4.9";

Expand Down
2 changes: 1 addition & 1 deletion pkgs/os-specific/linux/kernel/linux-beagleboard.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ let
modDirVersion = "4.14.12";
tag = "r23";
in
stdenv.lib.overrideDerivation (import ./generic.nix (args // rec {
stdenv.lib.overrideDerivation (buildLinux (args // rec {
version = "${modDirVersion}-ti-${tag}";
inherit modDirVersion;

Expand Down
2 changes: 1 addition & 1 deletion pkgs/os-specific/linux/kernel/linux-hardened-copperhead.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ let

modDirVersion = "${modVersion}-hardened";
in
import ./generic.nix (args // {
buildLinux (args // {
inherit modDirVersion;

version = "${version}-${revision}";
Expand Down
5 changes: 3 additions & 2 deletions pkgs/os-specific/linux/kernel/linux-mptcp.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
{ stdenv, buildPackages, hostPlatform, fetchFromGitHub, perl, buildLinux, ... } @ args:

import ./generic.nix (rec {
buildLinux (rec {
mptcpVersion = "0.93";
modDirVersion = "4.9.60";
version = "${modDirVersion}-mptcp_v${mptcpVersion}";
# autoModules= true;

extraMeta = {
branch = "4.4";
Expand Down Expand Up @@ -43,4 +44,4 @@ import ./generic.nix (rec {
TCP_CONG_BALIA m

'' + (args.extraConfig or "");
} // args // (args.argsOverride or {}))
} // args)
2 changes: 1 addition & 1 deletion pkgs/os-specific/linux/kernel/linux-rpi.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ let
modDirVersion = "4.9.59";
tag = "1.20171029";
in
stdenv.lib.overrideDerivation (import ./generic.nix (args // rec {
stdenv.lib.overrideDerivation (buildLinux (args // rec {
version = "${modDirVersion}-${tag}";
inherit modDirVersion;

Expand Down
2 changes: 1 addition & 1 deletion pkgs/os-specific/linux/kernel/linux-samus-4.12.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{ stdenv, buildPackages, hostPlatform, fetchFromGitHub, perl, buildLinux, ncurses, ... } @ args:

import ./generic.nix (args // rec {
buildLinux (args // rec {
version = "4.12.2";
extraMeta.branch = "4.12-2";

Expand Down
2 changes: 1 addition & 1 deletion pkgs/os-specific/linux/kernel/linux-testing-bcachefs.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{ stdenv, buildPackages, hostPlatform, fetchgit, perl, buildLinux, ... } @ args:

import ./generic.nix (args // rec {
buildLinux (args // rec {
version = "4.11.2017.08.23";
modDirVersion = "4.11.0";
extraMeta.branch = "master";
Expand Down
2 changes: 1 addition & 1 deletion pkgs/os-specific/linux/kernel/linux-testing.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{ stdenv, buildPackages, hostPlatform, fetchurl, perl, buildLinux, libelf, utillinux, ... } @ args:

import ./generic.nix (args // rec {
buildLinux (args // rec {
version = "4.15-rc9";
modDirVersion = "4.15.0-rc9";
extraMeta.branch = "4.15";
Expand Down
34 changes: 27 additions & 7 deletions pkgs/os-specific/linux/kernel/manual-config.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
{ buildPackages, runCommand, nettools, bc, perl, gmp, libmpc, mpfr, openssl
, ncurses ? null
, libelf
, utillinux
, writeTextFile, ubootTools
, hostPlatform
}:

let
Expand Down Expand Up @@ -34,7 +34,9 @@ in {
# Use defaultMeta // extraMeta
extraMeta ? {},
# Whether to utilize the controversial import-from-derivation feature to parse the config
allowImportFromDerivation ? false
allowImportFromDerivation ? false,

hostPlatform
}:

let
Expand Down Expand Up @@ -86,8 +88,6 @@ let
inherit src;

preUnpack = ''
mkdir build
export buildRoot="$(pwd)/build"
'';

patches = map (p: p.patch) kernelPatches;
Expand All @@ -102,7 +102,25 @@ let

configurePhase = ''
runHook preConfigure

mkdir build
export buildRoot="$(pwd)/build"

echo "manual-config configurePhase buildRoot=$buildRoot pwd=$PWD"

if [[ -z "$buildRoot" || ! -d "$buildRoot" ]]; then
echo "set $buildRoot to the build folder please"
exit 1
fi

if [ -f "$buildRoot/.config" ]; then
echo "Could not link $buildRoot/.config : file exists"
exit 1
fi
ln -sv ${configfile} $buildRoot/.config

# reads the existing .config file and prompts the user for options in
# the current kernel source that are not found in the file.
make $makeFlags "''${makeFlagsArray[@]}" oldconfig
runHook postConfigure

Expand All @@ -115,6 +133,8 @@ let

# Note: we can get rid of this once http://permalink.gmane.org/gmane.linux.kbuild.devel/13800 is merged.
buildFlagsArray+=("KBUILD_BUILD_TIMESTAMP=$(date -u -d @$SOURCE_DATE_EPOCH)")

cd $buildRoot
'';

buildFlags = [
Expand All @@ -136,7 +156,7 @@ let

postInstall = ''
mkdir -p $dev
cp $buildRoot/vmlinux $dev/
cp vmlinux $dev/
'' + (optionalString installsFirmware ''
mkdir -p $out/lib/firmware
'') + (if (platform ? kernelDTB && platform.kernelDTB) then ''
Expand All @@ -151,7 +171,7 @@ let
unlink $out/lib/modules/${modDirVersion}/source

mkdir -p $dev/lib/modules/${modDirVersion}/build
cp -dpR ../$sourceRoot $dev/lib/modules/${modDirVersion}/source
cp -dpR .. $dev/lib/modules/${modDirVersion}/source
Copy link
Contributor

@dezgeg dezgeg Jan 31, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm. Does this now copy way too much to $dev/lib/modules/${modDirVersion}/source?

Copy link
Member Author

@teto teto Jan 31, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a problem I realized while testing with nox-review, I moved the build folder to within source (aka someFolder/source/build) so that it looks more like other packages (cmake-based etc). It used to have this architecture:
someFolder
|--build
|--source
The change just accounts for this folder move.

To sum up, it should be exactly the same as before. my nox-review has been running for a day but it still hasn't met any error so I without further comments I consider the PR ready.

Copy link
Member Author

@teto teto Jan 31, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nevermind looks like I haven't pushed the latest changes. Need to rebase with the new kernel updates on master too :'(

cd $dev/lib/modules/${modDirVersion}/source

cp $buildRoot/{.config,Module.symvers} $dev/lib/modules/${modDirVersion}/build
Expand All @@ -170,7 +190,7 @@ let
# from drivers/ in the future; it adds 50M to keep all of its
# headers on 3.10 though.

chmod u+w -R ../source
chmod u+w -R ..
arch=$(cd $dev/lib/modules/${modDirVersion}/build/arch; ls)

# Remove unused arches
Expand Down
6 changes: 3 additions & 3 deletions pkgs/os-specific/linux/kernel/update.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,13 +50,13 @@ ls $NIXPKGS/pkgs/os-specific/linux/kernel | while read FILE; do
# Rewrite the expression
sed -i -e '/version = /d' -e '/modDirVersion = /d' $NIXPKGS/pkgs/os-specific/linux/kernel/$FILE
if grep -q '^[0-9]\+.[0-9]\+$' <<< "$V"; then
sed -i "\#import ./generic.nix (args // rec {#a \ modDirVersion = \"${V}.0\";" $NIXPKGS/pkgs/os-specific/linux/kernel/$FILE
sed -i "\#buildLinux (args // rec {#a \ modDirVersion = \"${V}.0\";" $NIXPKGS/pkgs/os-specific/linux/kernel/$FILE
fi
sed -i "\#import ./generic.nix (args // rec {#a \ version = \"$V\";" $NIXPKGS/pkgs/os-specific/linux/kernel/$FILE
sed -i "\#buildLinux (args // rec {#a \ version = \"$V\";" $NIXPKGS/pkgs/os-specific/linux/kernel/$FILE

# Commit the changes
git add -u $NIXPKGS/pkgs/os-specific/linux/kernel/$FILE
git commit -m "kernel: $OLDVER -> $V" >/dev/null 2>&1

echo "Updated $OLDVER -> $V"
done
2 changes: 1 addition & 1 deletion pkgs/top-level/all-packages.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13197,7 +13197,7 @@ with pkgs;

# A function to build a manually-configured kernel
linuxManualConfig = pkgs.buildLinux;
buildLinux = makeOverridable (callPackage ../os-specific/linux/kernel/manual-config.nix {});
buildLinux = makeOverridable (callPackage ../os-specific/linux/kernel/generic.nix {});

keyutils = callPackage ../os-specific/linux/keyutils { };

Expand Down