From 90f214db3edc3ff2ebae12038e2c7517729a480e Mon Sep 17 00:00:00 2001 From: Robert Scott Date: Sun, 14 Mar 2021 14:55:54 +0000 Subject: [PATCH 1/6] create 0089-collect-non-source-package-meta.md --- rfcs/0089-collect-non-source-package-meta.md | 103 +++++++++++++++++++ 1 file changed, 103 insertions(+) create mode 100644 rfcs/0089-collect-non-source-package-meta.md diff --git a/rfcs/0089-collect-non-source-package-meta.md b/rfcs/0089-collect-non-source-package-meta.md new file mode 100644 index 000000000..bec74be72 --- /dev/null +++ b/rfcs/0089-collect-non-source-package-meta.md @@ -0,0 +1,103 @@ +--- +feature: collect-non-source-package-meta +start-date: 2021-03-14 +author: Robert Scott +co-authors: (find a buddy later to help out with the RFC) +shepherd-team: (names, to be nominated and accepted by RFC steering committee) +shepherd-leader: (name to be appointed by RFC steering committee) +related-issues: (will contain links to implementation PRs) +--- + +# Summary +[summary]: #summary + +Collect and maintain a new `meta` attribute in packages allowing users to easily +identify and manage their preference for binary (more broadly "non-source") +packages. + +# Motivation +[motivation]: #motivation + +Different users have different expectations from a software distribution. We +acknowledge that much with the collection of license information and the +existence of the `allowUnfree` nixpkgs option, much as Debian maintains a +separate `-nonfree` repository. + +Similarly, there are a number of different reasons users may have to disfavour +those packages not built-from-source: + +- Transparency: an ever-growing concern with more focus than ever on + supply-chain attacks. +- Malleability: being able to conveniently override packages with patches or an + altered build process is a key advantage of Nix, and for nixpkgs maintainers + it's not generally possible to backport security fixes to binary packages. + +For some users, these concerns are enough to deter them from using Nix entirely. + +# Detailed design +[design]: #detailed-design + +Add a new `meta` attribute to non-source-built packages, `fromSource = false`. +Leave other packages as-is with the assumption of a missing attribute meaning +`true`. + +Add a mechanism to allow `.nixpkgs/config.nix` to specify +`allowNonSource = false` to prevent use of these packages in a similar manner +to `allowUnfree`. + +# Alternatives +[alternatives]: #alternatives + +I might have been tempted to collect the inverse, i.e. `isBinary = true` but +this runs into problems with clunky terminology. In my mind, the kind of package +that fails the transparency/malleability tests goes beyond what many people +would argue is "a binary". For instance, many (most?) java packages in nixpkgs +simply pull opaque `.jar`s - if not for their own app, they pull `.jar` +dependencies from maven. These are not transparent or malleable, but it's quite +an obtuse and disputable use of the term "binary" to describe them as such. + +I decided that those packages which _did_ pass these transparency/malleability +tests had more in common than those that don't: that they are "from source", a +form where users have as much ability to inspect and alter the result as the +original author did. + +There already exists a rather informally-applied convention of adding a `-bin` +suffix to the package names of "binary packages". This is non-ideal because: + +- It doesn't allow a user to filter the use of these packages in a better way + than simply not requesting a package with a `-bin` suffix. Binary-package + _dependencies_ of non-`-bin` packages will still be installed regardless. +- It falls into the terminology trap over the term "binary", and if we expanded + the definition of what a "binary" package is, *very many* packages in nixpkgs + would have to be renamed, causing not only visual clutter but possible + breakage and churn. + +If we _don't_ do anything about this, then I think we continue to signal to +users who have such concerns over the source of their software that +nixpkgs/NixOS isn't for them. Far from being a concern just for obscure +extremists, most Debian users would probably balk at our appetite for binary +packages. + +# Drawbacks +[drawbacks]: #drawbacks + +- Some maintainers may be upset by having their packages marked as + `fromSource = false`. +- It could spur us to disappear into endless navel-gazing conversations about + what really counts as "from source" and what doesn't. +- On the other hand, _not_ discussing where the line stands thoroughly enough + could cause the flag to be over-applied and thus become useless. Should we be + compiling all our fonts where e.g. fontforge files are available? If all of + these got marked as `fromSource = false`, all of a sudden users with + `allowNonSource = false` set may end up with no installable desktop. + +# Unresolved questions +[unresolved]: #unresolved-questions + +Exact attribute names are open for debate. + +# Future work +[future]: #future-work + +The author is willing to spend a significant amount of time finding and marking +non-source packages in nixpkgs. From a60c2f4a516578206dbc2ec2c6abfaee95521b37 Mon Sep 17 00:00:00 2001 From: Linus Heckemann Date: Thu, 12 Aug 2021 15:12:38 +0200 Subject: [PATCH 2/6] Add shepherd metadata --- rfcs/0089-collect-non-source-package-meta.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rfcs/0089-collect-non-source-package-meta.md b/rfcs/0089-collect-non-source-package-meta.md index bec74be72..a1654777d 100644 --- a/rfcs/0089-collect-non-source-package-meta.md +++ b/rfcs/0089-collect-non-source-package-meta.md @@ -3,8 +3,8 @@ feature: collect-non-source-package-meta start-date: 2021-03-14 author: Robert Scott co-authors: (find a buddy later to help out with the RFC) -shepherd-team: (names, to be nominated and accepted by RFC steering committee) -shepherd-leader: (name to be appointed by RFC steering committee) +shepherd-team: @asymmetric, @aforemny, @alyssais +shepherd-leader: @asymmetric related-issues: (will contain links to implementation PRs) --- From a2ad7439e14839aeb29d78d18af9a77975676997 Mon Sep 17 00:00:00 2001 From: Robert Scott Date: Sat, 4 Dec 2021 23:00:53 +0000 Subject: [PATCH 3/6] [RFC 0089] adopt `sourceProvenance` "multi-valued" approach --- rfcs/0089-collect-non-source-package-meta.md | 64 ++++++++++++-------- 1 file changed, 39 insertions(+), 25 deletions(-) diff --git a/rfcs/0089-collect-non-source-package-meta.md b/rfcs/0089-collect-non-source-package-meta.md index a1654777d..c1c1a2ea0 100644 --- a/rfcs/0089-collect-non-source-package-meta.md +++ b/rfcs/0089-collect-non-source-package-meta.md @@ -37,29 +37,43 @@ For some users, these concerns are enough to deter them from using Nix entirely. # Detailed design [design]: #detailed-design -Add a new `meta` attribute to non-source-built packages, `fromSource = false`. -Leave other packages as-is with the assumption of a missing attribute meaning -`true`. +Add a new `meta` attribute to non-source-built packages, `sourceProvenance`. +The value of this attribute being a list of at least one entriy from a +collection of possibilities maintained in `lib.sourceTypes`. These possibilities +can be of opaque type, but should have entries to represent at least "built from +source", "binary native code", "binary bytecode" and "binary firmware". + +Packages built from source can be left as-is with the assumption of a missing +attribute being the equivalent of `[ lib.sourceTypes.fromSource ]`. + +Multiple values present in a package's `sourceProvenance` would be used to +mean that the package contains parts that fall under each of these categories. +However, a "source type" not appearing in a package's `sourceProvenance` would +_not_ necessarily mean that the package _doesn't_ contain parts which fall +under that category - it could simply mean that a package hasn't been fully +assessed yet. See "Future work" for discussion of adding the ability to make +such a "comprehensive" declaration. Add a mechanism to allow `.nixpkgs/config.nix` to specify `allowNonSource = false` to prevent use of these packages in a similar manner -to `allowUnfree`. +to `allowUnfree`. An `allowNonSourcePredicate` parameter would allow the +distinction to be customized, but the default predicate should take into account +the possible hierarchical nature of `lib.sourceTypes` entries. # Alternatives [alternatives]: #alternatives -I might have been tempted to collect the inverse, i.e. `isBinary = true` but -this runs into problems with clunky terminology. In my mind, the kind of package -that fails the transparency/malleability tests goes beyond what many people -would argue is "a binary". For instance, many (most?) java packages in nixpkgs -simply pull opaque `.jar`s - if not for their own app, they pull `.jar` -dependencies from maven. These are not transparent or malleable, but it's quite -an obtuse and disputable use of the term "binary" to describe them as such. +The original proposal used a simple boolean attribute to declare whether the +package contains any binary parts, mostly in an attempt to avoid having +to go down the route of devising and debating an ontology of source types. This +was deemed by the shepherding meeting to not embrace extensibility enough. -I decided that those packages which _did_ pass these transparency/malleability -tests had more in common than those that don't: that they are "from source", a -form where users have as much ability to inspect and alter the result as the -original author did. +Another suggestion involved using a single value from a collection of "source +types" to describe the source provenance in an effort to avoid complexity, but +this appeared to have the disadvantages of requiring an ontology to be agreed +upon and still yet not providing sufficient flexibility to cover many cases. +The missing ability to express multiple provenances might even encourage +maintainers to proliferate source types that represent combinations of others. There already exists a rather informally-applied convention of adding a `-bin` suffix to the package names of "binary packages". This is non-ideal because: @@ -81,23 +95,23 @@ packages. # Drawbacks [drawbacks]: #drawbacks -- Some maintainers may be upset by having their packages marked as - `fromSource = false`. -- It could spur us to disappear into endless navel-gazing conversations about - what really counts as "from source" and what doesn't. -- On the other hand, _not_ discussing where the line stands thoroughly enough - could cause the flag to be over-applied and thus become useless. Should we be - compiling all our fonts where e.g. fontforge files are available? If all of - these got marked as `fromSource = false`, all of a sudden users with - `allowNonSource = false` set may end up with no installable desktop. +- It could spur us to disappear into endless navel-gazing conversations over + the `lib.sourceTypes` ontology. # Unresolved questions [unresolved]: #unresolved-questions -Exact attribute names are open for debate. +Exact attribute names and contents of `lib.sourceTypes` are open for debate. # Future work [future]: #future-work The author is willing to spend a significant amount of time finding and marking non-source packages in nixpkgs. + +Addition of a simple accompanying boolean flag could allow the meaning of the +`sourceProvenance` field to be switched to imply the declaration is +"comprehensive" and that "source types" missing from the declaration are not +present. This is something that could be added once a maintainer has thoroughly +inspected a package, but should not place extra burden on someone wanting to +simply flag up that they have spotted some binary element in the package. From 0387adc6704fcbf54f9b07e208bae5af51606364 Mon Sep 17 00:00:00 2001 From: Robert Scott Date: Mon, 6 Dec 2021 19:40:27 +0000 Subject: [PATCH 4/6] [RFC 0089] change switched -> changed Co-authored-by: asymmetric --- rfcs/0089-collect-non-source-package-meta.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0089-collect-non-source-package-meta.md b/rfcs/0089-collect-non-source-package-meta.md index c1c1a2ea0..2fa31bee8 100644 --- a/rfcs/0089-collect-non-source-package-meta.md +++ b/rfcs/0089-collect-non-source-package-meta.md @@ -110,7 +110,7 @@ The author is willing to spend a significant amount of time finding and marking non-source packages in nixpkgs. Addition of a simple accompanying boolean flag could allow the meaning of the -`sourceProvenance` field to be switched to imply the declaration is +`sourceProvenance` field to be changed to imply the declaration is "comprehensive" and that "source types" missing from the declaration are not present. This is something that could be added once a maintainer has thoroughly inspected a package, but should not place extra burden on someone wanting to From 32e773db8b177404bac147fa3ac4080b33c3b85f Mon Sep 17 00:00:00 2001 From: Robert Scott Date: Mon, 6 Dec 2021 19:41:02 +0000 Subject: [PATCH 5/6] [RFC 0089] typo fix Co-authored-by: asymmetric --- rfcs/0089-collect-non-source-package-meta.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0089-collect-non-source-package-meta.md b/rfcs/0089-collect-non-source-package-meta.md index 2fa31bee8..2df716226 100644 --- a/rfcs/0089-collect-non-source-package-meta.md +++ b/rfcs/0089-collect-non-source-package-meta.md @@ -38,7 +38,7 @@ For some users, these concerns are enough to deter them from using Nix entirely. [design]: #detailed-design Add a new `meta` attribute to non-source-built packages, `sourceProvenance`. -The value of this attribute being a list of at least one entriy from a +The value of this attribute being a list of at least one entry from a collection of possibilities maintained in `lib.sourceTypes`. These possibilities can be of opaque type, but should have entries to represent at least "built from source", "binary native code", "binary bytecode" and "binary firmware". From b292ab65a0542967bce65a59e42ca22db04403f7 Mon Sep 17 00:00:00 2001 From: Robert Scott Date: Wed, 12 Jan 2022 23:44:29 +0000 Subject: [PATCH 6/6] [RFC 0089] remove unclear "opaque type" clause --- rfcs/0089-collect-non-source-package-meta.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rfcs/0089-collect-non-source-package-meta.md b/rfcs/0089-collect-non-source-package-meta.md index 2df716226..952ad582d 100644 --- a/rfcs/0089-collect-non-source-package-meta.md +++ b/rfcs/0089-collect-non-source-package-meta.md @@ -40,8 +40,8 @@ For some users, these concerns are enough to deter them from using Nix entirely. Add a new `meta` attribute to non-source-built packages, `sourceProvenance`. The value of this attribute being a list of at least one entry from a collection of possibilities maintained in `lib.sourceTypes`. These possibilities -can be of opaque type, but should have entries to represent at least "built from -source", "binary native code", "binary bytecode" and "binary firmware". +should have entries to represent at least "built from source", "binary native +code", "binary bytecode" and "binary firmware". Packages built from source can be left as-is with the assumption of a missing attribute being the equivalent of `[ lib.sourceTypes.fromSource ]`.