cachi2 yarn followups: "Handling development dependencies"

[slimreaper35]

Introduction

Package managers often have a feature that allows specifying dependencies meant only for development. Yarn supports this through the devDependencies section in the package.json file. However, yarn has no clear indication or "dev" label in the lock file to identify these development-only dependencies.

STONEBLD-1900

Here is a snippet from the conversation with the lead maintainer of yarnpkg 📦 🧶 on Discord:

it doesn't need to be in the yarn.lock, since we don't even care about dev dependencies when
they are listed by a non-workspace package

if they are a workspace package we don't need to know whether they are dev dependencies
or not (for the purpose of the lockfile, at least), since we'll always need to install them

SBOM

The goal is to mark these development dependencies in the SBOM using a CycloneDX property. The snippet is from npm_bundled_lockfile3 test data bom.json.

"properties": [
  {
    "name": "cachi2:found_by",
    "value": "cachi2"
  },
  {
    "name": "cdx:npm:package:development",
    "value": "true"
  }
],

Cachito

As you might know, something similar already exists in cachito. There is a function that finds all non-development dependencies based on the BFS algorithm using the devDependencies section as a starting point from all top-level package.json files.

To copy the solution from cachito, we will need a couple of things:

1. Get the list of all development dependencies from package.json files from all workspaces.
2. Take all the records in the devDependencies sections and map them somehow to the output of
   yarn info --all --json --recursive --cache command which basically means map them to our internal Package dataclass.
3. Find all transitive dependencies.
4. Update generated SBOM.

In cachito it looks like this:

root_dep_ids = [
    f"{name}@{version}"
    for dep_type in ["dependencies", "peerDependencies", "optionalDependencies"]
    for package_json in all_package_jsons
    for name, version in package_json.get(dep_type, {}).items()
]

An example of package.json from cachi2 integration tests repository:

"devDependencies": {
  "@types/node": "^20.4.5",
  "@types/yargs": "^17",
  "emoji-regex": "https://github.com/mathiasbynens/emoji-regex/archive/refs/tags/v10.2.1.tar.gz",
  "typescript": "^5.1.6"
},

My concerns are that devDependencies are not declared to a specific version. Will we need to parse yarn.lock for that and is it actually worth doing it ?

---

Thank you for all your comments. 🙏

[taylormadore]

It appears that both yarn.lock and the "Dependencies" key in the output from yarn info -AR --json both have the descriptor and the resolution for each of the packages. It seems to me that the missing part is translating the name+version(range?) in package.json to a yarn package descriptor. Once we do that, it might not matter which source of information we use.

Or did you see something else missing?