Skip to content

libstore: avoid PathSet allocation in derivation unparse#14823

Open
Zaczero wants to merge 4 commits intoNixOS:masterfrom
Zaczero:zaczero/libstore-pathset
Open

libstore: avoid PathSet allocation in derivation unparse#14823
Zaczero wants to merge 4 commits intoNixOS:masterfrom
Zaczero:zaczero/libstore-pathset

Conversation

@Zaczero
Copy link
Member

@Zaczero Zaczero commented Dec 18, 2025

  • Derivation::unparse() built a temporary PathSet via printStorePathSet(inputSrcs) only to immediately serialize it, which was already flagged as // FIXME: slow.

  • This reduces allocations/copies on a hot path during derivation (un)parsing; in local microbenchmarks (BM_UnparseRealDerivationFile/hello) it improves runtime by ~8%.

Motivation

Context


Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

- Derivation::unparse() built a temporary PathSet via printStorePathSet(inputSrcs) only to immediately serialize it, which was already flagged as // FIXME: slow.
- Emit the quoted inputSrcs paths directly into the output string instead, keeping the exact on-disk format while avoiding the intermediate container and extra iteration.
- This reduces allocations/copies on a hot path during derivation (un)parsing; in local microbenchmarks (BM_UnparseRealDerivationFile/hello) it improves runtime by ~8%.
@Zaczero Zaczero requested a review from Ericson2314 as a code owner December 18, 2025 02:42
@xokdvium
Copy link
Contributor

xokdvium commented Dec 18, 2025

SGTM. I suppose we also skip the potential string escaping because derivation names cannot contain any characters that need escapes, right?

@Zaczero
Copy link
Member Author

Zaczero commented Dec 18, 2025

@xokdvium I don't think the original was escaping anything (I didn't review whether it's correct or not, I assume it is). It did build a set of strings and then used printUnquotedStrings to print each as-is. The new approach simply iterates and prints directly without allocations in this hot path.

Copy link
Member

@Ericson2314 Ericson2314 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See also my review in #14825 (review)

Here's what I would like

  1. Move concatStrings from util.hh to strings.hh. That is my mistake not doing that when I broke up most of util.hh. (Feel free to move some other things there while you are at it.

  2. Just as we have basicSplitStringInto vs basicSplitString in strings-inline.hh, we should have a version of concatStrings (and the underlying concatStringSep) that appends to an existing string.

  3. I think you can just change printStorePathSet to do the appending way, and that will benefit most call sites, and for the remainder you can either just make the initial empty string manually, or make a small overload wrapper that does it.

How does that sound? That allows us to have or cake (perf) and eat it (code reuse) too.

@Zaczero
Copy link
Member Author

Zaczero commented Dec 18, 2025

See also my review in #14825 (review)

Here's what I would like

  1. Move concatStrings from util.hh to strings.hh. That is my mistake not doing that when I broke up most of util.hh. (Feel free to move some other things there while you are at it.
  2. Just as we have basicSplitStringInto vs basicSplitString in strings-inline.hh, we should have a version of concatStrings (and the underlying concatStringSep) that appends to an existing string.
  3. I think you can just change printStorePathSet to do the appending way, and that will benefit most call sites, and for the remainder you can either just make the initial empty string manually, or make a small overload wrapper that does it.

How does that sound? That allows us to have or cake (perf) and eat it (code reuse) too.

Sounds good, I'll dig into it.

@Zaczero Zaczero marked this pull request as draft December 18, 2025 05:19
StoreDirConfig::printStorePath() previously built an intermediate string via storeDir + "/" and then appended the store-path basename, which can trigger an extra allocation/copy.
@github-actions github-actions bot added the repl The Read Eval Print Loop, "nix repl" command and debugger label Dec 18, 2025
@Zaczero Zaczero marked this pull request as ready for review December 18, 2025 07:19
@Zaczero Zaczero requested a review from edolstra as a code owner December 18, 2025 07:19
@Zaczero Zaczero requested a review from Ericson2314 December 18, 2025 07:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

repl The Read Eval Print Loop, "nix repl" command and debugger

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants