install: leave URL credentials out of isolated store entry names - #39014
Conversation
With the isolated linker, a package installed from a tarball or git URL gets a node_modules/.bun entry named after the whole URL, so a password in the userinfo or a token in the query string became part of a directory name (and of every realpath, stack trace and `bun pm` line derived from it). Tarball and repository URLs are now written into the store name without their userinfo and query string. When either part was present, the wyhash of the complete URL is appended in their place, so URLs that differ only in those parts still get separate entries. URLs without either part keep the names they had.
|
Warning Review limit reached
Next review available in: 28 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (4)
Comment |
|
Status: reproduced on 1.4.0-canary.1 with a local tarball server ( The one design call to look at is the third and fourth bullets under Fix: URLs with a userinfo or a query string are renamed once (this includes |
There was a problem hiding this comment.
I reviewed this PR and didn't find any bugs. Because it changes the on-disk store naming scheme (renaming every existing entry whose URL carries userinfo, including the very common git+ssh://git@… form) and hand-parses URL authority/query boundaries for a credential-redaction purpose, a maintainer look at those design choices would still be worthwhile.
What was reviewed:
fmt_store_urlparsing: authority-start via first:+//, authority-end via first/?#, userinfo via last@in authority, query via first?after host — matches RFC 3986 andbun_core::find_url_password; scp-likeuser@host:pathhandled byauthority_start = 0.- Uniqueness preserved: hash of the full URL is appended whenever anything is stripped, so
?v=1vs?v=2still separate; plain URLs keep their existing names byte-for-byte (no hash appended). - Consumers:
bun pm prune'ssplit_store_keysplits at the first non-leading@and the new names contain none in the version part;bun pm licensesand the global store re-derive names through the same formatter. bun_semver::StorePathFormatterfield change (str+buf→bytes) — the only external constructor wasString::fmt_store_path, updated in the same commit;install/lib.rshas its own unrelatedStorePathFormatter.
Extended reasoning...
Overview
The PR stops the isolated linker from writing URL-embedded credentials into directory names under node_modules/.bun. It adds fmt_store_url in src/install/resolution.rs (used for remote-tarball resolutions and, via src/install/repository.rs, for the repo part of git resolutions), which emits the URL with its userinfo and query string removed and, when either was present, appends a 16-hex wyhash of the complete URL. src/semver/lib.rs refactors StorePathFormatter to accept a raw byte slice via a free fmt_store_path so the two kept URL pieces are spelled with the exact same character mapping as before. ~220 lines of new tests in test/cli/install/isolated-install.test.ts cover tarball and git dependencies with plain URLs (name unchanged), passwords, query-string tokens, both together, query-string-only disambiguation, the global store, and lockfile round-tripping.
Security risks
The change is redactive: it removes secrets from disk paths rather than introducing any new trust boundary. The URL is still stored in full in the lockfile (unchanged, tested), so no functional auth is affected. The hand-rolled URL parsing is only used to derive a directory name — if it mis-splits, the worst case is a longer or shorter readable prefix, and the appended full-URL hash still keeps distinct resolutions in distinct directories. I did not find a way for two previously-distinct resolutions to collide onto the same store entry.
Level of scrutiny
Medium-high. The Rust change itself is small and self-contained (one new formatter, two call-site swaps, one struct-field refactor with no external constructors), but it is a deliberate change to the isolated store's on-disk naming, which has downstream consumers (bun pm prune, bun pm licenses, the global store, #38867's length bounding) and a one-time migration effect. In particular, every git+ssh://git@host/… dependency — a very common form where git@ is not secret — will be renamed with a hash suffix on the next install. The PR calls this out and argues it cannot be distinguished from a token-as-username, which is reasonable, but it is exactly the kind of tradeoff a maintainer should sign off on.
Other factors
Test coverage is thorough: exact expected names computed with Bun.hash, symlink targets, lockfile contents, runtime import, and second-install stability are all asserted; the two "plain URL" cases prove existing names are byte-identical. The PR description states the rest of isolated-install.test.ts, bun-pm-licenses.test.ts, and bun-prune.test.ts still pass. No prior review comments to address. Given the on-disk layout change and the credential-handling context, deferring to a human rather than auto-approving.
|
@coderabbitai review |
|
#39445) Stacked on #36463 (the base branch is that PR's branch, so the diff here is only the additions). Merging this into #36463 adds the behavior changes listed below; #36463 itself now covers the #38333 install batch, the optional-peer correction, and the TOML / `bun init` fixes, so this PR no longer touches those. ### Problem - These 1.3 to 1.4 behavior changes are not in the guide at `701b3e2a0`: - MySQL: the first `caching_sha2_password` connection over plain TCP is refused unless `allowPublicKeyRetrieval: true` (#31129; 1.3.14 requested the key automatically, `MySQLConnection.zig` in the 1.3.14 tag). SQL `tls` / `ssl` options now require TLS instead of falling back to plaintext, and `?ssl=` / `?ssl-mode=` are read (`shared.ts` 1.3.14 only read `?sslmode=`; #37669). - Install: `~/.npmrc` fallback when `XDG_CONFIG_HOME` is set (#36289), credentials in `--registry` / env / bunfig object URLs are sent and outrank same-host `.npmrc` tokens (#38796, #38824), `bun outdated` exits 1 on fetch failures (#38809), new `dedupe` / `up` commands shadow scripts of those names and `bun feedback` is removed (#38333, #38444), `workspace:` ranges inside registry packages (#37669), isolated store entry names (#39014). - Runtime: `module.enableCompileCache()` / `NODE_COMPILE_CACHE` implemented (#34660), `require()` / `import` not-found messages (#34660), `AbortError` message without the period (#39277; 1.3.14's `BunCommonStrings.h` has the period), GCM IV length (#34092), `mkdtemp("")` (#34908), vm options (#38381), `server.reload` (#38697), ICU 75/73 to 78 (#38013), Compression stream chunking (#38695), `Bun.SQL` sqlite bindings (#35950). - Bundler: `splitting` with `cjs` / `iife` is an error (#32685), block-scoped `enum` lowers to `let` (#34249), exports emitted ascending instead of descending (#35957; `doStep5.zig` in 1.3.14 used `sortDesc`), minified `$` (#35668). - The TOML integer bullet did not say what the limit or the fix is. ### Fix - Adds a MySQL public key section (plus a summary table row), a TLS note under the `PGSSLMODE` section, an `.npmrc` / credentials addendum to the `bunfig.toml` section, a `module.enableCompileCache()` section, and the rest as bullets in the existing lists. - `docs/pm/overrides.mdx`: one-line change adding a pointer to this guide in the existing `lockfileVersion` 3 limitation. (The base branch briefly had a duplicate "Nested overrides" section; it removed that itself in `8257d01acb`, and this PR was rebased over it.) - Verification: each runtime claim was run against `1.4.0-canary.1+8326d1bd3` (22 commits behind main; contains every change referenced), and each install or bundler claim was checked against the source on main, with the 1.3 side taken from the `bun-v1.3.14` tag where the PR body did not state it. The `/runtime/sql#mysql` and `/upgrade-to-1.4` links resolve. `prettier --check` passes. ### Not included on purpose - Lifecycle scripts no longer receiving `npm_package_name` / `npm_package_version` / `npm_package_json` / `npm_config_local_prefix` during `bun install`, and transitive `"*"` ranges no longer deduplicating onto the root's version: regressions with open fixes (#36690, #38110, #38770). They need either the fixes or a guide line before release. - Postgres `sslmode=prefer` / `allow` (including `PGSSLMODE=prefer`, which 1.4 newly reads) hangs until the connection timeout against a server without SSL because nothing sends the startup message after the `N` reply. Same code in 1.3.14; filed as a bug instead of documented. <details> <summary>Commands used to verify the runtime claims</summary> ``` timers/promises setTimeout with an aborted signal # "The operation was aborted" bun req.cjs # Cannot find module ... Require stack: bun b.mjs (import() of a missing package / relative file) # Cannot find package 'x' imported from /path, ERR_MODULE_NOT_FOUND bun a_static.mjs (unhandled static import) # printed line still: Cannot find package 'x' from '/path' process.versions.icu # 78.3 createCipheriv("aes-128-gcm", key, Buffer.alloc(129)) # ERR_CRYPTO_INVALID_IV DecompressionStream of a 1 MiB gzip member # 16 chunks of 65536 bytes new SQL("sqlite://:memory:") with ${[1,2]} / ${new Date()} # Binding expected ... fs.mkdtempSync("") # EINVAL vm.runInThisContext("1", []) # ERR_INVALID_ARG_TYPE NODE_COMPILE_CACHE=/tmp/cc bun cc.cjs # creates /tmp/cc/v1.4.0-x86_64-<sha>-<uid> NODE_DISABLE_COMPILE_CACHE=1 + enableCompileCache() # status 3 (DISABLED) bun dedupe / bun up with package.json scripts of those names # built-in command runs bun feedback # Script not found "feedback" Bun.build({ splitting: true, format: "cjs" }) # Code splitting is currently only supported ... bun build of a function-scoped enum and import * as ns # let Color; exports a, m, z new SQL({ url: "postgres://...", tls: true }) on a non-TLS server # ERR_POSTGRES_TLS_NOT_AVAILABLE Bun.TOML.parse("a = 9007199254740993") # Integer cannot be losslessly represented ... ``` </details> <details> <summary>Previous revision</summary> The first revision of this PR (`3c5611454a`) also rewrote the package manager section for #38333 / #38853 (nested overrides and `lockfileVersion: 3`, the optional-peer correction, `bun update`, `bunfig.toml` over `.npmrc`, `--filter`) and fixed the TOML date and `bun init` lines. #36463 picked those up in its own commits the same day, so this PR was rebased onto its new head and reduced to the items above. </details> <!-- robobun:evidence:begin --> --- **no test proof** · iteration 0 · docs-only change; test-proof not applicable <!-- robobun:evidence:end -->
…ile: tarballs relative to their folder package (#38867) This PR bundles three independent install fixes (each was reviewed on its own PR; folded here so they land together): 1. bound the resolution part of isolated store entry names (this PR's original change) 2. send credentials embedded in a tarball URL as Basic authorization (from #39025) 3. read `file:` tarballs relative to the `file:` folder package that declares them (from #39017) Rebased on main after #39014: the isolated-store test file now computes expected git entry names through the same resolution cut (`storeEntryName`), since a `git+http+++host+port+repo.git+<url hash>+<sha>` resolution passes 80 bytes. --- ## 1. install: bound the resolution part of isolated store entry names #### Problem - With `linker = "isolated"`, a trusted git dependency with any lifecycle script makes `bun install` fail on Windows with `error: Failed to run script prepare due to error ENOENT` (exit 1). The same project installs fine with the hoisted linker. This is the Windows failure of the isolated case in #38810's test, which that PR currently skips on Windows. - A store entry is named `<name>@<resolution>` (`StoreKeyFormatter`, `src/install/isolated_install/Store.rs`). For git, github, tarball and folder dependencies the resolution is the whole URL or path with separators turned into `+` (`src/install/repository.rs` `StorePathFormatter`, `src/install/resolution.rs` `StorePathFormatter`), plus the commit for git: a git dependency checked out from a temp directory gets an entry like `dep@git+file++++C++Users+AZUREU~1+AppData+Local+Temp+lc-repro+dep-repo+3c6955c70dbe1ff186a126c93c5e77c3ae7bd6b4` (111 bytes here, 170+ in #38810's test), and the name grows with the repository path. - The package directory, `<project>\node_modules\.bun\<entry>\node_modules\<name>`, is the cwd of the package's lifecycle scripts (`Scripts::get_list` -> `lifecycle_script_runner.rs` `SpawnOptions.cwd`). bun's own file operations use long-path-capable NT paths, so the entry is created and linked, but `CreateProcessW` does not accept a current directory longer than `MAX_PATH`, and the spawn (libuv's `uv_spawn` on Windows) reports that as ENOENT. Measured on Windows x64 with the released `1.4.0-canary.1`: a package directory of 238 characters runs the script, one of 268 characters fails as above. - The same unbounded name also fails on every platform once it passes `NAME_MAX`: a tarball URL or folder path of 250+ bytes makes the install fail with `ENAMETOOLONG: File name too long: failed to link package` (the case of #37470). #### Fix - `StoreKeyFormatter` now writes the resolution through `write_resolution`: a resolution of at most 80 bytes (`MAX_RESOLUTION_LEN`) is written unchanged; a longer one is written as its first 63 bytes (backed up to a UTF-8 character boundary) followed by `+` and the 16 hex digit wyhash of the full text (seed 0, so `Bun.hash(text)` reproduces it), which is 80 bytes again. The `name@` prefix and the `+<peer hash>` suffix are unchanged. A cut git entry looks like `dep@git+file++++C++Users+AZUREU~1+AppData+Local+Temp+lc-repro-aaaaa+ef7ad81a431ad661` (the 268 character case from the measurement above; its package directory is now 206 characters). - Why this is correct: - Every path that contains an entry name is formatted through this one `Display` impl: the project store directory and the dependency symlinks into it (`Installer.rs` `append_store_path` / `append_store_node_modules_path` / `link_to_hidden_node_modules`), the lifecycle script cwd (same `append_store_path`), the global store directory name and the entry hash seeded from the name (`Installer.rs` `append_global_store_entry_path`, `isolated_install.rs` entry hash), `bun pm prune`'s set of expected directories (`prune.rs` `push_store_entry_names`) and `bun pm licenses`'s lookup (`pm_licenses_command.rs` `BunStore::lookup`, via `fmt_store_key`). They all see the same cut name; nothing persists or parses the old one (the lockfile stores resolutions, and an existing entry is detected by re-deriving its name). - The cut name stays a function of the resolution alone, so repeated installs find the same entry, and two resolutions that agree on the first 63 bytes still get distinct entries through the hash. The sink hashes the text in whatever chunks the formatters write it in (the path formatters write one character at a time); `Wyhash`'s streaming form is chunk-invariant (`test_iterative_chunked_matches_oneshot` in `src/wyhash/lib.rs`), which is what lets the tests compute the expected names with `Bun.hash`. - The cut happens inside the resolution only, so `prune.rs` (`split_store_key`, `store_has_entries`, `store_link_target`, which split the name at `@`) and `pm licenses` (which matches `<key>` or `<key>+<peer hash>`) keep working without changes. - Resolutions of 80 bytes or less are byte for byte what they were, so versions and `github:` shorthands (`github+owner+repo+<sha>` is about 60 to 75 bytes) are unaffected on upgrade. Most full git URLs are over 80 bytes (`git+https+++github.com+` is 23 bytes and the commit adds 41), so those entries are re-created once under the cut name on the first install after upgrading, and the old directories stay until `bun pm prune`, as for any re-resolved entry. The cut keeps the URL itself readable: `git-pkg@git+file++++tmp+licenses-git-repo_eXITMt+491eb29762d8d19b570b43+f87e35b31f207dd4`. - 80 is the tunable here. The package directory is `<project> + 34 + 2 * <name> + <resolution>` (+17 with peers) characters, so with 80 it fits `MAX_PATH` whenever `<project> + 2 * <name>` is at most 145 (128 with peers); #38810's case (61 character temp dir, 25 character name) ends up at about 225 instead of about 290. It also makes the entry directory name fit `NAME_MAX` together with every suffix the installers append (`+<peer hash>`, `-<entry hash>`, `.tmp-<hex>`) for names up to 119 bytes, and with the peer suffix alone for names up to 157 bytes. For comparison, pnpm's `virtual-store-dir-max-length` defaults to 120 bytes for the whole directory name; 80 bytes of resolution plus a typical name lands in the same range. - #37470 (the same mechanism with a 200 byte limit on the whole `name@resolution`, for the `NAME_MAX` case only) is closed in favor of this PR: that limit does not help the `MAX_PATH` case (the failing names above are 130 to 180 bytes), and a cut through `name@` would no longer work with `bun pm prune` / `pm licenses` (see the `prune.rs` bullet above). Its repro installs with this branch and its deep-directory case is carried over below. What it covered and this PR does not is the name part: a name over 119 bytes (157 in the project store) can still produce an entry over `NAME_MAX` once its resolution is cut; if that should be covered too, it would be the same sink applied to the name separately, keeping the `@`. #36973 (documenting the layout) would need a sentence about this rule; this PR adds a short one to `docs/pm/isolated-installs.mdx`. #38810's Windows skip of its isolated case can be removed once both land. #39014 changes the same text (it removes URL credentials, and the hash is of the text as formatted), so landing it in the same release as this avoids renaming those entries twice. - Verification, `long store entry names` in `test/cli/install/isolated-install.test.ts`: - a folder resolution of exactly 80 bytes is kept verbatim and one of 81 bytes is cut to the name computed with `Bun.hash`; two resolutions of one package that only differ after the cut point get separate entries that each resolve to their own package; a second install finds the same entries - a cut entry with a resolved peer gets `+<peer hash>` appended after the cut name and links the peer - a folder name made of 2 byte characters, positioned so that byte 63 falls inside a character, is cut at byte 62 and still installs (without the character boundary loop this case panics with `a formatting trait implementation returned an error`; the test only asserts the shape because the spelling of non-ASCII bytes in these names is #32304's subject) - (carried over from #37470) a local tarball and a folder three 85 byte directories deep, whose unbounded entry names would be 277 bytes, install; the top-level symlinks and the `.bun/node_modules` fallback link point at the cut names, and a second install finds the same entries - a tarball URL of about 290 bytes installs with `install.globalStore` enabled: the local entry has the cut name, links to `links/<cut name>-<entry hash>`, and the package imports at runtime - the reported case: a trusted `git+file://` dependency whose repository directory name alone is 60 characters runs its postinstall script with the isolated linker; the entry has the cut name - Without the `Store.rs` change the first three tests and the git one fail on the entry name (on Windows the git one fails with `error: Failed to run script postinstall due to error ENOENT`, the reported message) and the deep-directory and tarball ones exit 1 with `ENAMETOOLONG`; all of them pass with `bun bd test` on Linux (the five original ones also on a local Windows x64 build; the deep-directory one on the Windows lanes in CI). #37470's repro from its description (`./x/../` x130 tarball spec) installs with this branch with an 84 byte entry. - Also run with the fix: the rest of `isolated-install.test.ts`, `bun-pm-licenses.test.ts` (its `git dependency is listed (isolated)` case now goes through a cut name on every platform, since the repo lives in a temp dir), `bun-prune.test.ts`, `public-hoist-pattern.test.ts`, `isolated-relink.test.ts`, `bun-install-native-binlink.test.ts`, `config-version.test.ts`, and the PowerShell reproduction from the report on Windows x64 (package directory of 268 characters before, 206 after, script runs). #### Background - Isolated linker: instead of hoisting, every package is materialized once under `node_modules/.bun/<entry>/node_modules/<name>` and everything else symlinks to it. `<entry>` is `<name>@<resolution>`, plus `+<16 hex>` when the package was installed for a specific set of peer dependencies. With `install.globalStore`, `node_modules/.bun/<entry>` is itself a symlink to `<cache>/links/<entry>-<16 hex entry hash>`. - Resolution: the lockfile's description of where a package came from. It is the version for registry packages and the spec itself (folder path, tarball URL, git URL plus commit) for everything else, which is why only those entries get long. - `MAX_PATH`: Windows' 260 character limit for paths passed to Win32 APIs. Applications can opt out of it for file operations (bun does, by using `\\?\` / NT paths), but the current directory handed to `CreateProcess` is still limited to it, and libuv maps the resulting Win32 error to ENOENT. - `NAME_MAX`: the 255 byte limit on one path component on Linux and macOS (255 UTF-16 units on NTFS); exceeding it fails with `ENAMETOOLONG` regardless of how long the whole path is. <details> <summary>Windows measurement with the released build (1.4.0-canary.1)</summary> Git repo and project created under `%TEMP%` with `linker = "isolated"`, `trustedDependencies: ["dep"]` and `"prepare": "echo prepared > prepared.txt"`; the temp directory name was lengthened to move the package directory across the limit. ``` store entry: dep@git+file++++C++Users+AZUREU~1+AppData+Local+Temp+lc-repro-aaaaaaaaaaaaaaaaaaa+dep-repo+d1a8f7c4111533cb16b52a9edcb086329e6472b5 pkg dir len: 238 prepared.txt exists: True exit: 0 store entry: dep@git+file++++C++Users+AZUREU~1+AppData+Local+Temp+lc-repro-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa+dep-repo+30b6ecc8d4d0e4fb2f3df2402556222bf6a5b7dc pkg dir len: 268 pkg dir exists: True prepared.txt exists: False error: Failed to run script prepare due to error ENOENT exit: 1 ``` Same second layout with this PR's build: ``` store entry: dep@git+file++++C++Users+AZUREU~1+AppData+Local+Temp+lc-repro-aaaaa+ef7ad81a431ad661 pkg dir len: 206 prepared.txt exists: True exit: 0 ``` </details> <!-- robobun:evidence:begin --> --- **no test proof** · iteration 1 · Platform-specific test(s) that do not run on this machine. Deferring to CI, which covers all platforms: test/cli/install/bun-install.test.ts test/cli/install/isolated-install.test.ts <!-- robobun:evidence:end --> --- ## 2. install: send credentials embedded in a tarball URL as Basic authorization (from #39025) #### Problem - A dependency declared as a tarball URL with credentials, `"no-deps": "http://carol:s3cret@127.0.0.1:PORT/cdn/no-deps-1.0.0.tgz"`, is downloaded without an `Authorization` header. A server that needs the credentials answers 401 and the install fails with `error: GET http://carol:s3cret@127.0.0.1:PORT/cdn/no-deps-1.0.0.tgz - 401` (bun 1.4.0 and `main`, hoisted and isolated linker). npm 11 installs the same package.json and sends `Authorization: Basic Y2Fyb2w6czNjcmV0` (`base64("carol:s3cret")`). - The username-only form (`http://token@127.0.0.1:PORT/x.tgz`) does not reach the server at all: the request goes to the hostname `token@127.0.0.1`. - Cause: `NetworkTask::for_tarball` (`src/install/NetworkTask.rs`) only ever attaches the registry scope's configured token or `_auth`, and only for npm packages whose tarball is on the registry's origin. Nothing reads the URL's userinfo, and the HTTP client does not either (it only turns the userinfo of a proxy URL into `Proxy-Authorization`). The misrouted username-only form comes from `bun_url::URL::parse`, which takes `token@127.0.0.1` for the hostname when the userinfo has no `:` and a port follows (tracked separately; this change no longer depends on how that parser splits the authority). - Found while fixing the isolated store names of these URLs (#39014), not from a user report. #### Fix - `for_tarball` splits the userinfo off the request URL before anything else looks at it (`split_url_userinfo`: the authority runs from `://` to the first `/`, `?` or `#`, the userinfo is everything in it up to the last `@`, so the `@` of `/@scope/pkg/-/pkg.tgz` is not one) and sends it as `Authorization: Basic base64(userinfo)` (`basic_authorization_from_userinfo`, which appends `:` when the userinfo has no password). The request URL is the URL without the userinfo. - Precedence: when the registry scope's credentials apply to the tarball (npm package, tarball on the registry origin, scope has a token or `_auth`), they are still sent and the URL's are not. This is npm's order as well: npm-registry-fetch sets the header from the config, and node only derives `Authorization` from the URL's `auth` when the request has no such header. - Why Basic of the userinfo as written: it is what npm sends, checked against npm 11.16 for `user:pass`, `user` (sent as `user:`), `:pass` and a percent-encoded password (sent undecoded); transcript below. It is also how bun already treats credentials written into a registry URL (#38796 stores the username and password bytes as given). pnpm was not available offline. Two spellings are deliberately not npm's: a second `:` inside the password is sent as written where npm percent-encodes it, and `:token@` in a tarball URL is Basic like npm rather than the Bearer that bun's registry URLs make of it; both are called out in the test table and in the code comment. - Why the URL is requested without the userinfo: `bun_url::URL::origin` includes the userinfo and the HTTP client compares origins to decide whether `Authorization` follows a redirect, so with the userinfo left in, a redirect to the same host would drop the header (verified: with only the header added, the redirect test below fails with `authorization: null` on the second hop). It also fixes the username-only form without touching `bun_url`, and the `GET <url> - 401` line now prints the URL without the credentials. The cross-host rule is unchanged: the HTTP client strips the header on a redirect to another origin, same as for the registry token (test included). - Behavior outside the request is unchanged: `package.json`, the lockfile, the task id and the cache key still use the URL as written. Documented in `docs/pm/cli/add.mdx`. - Verified with `test/cli/install/bun-install.test.ts`, `describe("credentials embedded in a tarball URL")`: 12 tests, 10 of which fail on the unfixed build (the two guards, scoped path and registry credentials taking precedence, pass on both). The rest of the file is unchanged: the remaining failures locally are the tests that need the public internet, identical with the unmodified binary. - `cargo check -p bun_install`, `cargo clippy -p bun_install`, rustfmt and prettier are clean. #### Background - Tarball dependency: a `package.json` entry whose version is an `http(s)://` URL ending in `.tgz`/`.tar.gz`/`.tar`. Bun downloads it directly; no registry manifest is involved, so the registry's configured credentials never applied to it (`Authorization::NoAuthorization` at the call sites in `PackageManagerEnqueue.rs`). Registry packages reach the same `for_tarball` through their manifest's `dist.tarball` URL with `AllowAuthorization`, which is the case where the registry scope's credentials can apply. - Userinfo: the `user:password@` part of a URL's authority (RFC 3986 section 3.2.1). HTTP never puts it on the wire; clients that honor it (npm through minipass-fetch, curl, browsers) convert it into `Authorization: Basic base64(user:password)`. - `NetworkTask::for_tarball` builds one HTTP request per tarball download: `url_buf` (the request URL) and `header_buf` (the headers, built with `HeaderBuilder` in two passes, `count` then `append`, because the buffer is allocated exactly once in between). Retries reuse the same request, so the header is sent on every attempt. - `bun_url::URL::parse` is the allocation-free splitter the HTTP client and `for_tarball` use; it is not a WHATWG parser. Its `origin` is a prefix of the input string, which is why it still contains the userinfo. <details> <summary>npm 11.16 against a local server logging the Authorization header (same tarball, same package.json shape)</summary> ``` userinfo in the dependency URL header npm sent carol:s3cret@ Basic Y2Fyb2w6czNjcmV0 = base64("carol:s3cret") carol@ Basic Y2Fyb2w6 = base64("carol:") :s3cret@ Basic OnMzY3JldA== = base64(":s3cret") carol:s3%40cret@ Basic Y2Fyb2w6czMlNDBjcmV0 = base64("carol:s3%40cret"), not decoded carol:s3:cret@ Basic Y2Fyb2w6czMlM0FjcmV0 = base64("carol:s3%3Acret"), bun sends base64("carol:s3:cret") carol:s3cret@ + 302 to same host Basic ... on both hops ``` bun 1.4.0-canary.1 (eabb96d) for the first row: the server logs `auth=null` and bun prints `error: GET http://carol:s3cret@127.0.0.1:PORT/cdn/no-deps-1.0.0.tgz - 401`. </details> --- ## 3. install: read `file:` tarballs relative to the `file:` folder package that declares them (from #39017) #### Problem - A `file:` folder dependency whose own package.json declares a local tarball fails to install (released 1.4.0 canary and main, both linkers): project package.json `{"dependencies":{"lib":"file:./vendor/lib"}}`, `vendor/lib/package.json` declaring `"tool": "file:./tool.tgz"`, `vendor/lib/tool.tgz` present: ``` error: ENOENT extracting tarball from tool error: tool@file:./tool.tgz failed to resolve ``` - The path is read as `<project>/tool.tgz`. If that file happens to exist it is installed as `tool` instead of the folder's copy, with no error. The directory form of the same declaration (`"tool": "file:./tool"`) is already resolved relative to `vendor/lib`. - Cause: `enqueue_local_tarball` (`src/install/PackageManager/PackageManagerEnqueue.rs`) picks the directory a local tarball path is relative to, and the only declarer it looked at was a workspace (`get_workspace_pkg_if_workspace_dep`). Every other declarer, including a `file:` folder package, fell through to the top-level dir. - The same choice was also wrong in the other direction: a root `overrides` / `resolutions` entry or catalog entry pointing a dependency at `file:./x.tgz` was read relative to the workspace when the dependency it applied to was declared by a workspace member, so `overrides: { bar: "file:./bar.tgz" }` with `bar.tgz` in the project root failed with the same ENOENT as soon as a workspace depended on `bar`. This is #25835 (`overrides`) and #25752 (`catalogs`); both reproduce as reported on the released build and install with this change. The directory form of an override is already resolved relative to the project (`Folder` arm of `get_or_put_resolved_package`). Fixes #25835 Fixes #25752 #### Fix - `enqueue_local_tarball` now takes the base directory from `local_tarball_base_dir`: the directory of the declaring package when it is a workspace or a `file:` folder package and that package's own specifier is the tarball path being read; the top-level dir in every other case. - Why the declaring package: a path in a package.json means a file next to that package.json, which is what npm does for `file:` and what bun already does for workspace declarers and for the directory form. A `file:` folder package is read from the project like a workspace is, and its `Resolution::Folder` payload is its directory relative to the top-level dir (`folder_resolver.rs`, `NewResolver { folder_path: rel }`), the same shape as a workspace's `Resolution::Workspace` payload, so both are joined the same way. The only other `Resolution::Folder` packages are the stubs created for `file:` directories declared by something other than the root or a workspace (`Folder` arm of `get_or_put_resolved_package`); those carry no dependency list, so they are never the declarer of an edge. - Why the "own specifier" condition: `overrides`, `resolutions` and catalogs are only parsed from the root package.json (`Package.rs`, `FEATURES.is_main`), and applying one leaves the declaring package's stored edge untouched (the replacement is local to `enqueue_dependency_with_main_and_success_fn`). So when the stored edge's specifier is not the path being read, the root wrote the path and the top-level dir is the only directory it can mean. Without this condition, root overrides applied to a folder-declared dependency, which work today only because of the bug, would start being read from the folder. - Why it is decided from the edge and not from the resolve pass's `version_was_replaced`: `enqueue_local_tarball` is also reached from `enqueue_tarball_for_reading` when a project with a `bun.lock` is installed into an empty cache. The lockfile row keeps the path as declared (`"tool": ["bar@./tool.tgz", ...]` under `"lib": [..., { "dependencies": { "tool": "file:./tool.tgz" } }]`), and the edge's declarer and specifier are available there too, so both passes compute the same directory. Lockfile format is unchanged and existing lockfiles keep working; the path in the row is joined onto a different directory only for declarers that previously failed or installed the wrong file. - Declarers extracted from the cache (registry, git, tarball packages) still fall through to the top-level dir, as before; #38986 is changing what happens to those separately. `Lockfile::get_parent_pkg_of_dependency` is the same helper that PR adds. - Left as is: the lockfile identity of a local tarball is still the path as written (`bar@./tool.tgz`), so two project packages declaring the same relative path to two different files still share one row and one read, the limitation workspaces already have. A package.json inside a folder dependency that worked around this bug by writing a project-relative path (`file:./vendor/lib/tool.tgz`) will now need the path relative to itself, which is what npm requires for it as well. - Verified with `test/cli/install/bun-install.test.ts`, `describe("file: tarball declared by a file: folder dependency")`: the folder's tarball is installed with the hoisted and with the isolated linker, and a root override supplying the path is read from the project. Each test plants a different tarball at the other candidate path and runs a fresh install followed by `--frozen-lockfile` into an emptied cache, so both the resolve pass and the install from `bun.lock` have to read the right file. `test/cli/install/bun-workspaces.test.ts`, `relative tarballs > from a root override / catalog entry applied to a workspace dependency`, covers #25835 and #25752 the same way. The four folder/workspace tests install the wrong tarball without the `src/` change (checked against a debug build of main and against the released build); the override-on-folder test passes before and after and pins that case. - Also run with this change: `bun-workspaces.test.ts` (74 pass), `overrides.test.ts` + `nested-overrides.test.ts` (151 pass), `bun-lock.test.ts` (40 pass), `bun-install.test.ts -t "tarball|tgz|file:|folder|override|resolutions"` (51 pass; `should treat non-GitHub http(s) URLs as tarballs` fails identically on the released build, it needs network), `isolated-install.test.ts`, `bun-add.test.ts` and `bun-install-registry.test.ts` filtered to tarball/file tests (all pass). `cargo clippy -p bun_install` is clean. #### Background - A `file:` dependency on a directory is a `Tag::Folder` dependency; one on a `.tgz` is a `Tag::Tarball` dependency with a local URI. The latter resolves to `Resolution::LocalTarball(<path as written>)`; the path string is both the task id for reading it and the package's identity in `bun.lock`. - bun reads the package.json of the root, of workspace members and of `file:` folder dependencies from disk (`Package::parse`), and stores for each of the latter two a `Resolution::Workspace` / `Resolution::Folder` whose payload is the package directory relative to the top-level dir. Packages from the registry, git or a tarball get their dependency lists from the manifest or the extracted archive instead and have no directory in the project while they resolve. - Dependency edges live in one flat buffer and every package owns a contiguous slice of it, which is how the package that declared an edge is found. The edge stores the specifier as declared; overrides, resolutions and catalogs replace it only for the duration of resolving that edge. - `enqueue_local_tarball` is called from two places: the `Tarball` arm of the resolve pass (no lockfile row yet, the tarball is read to learn the package's name and dependencies) and `enqueue_tarball_for_reading` during install (a row exists in `bun.lock`, but the extracted package is missing from the cache). It computes the on-disk path on the main thread and hands it to a thread pool task, which only reads the file. Closes #39025 Closes #39017 --------- Co-authored-by: Jarred Sumner <jarred@jarredsumner.com>
#39445) Stacked on #36463 (the base branch is that PR's branch, so the diff here is only the additions). Merging this into #36463 adds the behavior changes listed below; #36463 itself now covers the #38333 install batch, the optional-peer correction, and the TOML / `bun init` fixes, so this PR no longer touches those. ### Problem - These 1.3 to 1.4 behavior changes are not in the guide at `701b3e2a0`: - MySQL: the first `caching_sha2_password` connection over plain TCP is refused unless `allowPublicKeyRetrieval: true` (#31129; 1.3.14 requested the key automatically, `MySQLConnection.zig` in the 1.3.14 tag). SQL `tls` / `ssl` options now require TLS instead of falling back to plaintext, and `?ssl=` / `?ssl-mode=` are read (`shared.ts` 1.3.14 only read `?sslmode=`; #37669). - Install: `~/.npmrc` fallback when `XDG_CONFIG_HOME` is set (#36289), credentials in `--registry` / env / bunfig object URLs are sent and outrank same-host `.npmrc` tokens (#38796, #38824), `bun outdated` exits 1 on fetch failures (#38809), new `dedupe` / `up` commands shadow scripts of those names and `bun feedback` is removed (#38333, #38444), `workspace:` ranges inside registry packages (#37669), isolated store entry names (#39014). - Runtime: `module.enableCompileCache()` / `NODE_COMPILE_CACHE` implemented (#34660), `require()` / `import` not-found messages (#34660), `AbortError` message without the period (#39277; 1.3.14's `BunCommonStrings.h` has the period), GCM IV length (#34092), `mkdtemp("")` (#34908), vm options (#38381), `server.reload` (#38697), ICU 75/73 to 78 (#38013), Compression stream chunking (#38695), `Bun.SQL` sqlite bindings (#35950). - Bundler: `splitting` with `cjs` / `iife` is an error (#32685), block-scoped `enum` lowers to `let` (#34249), exports emitted ascending instead of descending (#35957; `doStep5.zig` in 1.3.14 used `sortDesc`), minified `$` (#35668). - The TOML integer bullet did not say what the limit or the fix is. ### Fix - Adds a MySQL public key section (plus a summary table row), a TLS note under the `PGSSLMODE` section, an `.npmrc` / credentials addendum to the `bunfig.toml` section, a `module.enableCompileCache()` section, and the rest as bullets in the existing lists. - `docs/pm/overrides.mdx`: one-line change adding a pointer to this guide in the existing `lockfileVersion` 3 limitation. (The base branch briefly had a duplicate "Nested overrides" section; it removed that itself in `8257d01acb`, and this PR was rebased over it.) - Verification: each runtime claim was run against `1.4.0-canary.1+8326d1bd3` (22 commits behind main; contains every change referenced), and each install or bundler claim was checked against the source on main, with the 1.3 side taken from the `bun-v1.3.14` tag where the PR body did not state it. The `/runtime/sql#mysql` and `/upgrade-to-1.4` links resolve. `prettier --check` passes. ### Not included on purpose - Lifecycle scripts no longer receiving `npm_package_name` / `npm_package_version` / `npm_package_json` / `npm_config_local_prefix` during `bun install`, and transitive `"*"` ranges no longer deduplicating onto the root's version: regressions with open fixes (#36690, #38110, #38770). They need either the fixes or a guide line before release. - Postgres `sslmode=prefer` / `allow` (including `PGSSLMODE=prefer`, which 1.4 newly reads) hangs until the connection timeout against a server without SSL because nothing sends the startup message after the `N` reply. Same code in 1.3.14; filed as a bug instead of documented. <details> <summary>Commands used to verify the runtime claims</summary> ``` timers/promises setTimeout with an aborted signal # "The operation was aborted" bun req.cjs # Cannot find module ... Require stack: bun b.mjs (import() of a missing package / relative file) # Cannot find package 'x' imported from /path, ERR_MODULE_NOT_FOUND bun a_static.mjs (unhandled static import) # printed line still: Cannot find package 'x' from '/path' process.versions.icu # 78.3 createCipheriv("aes-128-gcm", key, Buffer.alloc(129)) # ERR_CRYPTO_INVALID_IV DecompressionStream of a 1 MiB gzip member # 16 chunks of 65536 bytes new SQL("sqlite://:memory:") with ${[1,2]} / ${new Date()} # Binding expected ... fs.mkdtempSync("") # EINVAL vm.runInThisContext("1", []) # ERR_INVALID_ARG_TYPE NODE_COMPILE_CACHE=/tmp/cc bun cc.cjs # creates /tmp/cc/v1.4.0-x86_64-<sha>-<uid> NODE_DISABLE_COMPILE_CACHE=1 + enableCompileCache() # status 3 (DISABLED) bun dedupe / bun up with package.json scripts of those names # built-in command runs bun feedback # Script not found "feedback" Bun.build({ splitting: true, format: "cjs" }) # Code splitting is currently only supported ... bun build of a function-scoped enum and import * as ns # let Color; exports a, m, z new SQL({ url: "postgres://...", tls: true }) on a non-TLS server # ERR_POSTGRES_TLS_NOT_AVAILABLE Bun.TOML.parse("a = 9007199254740993") # Integer cannot be losslessly represented ... ``` </details> <details> <summary>Previous revision</summary> The first revision of this PR (`3c5611454a`) also rewrote the package manager section for #38333 / #38853 (nested overrides and `lockfileVersion: 3`, the optional-peer correction, `bun update`, `bunfig.toml` over `.npmrc`, `--filter`) and fixed the TOML date and `bun init` lines. #36463 picked those up in its own commits the same day, so this PR was rebased onto its new head and reduced to the items above. </details> <!-- robobun:evidence:begin --> --- **no test proof** · iteration 0 · docs-only change; test-proof not applicable <!-- robobun:evidence:end -->
Problem
--linker isolated, a dependency declared as a tarball URL with credentials, e.g."direct": "http://carol:s3cret@127.0.0.1:PORT/cdn/direct-1.0.0.tgz?token=npm_a1b2...", is installed into a store directory literally namednode_modules/.bun/no-deps@http+++carol+s3cret@127.0.0.1+PORT+cdn+direct-1.0.0.tgz+token=npm_a1b2...(reproduced on 1.4.0-canary.1 and current main). A git dependency such asgit+https://carol:s3cret@host/org/repo.gitgetsrepo@git+https+++carol+s3cret@host+org+repo.git+<commit>.import.meta.url,bun pm licensespaths, the--verboselink failure messages, andls node_modules/.bun. The output redaction in install: redact credentials in tarball and git URLs printed as resolutions and specifiers #38977 cannot cover this: these paths really exist on disk.<name>@<resolution store path>(src/install/isolated_install/Store.rsStoreKeyFormatter). For a remote tarball the store path was the whole URL with/ \ : # ?turned into+(src/install/resolution.rsStorePathFormatter, viabun_semver'sString::fmt_store_pathinsrc/semver/lib.rs), and for git it was the whole repository URL plus the commit (src/install/repository.rsStorePathFormatter). Nothing removed the userinfo or the query string.Fix
src/install/resolution.rs: newfmt_store_url, used for the remote tarball store path and, insrc/install/repository.rs, for the repository part of the git store path. It writes the URL without its userinfo and without its query string, and when either of them was present it appends+<16 hex wyhash of the complete URL>. The git commit suffix is unchanged and still follows. Examples:http://carol:s3cret@h/p.tgz?token=xbecomesno-deps@http+++h+p.tgz+<url hash>; the same URL without credentials staysno-deps@http+++h+p.tgz; a credentialed git URL becomesrepo@git+https+++host+org+repo.git+<url hash>+<commit>.src/semver/lib.rs:String::fmt_store_pathnow delegates to a byte-slicefmt_store_path, so the two kept pieces of the URL are spelled exactly the way the whole URL was before (one character mapping, no copy of it).user:password@, a bare token as the username,?token=/ signed URLs); host, path and scheme are not secret, so they stay readable. The userinfo is delimited with RFC 3986's rule, the same onefind_url_passwordinbun_coreuses: the authority runs fromscheme://(or from the start of an scp-likeuser@host:path) to the first/,?or#, and the userinfo is everything in it up to the last@. The whole userinfo goes, not only the password, becausehttps://TOKEN@host/...is a documented way of passing tokens.bun_url::URL::parsewas not used for this because it does not recognize the userinfo ofuser@host:portat all.pkg.tgz?v=1andpkg.tgz?v=2are different packages and must not share a directory, and even for git, where the commit usually disambiguates,resolvedcan be empty for packages migrated from pnpm/yarn lockfiles. Without either part there is nothing to disambiguate, so no hash is appended and every existing entry for a plain tarball,git+file://,github:or registry package keeps its name byte for byte.git+ssh://git@host/...form, whosegit@is not a secret but cannot be told apart from a token); the nextbun installlinks them under the new name andbun pm pruneremoves the old directories, since it removes every store entry the lockfile does not produce. The lockfile itself is untouched.bun pm prunesplits at@(src/install/prune.rssplit_store_key; the names now also contain no second@),bun pm licensesre-derives the name through the same formatter (src/runtime/cli/pm_licenses_command.rs), and the global store derives itslinks/<name>-<entry hash>directory from the same name, so it is fixed as well. install: bound isolated store entry names; tarball URL credentials; file: tarballs relative to their folder package #38867 (bounding long names) would apply on top of this name.test/cli/install/isolated-install.test.ts, describestore entry names of URL dependencies:Bun.hash, thenode_moduleslink points into it,bun.lockstill contains the full URL, the package imports at runtime, and a second install keeps the name?v=get two entries and each alias resolves to its own versioninstall.globalStore, thelinks/directory name is built from the credential-free namehttp://token@host:port/x.tgz) is exercised through the git cases only: the tarball downloader currently sends that URL with the userinfo still in the Host header and gets a 400, which is tracked separately (as is the fact that the downloader never sends URL credentials at all); neither affects this change+x=yname and now asserts the hashed one; its point (no literal?in the name, package resolves at runtime) is unchangedbun bd test. Also run with the change: the rest ofisolated-install.test.ts(75 pass),bun-pm-licenses.test.ts(79 pass, it asserts the unchanged name of a plain tarball entry),bun-prune.test.ts(109 pass),bun-install-git-deps.test.ts(7 pass),cargo clippyandcargo fmt --checkonbun_installandbun_semver, andtest/internal/source-lints.Background
node_modules/.bun/<entry>/node_modules/<name>and everything that depends on it gets a symlink to that directory.<entry>is<package name>@<store path of the resolution>, optionally followed by+<peer hash>; withinstall.globalStorethe entry is itself a symlink into<cache>/links/<entry>-<entry hash>.name@1.2.3; for tarball and git dependencies it is the URL as written in package.json (git: plus the resolved commit), which is what became the directory name here./,\,:,#and?with+(bun_semver'sStorePathFormatter);http://h/p.tgzreadshttp+++h+p.tgz.user:password@part of a URL's authority (scheme://userinfo@host:port/path?query#fragment).bun_wyhash::hash, whatBun.hash()computes), which is how the tests compute the expected names; the store already uses it for the peer hash suffix.Reproduction on the released build
package.jsonwith{"dependencies": {"direct": "http://carol:s3cret@127.0.0.1:PORT/cdn/direct-1.0.0.tgz?token=npm_a1b2c3d4e5f6"}}, aBun.serveanswering that path with a tarball,bunfig.tomlwithinstall.linker = "isolated":With this change the entry is
no-deps@http+++127.0.0.1+35767+cdn+direct-1.0.0.tgz+<16 hex>. The cache folder for the same tarball was already credential-free (@T@<hash>).