chore: handle Rekor 409 conflict during npm provenance publish

[lodekeeper]

Current recommendation

As of 2026-07-28 19:2x UTC, the latest unstable publish.yml runs have green npm-publish jobs again. I do not recommend merging this local pnpm patch while the current signal is green.

Keep this PR open as a ready emergency downstream patch and as the Lodestar pointer to the upstream sigstore-js fix. Merge it only if the Rekor 409 provenance failure recurs before the upstream fix is released/consumed, or if an RC/stable release is blocked by this exact retry-after-success failure mode.

Close/drop this PR once sigstore-js ships the upstream default change and npm/libnpmpublish/lerna-lite consume a version that enables the Rekor conflict recovery path without a Lodestar-local patch.

Problem

The Publish workflow intermittently failed during dev publishes with:

lerna-lite ERR! TLOG_CREATE_ENTRY_ERROR error creating tlog entry - (409) an equivalent entry already exists in the transparency log with UUID ...

The failure was observed across multiple unstable publishes on 2026-07-23 and 2026-07-24. Later unstable publishes are green again, which points to a transient Rekor/signing-path trigger rather than a deterministic Lodestar publishing regression.

Root cause

npm provenance uploads a signed entry to the Sigstore Rekor transparency log via @sigstore/sign (pulled in through lerna-lite -> libnpmpublish -> sigstore). The observed failure matches a retry-after-success path:

1. Rekor creates the entry server-side.
2. The client times out or retries after a transient failure.
3. The retry submits the same equivalent entry.
4. Rekor correctly returns 409 an equivalent entry already exists.
5. @sigstore/sign treats that 409 as fatal because the default Rekor witness config has fetchOnConflict: false.

@sigstore/sign already has the needed recovery behavior: on 409 it can fetch the existing entry and continue. The issue is that the high-level default disables it.

Emergency downstream patch

This PR uses pnpm patch sigstore to flip the Rekor witness default used by the Lodestar publish stack:

   new RekorWitness({
     rekorBaseURL: options.rekorURL,
-    fetchOnConflict: false,
+    fetchOnConflict: true,
     retry: options.retry ?? DEFAULT_RETRY,
     timeout: options.timeout ?? DEFAULT_TIMEOUT,
   })

On a 409 the client fetches the already-created Rekor entry and the publish proceeds. Provenance is preserved; the client uses the entry Rekor already recorded.

Upstream

Opened upstream sigstore-js tracking and fix:

• Issue: Recover Rekor duplicate-entry conflicts by default sigstore/sigstore-js#1708
• PR: fix(sign): fetch existing Rekor entries by default sigstore/sigstore-js#1709

Similar downstream reports:

• Release npm en échec sur cdtn-utils (Sigstore/Rekor TLOG 409) SocialGouv/code-du-travail-numerique#7419
• Release pipeline: detect & prevent git-ahead-of-npm drift (post-publish verification + graceful Rekor 409) apify/apify-shared-js#649

Alternatives considered

• Bump lerna-lite/sigstore -- no effect yet; fetchOnConflict: false is still the default in currently consumed versions.
• Disable provenance on dev -- drops supply-chain provenance; rejected.
• Re-run the job -- can work for dev publishes when a later commit changes the tarball/version, but is not a robust stable/RC release strategy if the same package/version is wedged after Rekor already recorded the entry.
• Increase timeout -- may reduce timeout-triggered retries, but does not make the Rekor create-entry operation idempotent across 5xx/429/retry-after-success cases.

Notes

• First patchedDependencies entry in the repo.
• Treat this as a hold/emergency patch unless the failure recurs or a release publish is blocked before upstream is available.

🤖 Generated with AI assistance

[matthewkeil]

@lodekeeper please identify the root cause of the failures. Something seems wrong with your analysis and a patch seems like the wrong fix. Why did it just start to fail on the last 4 runs?

[lodekeeper]

(Duplicate reply — apologies for the noise. See the follow-up comment below for the complete root-cause analysis and recommendation.)

[lodekeeper]

@matthewkeil You're right to push back — the patch treats a symptom, not the cause. Here's the root cause.

It's not our code. The (409) an equivalent entry already exists is a retry-after-success inside sigstore's Rekor client, not a genuine duplicate:

• @sigstore/sign@4.0.1 (external/fetch.js) sets DEFAULT_TIMEOUT = 5000 (5s/attempt) and DEFAULT_RETRY = {retries: 2}, and it retries the Rekor POST on a client-side timeout or a 408/429/≥500.
• When Rekor commits the entry server-side but takes >5s to respond (or blips a 5xx), the client aborts at 5s and re-POSTs the identical payload → Rekor returns 409 (a dup of the entry it just wrote).
• sigstore@4.0.0 (config.js:99) hardcodes fetchOnConflict: false on the RekorWitness, so @sigstore/sign skips its 409-recovery getEntry() path and throws TLOG_CREATE_ENTRY_ERROR. The failed first attempt logs to npm's proc-log rather than lerna-lite's logger, which is why the run output shows only the 409 and hides the timeout.

Why the last ~4 runs and not before:

• Those are 4 different commits, each the nightly publish.yml on push to unstable: 4606097ec4 (feat: add fork choice compliance test for pre-gloas forks #9670) → 845585420e → 3d99c9cb5c → 3c41f4be9a. Last green was d16fa2b13a (07-22 06:01).
• Signing deps are frozen: pnpm-lock.yaml last changed 2026-06-19 (a js-yaml bump) — sigstore / lerna-lite / libnpmpublish are all untouched. A deterministic dependency bug can't newly appear 07-23 on deps that froze 06-19.
• feat: add fork choice compliance test for pre-gloas forks #9670 (the first failure) touched only fork-choice + spec-test config — nothing in the publishing path.
• Every 409 UUID shares the same Rekor tree id 108e9186e8c5677a but has a different leaf hash → each run collides with its own in-run entry, which is the signature of retry-after-success, not a cross-commit collision.
• Every publish job, including the last green one, ran Node v24.18.0, so this is not the June keep-alive / CVE-2026-48931 premature-close regression that produced this exact 409 signature in other projects that week — a runtime bug can't pass then fail on the same version.

So the trigger is external: Rekor write latency crossed the 5s client timeout (or a transient 5xx/429) around 07-23. Sigstore's status page showed no incident, but elevated write latency wouldn't necessarily post there.

On the fix: fetchOnConflict: true (this PR) is sigstore's own escape hatch and is the correct idempotency handling for a benign 409 — but it papers over an external blip on supply-chain signing tooling. And fetchOnConflict: false is hardcoded at every sigstore version (4.x, 5.0.0), so no version bump fixes it either.

My recommendation: hold this PR and let the next dev publish be the test. If the Rekor latency was transient it self-heals with no patch at all; if it persists, the fix belongs upstream (default fetchOnConflict: true / don't retry the non-idempotent create / raise the 5s timeout) rather than a local pnpm patch we'd have to carry on signing tooling. Reverting to NPM_TOKEN isn't an option — ChainSafe publishing is OIDC/provenance-only. Note re-running the same sha won't recover: byte-identical tarball → same attestation → same 409.

[nflaig]

@lodekeeper there seem to be green ci runs on unstable again, please confirm, and reconsider if we need this pr

[lodekeeper]

Confirmed the state — but the "green again" read only holds for the last two runs; the npm-publish job still failed within the last hour with the exact same bug. Run-level status is misleading here because the Docker Hub jobs fail independently of npm publish, so here's the job-level breakdown of the last 5 publish.yml runs on unstable:

Run (UTC)	Commit	Publish to NPM Registry	Note
17:04	#9645 88c017b	✅ success	Docker also green
16:53	#9643 f6c521a	✅ success	run shows "failure" = Docker Hub amd64 build only, unrelated
16:37	#9654 bcf03e7	❌ failure	Rekor 409 TLOG_CREATE_ENTRY_ERROR @ 16:40:26Z, UUID 108e9186e8c5677a…
16:34	#9681	cancelled	—
16:28	#9419	✅ success	—

So the npm-publish job failed at 16:40:26Z — ~20 min before your comment — with the same signature and same Rekor tree UUID (108e9186e8c5677a…) we already documented. Two green runs after it isn't a fix.

This is the self-heal behavior, not a resolution: the dev version is SHA-deterministic, so a wedged run only clears when the next commit pushes a new tarball (new tlog leaf). Each publish is still a coin-flip on whether Rekor commits-but-responds-late and trips the retry-after-success 409 — 16:53 and 17:04 just didn't hit the slow-Rekor race; 16:37 did.

Recommendation: keep the PR / hold, don't close it.

1. The bug isn't gone — it recurred ~20 min ago.
2. fetchOnConflict: true is the correct idempotency handling for exactly this race (Rekor already committed our entry; we fetch it instead of erroring), not a symptom mask.
3. The self-heal only rescues dev publishes — you can push another commit for a new SHA. A stable/RC release publish can't: the version is fixed to the tag, so once this race wedges it, re-running keeps 409ing against the entry Rekor already wrote (identical tarball → identical leaf), and fetchOnConflict: false makes that fatal with no recourse short of a version bump. This patch is what protects that path.

Happy to file the upstream sigstore/rekor issue in parallel, but I wouldn't drop our defensive patch while the failure is still live on unstable.

[lodekeeper]

Upstream tracking/fix is now filed:

• sigstore/sigstore-js issue: Recover Rekor duplicate-entry conflicts by default sigstore/sigstore-js#1708
• sigstore/sigstore-js PR: fix(sign): fetch existing Rekor entries by default sigstore/sigstore-js#1709

The upstream PR changes @sigstore/sign's Rekor conflict behavior so create-entry 409 conflicts fetch the existing Rekor entry by default. It preserves explicit fetchOnConflict: false, adds a regression test, and includes a changeset.

I also updated this PR body to reference the upstream issue/PR and the similar downstream SocialGouv/code-du-travail-numerique#7419 + apify/apify-shared-js#649 reports.

🤖 Generated with AI assistance

[lodekeeper]

Fresh check after the latest unstable publish signal:

• the six newest publish.yml runs have green npm-publish jobs again: 30358898476, 30344094906, 30312548013, 30253409358, 30251492850, 30235593008
• the last confirmed npm-provenance failures I see are still the 2026-07-24 runs 30112910434 and 30113282236
• upstream fix(sign): fetch existing Rekor entries by default sigstore/sigstore-js#1709 is open, DCO-green, and has Matthew's approval

So my recommendation changes from “keep/hold because the failure is still live” to: do not merge this Lodestar-local patch while the current signal is green; keep the PR only as a ready emergency patch and pointer to the upstream fix.

The stable/RC concern is still the reason not to delete the work outright: if a release publish hits retry-after-success before upstream ships and is consumed, re-running the same package/version can keep colliding with the Rekor entry that already exists. But carrying the first local patchedDependencies entry for publishing infrastructure is heavy enough that we should prefer the upstream fix while CI is healthy.

I updated the PR body to reflect this hold/emergency-patch status instead of the older “dev publish is currently failing every run” framing.

🤖 Generated with AI assistance

[matthewkeil]

LGTM 🎸

[nflaig]

latest ci run on unstable failed due to this again, can we merge this?

[wemeetagain]

🎉 This PR is included in v1.46.0 🎉