Fix manifest validation issue#4541
Conversation
|
Sadly this isn't very easy to test because we don't have a simple reproducer, I only saw this on CI, not locally. One could try to jump on those machines with tmate, but I haven't had a chance to do that (CC @vchuravy in case you have spare time for this) |
reproduces for me: Important was to remove the |
|
Thanks. I assume you've checked out EnzymeAD/Enzyme.jl@ |
|
No need, I was on |
|
I can reproduce it locally and this PR fixes it. All I did between testing is delete the manifest. Same depot. |
IanButterworth
changed the title
|
Do you have an idea of could have suddenly caused this? Could have been a change in the registry? |
|
Not me, but Claude. Does the following make sense? Yes, this could absolutely be caused by registry state changes. Here's how: Registry-dependent scenarios that could trigger this bug:
Let me verify this by checking when LLVM_jll added Using "Fetch Web Page" Fetched https://raw.githubusercontent.com/JuliaRegistries/General/master/L/LLVM_jll/WeakDeps.toml Using "Fetch Web Page" Using "Fetch Web Page" Fetched https://github.com/JuliaPackaging/Yggdrasil/tree/master/L/LLVM Based on my analysis, yes, this bug is almost certainly dependent on registry state. Here's the detailed explanation: Why this is registry-dependent:The Root CauseThe bug occurs when:
Registry State Triggers
|
|
@giordano I'm holding off merging this because you mentioned you could test the theory that the registry change was what caused this to start failing |
|
While I'm looking into reproducing the issue with older registry, do we know a workaround for stable versions of Julia? This is now affecting CI of many Clima packages with a new error This is extremely disruptive. |
|
@KristofferC fyi |
|
I can't reproduce the issue if I check out JuliaRegistries/General@cf36cd2 (just a random commit from a few weeks ago). Looks like something broke in the registry. I'll now try to git bisect it. |
|
Ooof, I made a mistake: I wanted to reduce the tests in Enzyme, but I clearly deleted too much stuff: if I keep Enzyme's For what is worth, these were the script I used for trying and reproduce the error in a clean environment: #!/bin/bash
DIR="${HOME}/tmp/enzyme-general"
ENZYME_DIR="${DIR}/Enzyme"
GENERAL_DIR="${DIR}/General"
# Check out an old enough version of Enzyme
git clone https://github.com/EnzymeAD/Enzyme.jl "${ENZYME_DIR}"
git -C "${ENZYME_DIR}" checkout v0.13.90
# Remove all the tests, we don't want to run them, just keep a dummy runtests.jl
find "${ENZYME_DIR}/test" -name '*.jl' -delete
touch "${ENZYME_DIR}/test/runtests.jl"
git clone https://github.com/JuliaRegistries/General "${GENERAL_DIR}"
#!/bin/bash
DIR="${HOME}/tmp/enzyme-general"
export TMPDIR="${DIR}/tmp"
mkdir -p "${TMPDIR}"
DEPOT_PATH="$(mktemp -d)"
export JULIA_DEPOT_PATH="${DEPOT_PATH}:"
# Use git registry, clone it locally to speed it up.
export JULIA_PKG_SERVER=""
GENERAL="${DEPOT_PATH}/registries/General"
git clone --depth=1 "${DIR}/General" "${GENERAL}"
# Remove Enzyme's manifest
rm -f "${DIR}/Enzyme/Manifest.toml"
julia +1.13 --project="${DIR}/Enzyme" --color=yes --warn-overwrite=yes --depwarn=yes --inline=yes --startup-file=no --track-allocation=none --check-bounds=yes --compiled-modules=yes --depwarn=yes -e 'import Pkg; Pkg.test(;coverage=false, julia_args=["--check-bounds=yes", "--compiled-modules=yes", "--depwarn=yes"], test_args=["--verbose"], force_latest_compatible_version=false, allow_reresolve=true)' |
|
If this fixes things let's get it in sooner rather than later. Can work on the understanding of it later :P. Since we have a repo, someone could bisect Pkg I guess? |
(cherry picked from commit 831f8db)
(cherry picked from commit 831f8db)
(cherry picked from commit 831f8db)
4 participants
Fixes #4540
The issue isn't obvious to me, but this is the best Claude Opus 4.5 could come up with
Details
Bug Analysis: Missing Manifest Entry Error
Error Message
Also seen with:
Enzyme→ADTypesKey Observations
Root Cause
When building
deps_mapfor manifest entries,query_deps_for_versionreturns ALL deps including weak deps:But weak deps that weren't triggered (no extension trigger present) are NOT in
vers/pkgsbecause the resolver marked them as "uninstalled".Result:
entry.depscontains UUIDs that have no corresponding manifest entry.Why Existing Fix Doesn't Help
Commit
cdc17a0d7("allow having unknown weak dependencies") handles weak deps that are not in any registry. It filters them fromfixedbefore resolution.This bug is about weak deps that ARE registered but weren't resolved because the trigger wasn't needed.
The Fix
Filter
deps_mapto only include UUIDs that are actually inpkgs:Why Intermittent?
Unclear. Possibly related to:
@giordano