Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backports release 1.11 #55344

Merged
merged 59 commits into from
Aug 26, 2024
Merged

Backports release 1.11 #55344

merged 59 commits into from
Aug 26, 2024

Conversation

KristofferC
Copy link
Member

@KristofferC KristofferC commented Aug 2, 2024

Backported PRs:

Need manual backport:

Contains multiple commits, manual intervention needed:

Non-merged PRs with backport label:

IanButterworth and others added 16 commits August 2, 2024 11:56
I think this tool is there mainly to see what's taking so long, so
timing information is helpful.

(cherry picked from commit f2558c4)
As raised in
<JuliaLang/StyledStrings.jl#57 (comment)>,
when the eltype of an iterator is non-concrete, `_isannotated` will
return false. To properly check such cases, we need to see if any of the
elements of the iterator are annotated.

This is a bit of an interesting case, since:
- Most of the time it shouldn't be hit, we can reasonably expect most
iterables to infer as producing concrete types
- The eltype of the iterator is (generally) known at compile-time, and
so in any case other than the ambiguous non-concrete one, this check
remains able to be done at compile-time.

With this change, join always preserves annotations. The compromise made
is that iterators with non-concrete eltypes can result in join inferring
a union return type (i.e. type instability with non-concrete iterators),
but that doesn't seem too bad to me (I don't see how we could be
completely type stable without concrete types here).

(cherry picked from commit 462d7f4)
The docstring of `LinearAlgebra.reflectorApply!` is incorrect. It says
the function is applying `(I - τ*[1; x] * [1; x]')*A`, but in reality it
is applying `(I - conj(τ)*[1; x[2:end]]*[1; x[2:end]]')*A`. You can
check that by looking at the code, or running for example
```julia
using LinearAlgebra
T=ComplexF64;d=5;
τ=randn(T);x=randn(T,d);A=randn(T,d,d);
(I - conj(τ)*[1; x[2:end]]*[1; x[2:end]]')*A
LinearAlgebra.reflectorApply!(x,τ,A)
```

(cherry picked from commit 1ece299)
This will allow us to resolve the `Dates` at compile-time eventually. It
also fixes `TOML.Parser()` to return Dates types again.

(cherry picked from commit 0ffbae8)
`stack` does not require that the inner iterator defines `axes`, but the
code to assemble an error message assumed this.

Found here:
https://discourse.julialang.org/t/reduce-hcat-is-type-unstable/112800/3

(cherry picked from commit ae483c3)
Changes made:
- Use 0 for the thread ID to ensure it's always valid. The function
expects `0 <= tid < jl_n_threads` so 1 is incorrect if `jl_n_threads` is
1.
- After retrieving the affinity mask with `jl_getaffinity`, pass that
same mask back to `jl_setaffinity`. This ensures that the mask is always
valid. Using a mask of all ones results in `EINVAL` on FreeBSD. Based on
the discussion in #53402, this change may also fix Windows, so I've
tried reenabling it here.
- To check whether `jl_getaffinity` actually did something, we can check
that the mask is no longer all zeros after the call.

Fixes #54817

(cherry picked from commit 8a7e23d)
…l` wrapper correctly. (#54736)

(cherry picked from commit 34cd610)
This PR reverts the optimization from
748149e (part of #48167), while
keeping the fix for merging occurs_inv/occurs_cov, as that optimzation
makes no sense especially when typevar occurs both inside and outside
the inner intersection.

Close #55206

(cherry picked from commit fb6b790)
It can be very helpful, when struggling with LBT forwarding, to see what
functions were actually forwarded to a new library. This utility
function makes it easy to query which functions are forwarded to that
library.

(cherry picked from commit e0f2e29)
The following symbols:
`seed!, default_rng, Sampler, SamplerType, SamplerTrivial,
SamplerSimple` Are documented in the Julia documentation and unexported,
but not marked as public.

Co-authored-by: Max Horn <[email protected]>
(cherry picked from commit c66513f)
In 1.10 we have

```jl
julia> isleapyear(Year(1992))
false
```

which is semantically incorrect because `Year(1992)` is a duration (1992
years), not an instant. This PR restricts the currently unrestricted
argument to integers.

(cherry picked from commit fdecc59)
(cherry picked from commit b759fe2)
This includes support to properly forward MKL v2024's ILP64 CBLAS
symbols, which fixes this [Enzyme
issue](EnzymeAD/Enzyme.jl#1683)

(cherry picked from commit 602b582)
Addresses the `Matrix` cases from
#55296. This restores the
behavior to match that on v1.10, and preserves the structure of the
matrix on scaling by `NaN`. This behavior is consistent with the
strong-zero behavior for other structured matrix types, and the scaling
may be seen to be occurring in the vector space corresponding to the
filled elements.

After this,
```julia
julia> UpperTriangular(rand(2,2)) * NaN
2×2 UpperTriangular{Float64, Matrix{Float64}}:
 NaN    NaN
    ⋅   NaN
```
cc. @mikmoore

(cherry picked from commit 0ef8a91)
@KristofferC KristofferC added the release Release management and versioning. label Aug 2, 2024
@giordano
Copy link
Contributor

giordano commented Aug 2, 2024

I pushed a commit to add a NEWS entry about #45641 (ref: #54107 (comment)), basically a backport of #55282.

DilumAluthgeBot and others added 11 commits August 5, 2024 17:44
After #55180 we implicitly require an LLVM built with Zlib support, but
compiling Julia with `make USE_BINARYBUILDER_LLVM=0` builds an LLVM
without Zlib support, despite the fact we attempt to request it at

https://github.com/JuliaLang/julia/blob/996351f5f6651d1508aef3c35c7d37eb22a0fb1e/deps/llvm.mk#L97
This was first identified in #55337.

`ZLIB_LIBRARY` must be the path to the zlib library, but we currently
set it to the libdir where the library is installed (introduced in

https://github.com/JuliaLang/julia/blob/996351f5f6651d1508aef3c35c7d37eb22a0fb1e/deps/llvm.mk#L97
which is wrong. However, CMake is actually able to find Zlib correctly,
but then the check at
https://github.com/llvm/llvm-project/blob/46425b8d0fac3c529aa4a716d19abd7032e452f3/llvm/cmake/config-ix.cmake#L139-L141
uses the value of `ZLIB_LIBRARY` to list the Zlib to link for the test,
but being `ZLIB_LIBRARY` a directory, CMake doesn't see any valid Zlib
and thus tries to run the test without linking any Zlib, and the test
silently fails (they're silent only when `LLVM_ENABLE_ZLIB=ON`),
resulting in no usable Zlib available, even if found.

`ZLIB_ROOT` is the only [hint recommended by the CMake module
`FindZLIB`](https://cmake.org/cmake/help/latest/module/FindZLIB.html#hints).
This PR replaces a broken `ZLIB_LIBRARY` with an appropriate
`ZLIB_ROOT`. Also, we set `LLVM_ENABLE_ZLIB=FORCE_ON` which is the only
way to make CMake fail loudly if no usable Zlib is available, and avoid
going on with a non-usable build.

I confirm this fixes #55337 for me, it should likely address
JuliaCI/julia-buildkite#373 as well.

Also, options `COMPILER_RT_ENABLE_IOS`, `COMPILER_RT_ENABLE_WATCHOS`,
`COMPILER_RT_ENABLE_TVOS`, and `HAVE_HISTEDIT_H` don't exist anymore,
and they are removed.
This ensures that `tr` for a block `SymTridiagonal` symmetrizes the
diagonal elements.

(cherry picked from commit a163483)
If the LimitedAccuracy was supposed to resolve against the top-most
frame (or hypothetically a non-InferenceState frame), it would not have
a parentframe, preventing it from reaching the subsequent
poison_callstack line that is required for reliable inference (avoiding
caching bad results). This should restore the original intent of this
code (pre #48913)

(cherry picked from commit d1b1a5d)
…data (#55306)

Co-authored-by: Valentin Churavy <[email protected]>
Co-authored-by: Mosè Giordano <[email protected]>
Co-authored-by: Gabriel Baraldi <[email protected]>
(cherry picked from commit 1e1e710)
(cherry picked from commit 09e5c40)
In extreme cases, the compiler could mark this function for
concrete-eval, even though that is illegal unless the compiler has first
deleted this instruction. Otherwise the attempt to concrete-eval will
re-run the function repeatedly until it hits a StackOverflow.

Workaround to fix #55147

@aviatesk You might know how to solve this even better, using
post-optimization effect refinements? Since we should actually only
apply the refinement of terminates=false => terminates=true (and thus
allowing concrete eval) if the optimization occurs, and not just in
inference thinks the optimization would be legal.

---------

Co-authored-by: Shuhei Kadowaki <[email protected]>
Comparing objects by `==` will happily answer nonsense for malformed
type comparisons, such as `unwrap_unionall(A) == A`. Avoid forming that
query. Additionally, need to recourse through Vararg when examining type
structure to make decisions.

Fix #55076
Fix #55189

(cherry picked from commit 32423a8)
d-netto and others added 22 commits August 14, 2024 15:11
We have parallel sweeping on 1.11, so we should use atomics here.

This issue has already been fixed on master due to
#54961.

We can also backport #54961 to
release-1.11, but I have some preference to land this one since it's
considerably less disruptive and we're already on RC2.
In some ad-hoc testing, I had JIT about 19 MB of code and data, which
generated about 170 MB of debuginfo alongside it, and that debuginfo
then compressed to about 50 MB with this change, which simply compresses
the ObjectFile until it is actually required (which it very rarely is
needed).

(cherry picked from commit fe597c1)
More finely scope the `@inbounds` annotations to ensure neither `f` nor
`op` are erroneously `@inbounds`ed.

(cherry picked from commit 1dffd77)
ThreadSynchronizer is only for things that are very trivial, as there
are a lot of things they are forbidden from doing (such as waiting for a
Task to set it).

Happened to notice while reviewing
#55439 (review)
that this was still using the pre-v1.2 style lock, which makes this
mostly useless in v1.4+

(cherry picked from commit 2a4e2b1)
I was confused why #49121 was
re-occuring locally, until I noticed this file was not getting rebuilt.

(cherry picked from commit b4ebb00)
This brings the docstring closer to the actual implementation. In
particular, following the current docstring and defining
```julia
symmetric(::MyMatrix, uplo=:U)
```
leads to a method ambiguity, as `LinearAlgebra` defines
`symmetric(::AbstractMatrix, uplo::Symbol=:U)`.

(cherry picked from commit 8a19b74)
…55342)

This fixes the error message if the matrix is uninitialized. This is
because a `Bidiagonal` with `uplo == 'L'` may still be `istriu` if the
subdiaognal is zero. We only care about the band index in the error
message, and not the values.

(cherry picked from commit f2f188d)
Stop loading `ScopedValues` with `using` so folks use
`ScopedValues.with` or `using ScopedValues` rather than `Base.with`.

Implements
#55095 (comment)

~Have to bump the StyledStrings stdlib to include
JuliaLang/StyledStrings.jl#80 Done

---------

Co-authored-by: Dilum Aluthge <[email protected]>
(cherry picked from commit e1aefeb)
The `clamp` function was defined in Base.Math, but required to be in
Base now, so move it to intfuncs with other similar functions

Fixes #55279

(cherry picked from commit 3db1d19)
This adds the `terminfo` database to `deps/`, providing a better user
experience on systems that don't have `terminfo` on the system by
default. The database is built using BinaryBuilder but is not actually
platform-specific (it's built for `AnyPlatform`) and as such, this
fetches the artifact directly rather than adding a new JLL to stdlib,
and it requires no compilation.

A build flag, `WITH_TERMINFO`, is added here and assumed true by
default, allowing users to set `WITH_TERMINFO=0` in Make.user to avoid
bundling `terminfo` should they want to do so.

The lookup policy for `terminfo` entries is still compliant with what's
described in `terminfo(5)`; the bundled directory is taken to be the
first "compiled in" location, i.e. prepended to `@TERMINFO_DIRS@`. This
allows any user settings that exist locally, such as custom entries or
locations, to take precedence.

Fixes #55274

Co-authored-by: Mosè Giordano <[email protected]>
(cherry picked from commit e7e8768)
The Julia memory model is always inbounds for GEP.

This makes the code in #55090
look almost the same as it did before the change. Locally I wasn't able
to reproduce the regression, but given it's vectorized code I suspect it
is backend sensitive.

Fixes #55090

Co-authored-by: Zentrik <[email protected]>
(cherry picked from commit 7e1f0be)
…5359)

This should hopefully fix the failing tests.

Co-authored-by: Kristoffer Carlsson <[email protected]>
)

The `(::Diagonal) + (::Symmetric)` and analogous methods were
specialized in #35333 to return a
`Symmetric`, but these only work if the `Diagonal` is also symmetric.
This typically holds for arrays of numbers, but may not hold for
block-diagonal and other types for which symmetry isn't guaranteed. This
PR restricts the methods to arrays of `Number`s.

Fixes, e.g.:
```julia
julia> using StaticArrays, LinearAlgebra

julia> D = Diagonal(fill(SMatrix{2,2}(1:4), 2))
2×2 Diagonal{SMatrix{2, 2, Int64, 4}, Vector{SMatrix{2, 2, Int64, 4}}}:
 [1 3; 2 4]      ⋅
     ⋅       [1 3; 2 4]

julia> S = Symmetric(D)
2×2 Symmetric{AbstractMatrix, Diagonal{SMatrix{2, 2, Int64, 4}, Vector{SMatrix{2, 2, Int64, 4}}}}:
 [1 3; 3 4]      ⋅
     ⋅       [1 3; 3 4]

julia> S + D
2×2 Symmetric{AbstractMatrix, Diagonal{SMatrix{2, 2, Int64, 4}, Vector{SMatrix{2, 2, Int64, 4}}}}:
 [2 6; 6 8]      ⋅
     ⋅       [2 6; 6 8]

julia> S[1,1] + D[1,1]
2×2 SMatrix{2, 2, Int64, 4} with indices SOneTo(2)×SOneTo(2):
 2  6
 5  8

julia> (S + D)[1,1] == S[1,1] + D[1,1]
false
```
After this,
```julia
julia> S + D
2×2 Matrix{AbstractMatrix{Int64}}:
 [2 6; 5 8]  [0 0; 0 0]
 [0 0; 0 0]  [2 6; 5 8]
```

Even with `Number`s as elements, there might be an issue with `NaN`s
along the diagonal as `!issymmetric(NaN)`, but that may be a different
PR.

(cherry picked from commit 197295c)
There's a few reasons for making it public:
- It's already mentioned in the manual (#54211).
- It's the easiest way to deprecate a function that shouldn't be used
anymore at all.
- It's already widely used in the ecosystem:
https://juliahub.com/ui/Search?type=code&q=Base.depwarn(

I also moved the `@deprecate` docs into a new `Managing deprecations`
section because I felt it should go together with `Base.depwarn()`.
Might be worth backporting to 1.11?

(cherry picked from commit 442f9d5)
Currently, these are inferred as a 2-Tuple of possible return types
depending on `jobz`, but since `jobz` is usually a constant, we may
propagate it aggressively and have the return types inferred concretely.

(cherry picked from commit 686804d)
)

Stdlib: StyledStrings
URL: https://github.com/JuliaLang/StyledStrings.jl.git
Stdlib branch: main
Julia branch: master
Old commit: d7496d2
New commit: f6035eb
Julia version: 1.12.0-DEV
StyledStrings version: 1.11.0(Does not match)
Bump invoked by: @LilithHafner
Powered by:
[BumpStdlibs.jl](https://github.com/JuliaLang/BumpStdlibs.jl)

Diff:
JuliaLang/StyledStrings.jl@d7496d2...f6035eb

```
$ git log --oneline d7496d2..f6035eb
f6035eb Replace accidental Int64s with Ints
4fcd8bb Use const fields in parser State instead of refs
35a3cdf Load user-customisations lazily
9802b6c Load ScopedValues symbols from their source
9b9cf71 Use branches when choosing how to merge face attrs
eada2dc Avoid needlessly creating a new Face in get calls
c647af9 Avoid boxing mergedface by making it toplevel
a117008 Avoid creating strings for ansi_4bit_color_code
6863348 Improve type inference of face merging
f588218 Quick fix for 4d04102adf0d (Optimised SimpleColor)
4d04102 Optimise creation of a SimpleColor from a UInt32
6d3f44d Actually overload Base's escape_string
58507e5 Fully qualify method overloads, avoid importing
fc686f3 Explicitly test eachregion
c417262 Refactor eachregion to be O(n log n) not O(n^2)
f7af623 Use concrete refs in macro parser state struct
41b2446 Check for underline term capability flag
987f776 Treat printing as more than a nothing-return write
43fb018 Add types to some comprehensions
d3aa7e1 Improve inference with a function over a closure
6901610 Mention the importance of semantic names in docs
0be209b Better hint at the package capabilities in readme
37b9e4b Load no faces.toml when the DEPOT_PATH is empty
```

Co-authored-by: Dilum Aluthge <[email protected]>
(cherry picked from commit a23aee8)
Later code likes to see that the type is consistent with the cgval and the unbox.

(cherry picked from commit e1e5a46)
Due to limitations in the LLVM implementation, we are forced to emit
fairly bad code here. But we need to make sure it is still correct with
respect to GC rooting.

The PR 50833c8 also changed the meaning
of haspadding without changing all of the existing users to use the new
equivalent flag (haspadding || !isbitsegal), incurring additional
breakage here as well and needing more tests for that.

Fixes #54720

(cherry picked from commit 8bfef8f)
@KristofferC
Copy link
Member Author

@nanosoldier runtests()

@nanosoldier
Copy link
Collaborator

The package evaluation job you requested has completed - possible new issues were detected.
The full report is available.

@KristofferC
Copy link
Member Author

@nanosoldier runtests(["ZippedArrays", "CassetteOverlay", "PolyLog", "UnixTimes", "Suppressor", "TZJData", "DateFormats", "Overseer", "Loess", "Mocking", "JuliaInterpreter", "Memento", "DispatchDoctor", "FactorLoadingMatrices", "Dubins", "MixedStructTypes", "AbstractFFTs", "ArraysOfArrays", "ExponentialAction", "DataFlowTasks", "LogDensityProblemsAD", "FlexiMaps", "JSON3", "LogExpFunctions", "LinearMaps", "NPFinancial", "CatmullRom", "ParameterHandling", "AffineArithmetic", "LIBSVM", "SatelliteDynamics", "ChainRulesTestUtils", "Polynomials", "Roots", "SpecialFunctions", "FastChebInterp", "XsdToStruct", "TypeClasses", "ImageShow", "Term", "BasicBSpline", "StatsFuns", "DataSkimmer", "JpegTurbo", "TimeSeriesEcon", "UTCDateTimes", "Quaternionic", "TimeZones", "TimesDates", "TidierDates", "DataGraphs", "DiffImageRotation", "SpectralEnvelope", "BatchedTransformations", "LaxZonedDateTimes", "HigherOrderKernels", "Hankel", "CloudEvents", "EinExprs", "Neurthino", "EnergyExpressions", "LoggingFacilities", "FeatureTransforms", "Norg", "TimeStruct", "SPDX", "RadonKA", "ForwardDiffPullbacks", "DiffPointRasterisation", "GraphViz", "TotalVariation", "ElementarySymmetricFunctions", "PkgToSoftwareBOM", "OkFiles", "MeasureBase", "DataToolkitCommon", "ChunkedCSV", "CodingTheory", "OpenAPI", "GitForge", "MatterMost", "TestTools", "OpenPolicyAgent", "SunCalc", "AnovaGLM", "ChemicalIdentifiers", "MixedModelsDatasets", "Timestream", "RangeTrees", "AWSInfinidash", "TransformSpecifications", "LabelledArrays", "Azure", "CoDa", "FHIRClient", "Bigsimr", "SmallDatasetMaker", "PetroleumModels", "OMOPCDMCohortCreator", "StateSpaceEcon", "OndaEDFSchemas", "ReverseMcCormick", "RegressionTables", "Plasmo", "AlignedSpans", "BivariateCopulas", "ClimateModels", "JDF", "SQLCompose", "TimeDag", "Bootstrap", "RecoverPose", "RDatasets", "TypeDBClient", "AbstractLogic", "TexTables", "RateTables", "VLBILikelihoods", "FredData", "BestModelSubset", "XPORTA", "PWF", "ParetoSmooth", "ScikitLearn", "DistributionMeasures", "CoinbasePro", "CompositionalNetworks", "Powersense", "Trading", "EnergyModelsCO2", "EnergyModelsRenewableProducers", "EnergyModelsGeography", "SQLREPL", "Effects", "MLUtils", "OPFSampler", "OndaEDF", "NURBS", "Transits", "LocalSearchSolvers", "CryptoMarketData", "Ripserer", "KiteUtils", "StateSpaceModels", "GraphSignals", "WinchModels", "KitePodModels", "DTables", "NetDecOPF", "DecisionMakingPolicies", "BayesSizeAndShape", "BadukGoWeiqiTools", "HDMjl", "Gadfly", "FeatureSelectors", "AtmosphericModels", "WhittleLikelihoodInference", "GLFixedEffectModels", "ImageQualityIndexes", "RDataGet", "MITgcm", "TensorTrains", "OPFLearn", "TidierDB", "POMDPSolve", "SignalingDimension", "GenieAuthorisation", "InvariantPointAttention", "PlutoStaticHTML", "JumpProcesses", "TimeZoneFinder", "OrdinalMultinomialModels", "TaijaBase", "MixedModelsSim", "MixedModelsExtras", "LiftAndLearn", "MixedModelsPermutations", "AnovaMixedModels", "OperatorFlux", "MixedModels", "ITensorMPS", "QuasiCopula", "NextGP", "WaveOpticsPropagation", "ReachabilityAnalysis", "MetidaBioeq", "EverySingleStreet", "TidierFiles", "FluxTraining", "HydroPowerSimulations", "Images", "Consensus", "BloqadeDormandPrince", "GCIdentifier", "Knockoffs", "MarginalLogDensities", "MimiRFFSPs", "ArgoData", "UnfoldSim", "Plots", "AutocorrelationShell", "KiteModels", "Simpsons", "EntropyHub", "BloqadeWaveforms", "PhyNEST", "TrillionDollarWords", "SimpleCrop", "ClimateTools", "LeafGasExchange", "Garlic", "BloqadeKrylov", "ClimatePlots", "EqualitySampler", "Lighthouse", "IonSim", "SwissVAMyKnife", "GasChromatographySimulator", "GeometricIntegrators", "RetentionParameterEstimator", "QuantumAnnealingAnalytics"])

@nanosoldier
Copy link
Collaborator

The package evaluation job you requested has completed - possible new issues were detected.
The full report is available.

@KristofferC
Copy link
Member Author

gnome-shell-screenshot-7y9y6y

🎉

@KristofferC KristofferC merged commit 0e42398 into release-1.11 Aug 26, 2024
8 checks passed
@KristofferC KristofferC deleted the backports-release-1.11 branch August 26, 2024 08:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release Release management and versioning.
Projects
None yet
Development

Successfully merging this pull request may close these issues.