Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update feature branch #2

Merged
merged 156 commits into from
Oct 19, 2024
Merged

update feature branch #2

merged 156 commits into from
Oct 19, 2024

Conversation

KronosTheLate
Copy link
Owner

No description provided.

aviatesk and others added 30 commits October 2, 2024 17:59
The current implementation is wrong, causing it to display inappropriate
hints like the following:
```julia
julia> s = Some("foo");

julia> s[] = "bar"
ERROR: MethodError: no method matching setindex!(::Some{String}, ::String)
The function `setindex!` exists, but no method is defined for this combination of argument types.
You attempted to index the type String, rather than an instance of the type. Make sure you create the type using its constructor: d = String([...]) rather than d = String
Stacktrace:
 [1] top-level scope
   @ REPL[2]:1
```
In particular, the implementation of `overdub_generator54341` was
dangerous. This fixes it up.
Revise wants to know what file a module's `module` definition is in.
Currently it does this by looking at the source location for the
implicitly generated `eval` method. This is terrible for two reasons:

1. The method may not exist if the module is a baremodule (which is not
particularly common, which is probably why we haven't seen it).
2. The fact that the implicitly generated `eval` method has this
location information is an implementation detail that I'd like to get
rid of (JuliaLang#55949).

This PR adds explicit file/line info to `Module`, so that Revise doesn't
have to use the hack anymore.
I ran into this edge case. I though it should be documented.
---------

Co-authored-by: Lilith Orion Hafner <[email protected]>
…ang#55968)

While building the 1.11.0-rc4 in Homebrew[^1] in preparation for 1.11.0
release (and to confirm Sequoia successfully builds) I noticed some odd
linkage for our Linux builds, which included of:

1. LLVM libraries were linking to `libedit.so`, e.g.
    ```
    Dynamic Section:
      NEEDED       libedit.so.0
      NEEDED       libz.so.1
      NEEDED       libzstd.so.1
      NEEDED       libstdc++.so.6
      NEEDED       libm.so.6
      NEEDED       libgcc_s.so.1
      NEEDED       libc.so.6
      NEEDED       ld-linux-x86-64.so.2
      SONAME       libLLVM-16jl.so
    ```
    CMakeCache.txt showed
    ```
    //Use libedit if available.
    LLVM_ENABLE_LIBEDIT:BOOL=ON
    ```
Which might be overriding `HAVE_LIBEDIT` at
https://github.com/JuliaLang/llvm-project/blob/julia-release/16.x/llvm/cmake/config-ix.cmake#L222-L225.
So just added `LLVM_ENABLE_LIBEDIT`

2. Wasn't sure if there was a reason for this but `libccalllazy*` had
mismatched SONAME:
    ```console
    ❯ objdump -p lib/julia/libccalllazy* | rg '\.so'
    lib/julia/libccalllazybar.so:	file format elf64-x86-64
      NEEDED       ccalllazyfoo.so
      SONAME       ccalllazybar.so
    lib/julia/libccalllazyfoo.so:	file format elf64-x86-64
      SONAME       ccalllazyfoo.so
    ```
    Modifying this, but can drop if intentional.

---

[^1]: Homebrew/homebrew-core#192116
…Lang#55970)

Hi everyone! First PR to Julia here.

It was noticed in a Slack thread yesterday
that `copy!(A, I)` doesn't work, but `copyto!(A, I)` does. This PR adds
the missing method for `copy!(::AbstractMatrix, ::UniformScaling)`,
which simply defers to `copyto!`, and corresponding tests.

I added a `compat` notice for Julia 1.12.

---------

Co-authored-by: Lilith Orion Hafner <[email protected]>
Closes JuliaLang#40009 which was left open because of the needs news tag.

---------

Co-authored-by: Ian Butterworth <[email protected]>
The test wants to assert that `Module` is not resolved in `Main`, but
other tests do resolve this identifier, so the test can fail depending
on test order (and I've been seeing such failures on CI recently). Fix
that by running the test in a fresh subprocess.
JuliaLang#55976)

It appears that inlining.jl was not updated in JuliaLang#54341.
Specifically, using `nargs`/`isva` from `mi.def::Method` in
`ir_prepare_inlining!` causes the following error to occur:
```julia
function generate_lambda_ex(world::UInt, source::LineNumberNode,
                            argnames, spnames, @nospecialize body)
    stub = Core.GeneratedFunctionStub(identity, Core.svec(argnames...), Core.svec(spnames...))
    return stub(world, source, body)
end
function overdubbee54341(a, b)
    return a + b
end
const overdubee_codeinfo54341 = code_lowered(overdubbee54341, Tuple{Any, Any})[1]
function overdub_generator54341(world::UInt, source::LineNumberNode, selftype, fargtypes)
    if length(fargtypes) != 2
        return generate_lambda_ex(world, source,
            (:overdub54341, :args), (), :(error("Wrong number of arguments")))
    else
        return copy(overdubee_codeinfo54341)
    end
end
@eval function overdub54341(args...)
    $(Expr(:meta, :generated, overdub_generator54341))
    $(Expr(:meta, :generated_only))
end
topfunc(x) = overdub54341(x, 2)
```
```julia
julia> topfunc(1)
Internal error: during type inference of
topfunc(Int64)
Encountered unexpected error in runtime:
BoundsError(a=Array{Any, 1}(dims=(2,), mem=Memory{Any}(8, 0x10632e780)[SSAValue(2), SSAValue(3), #<null>, #<null>, #<null>, #<null>, #<null>, #<null>]), i=(3,))
throw_boundserror at ./essentials.jl:14
getindex at ./essentials.jl:909 [inlined]
ssa_substitute_op! at ./compiler/ssair/inlining.jl:1798
ssa_substitute_op! at ./compiler/ssair/inlining.jl:1852
ir_inline_item! at ./compiler/ssair/inlining.jl:386
...
```

This commit updates the abstract interpretation and inlining algorithm
to use the `nargs`/`isva` values held by `CodeInfo`. Similar
modifications have also been made to EscapeAnalysis.jl.

@nanosoldier `runbenchmarks("inference", vs=":master")`
Similar to the `vscode` config directory, we may ignore the `zed`
directory as well.
`merge_env` and `final_merge_env` could be skipped
for emptiness test or if we know there's only 1 valid Union state.
This only covers the simplest cases. We might want a full dependence analysis and keep env length minimum in the future.
JuliaLang#55977 looked simple but wasn't
quite right because of a bad pattern in the lock conflicts report
section.

So fix and add tests.
`EnterNode.catch_dest` can now be `0` after the `try`/`catch` elision
feature implemented in JuliaLang#52527, and we actually need to
adjust `EscapeAnalysis.compute_frameinfo` too.
Seeing what this will look like, since it has a number of features
(delayed compilation, concurrent compilation) that are starting to
become important, so it would be nice to switch to only supporting one
common implementation of memory management.

Refs JuliaLang#50248

I am expecting llvm/llvm-project#63236 may
cause some problems, since we reconfigured some CI machines to minimize
that issue, but it is still likely relevant.
…g#55908)

Iterate over the list of existing loaded modules for PkgId whenever
loading a new module for PkgId, so that we will use that existing
build_id content if it otherwise passes the other stale_checks.
This PR tries to reduce the allocation caused by `save_env` by
skipping `alloc_env` and truncating env size when possible.
…Lang#55991)

When the finalizer elision pass is used, load forwarding is not
performed currently, regardless of whether the pass succeeds or not. But
this is not necessary, and by keeping the `setfield!` call, we can
safely forward `getfield` even if finalizer elision is tried.
This doesn't switch the default to JITLink, but does some updates to
get JITLink working (and more efficient) across all of our supported
platforms.

Refs JuliaLang#50248
Keno and others added 29 commits October 17, 2024 01:20
It makes sense that we originally added this to the compiler, but these
annotations are really a runtime feature that the compiler simply reads
to allow it to make additional assumptions. The runtime should not
semantically depend on the compiler for this, so move these definitions
to expr.jl. The practical effect of this right now is that Base gains a
second copy of this code. Post JuliaLang#56128, the compiler will use the Base
copy of this.

Split out from JuliaLang#56128.
Also, add an aggressive constprop annotation to `generic_matvecmul!`.

Together, these improve performance:
```julia
julia> A = rand(Int,100,100);

julia> @Btime $A' * $A';
  290.203 μs (405 allocations: 175.98 KiB) # v"1.12.0-DEV.1364"
  270.008 μs (5 allocations: 79.11 KiB) # This PR
```
With this, the error message in
https://buildkite.com/julialang/julia-master/builds/41054#019294ca-e2c5-41f2-a897-e2959715f154
would become
```julia
Error in testset errorshow:
Test Failed at /home/jishnu/juliaPR/usr/share/julia/test/errorshow.jl:226
  Expression: typeof(err) === $(Expr(:escape, :MethodError))
   Evaluated: StackOverflowError === MethodError
     Context: expr = :(+())
```
Having the failing expression displayed makes it easier to locate the
source of the error.
…butes (JuliaLang#52946)

I doubt this will make too much of a difference since we don't use too
many libfuncs, but it's also quite a cheap pass if it makes any
difference

---------

Co-authored-by: Valentin Churavy <[email protected]>
This is a relatively independent part of the bindings partition branch,
extending the module usings list to gain `min_world` and `max_world`
`size_t`s. These are always `0` and `(size_t)-1` respectively in this
PR, which handles the GC and serialization implications of this change.
The semantic part will come later.
…rns (JuliaLang#55793)

This adds 3 new types, to conveniently express 3 common concurrent code
patterns:

 - `PerProcess`: an action that must be taken once per process
 - `PerThread`: an action that must be taken once per thread id
 - `PerTask`: an action that must be take once per task object

The PerProcess object should replace `__init__` or similar hand rolled
implementations of this.
The PerThread object should replace code that used to use `nthreads()`
to implement a much less correct version of this (though this is not
recommended in most new code, some foreign libraries may need this to
interact well with C).
The PerTask object is simply a thin wrapper over `task_local_storage()`.
Fixes: JuliaLang#56219

I am not really sure why we have a test for this, but we need to make
the test happy
This fixes
```julia
julia> using LinearAlgebra, StaticArrays

julia> M = Matrix{BigInt}(undef, 2, 2); M[1,1] = M[2,2] = M[1,2] = 3;

julia> S = SizedMatrix{2,2}(M)
2×2 SizedMatrix{2, 2, BigInt, 2, Matrix{BigInt}} with indices SOneTo(2)×SOneTo(2):
   3     3
 #undef  3

julia> triu(S)
ERROR: UndefRefError: access to undefined reference
Stacktrace:
 [1] getindex
   @ ./essentials.jl:907 [inlined]
 [2] getindex
   @ ~/.julia/packages/StaticArrays/MSJcA/src/SizedArray.jl:92 [inlined]
 [3] copyto_unaliased!
   @ ./abstractarray.jl:1086 [inlined]
 [4] copyto!(dest::SizedMatrix{2, 2, BigInt, 2, Matrix{BigInt}}, src::SizedMatrix{2, 2, BigInt, 2, Matrix{BigInt}})
   @ Base ./abstractarray.jl:1066
 [5] copymutable
   @ ./abstractarray.jl:1200 [inlined]
 [6] triu(M::SizedMatrix{2, 2, BigInt, 2, Matrix{BigInt}})
   @ LinearAlgebra ~/.julia/juliaup/julia-nightly/share/julia/stdlib/v1.12/LinearAlgebra/src/generic.jl:413
 [7] top-level scope
   @ REPL[11]:1
```
After this PR:
```julia
julia> triu(S)
2×2 SizedMatrix{2, 2, BigInt, 2, Matrix{BigInt}} with indices SOneTo(2)×SOneTo(2):
 3  3
 0  3
```
Only the indices that need to be copied are accessed, and the others are
written to without being read.
…56149)

Fixes https://github.com/JuliaLang/julia/issues/56134
After this,
```julia
julia> using LinearAlgebra

julia> A = hermitianpart(rand(4, 4))
4×4 Hermitian{Float64, Matrix{Float64}}:
 0.387617  0.277226  0.67629   0.60678
 0.277226  0.894101  0.388416  0.489141
 0.67629   0.388416  0.100907  0.619955
 0.60678   0.489141  0.619955  0.452605

julia> B = UpperTriangular(A)
4×4 UpperTriangular{Float64, Hermitian{Float64, Matrix{Float64}}}:
 0.387617  0.277226  0.67629   0.60678
  ⋅        0.894101  0.388416  0.489141
  ⋅         ⋅        0.100907  0.619955
  ⋅         ⋅         ⋅        0.452605

julia> B - B'
4×4 Matrix{Float64}:
  0.0        0.277226   0.67629   0.60678
 -0.277226   0.0        0.388416  0.489141
 -0.67629   -0.388416   0.0       0.619955
 -0.60678   -0.489141  -0.619955  0.0
```
This preserves the band structure of the parent, if any:
```julia
julia> U = UpperTriangular(Diagonal(ones(4)))
4×4 UpperTriangular{Float64, Diagonal{Float64, Vector{Float64}}}:
 1.0  0.0  0.0  0.0
  ⋅   1.0  0.0  0.0
  ⋅    ⋅   1.0  0.0
  ⋅    ⋅    ⋅   1.0

julia> U - U'
4×4 Diagonal{Float64, Vector{Float64}}:
 0.0   ⋅    ⋅    ⋅ 
  ⋅   0.0   ⋅    ⋅ 
  ⋅    ⋅   0.0   ⋅ 
  ⋅    ⋅    ⋅   0.0
```
This doesn't fully work with partly initialized matrices, and would need
JuliaLang#55312 for that.

The abstract triangular methods now construct matrices using
`similar(parent(U), size(U))` so that the destinations are fully
mutable.
```julia
julia> @invoke B::LinearAlgebra.AbstractTriangular - B'::LinearAlgebra.AbstractTriangular
4×4 Matrix{Float64}:
  0.0        0.277226   0.67629   0.60678
 -0.277226   0.0        0.388416  0.489141
 -0.67629   -0.388416   0.0       0.619955
 -0.60678   -0.489141  -0.619955  0.0
```

---------

Co-authored-by: Daniel Karrasch <[email protected]>
…g#56189)

Fix-up this pass a bit to correctly handle fall-through terminators that
cannot have their BasicBlock extended
(e.g. `Expr(:leave, ...)`)
…th memset since aggregate stores are bad (JuliaLang#55879)

This fixes the issues found in slack in the reinterprets of
```julia
julia> split128_v2(x::UInt128) = (first(reinterpret(NTuple{2, UInt}, x)), last(reinterpret(NTuple{2, UInt}, x)))
split128_v2 (generic function with 1 method)

julia> split128(x::UInt128) = reinterpret(NTuple{2, UInt}, x)
split128 (generic function with 1 method)

@code_native  split128(UInt128(5))
	push	rbp
	mov	rbp, rsp
	mov	rax, rdi
	mov	qword ptr [rdi + 8], rdx
	mov	qword ptr [rdi], rsi
	pop	rbp
	ret

@code_native  split128_v2(UInt128(5))
	push	rbp
	mov	rbp, rsp
	mov	rax, rdi
	mov	qword ptr [rdi], rsi
	mov	qword ptr [rdi + 8], rdx
	pop	rbp
	ret
```

vs on master where
```julia
julia> @code_native  split128(UInt128(5))

	push	rbp
	mov	rbp, rsp
	mov	eax, esi
	shr	eax, 8
	mov	ecx, esi
	shr	ecx, 16
	mov	r8, rsi
	mov	r9, rsi
	vmovd	xmm0, esi
	vpinsrb	xmm0, xmm0, eax, 1
	mov	rax, rsi
	vpinsrb	xmm0, xmm0, ecx, 2
	mov	rcx, rsi
	shr	esi, 24
	vpinsrb	xmm0, xmm0, esi, 3
	shr	r8, 32
	vpinsrb	xmm0, xmm0, r8d, 4
	shr	r9, 40
	vpinsrb	xmm0, xmm0, r9d, 5
	shr	rax, 48
	vpinsrb	xmm0, xmm0, eax, 6
	shr	rcx, 56
	vpinsrb	xmm0, xmm0, ecx, 7
	vpinsrb	xmm0, xmm0, edx, 8
	mov	eax, edx
	shr	eax, 8
	vpinsrb	xmm0, xmm0, eax, 9
	mov	eax, edx
	shr	eax, 16
	vpinsrb	xmm0, xmm0, eax, 10
	mov	eax, edx
	shr	eax, 24
	vpinsrb	xmm0, xmm0, eax, 11
	mov	rax, rdx
	shr	rax, 32
	vpinsrb	xmm0, xmm0, eax, 12
	mov	rax, rdx
	shr	rax, 40
	vpinsrb	xmm0, xmm0, eax, 13
	mov	rax, rdx
	shr	rax, 48
	vpinsrb	xmm0, xmm0, eax, 14
	mov	rax, rdi
	shr	rdx, 56
	vpinsrb	xmm0, xmm0, edx, 15
	vmovdqu	xmmword ptr [rdi], xmm0
	pop	rbp
	ret
```
Currently the header doesn't print for `Pkg.test` with coverage on

```
  [8dfed614] Test v1.11.0
   1077.2 ms  ✓ RequiredInterfaces
  1 dependency successfully precompiled in 1 seconds. 8 already precompiled.
```
Adds a new internal function `_take!(dst::Array{T,N}, src::Array{T,N})` for
doing an efficient `copyto!` equivalent. Previously it was assumed that
`compact` did this automatically, which wasn't a great assumption.

Fixes JuliaLang#56078
CacheFlags could get set, but were never propagated to the target
process, so the result would be unusable. Additionally, the debug and
optimization levels were not synchronized with the sysimg, causing a
regression in pkgimage usability after moving out stdlibs.

Fixes JuliaLang#56207
Fixes JuliaLang#56054
Fixes JuliaLang#56206
This change by itself doesn't do anything significant on `master`, but
when backported to the v1.11 branch it'll address JuliaLang#56177. However it'd
be great if someone could tell _why_ this fixes that issue, because it
looks very unrelated.

---------

Co-authored-by: Ian Butterworth <[email protected]>
…ncy edge tracking (JuliaLang#56179)

Disjoint content can be LLVM optimized in parallel now, since codegen
no longer has any ability to handle recursion, and compilation should
even be able to run in parallel with the GC also. Removes any remaining
global state, since that is unsafe.

Adds a C++ shim for concurrent gc support in conjunction with
using a `std::unique_lock` to DRY code.

Fix RuntimeDyld implementation:
Since we use the ForwardingMemoryManger instead of making a new
RTDyldMemoryManager object every time, we need to reference count the
finalizeMemory calls so that we only call that at the end of relocating
everything when everything is ready. We already happen to conveniently
have a shared_ptr here, so just use that instead of inventing a
duplicate counter.

Fixes many OC bugs, including mostly fixing JuliaLang#55035, since this bug is
just that much harder to express in the more constrained API.
Falling back to the older serial precompilation process is basically a
bug (except for if a manifest hasn't been resolved) so
JuliaLang#52619 added more info on why
it's been hit so we have a chance of fixing issues that are otherwise
very difficult to recreate.

However "invalid header" which usually just means it was made by a
different julia version appears to sound too alarming to users.
https://discourse.julialang.org/t/cache-misses-when-using-packages-since-upgrading-to-1-11/121445

So soften it there and in error messages, given it seems a better
description.

Suggested by @giordano in
https://discourse.julialang.org/t/cache-misses-when-using-packages-since-upgrading-to-1-11/121445/4?u=ianshmean
…st_throws (JuliaLang#56231)

Fix JuliaLang#54082 

Arguably this was a breaking change (as a consequence of
JuliaLang#51979). But regardless, it seems
like useful functionality to have a public API for testing that an
`UndefVarError` was thrown for the expected variable name (regardless of
scope). This is particularly useful if you don't know what the scope is
(for example, in my use-case i want to test that a specific
`UndefVarError` is thrown from a module with a `gensym`'d name).

Pre-v1.11 the syntax for this was 
```julia
@test_throws UndefVarError(:x) foo()
```
but that stopped working in v1.11 when `UndefVarError` got a second
field (in fact in v1.11.1 this is an error when before it would pass)

This PR restores that functionality.

We might want to backport it to v1.11.x so that v1.11 isn't the only
version that doesn't support this.
This PR contains some refactoring of common functions that were moved to
`gc-common.c` and should be shared between MMTk and Julia's stock GC.
merger actual master into mine
@KronosTheLate KronosTheLate merged commit 85a1707 into improve_docstring_filesize Oct 19, 2024
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.