Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpart: Fully switch to partitioned semantics #57253

Merged
merged 1 commit into from
Feb 6, 2025
Merged

Conversation

Keno
Copy link
Member

@Keno Keno commented Feb 4, 2025

This is the final PR in the binding partitions series (modulo bugs and tweaks), i.e. it closes #54654 and thus closes #40399, which was the original design sketch.

This thus activates the full designed semantics for binding partitions, in particular allowing safe replacement of const bindings. It in particular allows struct redefinitions. This thus closes timholy/Revise.jl#18 and also closes #38584.

The biggest semantic change here is probably that this gets rid of the notion of "resolvedness" of a binding. Previously, a lot of the behavior of our implementation depended on when bindings were "resolved", which could happen at basically an arbitrary point (in the compiler, in REPL completion, in a different thread), making a lot of the semantics around bindings ill- or at least implementation-defined. There are several related issues in the bugtracker, so this closes #14055 closes #44604 closes #46354 closes #30277

It is also the last step to close #24569.
It also supports bindings for undef->defined transitions and thus closes #53958 closes #54733 - however, this is not activated yet for performance reasons and may need some further optimization.

Since resolvedness no longer exists, we need to replace it with some hopefully more well-defined semantics. I will describe the semantics below, but before I do I will make two notes:

  1. There are a number of cases where these semantics will behave slightly differently than the old semantics absent some other task going around resolving random bindings.
  2. The new behavior (except for the replacement stuff) was generally permissible under the old semantics if the bindings happened to be resolved at the right time.

With all that said, there are essentially three "strengths" of bindings:

  1. Implicit Bindings: Anything implicitly obtained from using Mod, "no binding", plus slightly more exotic corner cases around conflicts

  2. Weakly declared bindings: Declared using global sym and nothing else

  3. Strongly declared bindings: Declared using global sym::T, const sym=val, import Mod: sym, using Mod: sym or as an implicit strong global declaration in sym=val, where sym is known to be global (either by being at toplevle or as global sym=val inside a function).

In general, you always allowed to syntactically replace a weaker binding by a stronger one (although the runtime permits arbitrary binding deletion now, this is just a syntactic constraint to catch errors). Second, any implicit binding can be replaced by other implicit bindings as the result of changing the using'ed module. And lastly, any constants may be replaced by any other constants (irrespective of type).

We do not currently allow replacing globals, but may consider changing that in 1.13.

This is mostly how things used to work, as well in the absence of any stray external binding resolutions. The most prominent difference is probably this one:

set_foo!() = global foo = 1

In the above terminology, this now always declares a "strongly declared binding", whereas before it declared a "weakly declared binding" that would become strongly declared on first write to the global (unless of course somebody had created a different strongly declared global in the meantime). To see the difference, this is now disallowed:

julia> set_foo!() = global foo = 1
set_foo! (generic function with 1 method)

julia> const foo = 1
ERROR: cannot declare Main.foo constant; it was already declared global
Stacktrace:
 [1] top-level scope
   @ REPL[2]:1

Before it would depend on the order of binding resolution (although it just crashes on current master for some reason - whoops, probably my fault).

Another major change is the ambiguousness of imports. In:

module M1; export x; x=1; end
module M2; export x; x=2; end
using .M1, .M2

the binding Main.x is now always ambiguous and will throw on access. Before which binding you get, would depend on resolution order. To choose one, use an explicit import (which was the behavior you would previously get if neither binding was resolved before both imports).

@Keno Keno force-pushed the kf/fullpartitioning branch from c266bc2 to 9697863 Compare February 4, 2025 10:34
@Keno Keno added the backport 1.12 Change should be backported to release-1.12 label Feb 4, 2025
@Keno Keno force-pushed the kf/fullpartitioning branch from 9697863 to 1606e82 Compare February 4, 2025 10:48
@KristofferC KristofferC mentioned this pull request Feb 4, 2025
32 tasks
@Keno Keno force-pushed the kf/fullpartitioning branch 4 times, most recently from 9f830d2 to 58a567d Compare February 5, 2025 10:25
@Keno
Copy link
Member Author

Keno commented Feb 5, 2025

This should hopefully be passing tests now, modulo the revise tests, which need JuliaDebug/JuliaInterpreter.jl#658. There are some additional performance optimizations to (mostly to call invalidation less often), as well as a missing invalidation when somebody changes the export set that a precompile'd package sees. However, those don't affect the semantics and this PR is already large, so I'm planning to do those in a follow-up.

This is the final PR in the binding partitions series (modulo bugs and tweaks),
i.e. it closes #54654 and thus closes #40399, which was the original design
sketch.

This thus activates the full designed semantics for binding partitions,
in particular allowing safe replacement of const bindings. It in particular
allows struct redefinitions. This thus closes timholy/Revise.jl#18 and
also closes #38584.

The biggest semantic change here is probably that this gets rid of the
notion of "resolvedness" of a binding. Previously, a lot of the behavior
of our implementation depended on when bindings were "resolved", which
could happen at basically an arbitrary point (in the compiler, in REPL
completion, in a different thread), making a lot of the semantics around
bindings ill- or at least implementation-defined. There are several related
issues in the bugtracker, so this closes #14055 #44604 #46354 #30277

It is also the last step to close #24569.
It also supports bindings for undef->defined transitions and thus
closes #53958 #54733 - however, this is not activated yet for performance
reasons and may need some further optimization.

Since resolvedness no longer exists, we need to replace it with some
hopefully more well-defined semantics. I will describe the semantics
below, but before I do I will make two notes:

1. There are a number of cases where these semantics will behave
   slightly differently than the old semantics absent some other
   task going around resolving random bindings.
2. The new behavior (except for the replacement stuff) was generally
   permissible under the old semantics if the bindings
   happened to be resolved at the right time.

With all that said, there are essentially three "strengths" of bindings:

1. Implicit Bindings: Anything implicitly obtained from `using Mod`, "no binding",
   plus slightly more exotic corner cases around conflicts

2. Weakly declared bindings: Declared using `global sym` and nothing else

3. Strongly declared bindings: Declared using `global sym::T`, `const sym=val`,
   `import Mod: sym`, `using Mod: sym` or as an implicit strong global declaration
   in `sym=val`, where `sym` is known to be global (either by being at toplevle
   or as `global sym=val` inside a function).

In general, you always allowed to syntactically replace a weaker binding
by a stronger one (although the runtime permits arbitrary binding deletion
now, this is just a syntactic constraint to catch errors).
Second, any implicit binding can be replaced by other implicit
bindings as the result of changing the `using`'ed module.
And lastly, any constants may be replaced by any other constants
(irrespective of type).

We do not currently allow replacing globals, but may consider changing that
in 1.13.

This is mostly how things used to work, as well in the absence of any stray
external binding resolutions. The most prominent difference is probably this one:

```
set_foo!() = global foo = 1
```

In the above terminology, this now always declares a "strongly declared binding",
whereas before it declared a "weakly declared binding" that would become
strongly declared on first write to the global (unless of course somebody
had created a different strongly declared global in the meantime). To see
the difference, this is now disallowed:

```
julia> set_foo!() = global foo = 1
set_foo! (generic function with 1 method)

julia> const foo = 1
ERROR: cannot declare Main.foo constant; it was already declared global
Stacktrace:
 [1] top-level scope
   @ REPL[2]:1
```

Before it would depend on the order of binding resolution (although it
just crashes on current master for some reason - whoops, probably my
fault).

Another major change is the ambiguousness of imports. In:
```
module M1; export x; x=1; end
module M2; export x; x=2; end
using .M1, .M2
```
the binding `Main.x` is now always ambiguous and will throw on access.
Before which binding you get, would depend on resolution order. To choose
one, use an explicit import (which was the behavior you would previously
get if neither binding was resolved before both imports).
@Keno Keno force-pushed the kf/fullpartitioning branch from 58a567d to e2d7d83 Compare February 6, 2025 01:19
@Keno Keno added the merge me PR is reviewed. Merge when all tests are passing label Feb 6, 2025
@Keno Keno merged commit 888cf03 into master Feb 6, 2025
5 of 8 checks passed
@Keno Keno deleted the kf/fullpartitioning branch February 6, 2025 06:14
@inkydragon inkydragon removed the merge me PR is reviewed. Merge when all tests are passing label Feb 6, 2025
KristofferC pushed a commit that referenced this pull request Feb 6, 2025
This is the final PR in the binding partitions series (modulo bugs and
tweaks), i.e. it closes #54654 and thus closes #40399, which was the
original design sketch.

This thus activates the full designed semantics for binding partitions,
in particular allowing safe replacement of const bindings. It in
particular allows struct redefinitions. This thus closes
timholy/Revise.jl#18 and also closes #38584.

The biggest semantic change here is probably that this gets rid of the
notion of "resolvedness" of a binding. Previously, a lot of the behavior
of our implementation depended on when bindings were "resolved", which
could happen at basically an arbitrary point (in the compiler, in REPL
completion, in a different thread), making a lot of the semantics around
bindings ill- or at least implementation-defined. There are several
related issues in the bugtracker, so this closes #14055 closes #44604
closes #46354 closes #30277

It is also the last step to close #24569.
It also supports bindings for undef->defined transitions and thus closes
#53958 closes #54733 - however, this is not activated yet for
performance reasons and may need some further optimization.

Since resolvedness no longer exists, we need to replace it with some
hopefully more well-defined semantics. I will describe the semantics
below, but before I do I will make two notes:

1. There are a number of cases where these semantics will behave
slightly differently than the old semantics absent some other task going
around resolving random bindings.
2. The new behavior (except for the replacement stuff) was generally
permissible under the old semantics if the bindings happened to be
resolved at the right time.

With all that said, there are essentially three "strengths" of bindings:

1. Implicit Bindings: Anything implicitly obtained from `using Mod`, "no
binding", plus slightly more exotic corner cases around conflicts

2. Weakly declared bindings: Declared using `global sym` and nothing
else

3. Strongly declared bindings: Declared using `global sym::T`, `const
sym=val`, `import Mod: sym`, `using Mod: sym` or as an implicit strong
global declaration in `sym=val`, where `sym` is known to be global
(either by being at toplevle or as `global sym=val` inside a function).

In general, you always allowed to syntactically replace a weaker binding
by a stronger one (although the runtime permits arbitrary binding
deletion now, this is just a syntactic constraint to catch errors).
Second, any implicit binding can be replaced by other implicit bindings
as the result of changing the `using`'ed module. And lastly, any
constants may be replaced by any other constants (irrespective of type).

We do not currently allow replacing globals, but may consider changing
that in 1.13.

This is mostly how things used to work, as well in the absence of any
stray external binding resolutions. The most prominent difference is
probably this one:

```
set_foo!() = global foo = 1
```

In the above terminology, this now always declares a "strongly declared
binding", whereas before it declared a "weakly declared binding" that
would become strongly declared on first write to the global (unless of
course somebody had created a different strongly declared global in the
meantime). To see the difference, this is now disallowed:

```
julia> set_foo!() = global foo = 1
set_foo! (generic function with 1 method)

julia> const foo = 1
ERROR: cannot declare Main.foo constant; it was already declared global
Stacktrace:
 [1] top-level scope
   @ REPL[2]:1
```

Before it would depend on the order of binding resolution (although it
just crashes on current master for some reason - whoops, probably my
fault).

Another major change is the ambiguousness of imports. In:
```
module M1; export x; x=1; end
module M2; export x; x=2; end
using .M1, .M2
```
the binding `Main.x` is now always ambiguous and will throw on access.
Before which binding you get, would depend on resolution order. To
choose one, use an explicit import (which was the behavior you would
previously get if neither binding was resolved before both imports).

(cherry picked from commit 888cf03)
KristofferC pushed a commit that referenced this pull request Feb 6, 2025
This is the final PR in the binding partitions series (modulo bugs and
tweaks), i.e. it closes #54654 and thus closes #40399, which was the
original design sketch.

This thus activates the full designed semantics for binding partitions,
in particular allowing safe replacement of const bindings. It in
particular allows struct redefinitions. This thus closes
timholy/Revise.jl#18 and also closes #38584.

The biggest semantic change here is probably that this gets rid of the
notion of "resolvedness" of a binding. Previously, a lot of the behavior
of our implementation depended on when bindings were "resolved", which
could happen at basically an arbitrary point (in the compiler, in REPL
completion, in a different thread), making a lot of the semantics around
bindings ill- or at least implementation-defined. There are several
related issues in the bugtracker, so this closes #14055 closes #44604
closes #46354 closes #30277

It is also the last step to close #24569.
It also supports bindings for undef->defined transitions and thus closes
#53958 closes #54733 - however, this is not activated yet for
performance reasons and may need some further optimization.

Since resolvedness no longer exists, we need to replace it with some
hopefully more well-defined semantics. I will describe the semantics
below, but before I do I will make two notes:

1. There are a number of cases where these semantics will behave
slightly differently than the old semantics absent some other task going
around resolving random bindings.
2. The new behavior (except for the replacement stuff) was generally
permissible under the old semantics if the bindings happened to be
resolved at the right time.

With all that said, there are essentially three "strengths" of bindings:

1. Implicit Bindings: Anything implicitly obtained from `using Mod`, "no
binding", plus slightly more exotic corner cases around conflicts

2. Weakly declared bindings: Declared using `global sym` and nothing
else

3. Strongly declared bindings: Declared using `global sym::T`, `const
sym=val`, `import Mod: sym`, `using Mod: sym` or as an implicit strong
global declaration in `sym=val`, where `sym` is known to be global
(either by being at toplevle or as `global sym=val` inside a function).

In general, you always allowed to syntactically replace a weaker binding
by a stronger one (although the runtime permits arbitrary binding
deletion now, this is just a syntactic constraint to catch errors).
Second, any implicit binding can be replaced by other implicit bindings
as the result of changing the `using`'ed module. And lastly, any
constants may be replaced by any other constants (irrespective of type).

We do not currently allow replacing globals, but may consider changing
that in 1.13.

This is mostly how things used to work, as well in the absence of any
stray external binding resolutions. The most prominent difference is
probably this one:

```
set_foo!() = global foo = 1
```

In the above terminology, this now always declares a "strongly declared
binding", whereas before it declared a "weakly declared binding" that
would become strongly declared on first write to the global (unless of
course somebody had created a different strongly declared global in the
meantime). To see the difference, this is now disallowed:

```
julia> set_foo!() = global foo = 1
set_foo! (generic function with 1 method)

julia> const foo = 1
ERROR: cannot declare Main.foo constant; it was already declared global
Stacktrace:
 [1] top-level scope
   @ REPL[2]:1
```

Before it would depend on the order of binding resolution (although it
just crashes on current master for some reason - whoops, probably my
fault).

Another major change is the ambiguousness of imports. In:
```
module M1; export x; x=1; end
module M2; export x; x=2; end
using .M1, .M2
```
the binding `Main.x` is now always ambiguous and will throw on access.
Before which binding you get, would depend on resolution order. To
choose one, use an explicit import (which was the behavior you would
previously get if neither binding was resolved before both imports).

(cherry picked from commit 888cf03)
@inkydragon
Copy link
Member

Seems this pr broken x86_64-linux-gnummtk build.

https://buildkite.com/julialang/julia-master/builds?branch=master

KristofferC added a commit that referenced this pull request Feb 13, 2025
Backported PRs:
- [x] #57142 <!-- Add reference to time_ns in time -->
- [x] #57241 <!-- Handle `waitpid` race condition when `SIGCHLD` is set
to `SIG_IGN` -->
- [x] #57249 <!-- restore non-freebsd-unix fix for profiling -->
- [x] #57211 <!-- Ensure read/readavailable for BufferStream are
threadsafe -->
- [x] #57262 <!-- edit NEWS for v1.12 -->
- [x] #57226 <!-- cfunction: reimplement, as originally planned, for
reliable performance -->
- [x] #57253 <!-- bpart: Fully switch to partitioned semantics -->
- [x] #57273 <!-- fix "Right arrow autocompletes at line end"
implementation -->
- [x] #57280 <!-- dep: Update JuliaSyntax -->
- [x] #57229 <!-- staticdata: Close data race after backedge insertion
-->
- [x] #57298 <!-- Updating binding version to fix MMTk CI -->
- [x] #57248 <!-- improve concurrency safety for `Compiler.finish!` -->
- [x] #57312 <!-- Profile.print: de-focus sleeping frames as gray -->
- [x] #57289 <!-- Make `OncePerX` subtype `Function` -->
- [x] #57310 <!-- Make ptls allocations at least 128 byte aligned -->
- [x] #57311 <!-- Add a warning for auto-import of types -->
- [x] #57338 <!-- fix typo in Float32 random number generation -->
- [x] #57293 <!-- Fix getfield_tfunc when order or boundscheck is Vararg
-->
- [x] #57349 <!-- docs: fix-up world-age handling for META access -->
- [x] #57344 <!-- Add missing type asserts when taking the queue out of
the task struct -->
- [x] #57348 <!-- 🤖 [master] Bump the SparseArrays stdlib from 212981b
to 72c7cac -->
- [x] #55040 <!-- Allow macrocall as function sig -->
- [x] #57299 <!-- Add missing latestworld after parameterized type alias
-->
@KristofferC KristofferC removed the backport 1.12 Change should be backported to release-1.12 label Feb 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment