-
Notifications
You must be signed in to change notification settings - Fork 734
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
core: fix missed register_callsite
error
#2938
Conversation
f8db879
to
8b6dca6
Compare
There are 2 triggers which will cause a subscriber to receive a call to `Subscriber::register_callsite` for a specific callsite. 1. The first time the event or span at that callsite is executed. 2. When a new subscriber is added or removed (for example, calls to `set_default` or `with_default`) It is trigger (2) that will cause a new subscriber to receive `Subscriber::register_callsite` for all the callsites which had already been registered before it became active. When a callsite is registered for trigger (1), the callsite starts in state `UNREGISTERED`. The first thread to encounter the callsite will transition it to `REGISTERING` and determine the overall interest for the callsite by registering with all known dispatchers (which will call into `Subscriber::register_callsite`). Once that is complete, the callsite is added to the list of all known callsites and its state is transitioned to `REGISTERED`. is (re)built for all known dispatchers. The callsite starts in state `UNREGISTERED`. The This calls down into `Subscriber::register_callsite` for each subscriber. Once that is complete, the callsite is added to the global list of known callsites. While the callsite interest is being rebuilt, other threads that encounter the callsite will be given `Interest::sometimes()` until the registration is complete. However, if a new subscriber is added during this window, all the interest for all callsites will be rebuilt, but because the new callsite (in state `REGISTERING`) won't be included because it isn't yet in the global list of callsites. This can cause a case where that new subscriber being added won't receive `Subscriber::register_callsite` before it receives the subsequent call to `Subscriber::event` or `Subscriber::new_span`. The documentation on [Registering Callsites] is not very explicit on this point, but it does suggest that `Subscriber::register_callsite` will be called before the call to either `Subscriber::event` or `Subscriber::new_span`, and the current behavior can break this implicit contract. [Registering Callsites]: https://docs.rs/tracing-core/0.1.32/tracing_core/callsite/index.html#registering-callsites This change swaps the order of rebuilding the callsite interest and adding the callsite to the global list so that the callsite gets pushed first, avoiding this window in which a subscriber won't get a call to `register_callsite`. As such, a callsite may have its interest read before it is set. In this case, the existing implementation will return `Interest::sometimes()` for the `DefaultCallsite` implementation. Other implementations (outside of the `tracing` project) may perform this differently, but in this case, there is no documented guarantee regarding the ordering. A regression test is included which provokes the race condition 100% of the time before the changes in this fix. Fixes: #2743
8b6dca6
to
51d7d9e
Compare
@hawkw I'd love to get your opinion on the change I've made in the The test in |
Thanks for the ping! I'll try to give it a look today or tomorrow. |
@hds you mentioned the docs not being explicit on the topic, could you maybe clarify them? You just might be the second most knowledgeable person on the topic now. |
### Added - Add index API for `Field` ([#2820]) - allow `&[u8]` to be recorded as event/span field ([#2954]) ### Changed - Use const `thread_local`s when possible ([#2838]) ### Fixed - Fix missed `register_callsite` error ([#2938]) - Do not add `valuable/std` feature as dependency unless `valuable` is used ([#3002]) - prefix macro calls with ::core to avoid clashing with local macros ([#3024]) ### Documented - Fix incorrect (incorrectly updated) docs for LevelFilter ([#2767]) [#2767]: #2767 [#2820]: #2820 [#2838]: #2838 [#2938]: #2938 [#3002]: #3002 [#3024]: #3024
# 0.1.33 (November 25, 2024) ### Added - Add index API for `Field` ([#2820]) - allow `&[u8]` to be recorded as event/span field ([#2954]) ### Changed - Use const `thread_local`s when possible ([#2838]) ### Fixed - Fix missed `register_callsite` error ([#2938]) - Do not add `valuable/std` feature as dependency unless `valuable` is used ([#3002]) - prefix macro calls with ::core to avoid clashing with local macros ([#3024]) ### Documented - Fix incorrect (incorrectly updated) docs for LevelFilter ([#2767]) [#2767]: #2767 [#2820]: #2820 [#2838]: #2838 [#2938]: #2938 [#3002]: #3002 [#3024]: #3024 # Please enter the commit message for your changes. Lines starting # with '#' will be kept; you may remove them yourself if you want to. # An empty message aborts the commit. # # Date: Sat Nov 23 23:28:42 2024 +0100 # # On branch hds/tracing-core-0.1.33 # Changes to be committed: # modified: tracing-core/CHANGELOG.md # modified: tracing-core/Cargo.toml #
# 0.1.33 (November 25, 2024) ### Added - Add index API for `Field` ([#2820]) - allow `&[u8]` to be recorded as event/span field ([#2954]) ### Changed - Use const `thread_local`s when possible ([#2838]) ### Fixed - Fix missed `register_callsite` error ([#2938]) - Do not add `valuable/std` feature as dependency unless `valuable` is used ([#3002]) - prefix macro calls with ::core to avoid clashing with local macros ([#3024]) ### Documented - Fix incorrect (incorrectly updated) docs for LevelFilter ([#2767]) [#2767]: #2767 [#2820]: #2820 [#2838]: #2838 [#2938]: #2938 [#3002]: #3002 [#3024]: #3024
# 0.1.33 (November 25, 2024) ### Added - Add index API for `Field` ([#2820]) - allow `&[u8]` to be recorded as event/span field ([#2954]) ### Changed - Use const `thread_local`s when possible ([#2838]) ### Fixed - Fix missed `register_callsite` error ([#2938]) - Do not add `valuable/std` feature as dependency unless `valuable` is used ([#3002]) - prefix macro calls with ::core to avoid clashing with local macros ([#3024]) ### Documented - Fix incorrect (incorrectly updated) docs for LevelFilter ([#2767]) [#2767]: #2767 [#2820]: #2820 [#2838]: #2838 [#2938]: #2938 [#2954]: #2954 [#3002]: #3002 [#3024]: #3024
# 0.1.33 (November 25, 2024) ### Added - Add index API for `Field` ([#2820]) - allow `&[u8]` to be recorded as event/span field ([#2954]) ### Changed - Use const `thread_local`s when possible ([#2838]) ### Fixed - Fix missed `register_callsite` error ([#2938]) - Do not add `valuable/std` feature as dependency unless `valuable` is used ([#3002]) - prefix macro calls with ::core to avoid clashing with local macros ([#3024]) ### Documented - Fix incorrect (incorrectly updated) docs for LevelFilter ([#2767]) Thanks to new contributor @maddiemort for contributing to this release! [#2767]: #2767 [#2820]: #2820 [#2838]: #2838 [#2938]: #2938 [#2954]: #2954 [#3002]: #3002 [#3024]: #3024
# 0.1.33 (November 25, 2024) ### Added - Add index API for `Field` ([#2820]) - allow `&[u8]` to be recorded as event/span field ([#2954]) ### Changed - Bump MSRV to 1.63 ([#2793]) - Use const `thread_local`s when possible ([#2838]) ### Fixed - Fix missed `register_callsite` error ([#2938]) - Do not add `valuable/std` feature as dependency unless `valuable` is used ([#3002]) - prefix macro calls with ::core to avoid clashing with local macros ([#3024]) ### Documented - Fix incorrect (incorrectly updated) docs for LevelFilter ([#2767]) Thanks to new contributor @maddiemort for contributing to this release! [#2767]: #2767 [#2793]: #2793 [#2820]: #2820 [#2838]: #2838 [#2938]: #2938 [#2954]: #2954 [#3002]: #3002 [#3024]: #3024
# 0.1.33 (November 25, 2024) ### Added - Add index API for `Field` ([#2820]) - allow `&[u8]` to be recorded as event/span field ([#2954]) ### Changed - Bump MSRV to 1.63 ([#2793]) - Use const `thread_local`s when possible ([#2838]) ### Fixed - Fix missed `register_callsite` error ([#2938]) - Do not add `valuable/std` feature as dependency unless `valuable` is used ([#3002]) - prefix macro calls with ::core to avoid clashing with local macros ([#3024]) ### Documented - Fix incorrect (incorrectly updated) docs for LevelFilter ([#2767]) Thanks to new contributor @maddiemort for contributing to this release! [#2767]: #2767 [#2793]: #2793 [#2820]: #2820 [#2838]: #2838 [#2938]: #2938 [#2954]: #2954 [#3002]: #3002 [#3024]: #3024
# 0.1.41 (November 25, 2024) [[[crates.io][crate-0.1.41]]] | [[[docs.rs][docs-0.1.41]]] This release updates the `tracing-core` dependency to [v0.1.33][core-0.1.33] and the `tracing-attributes` dependency to [v0.1.28][attrs-0.1.28]. ### Added - **core**: Add index API for `Field` ([#2820]) - **core**: Allow `&[u8]` to be recorded as event/span field ([#2954]) ### Changed - Bump MSRV to 1.63 ([#2793]) - **core**: Use const `thread_local`s when possible ([#2838]) ### Fixed - Removed core imports in macros ([#2762]) - **attributes**: Added missing RecordTypes for instrument ([#2781]) - **attributes**: Change order of async and unsafe modifier ([#2864]) - Fix missing field prefixes ([#2878]) - **attributes**: Extract match scrutinee ([#2880]) - Fix non-simple macro usage without message ([#2879]) - Fix event macros with constant field names in the first position ([#2883]) - Allow field path segments to be keywords ([#2925]) - **core**: Fix missed `register_callsite` error ([#2938]) - **attributes**: Support const values for `target` and `name` ([#2941]) - Prefix macro calls with ::core to avoid clashing with local macros ([#3024]) [#2762]: #2762 [#2781]: #2781 [#2793]: #2793 [#2820]: #2820 [#2838]: #2838 [#2864]: #2864 [#2878]: #2878 [#2879]: #2879 [#2880]: #2880 [#2883]: #2883 [#2925]: #2925 [#2938]: #2938 [#2941]: #2941 [#2954]: #2954 [#3024]: #3024 [attrs-0.1.28]: https://github.com/tokio-rs/tracing/releases/tag/tracing-attributes-0.1.28 [core-0.1.33]: https://github.com/tokio-rs/tracing/releases/tag/tracing-core-0.1.33 [docs-0.1.41]: https://docs.rs/tracing/0.1.41/tracing/ [crate-0.1.41]: https://crates.io/crates/tracing/0.1.41
# 0.1.41 (November 25, 2024) [[[crates.io][crate-0.1.41]]] | [[[docs.rs][docs-0.1.41]]] This release updates the `tracing-core` dependency to [v0.1.33][core-0.1.33] and the `tracing-attributes` dependency to [v0.1.28][attrs-0.1.28]. ### Added - **core**: Add index API for `Field` ([#2820]) - **core**: Allow `&[u8]` to be recorded as event/span field ([#2954]) ### Changed - Bump MSRV to 1.63 ([#2793]) - **core**: Use const `thread_local`s when possible ([#2838]) ### Fixed - Removed core imports in macros ([#2762]) - **attributes**: Added missing RecordTypes for instrument ([#2781]) - **attributes**: Change order of async and unsafe modifier ([#2864]) - Fix missing field prefixes ([#2878]) - **attributes**: Extract match scrutinee ([#2880]) - Fix non-simple macro usage without message ([#2879]) - Fix event macros with constant field names in the first position ([#2883]) - Allow field path segments to be keywords ([#2925]) - **core**: Fix missed `register_callsite` error ([#2938]) - **attributes**: Support const values for `target` and `name` ([#2941]) - Prefix macro calls with ::core to avoid clashing with local macros ([#3024]) [#2762]: #2762 [#2781]: #2781 [#2793]: #2793 [#2820]: #2820 [#2838]: #2838 [#2864]: #2864 [#2878]: #2878 [#2879]: #2879 [#2880]: #2880 [#2883]: #2883 [#2925]: #2925 [#2938]: #2938 [#2941]: #2941 [#2954]: #2954 [#3024]: #3024 [attrs-0.1.28]: https://github.com/tokio-rs/tracing/releases/tag/tracing-attributes-0.1.28 [core-0.1.33]: https://github.com/tokio-rs/tracing/releases/tag/tracing-core-0.1.33 [docs-0.1.41]: https://docs.rs/tracing/0.1.41/tracing/ [crate-0.1.41]: https://crates.io/crates/tracing/0.1.41
# 0.1.41 (November 25, 2024) [[[crates.io][crate-0.1.41]]] | [[[docs.rs][docs-0.1.41]]] This release updates the `tracing-core` dependency to [v0.1.33][core-0.1.33] and the `tracing-attributes` dependency to [v0.1.28][attrs-0.1.28]. ### Added - **core**: Add index API for `Field` ([#2820]) - **core**: Allow `&[u8]` to be recorded as event/span field ([#2954]) ### Changed - Bump MSRV to 1.63 ([#2793]) - **core**: Use const `thread_local`s when possible ([#2838]) ### Fixed - Removed core imports in macros ([#2762]) - **attributes**: Added missing RecordTypes for instrument ([#2781]) - **attributes**: Change order of async and unsafe modifier ([#2864]) - Fix missing field prefixes ([#2878]) - **attributes**: Extract match scrutinee ([#2880]) - Fix non-simple macro usage without message ([#2879]) - Fix event macros with constant field names in the first position ([#2883]) - Allow field path segments to be keywords ([#2925]) - **core**: Fix missed `register_callsite` error ([#2938]) - **attributes**: Support const values for `target` and `name` ([#2941]) - Prefix macro calls with ::core to avoid clashing with local macros ([#3024]) [#2762]: #2762 [#2781]: #2781 [#2793]: #2793 [#2820]: #2820 [#2838]: #2838 [#2864]: #2864 [#2878]: #2878 [#2879]: #2879 [#2880]: #2880 [#2883]: #2883 [#2925]: #2925 [#2938]: #2938 [#2941]: #2941 [#2954]: #2954 [#3024]: #3024 [attrs-0.1.28]: https://github.com/tokio-rs/tracing/releases/tag/tracing-attributes-0.1.28 [core-0.1.33]: https://github.com/tokio-rs/tracing/releases/tag/tracing-core-0.1.33 [docs-0.1.41]: https://docs.rs/tracing/0.1.41/tracing/ [crate-0.1.41]: https://crates.io/crates/tracing/0.1.41
# 0.1.41 (November 25, 2024) [ [crates.io][crate-0.1.41] ] | [ [docs.rs][docs-0.1.41] ] This release updates the `tracing-core` dependency to [v0.1.33][core-0.1.33] and the `tracing-attributes` dependency to [v0.1.28][attrs-0.1.28]. ### Added - **core**: Add index API for `Field` ([#2820]) - **core**: Allow `&[u8]` to be recorded as event/span field ([#2954]) ### Changed - Bump MSRV to 1.63 ([#2793]) - **core**: Use const `thread_local`s when possible ([#2838]) ### Fixed - Removed core imports in macros ([#2762]) - **attributes**: Added missing RecordTypes for instrument ([#2781]) - **attributes**: Change order of async and unsafe modifier ([#2864]) - Fix missing field prefixes ([#2878]) - **attributes**: Extract match scrutinee ([#2880]) - Fix non-simple macro usage without message ([#2879]) - Fix event macros with constant field names in the first position ([#2883]) - Allow field path segments to be keywords ([#2925]) - **core**: Fix missed `register_callsite` error ([#2938]) - **attributes**: Support const values for `target` and `name` ([#2941]) - Prefix macro calls with ::core to avoid clashing with local macros ([#3024]) [#2762]: #2762 [#2781]: #2781 [#2793]: #2793 [#2820]: #2820 [#2838]: #2838 [#2864]: #2864 [#2878]: #2878 [#2879]: #2879 [#2880]: #2880 [#2883]: #2883 [#2925]: #2925 [#2938]: #2938 [#2941]: #2941 [#2954]: #2954 [#3024]: #3024 [attrs-0.1.28]: https://github.com/tokio-rs/tracing/releases/tag/tracing-attributes-0.1.28 [core-0.1.33]: https://github.com/tokio-rs/tracing/releases/tag/tracing-core-0.1.33 [docs-0.1.41]: https://docs.rs/tracing/0.1.41/tracing/ [crate-0.1.41]: https://crates.io/crates/tracing/0.1.41
# 0.1.41 (November 25, 2024) [ [crates.io][crate-0.1.41] ] | [ [docs.rs][docs-0.1.41] ] This release updates the `tracing-core` dependency to [v0.1.33][core-0.1.33] and the `tracing-attributes` dependency to [v0.1.28][attrs-0.1.28]. ### Added - **core**: Add index API for `Field` ([#2820]) - **core**: Allow `&[u8]` to be recorded as event/span field ([#2954]) ### Changed - Bump MSRV to 1.63 ([#2793]) - **core**: Use const `thread_local`s when possible ([#2838]) ### Fixed - Removed core imports in macros ([#2762]) - **attributes**: Added missing RecordTypes for instrument ([#2781]) - **attributes**: Change order of async and unsafe modifier ([#2864]) - Fix missing field prefixes ([#2878]) - **attributes**: Extract match scrutinee ([#2880]) - Fix non-simple macro usage without message ([#2879]) - Fix event macros with constant field names in the first position ([#2883]) - Allow field path segments to be keywords ([#2925]) - **core**: Fix missed `register_callsite` error ([#2938]) - **attributes**: Support const values for `target` and `name` ([#2941]) - Prefix macro calls with ::core to avoid clashing with local macros ([#3024]) [#2762]: #2762 [#2781]: #2781 [#2793]: #2793 [#2820]: #2820 [#2838]: #2838 [#2864]: #2864 [#2878]: #2878 [#2879]: #2879 [#2880]: #2880 [#2883]: #2883 [#2925]: #2925 [#2938]: #2938 [#2941]: #2941 [#2954]: #2954 [#3024]: #3024 [attrs-0.1.28]: https://github.com/tokio-rs/tracing/releases/tag/tracing-attributes-0.1.28 [core-0.1.33]: https://github.com/tokio-rs/tracing/releases/tag/tracing-core-0.1.33 [docs-0.1.41]: https://docs.rs/tracing/0.1.41/tracing/ [crate-0.1.41]: https://crates.io/crates/tracing/0.1.41
# 0.1.41 (November 27, 2024) [ [crates.io][crate-0.1.41] ] | [ [docs.rs][docs-0.1.41] ] This release updates the `tracing-core` dependency to [v0.1.33][core-0.1.33] and the `tracing-attributes` dependency to [v0.1.28][attrs-0.1.28]. ### Added - **core**: Add index API for `Field` ([#2820]) - **core**: Allow `&[u8]` to be recorded as event/span field ([#2954]) ### Changed - Bump MSRV to 1.63 ([#2793]) - **core**: Use const `thread_local`s when possible ([#2838]) ### Fixed - Removed core imports in macros ([#2762]) - **attributes**: Added missing RecordTypes for instrument ([#2781]) - **attributes**: Change order of async and unsafe modifier ([#2864]) - Fix missing field prefixes ([#2878]) - **attributes**: Extract match scrutinee ([#2880]) - Fix non-simple macro usage without message ([#2879]) - Fix event macros with constant field names in the first position ([#2883]) - Allow field path segments to be keywords ([#2925]) - **core**: Fix missed `register_callsite` error ([#2938]) - **attributes**: Support const values for `target` and `name` ([#2941]) - Prefix macro calls with ::core to avoid clashing with local macros ([#3024]) [#2762]: #2762 [#2781]: #2781 [#2793]: #2793 [#2820]: #2820 [#2838]: #2838 [#2864]: #2864 [#2878]: #2878 [#2879]: #2879 [#2880]: #2880 [#2883]: #2883 [#2925]: #2925 [#2938]: #2938 [#2941]: #2941 [#2954]: #2954 [#3024]: #3024 [attrs-0.1.28]: https://github.com/tokio-rs/tracing/releases/tag/tracing-attributes-0.1.28 [core-0.1.33]: https://github.com/tokio-rs/tracing/releases/tag/tracing-core-0.1.33 [docs-0.1.41]: https://docs.rs/tracing/0.1.41/tracing/ [crate-0.1.41]: https://crates.io/crates/tracing/0.1.41
# 0.1.41 (November 27, 2024) [ [crates.io][crate-0.1.41] ] | [ [docs.rs][docs-0.1.41] ] This release updates the `tracing-core` dependency to [v0.1.33][core-0.1.33] and the `tracing-attributes` dependency to [v0.1.28][attrs-0.1.28]. ### Added - **core**: Add index API for `Field` ([#2820]) - **core**: Allow `&[u8]` to be recorded as event/span field ([#2954]) ### Changed - Bump MSRV to 1.63 ([#2793]) - **core**: Use const `thread_local`s when possible ([#2838]) ### Fixed - Removed core imports in macros ([#2762]) - **attributes**: Added missing RecordTypes for instrument ([#2781]) - **attributes**: Change order of async and unsafe modifier ([#2864]) - Fix missing field prefixes ([#2878]) - **attributes**: Extract match scrutinee ([#2880]) - Fix non-simple macro usage without message ([#2879]) - Fix event macros with constant field names in the first position ([#2883]) - Allow field path segments to be keywords ([#2925]) - **core**: Fix missed `register_callsite` error ([#2938]) - **attributes**: Support const values for `target` and `name` ([#2941]) - Prefix macro calls with ::core to avoid clashing with local macros ([#3024]) [#2762]: #2762 [#2781]: #2781 [#2793]: #2793 [#2820]: #2820 [#2838]: #2838 [#2864]: #2864 [#2878]: #2878 [#2879]: #2879 [#2880]: #2880 [#2883]: #2883 [#2925]: #2925 [#2938]: #2938 [#2941]: #2941 [#2954]: #2954 [#3024]: #3024 [attrs-0.1.28]: https://github.com/tokio-rs/tracing/releases/tag/tracing-attributes-0.1.28 [core-0.1.33]: https://github.com/tokio-rs/tracing/releases/tag/tracing-core-0.1.33 [docs-0.1.41]: https://docs.rs/tracing/0.1.41/tracing/ [crate-0.1.41]: https://crates.io/crates/tracing/0.1.41
Note: This is a change against the
v0.1.x
branch.Motivation
There are 2 triggers which will cause a subscriber to receive a call to
Subscriber::register_callsite
for a specific callsite.set_default
orwith_default
)It is trigger (2) that will cause a new subscriber to receive
Subscriber::register_callsite
for all the callsites which had alreadybeen registered before it became active.
When a callsite is registered for trigger (1), the callsite starts in
state
UNREGISTERED
.The first thread to encounter the callsite will transition it to
REGISTERING
and determine the overall interest for the callsite byregistering with all known dispatchers (which will call into
Subscriber::register_callsite
).Once that is complete, the callsite is added to the list of all known
callsites and its state is transitioned to
REGISTERED
.is (re)built for all known dispatchers. The callsite starts in state
UNREGISTERED
. The This calls down intoSubscriber::register_callsite
for each subscriber. Once that iscomplete, the callsite is added to the global list of known callsites.
While the callsite interest is being rebuilt, other threads that
encounter the callsite will be given
Interest::sometimes()
until theregistration is complete. However, if a new subscriber is added during
this window, all the interest for all callsites will be rebuilt, but
because the new callsite (in state
REGISTERING
) won't be includedbecause it isn't yet in the global list of callsites.
This can cause a case where that new subscriber being added won't
receive
Subscriber::register_callsite
before it receives the subsequentcall to
Subscriber::event
orSubscriber::new_span
.The documentation on Registering Callsites is not very explicit on
this point, but it does suggest that
Subscriber::register_callsite
will be called before the call to either
Subscriber::event
orSubscriber::new_span
, and the current behavior can break this implicitcontract.
Solution
This change swaps the order of rebuilding the callsite interest and
adding the callsite to the global list so that the callsite gets pushed
first, avoiding this window in which a subscriber won't get a call to
register_callsite
.As such, a callsite may have its interest read before it is set. In this
case, the existing implementation will return
Interest::sometimes()
for the
DefaultCallsite
implementation. Other implementations (outsideof the
tracing
project) may perform this differently, but in thiscase, there is no documented guarantee regarding the ordering.
A regression test is included which provokes the race condition 100% of
the time before the changes in this fix.
Fixes: #2743