Wrap the Rust HTTP client with `make_deferred_yieldable` #18903

MadLittleMods · 2025-09-09T03:58:28Z

Wrap the Rust HTTP client with make_deferred_yieldable so downstream usage doesn't need to use PreserveLoggingContext() or make_deferred_yieldable.

it seems like we should have some wrapper around it that uses make_deferred_yieldable(...) to make things right so we don't have to do this in the downstream code.

-- @MadLittleMods, #18357 (comment)

Spawning from wanting to remove PreserveLoggingContext() from the codebase and thinking that we shouldn't have to pollute all downstream usage with PreserveLoggingContext() or make_deferred_yieldable

Part of #18905 (Remove sentinel logcontext where we log in Synapse)

Dev notes

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
Code style is correct (run the linters)

So downstream usage doesn't need to use `PreserveLoggingContext()` or `make_deferred_yieldable` Spawning from #18870 and #18357 (comment)

MadLittleMods · 2025-09-09T04:01:42Z

synapse/synapse_rust_wrapper/http_client.py

+from synapse.types import ISynapseReactor
+
+
+class HttpClient:


I think this solution is awful.

What's a better way we can wrap the Rust code to do the right thing?

Call Python from the Rust code?

Write an equivalent wrapper in Rust? (not sure how to approach)

Better way to organize all of this? I initially tried to rename http_client.pyi to internal_http_client.pyi and add a new http_client.py with this wrapper all within synapse/synapse_rust/ but the native Python doesn't get picked up.

I think calling make_deferred_yieldable from the Rust side (and therefore not having the wrapper) would be a reasonable alternative, but I also don't think this wrapper approach is too bad — at least it's extremely clear. (Maybe a bit easy to forget about using it?)

If you want a sketch of calling Python code from Rust, here's how I called _get_size_of for my caching fun, whilst only importing it once:

static GETSIZEOF: OnceLock<pyo3::Py<pyo3::PyAny>> = OnceLock::new(); fn get_size_of(py: Python<'_>, obj: &Bound<'_, PyAny>) -> u64 { let getsizeof = GETSIZEOF.get_or_init(|| { let sys = PyModule::import(py, "synapse.util.caches.lrucache").unwrap(); let func = sys.getattr("_get_size_of").unwrap().unbind(); func }); let size: u64 = getsizeof.call1(py, (obj,)).unwrap().extract(py).unwrap(); size }

One thing I did when I did the pyo3-twisted bridge POC was to store and restore the contextvars during the future execution. If we switched to storing the logging context in contextvars, it would solve the logcontext being lost, but maybe not the resource usage metering. You can still probably take inspiration from that?

I think calling make_deferred_yieldable from the Rust side (and therefore not having the wrapper) would be a reasonable alternative

This would be my choice if we can get it working.

(haven't tried yet since I'm trying to write a test to exercise this part of the code easily, #18903 (comment))

reivilibre

OK to merge or otherwise would be happy to see the wrapperless approach if you have the energy for it

reivilibre · 2025-09-09T08:59:30Z

synapse/synapse_rust_wrapper/http_client.py

+from synapse.types import ISynapseReactor
+
+
+class HttpClient:


I think calling make_deferred_yieldable from the Rust side (and therefore not having the wrapper) would be a reasonable alternative, but I also don't think this wrapper approach is too bad — at least it's extremely clear. (Maybe a bit easy to forget about using it?)

If you want a sketch of calling Python code from Rust, here's how I called _get_size_of for my caching fun, whilst only importing it once:

static GETSIZEOF: OnceLock<pyo3::Py<pyo3::PyAny>> = OnceLock::new(); fn get_size_of(py: Python<'_>, obj: &Bound<'_, PyAny>) -> u64 { let getsizeof = GETSIZEOF.get_or_init(|| { let sys = PyModule::import(py, "synapse.util.caches.lrucache").unwrap(); let func = sys.getattr("_get_size_of").unwrap().unbind(); func }); let size: u64 = getsizeof.call1(py, (obj,)).unwrap().extract(py).unwrap(); size }

…ve-preserve-logging-context

MadLittleMods · 2025-09-09T19:19:13Z

tests/synapse_rust/test_http_client.py

+        # This triggers the server startup hooks, which starts the Tokio thread pool
+        reactor.run()


I'm trying to add a test for this but running into Tokio runtime is not running

$ SYNAPSE_TEST_LOG_LEVEL=INFO poetry run trial tests.synapse_rust.test_http_client The "poetry.dev-dependencies" section is deprecated and will be removed in a future version. Use "poetry.group.dev.dependencies" instead. tests.synapse_rust.test_http_client HttpClientTestCase test_logging_context ... [FAIL] =============================================================================== [FAIL] Traceback (most recent call last): File "synapse/tests/synapse_rust/test_http_client.py", line 87, in test_logging_context self.get_success(self.till_deferred_has_result(asdf())) File "synapse/tests/unittest.py", line 693, in get_success return self.successResultOf(deferred) File "virtualenvs/matrix-synapse-xCtC9ulO-py3.13/lib/python3.13/site-packages/twisted/trial/_synctest.py", line 732, in successResultOf self.fail( twisted.trial.unittest.FailTest: Success result expected on <Deferred at 0x7f928e6f5010 current result: None>, found failure result instead: Traceback (most recent call last): File "virtualenvs/matrix-synapse-xCtC9ulO-py3.13/lib/python3.13/site-packages/twisted/internet/defer.py", line 1857, in _inlineCallbacks result = context.run(gen.send, result) File "synapse/tests/synapse_rust/test_http_client.py", line 82, in asdf await self._rust_http_client.get( builtins.RuntimeError: Tokio runtime is not running tests.synapse_rust.test_http_client.HttpClientTestCase.test_logging_context ------------------------------------------------------------------------------- Ran 1 tests in 0.146s FAILED (failures=1)

I can see that the tokio runtime is supposed to start when the Twisted reactor starts:

synapse/rust/src/http_client.rs

Lines 115 to 117 in c68c5dd

// Attach the runtime to the reactor, starting it when the reactor is

// running, stopping it when the reactor is shutting down

reactor.call_method1("callWhenRunning", (runtime.getattr("start")?,))?;

(introduced in #18691)

If I add in my own hook in the test itself hs.get_reactor().callWhenRunning(print, "asdf"), it prints.

My test is based off of the existing MasAuthDelegation tests but those actually seem to work. Perhaps because the Rust action is happening within the Synapse code instead of in the test code where there might be enough indirection delay for tokio to startup fully.

I can't figure out how to wait that little bit longer for tokio to startup.

@sandhose Any insight? 🙇

…ve-preserve-logging-context

MadLittleMods added 3 commits September 8, 2025 22:56

Wrap the Rust HTTP client with make_deferred_yieldable

986e15c

So downstream usage doesn't need to use `PreserveLoggingContext()` or `make_deferred_yieldable` Spawning from #18870 and #18357 (comment)

Add chagnelog

5e555f3

Docstring for why we have a wrapper

4e0085c

MadLittleMods commented Sep 9, 2025

View reviewed changes

MadLittleMods marked this pull request as ready for review September 9, 2025 04:04

MadLittleMods requested a review from a team as a code owner September 9, 2025 04:04

This was referenced Sep 9, 2025

Speed up MAS token introspection #18357

Merged

Remove sentinel logcontext where we log in setup, start and exit #18870

Merged

reivilibre approved these changes Sep 9, 2025

View reviewed changes

Merge branch 'develop' into madlittlemods/18357-rust-http-client-remo…

763cdb9

…ve-preserve-logging-context

MadLittleMods mentioned this pull request Sep 9, 2025

replace loggingcontexts with contextvars #10342

Open

WIP: Add test

ce6f16d

MadLittleMods commented Sep 9, 2025

View reviewed changes

This was referenced Sep 10, 2025

Fix server_name in logging context for multiple Synapse instances in one process #18868

Draft

Remove sentinel logcontext where we log in Synapse #18905

Open

MadLittleMods added the A-Logging label Sep 10, 2025

Merge branch 'develop' into madlittlemods/18357-rust-http-client-remo…

d4558df

…ve-preserve-logging-context

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Wrap the Rust HTTP client with `make_deferred_yieldable` #18903

Wrap the Rust HTTP client with `make_deferred_yieldable` #18903

MadLittleMods commented Sep 9, 2025 •

edited

Loading

Uh oh!

MadLittleMods Sep 9, 2025 •

edited

Loading

Uh oh!

reivilibre Sep 9, 2025

Uh oh!

sandhose Sep 9, 2025

Uh oh!

MadLittleMods Sep 9, 2025

Uh oh!

reivilibre left a comment

Uh oh!

reivilibre Sep 9, 2025

Uh oh!

MadLittleMods Sep 9, 2025 •

edited

Loading

Uh oh!

MadLittleMods Sep 15, 2025

Uh oh!

Uh oh!

		# This triggers the server startup hooks, which starts the Tokio thread pool
		reactor.run()

	// Attach the runtime to the reactor, starting it when the reactor is
	// running, stopping it when the reactor is shutting down
	reactor.call_method1("callWhenRunning", (runtime.getattr("start")?,))?;

Wrap the Rust HTTP client with make_deferred_yieldable #18903

Are you sure you want to change the base?

Wrap the Rust HTTP client with make_deferred_yieldable #18903

Conversation

MadLittleMods commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dev notes

Pull Request Checklist

Uh oh!

MadLittleMods Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

reivilibre Sep 9, 2025

Choose a reason for hiding this comment

Uh oh!

sandhose Sep 9, 2025

Choose a reason for hiding this comment

Uh oh!

MadLittleMods Sep 9, 2025

Choose a reason for hiding this comment

Uh oh!

reivilibre left a comment

Choose a reason for hiding this comment

Uh oh!

reivilibre Sep 9, 2025

Choose a reason for hiding this comment

Uh oh!

MadLittleMods Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MadLittleMods Sep 15, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Wrap the Rust HTTP client with `make_deferred_yieldable` #18903

Wrap the Rust HTTP client with `make_deferred_yieldable` #18903

MadLittleMods commented Sep 9, 2025 •

edited

Loading

MadLittleMods Sep 9, 2025 •

edited

Loading

MadLittleMods Sep 9, 2025 •

edited

Loading