Split homeserver creation and setup #19015

MadLittleMods · 2025-10-03T22:34:53Z

Split homeserver creation and setup

Background

As part of Element's plan to support a light form of vhosting (virtual host) (multiple instances of Synapse in the same Python process), we're currently diving into the details and implications of running multiple instances of Synapse in the same Python process.

"Clean tenant provisioning" tracked internally by https://github.com/element-hq/synapse-small-hosts/issues/221

Partial startup problem

In the context of Synapse Pro for Small Hosts, since the Twisted reactor is already running (from the multi_synapse shard process itself), when provisioning a homeserver tenant, the reactor.callWhenRunning(...) callbacks will be invoked immediately. This includes the Synapse's start callback which sets up everything (including listeners, background tasks, etc). If we encounter an error at this point, we are partially setup but the exception will bubble back to us without us having a handle to the homeserver yet so we can't call hs.shutdown() and clean everything up.

What does this PR do?

Structures Synapse so we split creating the homeserver instance from setting everything up. This way we have access to hs if anything goes wrong during setup and can subsequently hs.shutdown() to clean everything up.

Dev notes

Refactor with start_background_tasks: #18886

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
Code style is correct (run the linters)

As we already have the reactor running in the case of Synapse Pro for Small Hosts and `setup()` should handle everything

…sks`

…round_tasks`" This reverts commit 211765f. (do this in another PR)

MadLittleMods · 2025-10-03T22:50:27Z

synapse/app/homeserver.py

+        # TODO: This should be moved to `SynapseHomeServer.start_background_tasks` (this
+        # way it matches the behavior of only running on `main`)


This is a future TODO

MadLittleMods · 2025-10-03T22:50:48Z

synapse/app/homeserver.py


-    async def start() -> None:
+    async def _start_when_reactor_running() -> None:
+        # TODO: Feels like this should be moved somewhere else.


This is a future TODO

MadLittleMods · 2025-10-07T23:06:27Z

synapse/app/homeserver.py

            "You have specified `worker_app` in the config but are attempting to start a non-worker "
            "instance. Please use `python -m synapse.app.generic_worker` instead (or remove the option if this is the main process)."
        )
-        sys.exit(1)


Removed this exit as it's unreachable since we raise just above

MadLittleMods · 2025-10-09T17:13:08Z

synapse/app/homeserver.py


 def start_reactor(
-    hs: HomeServer,
+    config: HomeServerConfig,


To document why I made the change to pass in the HomeServerConfig instead of hs to start_reactor(...) (as I was having trouble remembering myself the day after):

start_reactor(...) doesn't really have anything to do with the homeserver itself. This is just concerned with starting the Twisted event-loop that could be driving any application. And ideally, we wouldn't even pass something called HomeServerConfig as this is more just ReactorConfig. Currently just done out of convenience.

changelog.d/19015.misc

MadLittleMods · 2025-10-09T18:12:27Z

Thanks for the review @reivilibre 🐠

…HomeServerConfig` to do cross-config class validation (#19027) This means we can move the open registration config validation from `setup()` to `HomeServerConfig.validate_config()` (much more sane). Spawning from looking at this area of code in #19015

@frastefanini

Deployments that make use of the [synapse-s3-storage-provider](https://github.com/matrix-org/synapse-s3-storage-provider) module must upgrade to [v1.6.0](https://github.com/matrix-org/synapse-s3-storage-provider/releases/tag/v1.6.0). Using older versions of the module with this release of Synapse will prevent users from being able to upload or download media. No significant changes since 1.140.0rc1. - Add [a new Media Query by ID Admin API](https://element-hq.github.io/synapse/v1.140/admin_api/media_admin_api.html#query-a-piece-of-media-by-id) that allows server admins to query and investigate the metadata of local or cached remote media via the `origin/media_id` identifier found in a [Matrix Content URI](https://spec.matrix.org/v1.14/client-server-api/#matrix-content-mxc-uris). ([\element-hq#18911](element-hq#18911)) - Add [a new Fetch Event Admin API](https://element-hq.github.io/synapse/v1.140/admin_api/fetch_event.html) to fetch an event by ID. ([\element-hq#18963](element-hq#18963)) - Update [MSC4284: Policy Servers](matrix-org/matrix-spec-proposals#4284) implementation to support signatures when available. ([\element-hq#18934](element-hq#18934)) - Add experimental implementation of the `GET /_matrix/client/v1/rtc/transports` endpoint for the latest draft of [MSC4143: MatrixRTC](matrix-org/matrix-spec-proposals#4143). ([\element-hq#18967](element-hq#18967)) - Expose a `defer_to_threadpool` function in the Synapse Module API that allows modules to run a function on a separate thread in a custom threadpool. ([\element-hq#19032](element-hq#19032)) - Fix room upgrade `room_config` argument and documentation for `user_may_create_room` spam-checker callback. ([\element-hq#18721](element-hq#18721)) - Compute a user's last seen timestamp from their devices' last seen timestamps instead of IPs, because the latter are automatically cleared according to `user_ips_max_age`. ([\element-hq#18948](element-hq#18948)) - Fix bug where ephemeral events were not filtered by room ID. Contributed by @frastefanini. ([\element-hq#19002](element-hq#19002)) - Update Synapse main process version string to include git info. ([\element-hq#19011](element-hq#19011)) - Explain how `Deferred` callbacks interact with logcontexts. ([\element-hq#18914](element-hq#18914)) - Fix documentation for `rc_room_creation` and `rc_reports` to clarify that a `per_user` rate limit is not supported. ([\element-hq#18998](element-hq#18998)) - Remove deprecated `LoggingContext.set_current_context`/`LoggingContext.current_context` methods which already have equivalent bare methods in `synapse.logging.context`. ([\element-hq#18989](element-hq#18989)) - Drop support for unstable field names from the long-accepted [MSC2732](matrix-org/matrix-spec-proposals#2732) (Olm fallback keys) proposal. ([\element-hq#18996](element-hq#18996)) - Cleanly shutdown `SynapseHomeServer` object, allowing artifacts of embedded small hosts to be properly garbage collected. ([\element-hq#18828](element-hq#18828)) - Update OEmbed providers to use 'X' instead of 'Twitter' in URL previews, following a rebrand. Contributed by @HammyHavoc. ([\element-hq#18767](element-hq#18767)) - Fix `server_name` in logging context for multiple Synapse instances in one process. ([\element-hq#18868](element-hq#18868)) - Wrap the Rust HTTP client with `make_deferred_yieldable` so it follows Synapse logcontext rules. ([\element-hq#18903](element-hq#18903)) - Fix the GitHub Actions workflow that moves issues labeled "X-Needs-Info" to the "Needs info" column on the team's internal triage board. ([\element-hq#18913](element-hq#18913)) - Disconnect background process work from request trace. ([\element-hq#18932](element-hq#18932)) - Reduce overall number of calls to `_get_e2e_cross_signing_signatures_for_devices` by increasing the batch size of devices the query is called with, reducing DB load. ([\element-hq#18939](element-hq#18939)) - Update error code used when an appservice tries to masquerade as an unknown device using [MSC4326](matrix-org/matrix-spec-proposals#4326). Contributed by @tulir @ Beeper. ([\element-hq#18947](element-hq#18947)) - Fix `no active span when trying to log` tracing error on startup (when OpenTracing is enabled). ([\element-hq#18959](element-hq#18959)) - Fix `run_coroutine_in_background(...)` incorrectly handling logcontext. ([\element-hq#18964](element-hq#18964)) - Add debug logs wherever we change current logcontext. ([\element-hq#18966](element-hq#18966)) - Update dockerfile metadata to fix broken link; point to documentation website. ([\element-hq#18971](element-hq#18971)) - Note that the code is additionally licensed under the [Element Commercial license](https://github.com/element-hq/synapse/blob/develop/LICENSE-COMMERCIAL) in SPDX expression field configs. ([\element-hq#18973](element-hq#18973)) - Fix logcontext handling in `timeout_deferred` tests. ([\element-hq#18974](element-hq#18974)) - Remove internal `ReplicationUploadKeysForUserRestServlet` as a follow-up to the work in element-hq#18581 that moved device changes off the main process. ([\element-hq#18988](element-hq#18988)) - Switch task scheduler from raw logcontext manipulation to using the dedicated logcontext utils. ([\element-hq#18990](element-hq#18990)) - Remove `MockClock()` in tests. ([\element-hq#18992](element-hq#18992)) - Switch back to our own custom `LogContextScopeManager` instead of OpenTracing's `ContextVarsScopeManager` which was causing problems when using the experimental `SYNAPSE_ASYNC_IO_REACTOR` option with tracing enabled. ([\element-hq#19007](element-hq#19007)) - Remove `version_string` argument from `HomeServer` since it's always the same. ([\element-hq#19012](element-hq#19012)) - Remove duplicate call to `hs.start_background_tasks()` introduced from a bad merge. ([\element-hq#19013](element-hq#19013)) - Split homeserver creation (`create_homeserver`) and setup (`setup`). ([\element-hq#19015](element-hq#19015)) - Swap near-end-of-life `macos-13` GitHub Actions runner for the `macos-15-intel` variant. ([\element-hq#19025](element-hq#19025)) - Introduce `RootConfig.validate_config()` which can be subclassed in `HomeServerConfig` to do cross-config class validation. ([\element-hq#19027](element-hq#19027)) - Allow any command of the `release.py` script to accept a `--gh-token` argument. ([\element-hq#19035](element-hq#19035)) * Bump Swatinem/rust-cache from 2.8.0 to 2.8.1. ([\element-hq#18949](element-hq#18949)) * Bump actions/cache from 4.2.4 to 4.3.0. ([\element-hq#18983](element-hq#18983)) * Bump anyhow from 1.0.99 to 1.0.100. ([\element-hq#18950](element-hq#18950)) * Bump authlib from 1.6.3 to 1.6.4. ([\element-hq#18957](element-hq#18957)) * Bump authlib from 1.6.4 to 1.6.5. ([\element-hq#19019](element-hq#19019)) * Bump bcrypt from 4.3.0 to 5.0.0. ([\element-hq#18984](element-hq#18984)) * Bump docker/login-action from 3.5.0 to 3.6.0. ([\element-hq#18978](element-hq#18978)) * Bump lxml from 6.0.0 to 6.0.2. ([\element-hq#18979](element-hq#18979)) * Bump phonenumbers from 9.0.13 to 9.0.14. ([\element-hq#18954](element-hq#18954)) * Bump phonenumbers from 9.0.14 to 9.0.15. ([\element-hq#18991](element-hq#18991)) * Bump prometheus-client from 0.22.1 to 0.23.1. ([\element-hq#19016](element-hq#19016)) * Bump pydantic from 2.11.9 to 2.11.10. ([\element-hq#19017](element-hq#19017)) * Bump pygithub from 2.7.0 to 2.8.1. ([\element-hq#18952](element-hq#18952)) * Bump regex from 1.11.2 to 1.11.3. ([\element-hq#18981](element-hq#18981)) * Bump serde from 1.0.224 to 1.0.226. ([\element-hq#18953](element-hq#18953)) * Bump serde from 1.0.226 to 1.0.228. ([\element-hq#18982](element-hq#18982)) * Bump setuptools-rust from 1.11.1 to 1.12.0. ([\element-hq#18980](element-hq#18980)) * Bump twine from 6.1.0 to 6.2.0. ([\element-hq#18985](element-hq#18985)) * Bump types-pyyaml from 6.0.12.20250809 to 6.0.12.20250915. ([\element-hq#19018](element-hq#19018)) * Bump types-requests from 2.32.4.20250809 to 2.32.4.20250913. ([\element-hq#18951](element-hq#18951)) * Bump typing-extensions from 4.14.1 to 4.15.0. ([\element-hq#18956](element-hq#18956))

MadLittleMods added 9 commits October 3, 2025 14:34

Split homeserver creation and setup

73ddfba

Move start related things to run()

3521c0e

Rename run -> start to match start things inside

e651544

It's better if we do everything in setup

29457a2

As we already have the reactor running in the case of Synapse Pro for Small Hosts and `setup()` should handle everything

Add plans

b1f4478

Move db_pool start_doing_background_updates to `start_background_ta…

211765f

…sks`

Add docstring

2cf1f40

Revert "Move db_pool start_doing_background_updates to `start_backg…

0967fdf

…round_tasks`" This reverts commit 211765f. (do this in another PR)

Add changelog

0f50d26

MadLittleMods commented Oct 3, 2025

View reviewed changes

MadLittleMods added 5 commits October 7, 2025 17:37

Merge branch 'develop' into maddlittlemods/split-hs-creation-and-setup

e4857c0

Point out what not to do

ad3d981

Refactor in tests

18b42fb

Update docstrings

32e5f4b

No need to pass in the whole homeserver (just pass homeserver config)

16c7967

MadLittleMods commented Oct 7, 2025

View reviewed changes

MadLittleMods marked this pull request as ready for review October 7, 2025 23:41

MadLittleMods requested a review from a team as a code owner October 7, 2025 23:41

MadLittleMods mentioned this pull request Oct 7, 2025

Introduce RootConfig.validate_config() which can be subclassed in HomeServerConfig to do cross-config class validation #19027

Merged

3 tasks

Merge branch 'develop' into maddlittlemods/split-hs-creation-and-setup

614307b

reivilibre approved these changes Oct 9, 2025

View reviewed changes

MadLittleMods commented Oct 9, 2025

View reviewed changes

Merge branch 'develop' into maddlittlemods/split-hs-creation-and-setup

b2a82e0

MadLittleMods commented Oct 9, 2025

View reviewed changes

changelog.d/19015.misc Show resolved Hide resolved

MadLittleMods merged commit 715cc5e into develop Oct 9, 2025
44 checks passed

MadLittleMods deleted the maddlittlemods/split-hs-creation-and-setup branch October 9, 2025 18:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Split homeserver creation and setup #19015

Split homeserver creation and setup #19015

Uh oh!

MadLittleMods commented Oct 3, 2025 •

edited

Loading

Uh oh!

MadLittleMods Oct 3, 2025

Uh oh!

MadLittleMods Oct 3, 2025

Uh oh!

MadLittleMods Oct 7, 2025

Uh oh!

MadLittleMods Oct 9, 2025

Uh oh!

Uh oh!

Uh oh!

MadLittleMods commented Oct 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		# TODO: This should be moved to `SynapseHomeServer.start_background_tasks` (this
		# way it matches the behavior of only running on `main`)

Split homeserver creation and setup #19015

Split homeserver creation and setup #19015

Uh oh!

Conversation

MadLittleMods commented Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Background

Partial startup problem

What does this PR do?

Dev notes

Pull Request Checklist

Uh oh!

MadLittleMods Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

MadLittleMods Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

MadLittleMods Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

MadLittleMods Oct 9, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

MadLittleMods commented Oct 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

MadLittleMods commented Oct 3, 2025 •

edited

Loading