Merge tag 'v1.53.0' into babolivier/dinsic_1.58

Synapse 1.53.0 (2022-02-22)
===========================

No significant changes.

Synapse 1.53.0rc1 (2022-02-15)
==============================

Features
--------

- Add experimental support for sending to-device messages to application services, as specified by [MSC2409](matrix-org/matrix-spec-proposals#2409). ([\#11215](matrix-org/synapse#11215), [\#11966](matrix-org/synapse#11966))
- Remove account data (including client config, push rules and ignored users) upon user deactivation. ([\#11655](matrix-org/synapse#11655))
- Experimental support for [MSC3666](matrix-org/matrix-spec-proposals#3666): including bundled aggregations in server side search results. ([\#11837](matrix-org/synapse#11837))
- Enable cache time-based expiry by default. The `expiry_time` config flag has been superseded by `expire_caches` and `cache_entry_ttl`. ([\#11849](matrix-org/synapse#11849))
- Add a callback to allow modules to allow or forbid a 3PID (email address, phone number) from being associated to a local account. ([\#11854](matrix-org/synapse#11854))
- Stabilize support and remove unstable endpoints for [MSC3231](matrix-org/matrix-spec-proposals#3231). Clients must switch to the stable identifier and endpoint. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#stablisation-of-msc3231) for more information. ([\#11867](matrix-org/synapse#11867))
- Allow modules to retrieve the current instance's server name and worker name. ([\#11868](matrix-org/synapse#11868))
- Use a dedicated configurable rate limiter for 3PID invites. ([\#11892](matrix-org/synapse#11892))
- Support the stable API endpoint for [MSC3283](matrix-org/matrix-spec-proposals#3283): new settings in `/capabilities` endpoint. ([\#11933](matrix-org/synapse#11933), [\#11989](matrix-org/synapse#11989))
- Support the `dir` parameter on the `/relations` endpoint, per [MSC3715](matrix-org/matrix-spec-proposals#3715). ([\#11941](matrix-org/synapse#11941))
- Experimental implementation of [MSC3706](matrix-org/matrix-spec-proposals#3706): extensions to `/send_join` to support reduced response size. ([\#11967](matrix-org/synapse#11967))

Bugfixes
--------

- Fix [MSC2716](matrix-org/matrix-spec-proposals#2716) historical messages backfilling in random order on remote homeservers. ([\#11114](matrix-org/synapse#11114))
- Fix a bug introduced in Synapse 1.51.0 where incoming federation transactions containing at least one EDU would be dropped if debug logging was enabled for `synapse.8631_debug`. ([\#11890](matrix-org/synapse#11890))
- Fix a long-standing bug where some unknown endpoints would return HTML error pages instead of JSON `M_UNRECOGNIZED` errors. ([\#11930](matrix-org/synapse#11930))
- Implement an allow list of content types for which we will attempt to preview a URL. This prevents Synapse from making useless longer-lived connections to streaming media servers. ([\#11936](matrix-org/synapse#11936))
- Fix a long-standing bug where pagination tokens from `/sync` and `/messages` could not be provided to the `/relations` API. ([\#11952](matrix-org/synapse#11952))
- Require that modules register their callbacks using keyword arguments. ([\#11975](matrix-org/synapse#11975))
- Fix a long-standing bug where `M_WRONG_ROOM_KEYS_VERSION` errors would not include the specced `current_version` field. ([\#11988](matrix-org/synapse#11988))

Improved Documentation
----------------------

- Fix typo in User Admin API: unpind -> unbind. ([\#11859](matrix-org/synapse#11859))
- Document images returned by the User List Media Admin API can include those generated by URL previews. ([\#11862](matrix-org/synapse#11862))
- Remove outdated MSC1711 FAQ document. ([\#11907](matrix-org/synapse#11907))
- Correct the structured logging configuration example. Contributed by Brad Jones. ([\#11946](matrix-org/synapse#11946))
- Add information on the Synapse release cycle. ([\#11954](matrix-org/synapse#11954))
- Fix broken link in the README to the admin API for password reset. ([\#11955](matrix-org/synapse#11955))

Deprecations and Removals
-------------------------

- Drop support for `webclient` listeners and configuring `web_client_location` to a non-HTTP(S) URL. Deprecated configurations are a configuration error. ([\#11895](matrix-org/synapse#11895))
- Remove deprecated `user_may_create_room_with_invites` spam checker callback. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#removal-of-user_may_create_room_with_invites) for more information. ([\#11950](matrix-org/synapse#11950))
- No longer build `.deb` packages for Ubuntu 21.04 Hirsute Hippo, which has now EOLed. ([\#11961](matrix-org/synapse#11961))

Internal Changes
----------------

- Enhance user registration test helpers to make them more useful for tests involving application services and devices. ([\#11615](matrix-org/synapse#11615), [\#11616](matrix-org/synapse#11616))
- Improve performance when fetching bundled aggregations for multiple events. ([\#11660](matrix-org/synapse#11660), [\#11752](matrix-org/synapse#11752))
- Fix type errors introduced by new annotations in the Prometheus Client library. ([\#11832](matrix-org/synapse#11832))
- Add missing type hints to replication code. ([\#11856](matrix-org/synapse#11856), [\#11938](matrix-org/synapse#11938))
- Ensure that `opentracing` scopes are activated and closed at the right time. ([\#11869](matrix-org/synapse#11869))
- Improve opentracing for incoming federation requests. ([\#11870](matrix-org/synapse#11870))
- Improve internal docstrings in `synapse.util.caches`. ([\#11876](matrix-org/synapse#11876))
- Do not needlessly clear the `get_users_in_room` and `get_users_in_room_with_profiles` caches when any room state changes. ([\#11878](matrix-org/synapse#11878))
- Convert `ApplicationServiceTestCase` to use `simple_async_mock`. ([\#11880](matrix-org/synapse#11880))
- Remove experimental changes to the default push rules which were introduced in Synapse 1.19.0 but never enabled. ([\#11884](matrix-org/synapse#11884))
- Disable coverage calculation for olddeps build. ([\#11888](matrix-org/synapse#11888))
- Preparation to support sending device list updates to application services. ([\#11905](matrix-org/synapse#11905))
- Add a test that checks users receive their own device list updates down `/sync`. ([\#11909](matrix-org/synapse#11909))
- Run Complement tests sequentially. ([\#11910](matrix-org/synapse#11910))
- Various refactors to the application service notifier code. ([\#11911](matrix-org/synapse#11911), [\#11912](matrix-org/synapse#11912))
- Tests: replace mocked `Authenticator` with the real thing. ([\#11913](matrix-org/synapse#11913))
- Various refactors to the typing notifications code. ([\#11914](matrix-org/synapse#11914))
- Use the proper type for the `Content-Length` header in the `UploadResource`. ([\#11927](matrix-org/synapse#11927))
- Remove an unnecessary ignoring of type hints due to fixes in upstream packages. ([\#11939](matrix-org/synapse#11939))
- Add missing type hints. ([\#11953](matrix-org/synapse#11953))
- Fix an import cycle in `synapse.event_auth`. ([\#11965](matrix-org/synapse#11965))
- Unpin `frozendict` but exclude the known bad version 2.1.2. ([\#11969](matrix-org/synapse#11969))
- Prepare for rename of default Complement branch. ([\#11971](matrix-org/synapse#11971))
- Fetch Synapse's version using a helper from `matrix-common`. ([\#11979](matrix-org/synapse#11979))

Author: babolivier

.ci/scripts/test_old_deps.sh

@@ -15,4 +15,4 @@ export LANG="C.UTF-8"
 # Prevent virtualenv from auto-updating pip to an incompatible version
 export VIRTUALENV_NO_DOWNLOAD=1
 
-exec tox -e py3-old,combine
+exec tox -e py3-old

.github/workflows/tests.yml

@@ -338,7 +338,7 @@ jobs:
           path: synapse
 
       # Attempt to check out the same branch of Complement as the PR. If it
-      # doesn't exist, fallback to master.
+      # doesn't exist, fallback to HEAD.
       - name: Checkout complement
         shell: bash
         run: |
@@ -351,8 +351,8 @@ jobs:
           #    for pull requests, otherwise GITHUB_REF).
           # 2. Attempt to use the base branch, e.g. when merging into release-vX.Y
           #    (GITHUB_BASE_REF for pull requests).
-          # 3. Use the default complement branch ("master").
-          for BRANCH_NAME in "$GITHUB_HEAD_REF" "$GITHUB_BASE_REF" "${GITHUB_REF#refs/heads/}" "master"; do
+          # 3. Use the default complement branch ("HEAD").
+          for BRANCH_NAME in "$GITHUB_HEAD_REF" "$GITHUB_BASE_REF" "${GITHUB_REF#refs/heads/}" "HEAD"; do
             # Skip empty branch names and merge commits.
             if [[ -z "$BRANCH_NAME" || $BRANCH_NAME =~ ^refs/pull/.* ]]; then
               continue
@@ -376,7 +376,7 @@ jobs:
       # Run Complement
       - run: |
           set -o pipefail
-          go test -v -json -tags synapse_blacklist,msc2403 ./tests/... 2>&1 | gotestfmt
+          go test -v -json -p 1 -tags synapse_blacklist,msc2403 ./tests/... 2>&1 | gotestfmt
         shell: bash
         name: Run Complement Tests
         env:

CHANGES.md

@@ -1,14 +1,100 @@
+Synapse 1.53.0 (2022-02-22)
+===========================
+
+No significant changes.
+
+
+Synapse 1.53.0rc1 (2022-02-15)
+==============================
+
+Features
+--------
+
+- Add experimental support for sending to-device messages to application services, as specified by [MSC2409](https://github.com/matrix-org/matrix-doc/pull/2409). ([\#11215](https://github.com/matrix-org/synapse/issues/11215), [\#11966](https://github.com/matrix-org/synapse/issues/11966))
+- Remove account data (including client config, push rules and ignored users) upon user deactivation. ([\#11655](https://github.com/matrix-org/synapse/issues/11655))
+- Experimental support for [MSC3666](https://github.com/matrix-org/matrix-doc/pull/3666): including bundled aggregations in server side search results. ([\#11837](https://github.com/matrix-org/synapse/issues/11837))
+- Enable cache time-based expiry by default. The `expiry_time` config flag has been superseded by `expire_caches` and `cache_entry_ttl`. ([\#11849](https://github.com/matrix-org/synapse/issues/11849))
+- Add a callback to allow modules to allow or forbid a 3PID (email address, phone number) from being associated to a local account. ([\#11854](https://github.com/matrix-org/synapse/issues/11854))
+- Stabilize support and remove unstable endpoints for [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231). Clients must switch to the stable identifier and endpoint. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#stablisation-of-msc3231) for more information. ([\#11867](https://github.com/matrix-org/synapse/issues/11867))
+- Allow modules to retrieve the current instance's server name and worker name. ([\#11868](https://github.com/matrix-org/synapse/issues/11868))
+- Use a dedicated configurable rate limiter for 3PID invites. ([\#11892](https://github.com/matrix-org/synapse/issues/11892))
+- Support the stable API endpoint for [MSC3283](https://github.com/matrix-org/matrix-doc/pull/3283): new settings in `/capabilities` endpoint. ([\#11933](https://github.com/matrix-org/synapse/issues/11933), [\#11989](https://github.com/matrix-org/synapse/issues/11989))
+- Support the `dir` parameter on the `/relations` endpoint, per [MSC3715](https://github.com/matrix-org/matrix-doc/pull/3715). ([\#11941](https://github.com/matrix-org/synapse/issues/11941))
+- Experimental implementation of [MSC3706](https://github.com/matrix-org/matrix-doc/pull/3706): extensions to `/send_join` to support reduced response size. ([\#11967](https://github.com/matrix-org/synapse/issues/11967))
+
+
+Bugfixes
+--------
+
+- Fix [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) historical messages backfilling in random order on remote homeservers. ([\#11114](https://github.com/matrix-org/synapse/issues/11114))
+- Fix a bug introduced in Synapse 1.51.0 where incoming federation transactions containing at least one EDU would be dropped if debug logging was enabled for `synapse.8631_debug`. ([\#11890](https://github.com/matrix-org/synapse/issues/11890))
+- Fix a long-standing bug where some unknown endpoints would return HTML error pages instead of JSON `M_UNRECOGNIZED` errors. ([\#11930](https://github.com/matrix-org/synapse/issues/11930))
+- Implement an allow list of content types for which we will attempt to preview a URL. This prevents Synapse from making useless longer-lived connections to streaming media servers. ([\#11936](https://github.com/matrix-org/synapse/issues/11936))
+- Fix a long-standing bug where pagination tokens from `/sync` and `/messages` could not be provided to the `/relations` API. ([\#11952](https://github.com/matrix-org/synapse/issues/11952))
+- Require that modules register their callbacks using keyword arguments. ([\#11975](https://github.com/matrix-org/synapse/issues/11975))
+- Fix a long-standing bug where `M_WRONG_ROOM_KEYS_VERSION` errors would not include the specced `current_version` field. ([\#11988](https://github.com/matrix-org/synapse/issues/11988))
+
+
+Improved Documentation
+----------------------
+
+- Fix typo in User Admin API: unpind -> unbind. ([\#11859](https://github.com/matrix-org/synapse/issues/11859))
+- Document images returned by the User List Media Admin API can include those generated by URL previews. ([\#11862](https://github.com/matrix-org/synapse/issues/11862))
+- Remove outdated MSC1711 FAQ document. ([\#11907](https://github.com/matrix-org/synapse/issues/11907))
+- Correct the structured logging configuration example. Contributed by Brad Jones. ([\#11946](https://github.com/matrix-org/synapse/issues/11946))
+- Add information on the Synapse release cycle. ([\#11954](https://github.com/matrix-org/synapse/issues/11954))
+- Fix broken link in the README to the admin API for password reset. ([\#11955](https://github.com/matrix-org/synapse/issues/11955))
+
+
+Deprecations and Removals
+-------------------------
+
+- Drop support for `webclient` listeners and configuring `web_client_location` to a non-HTTP(S) URL. Deprecated configurations are a configuration error. ([\#11895](https://github.com/matrix-org/synapse/issues/11895))
+- Remove deprecated `user_may_create_room_with_invites` spam checker callback. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#removal-of-user_may_create_room_with_invites) for more information. ([\#11950](https://github.com/matrix-org/synapse/issues/11950))
+- No longer build `.deb` packages for Ubuntu 21.04 Hirsute Hippo, which has now EOLed. ([\#11961](https://github.com/matrix-org/synapse/issues/11961))
+
+
+Internal Changes
+----------------
+
+- Enhance user registration test helpers to make them more useful for tests involving application services and devices. ([\#11615](https://github.com/matrix-org/synapse/issues/11615), [\#11616](https://github.com/matrix-org/synapse/issues/11616))
+- Improve performance when fetching bundled aggregations for multiple events. ([\#11660](https://github.com/matrix-org/synapse/issues/11660), [\#11752](https://github.com/matrix-org/synapse/issues/11752))
+- Fix type errors introduced by new annotations in the Prometheus Client library. ([\#11832](https://github.com/matrix-org/synapse/issues/11832))
+- Add missing type hints to replication code. ([\#11856](https://github.com/matrix-org/synapse/issues/11856), [\#11938](https://github.com/matrix-org/synapse/issues/11938))
+- Ensure that `opentracing` scopes are activated and closed at the right time. ([\#11869](https://github.com/matrix-org/synapse/issues/11869))
+- Improve opentracing for incoming federation requests. ([\#11870](https://github.com/matrix-org/synapse/issues/11870))
+- Improve internal docstrings in `synapse.util.caches`. ([\#11876](https://github.com/matrix-org/synapse/issues/11876))
+- Do not needlessly clear the `get_users_in_room` and `get_users_in_room_with_profiles` caches when any room state changes. ([\#11878](https://github.com/matrix-org/synapse/issues/11878))
+- Convert `ApplicationServiceTestCase` to use `simple_async_mock`. ([\#11880](https://github.com/matrix-org/synapse/issues/11880))
+- Remove experimental changes to the default push rules which were introduced in Synapse 1.19.0 but never enabled. ([\#11884](https://github.com/matrix-org/synapse/issues/11884))
+- Disable coverage calculation for olddeps build. ([\#11888](https://github.com/matrix-org/synapse/issues/11888))
+- Preparation to support sending device list updates to application services. ([\#11905](https://github.com/matrix-org/synapse/issues/11905))
+- Add a test that checks users receive their own device list updates down `/sync`. ([\#11909](https://github.com/matrix-org/synapse/issues/11909))
+- Run Complement tests sequentially. ([\#11910](https://github.com/matrix-org/synapse/issues/11910))
+- Various refactors to the application service notifier code. ([\#11911](https://github.com/matrix-org/synapse/issues/11911), [\#11912](https://github.com/matrix-org/synapse/issues/11912))
+- Tests: replace mocked `Authenticator` with the real thing. ([\#11913](https://github.com/matrix-org/synapse/issues/11913))
+- Various refactors to the typing notifications code. ([\#11914](https://github.com/matrix-org/synapse/issues/11914))
+- Use the proper type for the `Content-Length` header in the `UploadResource`. ([\#11927](https://github.com/matrix-org/synapse/issues/11927))
+- Remove an unnecessary ignoring of type hints due to fixes in upstream packages. ([\#11939](https://github.com/matrix-org/synapse/issues/11939))
+- Add missing type hints. ([\#11953](https://github.com/matrix-org/synapse/issues/11953))
+- Fix an import cycle in `synapse.event_auth`. ([\#11965](https://github.com/matrix-org/synapse/issues/11965))
+- Unpin `frozendict` but exclude the known bad version 2.1.2. ([\#11969](https://github.com/matrix-org/synapse/issues/11969))
+- Prepare for rename of default Complement branch. ([\#11971](https://github.com/matrix-org/synapse/issues/11971))
+- Fetch Synapse's version using a helper from `matrix-common`. ([\#11979](https://github.com/matrix-org/synapse/issues/11979))
+
+
 Synapse 1.52.0 (2022-02-08)
 ===========================
 
 No significant changes since 1.52.0rc1.
 
-During the making of this release, the developers of Twisted have released
-[Twisted 22.1.0](https://github.com/twisted/twisted/releases/tag/twisted-22.1.0), which
-fixes [a security issue](https://github.com/twisted/twisted/security/advisories/GHSA-92x2-jw7w-xvvx)
-within Twisted. We do not believe Synapse to be vulnerable to any security problem caused
-by this issue, though we advise server administrators to update their local version of
-Twisted if they can.
+Note that [Twisted 22.1.0](https://github.com/twisted/twisted/releases/tag/twisted-22.1.0)
+has recently been released, which fixes a [security issue](https://github.com/twisted/twisted/security/advisories/GHSA-92x2-jw7w-xvvx)
+within the Twisted library. We do not believe Synapse is affected by this vulnerability,
+though we advise server administrators who installed Synapse via pip to upgrade Twisted
+with `pip install --upgrade Twisted` as a matter of good practice. The Docker image
+`matrixdotorg/synapse` and the Debian packages from `packages.matrix.org` are using the
+updated library.
 
 
 Synapse 1.52.0rc1 (2022-02-01)

README.rst

@@ -246,7 +246,7 @@ Password reset
 ==============
 
 Users can reset their password through their client. Alternatively, a server admin
-can reset a users password using the `admin API <docs/admin_api/user_admin_api.rst#reset-password>`_
+can reset a users password using the `admin API <docs/admin_api/user_admin_api.md#reset-password>`_
 or by directly editing the database as shown below.
 
 First calculate the hash of the new password::

debian/changelog

@@ -1,3 +1,15 @@
+matrix-synapse-py3 (1.53.0) stable; urgency=medium
+
+  * New synapse release 1.53.0.
+
+ -- Synapse Packaging team <[email protected]>  Tue, 22 Feb 2022 11:32:06 +0000
+
+matrix-synapse-py3 (1.53.0~rc1) stable; urgency=medium
+
+  * New synapse release 1.53.0~rc1.
+
+ -- Synapse Packaging team <[email protected]>  Tue, 15 Feb 2022 10:40:50 +0000
+
 matrix-synapse-py3 (1.52.0) stable; urgency=medium
 
   * New synapse release 1.52.0.