From 3eb300a96d4d6f337dc13dc47002d7480ea073fc Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Tue, 9 May 2023 16:37:07 +0300 Subject: [PATCH 01/11] Start writing spec for MSC2659: application service ping endpoint --- content/application-service-api.md | 42 ++++++ data/api/application-service/ping.yaml | 72 ++++++++++ data/api/client-server/appservice_ping.yaml | 148 ++++++++++++++++++++ 3 files changed, 262 insertions(+) create mode 100644 data/api/application-service/ping.yaml create mode 100644 data/api/client-server/appservice_ping.yaml diff --git a/content/application-service-api.md b/content/application-service-api.md index 7a14869ae..e433186bf 100644 --- a/content/application-service-api.md +++ b/content/application-service-api.md @@ -207,6 +207,48 @@ processed the events. {{% http-api spec="application-service" api="transactions" %}} +#### Pinging + +{{% added-in v="1.7" %}} + +The application service API includes a ping mechanism to allow +appservices to ensure that the homeserver can reach the appservice. +Appservices may use this mechanism to detect misconfigurations and +report them appropriately. + +Implementations using this mechanism should take care to not fail +entirely in the event of temporary issues, e.g. gracefully handling +cases where the appservice is started before the homeserver. + +The mechanism works as follows: + +``` + Typical + AS ---> HS : /_matrix/client/v1/appservice/{appserviceId}/ping {"transaction_id": "meow"} + HS ---> AS : /_matrix/app/v1/ping {"transaction_id": "meow"} + HS <--- AS : 200 OK {} + AS <--- HS : 200 OK {"duration_ms": 123} +``` + +``` + Incorrect hs_token + AS ---> HS : /_matrix/client/v1/appservice/{appserviceId}/ping {"transaction_id": "meow"} + HS ---> AS : /_matrix/app/v1/ping {"transaction_id": "meow"} + HS <--- AS : 401 Unauthorized {"errcode": "M_UNKNOWN_TOKEN"} + AS <--- HS : 502 Bad Gateway {"errcode": "M_BAD_STATUS", "status": 401, "body": "{\"errcode\": \"M_UNKNOWN_TOKEN\"}"} +``` + +``` + Can't connect to appservice + AS ---> HS : /_matrix/client/v1/appservice/{appserviceId}/ping {"transaction_id": "meow"} + HS -/-> AS : /_matrix/app/v1/ping {"transaction_id": "meow"} + AS <--- HS : 502 Bad Gateway {"errcode": "M_CONNECTION_FAILED"} +``` + +{{% http-api spec="client-server" api="appservice_ping" %}} + +{{% http-api spec="application-service" api="ping" %}} + #### Querying The application service API includes two querying APIs: for room aliases diff --git a/data/api/application-service/ping.yaml b/data/api/application-service/ping.yaml new file mode 100644 index 000000000..f6e7bf66d --- /dev/null +++ b/data/api/application-service/ping.yaml @@ -0,0 +1,72 @@ +# Copyright 2023 Tulir Asokan +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +swagger: '2.0' +info: + title: "Matrix Application Service API" + version: "1.0.0" +host: localhost:8008 +schemes: + - https + - http +basePath: /_matrix/app/v1 +produces: + - application/json +securityDefinitions: + $ref: definitions/security.yaml +paths: + "/ping": + post: + x-addedInMatrixVersion: "1.7" + summary: Ping the application service + description: |- + This API is called by the homeserver to ensure that the connection works + and the `hs_token` the homeserver has is correct. + + Currently this is only called by the homeserver as a direct result of + the application service calling the appservice ping endpoint in the + client API. + operationId: ping + security: + - homeserverAccessToken: [] + parameters: + - in: path + name: txnId + type: string + description: |- + The transaction ID for this set of events. Homeservers generate + these IDs and they are used to ensure idempotency of requests. + required: true + x-example: "35" + - in: body + name: body + description: Ping body with optional transaction ID. + schema: + type: object + example: { + "transaction_id": "mautrix-go_1683636478256400935_123" + } + properties: + transaction_id: + type: string + description: |- + A transaction ID for the ping, copied directly from the + client API appservice ping call. + required: ["events"] + responses: + 200: + description: The hs_token is valid and the ping request was successful. + examples: + application/json: {} + schema: + type: object diff --git a/data/api/client-server/appservice_ping.yaml b/data/api/client-server/appservice_ping.yaml new file mode 100644 index 000000000..c5e54f13c --- /dev/null +++ b/data/api/client-server/appservice_ping.yaml @@ -0,0 +1,148 @@ +# Copyright 2023 Tulir Asokan +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +swagger: '2.0' +info: + title: "Matrix Client-Server Application Service Ping API" + version: "1.0.0" +host: localhost:8008 +schemes: + - https + - http +basePath: /_matrix/client/v1 +consumes: + - application/json +produces: + - application/json +securityDefinitions: + # Note: this is the same access_token definition used elsewhere in the client + # server API, however this expects an access token for an application service. + $ref: definitions/security.yaml +paths: + "/appservice/{appserviceId}/ping": + post: + x-addedInMatrixVersion: "1.7" + summary: |- + Ask the homeserver to ping the application service to ensure the connection works. + description: |- + This API asks the homeserver to call the `/_matrix/app/v1/ping` endpoint + on the application service to ensure that the homeserver can communicate + with the application service. + + This API requires the use of an application service access token (`as_token`) + instead of a typical client's access_token. This API cannot be invoked by + users who are not identified as application services. Additionally, the + appservice ID in the path must be the same as the appservice whose `as_token` + is being used. + operationId: pingAppservice + parameters: + - in: path + type: string + name: appserviceId + description: |- + The appservice ID of the appservice to ping. This must be the same + as the appservice whose `as_token` is being used to authenticate the + request. + required: true + x-example: "telegram" + - in: body + name: body + required: true + schema: + type: object + properties: + transaction_id: + type: string + description: |- + An optional transaction ID that is passed through to the `/_matrix/app/v1/ping` call. + example: "mautrix-go_1683636478256400935_123" + security: + # again, this is the appservice's token - not a typical client's + - accessToken: [] + responses: + 200: + description: The ping was successful + schema: + type: object + properties: + duration_ms: + type: integer + description: |- + The duration that the `/_matrix/app/v1/ping` request took + from the homeserver's point of view. + examples: + application/json: {"duration_ms": 123} + 400: + description: The application service doesn't have a URL configured. The errcode is `M_URL_NOT_SET`. + schema: + $ref: "definitions/errors/error.yaml" + examples: + application/json: { + "errcode": "M_URL_NOT_SET", + "error": "Application service doesn't have a URL configured" + } + 403: + description: The access token used to authenticate the request doesn't belong to an appservice, or belongs to a different appservice than the one in the path. The errcode is `M_FORBIDDEN`. + schema: + $ref: "definitions/errors/error.yaml" + examples: + application/json: { + "errcode": "M_FORBIDDEN", + "error": "Provided access token is not the appservice's as_token" + } + 502: + description: |- + The application service returned a bad status, or the connection failed. + The errcode is `M_BAD_STATUS` or `M_CONNECTION_FAILED`. + + For bad statuses, the response may include `status` and `body` + fields containing the HTTP status code and response body text + respectively to aid with debugging. + schema: + type: object + title: Error + description: A Matrix-level Error + properties: + errcode: + type: string + description: An error code. + enum: [M_BAD_STATUS, M_CONNECTION_FAILED] + error: + type: string + description: A human-readable error message. + example: Ping returned status 401 + status: + type: integer + description: The HTTP status code returned by the appservice. + example: 401 + body: + type: string + description: The HTTP response body returned by the appservice. + example: "{\"errcode\": \"M_UNKNOWN_TOKEN\"}" + required: ["errcode"] + examples: + application/json: { + "errcode": "M_BAD_STATUS", + "error": "Ping returned status 401", + "status": 401, + "body": "{\"errcode\": \"M_UNKNOWN_TOKEN\"}" + } + 504: + description: The connection to the application service timed out. The errcode is `M_CONNECTION_TIMEOUT`. + schema: + $ref: "definitions/errors/error.yaml" + examples: + application/json: { + "errcode": "M_CONNECTION_TIMEOUT", + "error": "Connection to application service timed out" + } From 37ded93d412199698456ecd0e6aab5ee2044fbec Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Tue, 9 May 2023 16:40:50 +0300 Subject: [PATCH 02/11] Add newsfragment --- changelogs/application_service/newsfragments/1516.feature | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/application_service/newsfragments/1516.feature diff --git a/changelogs/application_service/newsfragments/1516.feature b/changelogs/application_service/newsfragments/1516.feature new file mode 100644 index 000000000..ee222fb06 --- /dev/null +++ b/changelogs/application_service/newsfragments/1516.feature @@ -0,0 +1 @@ +Add homeserver->appservice ping mechanism, as per [MSC2659](https://github.com/matrix-org/matrix-spec-proposals/pull/2659). Contributed by @tulir at @beeper. From 01183f5dc5c5b1771ef6109763559047d7b8eb93 Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Tue, 9 May 2023 16:43:28 +0300 Subject: [PATCH 03/11] Remove leftover parameter --- data/api/application-service/ping.yaml | 8 -------- 1 file changed, 8 deletions(-) diff --git a/data/api/application-service/ping.yaml b/data/api/application-service/ping.yaml index f6e7bf66d..e2e0584e9 100644 --- a/data/api/application-service/ping.yaml +++ b/data/api/application-service/ping.yaml @@ -40,14 +40,6 @@ paths: security: - homeserverAccessToken: [] parameters: - - in: path - name: txnId - type: string - description: |- - The transaction ID for this set of events. Homeservers generate - these IDs and they are used to ensure idempotency of requests. - required: true - x-example: "35" - in: body name: body description: Ping body with optional transaction ID. From 619e37e4c50b9bc2e5894185b73d7b9132e93a04 Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Tue, 9 May 2023 16:44:40 +0300 Subject: [PATCH 04/11] Remove another leftover bit --- data/api/application-service/ping.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/data/api/application-service/ping.yaml b/data/api/application-service/ping.yaml index e2e0584e9..1de4094f1 100644 --- a/data/api/application-service/ping.yaml +++ b/data/api/application-service/ping.yaml @@ -54,7 +54,6 @@ paths: description: |- A transaction ID for the ping, copied directly from the client API appservice ping call. - required: ["events"] responses: 200: description: The hs_token is valid and the ping request was successful. From 08768b4a6587275369510bf0075d39f33bfbb803 Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Tue, 23 May 2023 11:46:45 +0300 Subject: [PATCH 05/11] Add .new fragment --- changelogs/application_service/newsfragments/1516.new | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelogs/application_service/newsfragments/1516.new diff --git a/changelogs/application_service/newsfragments/1516.new b/changelogs/application_service/newsfragments/1516.new new file mode 100644 index 000000000..c62440755 --- /dev/null +++ b/changelogs/application_service/newsfragments/1516.new @@ -0,0 +1 @@ +Add `POST /_matrix/app/v1/ping` and `POST /_matrix/client/v1/appservice/{appserviceId}/ping` endpoints as per [MSC2659](https://github.com/matrix-org/matrix-spec-proposals/pull/2659). From 7e1c1a76e742ec741ea5d109f568a9691cc4fc96 Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Tue, 23 May 2023 11:47:56 +0300 Subject: [PATCH 06/11] Update flow formatting --- content/application-service-api.md | 31 ++++++++++++++++-------------- 1 file changed, 17 insertions(+), 14 deletions(-) diff --git a/content/application-service-api.md b/content/application-service-api.md index e433186bf..85223bedb 100644 --- a/content/application-service-api.md +++ b/content/application-service-api.md @@ -222,27 +222,30 @@ cases where the appservice is started before the homeserver. The mechanism works as follows: +**Typical** + ``` - Typical - AS ---> HS : /_matrix/client/v1/appservice/{appserviceId}/ping {"transaction_id": "meow"} - HS ---> AS : /_matrix/app/v1/ping {"transaction_id": "meow"} - HS <--- AS : 200 OK {} - AS <--- HS : 200 OK {"duration_ms": 123} +AS ---> HS : /_matrix/client/v1/appservice/{appserviceId}/ping {"transaction_id": "meow"} + HS ---> AS : /_matrix/app/v1/ping {"transaction_id": "meow"} + HS <--- AS : 200 OK {} +AS <--- HS : 200 OK {"duration_ms": 123} ``` +**Incorrect `hs_token`** + ``` - Incorrect hs_token - AS ---> HS : /_matrix/client/v1/appservice/{appserviceId}/ping {"transaction_id": "meow"} - HS ---> AS : /_matrix/app/v1/ping {"transaction_id": "meow"} - HS <--- AS : 401 Unauthorized {"errcode": "M_UNKNOWN_TOKEN"} - AS <--- HS : 502 Bad Gateway {"errcode": "M_BAD_STATUS", "status": 401, "body": "{\"errcode\": \"M_UNKNOWN_TOKEN\"}"} +AS ---> HS : /_matrix/client/v1/appservice/{appserviceId}/ping {"transaction_id": "meow"} + HS ---> AS : /_matrix/app/v1/ping {"transaction_id": "meow"} + HS <--- AS : 401 Unauthorized {"errcode": "M_UNKNOWN_TOKEN"} +AS <--- HS : 502 Bad Gateway {"errcode": "M_BAD_STATUS", "status": 401, "body": "{\"errcode\": \"M_UNKNOWN_TOKEN\"}"} ``` +**Can't connect to appservice** + ``` - Can't connect to appservice - AS ---> HS : /_matrix/client/v1/appservice/{appserviceId}/ping {"transaction_id": "meow"} - HS -/-> AS : /_matrix/app/v1/ping {"transaction_id": "meow"} - AS <--- HS : 502 Bad Gateway {"errcode": "M_CONNECTION_FAILED"} +AS ---> HS : /_matrix/client/v1/appservice/{appserviceId}/ping {"transaction_id": "meow"} + HS -/-> AS : /_matrix/app/v1/ping {"transaction_id": "meow"} +AS <--- HS : 502 Bad Gateway {"errcode": "M_CONNECTION_FAILED"} ``` {{% http-api spec="client-server" api="appservice_ping" %}} From e02c69de83ff235b81f4b6c27596aba18b4a4e39 Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Tue, 23 May 2023 11:48:37 +0300 Subject: [PATCH 07/11] Apply suggestions from code review Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> --- data/api/application-service/ping.yaml | 2 +- data/api/client-server/appservice_ping.yaml | 14 ++++++++------ 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/data/api/application-service/ping.yaml b/data/api/application-service/ping.yaml index 1de4094f1..c4f12b4f0 100644 --- a/data/api/application-service/ping.yaml +++ b/data/api/application-service/ping.yaml @@ -56,7 +56,7 @@ paths: client API appservice ping call. responses: 200: - description: The hs_token is valid and the ping request was successful. + description: The provided `hs_token` is valid and the ping request was successful. examples: application/json: {} schema: diff --git a/data/api/client-server/appservice_ping.yaml b/data/api/client-server/appservice_ping.yaml index c5e54f13c..4075e901f 100644 --- a/data/api/client-server/appservice_ping.yaml +++ b/data/api/client-server/appservice_ping.yaml @@ -35,12 +35,13 @@ paths: summary: |- Ask the homeserver to ping the application service to ensure the connection works. description: |- - This API asks the homeserver to call the `/_matrix/app/v1/ping` endpoint - on the application service to ensure that the homeserver can communicate + This API asks the homeserver to call the + [`/_matrix/app/v1/ping`](#post_matrixappv1ping) endpoint on the + application service to ensure that the homeserver can communicate with the application service. This API requires the use of an application service access token (`as_token`) - instead of a typical client's access_token. This API cannot be invoked by + instead of a typical client's access token. This API cannot be invoked by users who are not identified as application services. Additionally, the appservice ID in the path must be the same as the appservice whose `as_token` is being used. @@ -71,15 +72,16 @@ paths: - accessToken: [] responses: 200: - description: The ping was successful + description: The ping was successful. schema: type: object properties: duration_ms: type: integer description: |- - The duration that the `/_matrix/app/v1/ping` request took - from the homeserver's point of view. + The duration in milliseconds that the + [`/_matrix/app/v1/ping`](#post_matrixappv1ping) + request took from the homeserver's point of view. examples: application/json: {"duration_ms": 123} 400: From d50512e1c62cbf5a5ce5ad0634ef06173cb23db9 Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Tue, 23 May 2023 11:54:43 +0300 Subject: [PATCH 08/11] Explicitly link to endpoint in AS-side /ping doc --- data/api/application-service/ping.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/data/api/application-service/ping.yaml b/data/api/application-service/ping.yaml index c4f12b4f0..18682f113 100644 --- a/data/api/application-service/ping.yaml +++ b/data/api/application-service/ping.yaml @@ -34,8 +34,8 @@ paths: and the `hs_token` the homeserver has is correct. Currently this is only called by the homeserver as a direct result of - the application service calling the appservice ping endpoint in the - client API. + the application service calling + [`POST /_matrix/client/v1/appservice/{appserviceId}/ping`](#post_matrixclientv1appserviceappserviceidping). operationId: ping security: - homeserverAccessToken: [] @@ -53,7 +53,7 @@ paths: type: string description: |- A transaction ID for the ping, copied directly from the - client API appservice ping call. + `POST /_matrix/client/v1/appservice/{appserviceId}/ping` call. responses: 200: description: The provided `hs_token` is valid and the ping request was successful. From 13e6b5c1c0b42cff14490f618617a5f34e484e5b Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Tue, 23 May 2023 11:56:05 +0300 Subject: [PATCH 09/11] Move CS ping to client-server API extensions --- content/application-service-api.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/content/application-service-api.md b/content/application-service-api.md index 85223bedb..df480d376 100644 --- a/content/application-service-api.md +++ b/content/application-service-api.md @@ -248,7 +248,9 @@ AS ---> HS : /_matrix/client/v1/appservice/{appserviceId}/ping {"transaction_id" AS <--- HS : 502 Bad Gateway {"errcode": "M_CONNECTION_FAILED"} ``` -{{% http-api spec="client-server" api="appservice_ping" %}} +The `/_matrix/app/v1/ping` endpoint is described here. The +[`/_matrix/client/v1/appservice/{appserviceId}/ping`](#post_matrixclientv1appserviceappserviceidping) +endpoint is under the Client-Server API extensions section below. {{% http-api spec="application-service" api="ping" %}} @@ -433,6 +435,15 @@ an application service-defined namespace will receive the same `M_EXCLUSIVE` error code, but only if the application service has defined the namespace as `exclusive`. +#### Pinging + +{{% added-in v="1.7" %}} + +This is the client-server API companion endpoint for the +[pinging](#pinging) mechanism described above. + +{{% http-api spec="client-server" api="appservice_ping" %}} + #### Using `/sync` and `/events` Application services wishing to use `/sync` or `/events` from the From 8431201442355e97b2f94fcf89eb4dc3f2290e96 Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Tue, 23 May 2023 12:03:39 +0300 Subject: [PATCH 10/11] Change error in example flow --- content/application-service-api.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/application-service-api.md b/content/application-service-api.md index df480d376..10a7d9c53 100644 --- a/content/application-service-api.md +++ b/content/application-service-api.md @@ -236,8 +236,8 @@ AS <--- HS : 200 OK {"duration_ms": 123} ``` AS ---> HS : /_matrix/client/v1/appservice/{appserviceId}/ping {"transaction_id": "meow"} HS ---> AS : /_matrix/app/v1/ping {"transaction_id": "meow"} - HS <--- AS : 401 Unauthorized {"errcode": "M_UNKNOWN_TOKEN"} -AS <--- HS : 502 Bad Gateway {"errcode": "M_BAD_STATUS", "status": 401, "body": "{\"errcode\": \"M_UNKNOWN_TOKEN\"}"} + HS <--- AS : 403 Forbidden {"errcode": "M_FORBIDDEN"} +AS <--- HS : 502 Bad Gateway {"errcode": "M_BAD_STATUS", "status": 403, "body": "{\"errcode\": \"M_FORBIDDEN\"}"} ``` **Can't connect to appservice** From ff739e82723bf39c960a323455a73b6d789ac132 Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Tue, 23 May 2023 12:07:00 +0300 Subject: [PATCH 11/11] Note that error fields have been omitted --- content/application-service-api.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/content/application-service-api.md b/content/application-service-api.md index 10a7d9c53..f3db06cf3 100644 --- a/content/application-service-api.md +++ b/content/application-service-api.md @@ -220,7 +220,8 @@ Implementations using this mechanism should take care to not fail entirely in the event of temporary issues, e.g. gracefully handling cases where the appservice is started before the homeserver. -The mechanism works as follows: +The mechanism works as follows (note: the human-readable `error` fields +have been omitted for brevity): **Typical**