From ce59930ab144b1f468acf68df15cf09e790c8958 Mon Sep 17 00:00:00 2001 From: Nick Barrett Date: Thu, 4 Aug 2022 16:35:54 +0100 Subject: [PATCH 01/16] Add media download redirect proposal --- proposals/xxx-media-download-redirect.md | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 proposals/xxx-media-download-redirect.md diff --git a/proposals/xxx-media-download-redirect.md b/proposals/xxx-media-download-redirect.md new file mode 100644 index 0000000000..21bfe305bd --- /dev/null +++ b/proposals/xxx-media-download-redirect.md @@ -0,0 +1,8 @@ +# MSCXXX: Media Download Redirects + +Currently the media download endpoints must return either a 200 with content or error responses. This means the media server instance must stream the data from wherever it is stored, which is likely not local to itself. Allowing redirects on these endpoints would +make it possible for the media repo to tell clients/servers to pull data direct from the source, eg. a CDN. + +## Proposal + +This MSC simply proposes that a 307 redirect code is allowed and followed according to the `Location` header. It is possible some clients would already follow these which needs to be confirmed. From 964c19e2c062d946206b5b691e07e5e50a44bdb8 Mon Sep 17 00:00:00 2001 From: Nick Barrett Date: Thu, 4 Aug 2022 16:39:36 +0100 Subject: [PATCH 02/16] Add urls --- proposals/xxx-media-download-redirect.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/proposals/xxx-media-download-redirect.md b/proposals/xxx-media-download-redirect.md index 21bfe305bd..f4064d624f 100644 --- a/proposals/xxx-media-download-redirect.md +++ b/proposals/xxx-media-download-redirect.md @@ -5,4 +5,8 @@ make it possible for the media repo to tell clients/servers to pull data direct ## Proposal -This MSC simply proposes that a 307 redirect code is allowed and followed according to the `Location` header. It is possible some clients would already follow these which needs to be confirmed. +This MSC simply proposes that a 307 redirect code is allowed and followed according to the `Location` header. It is possible some clients would already follow these which needs to be confirmed. Specifc endpoints in question: + ++ `/_matrix/media/v3/download/{serverName}/{mediaId}` ++ `/_matrix/media/v3/download/{serverName}/{mediaId}/{fileName}` ++ `/_matrix/media/v3/thumbnail/{serverName}/{mediaId}` From 088789bbca2284adb083139012ec926987e5f40a Mon Sep 17 00:00:00 2001 From: Nick Barrett Date: Thu, 4 Aug 2022 16:39:44 +0100 Subject: [PATCH 03/16] Rename --- ...dia-download-redirect.md => 3860-media-download-redirect.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename proposals/{xxx-media-download-redirect.md => 3860-media-download-redirect.md} (95%) diff --git a/proposals/xxx-media-download-redirect.md b/proposals/3860-media-download-redirect.md similarity index 95% rename from proposals/xxx-media-download-redirect.md rename to proposals/3860-media-download-redirect.md index f4064d624f..c07914761d 100644 --- a/proposals/xxx-media-download-redirect.md +++ b/proposals/3860-media-download-redirect.md @@ -1,4 +1,4 @@ -# MSCXXX: Media Download Redirects +# MSC3860: Media Download Redirects Currently the media download endpoints must return either a 200 with content or error responses. This means the media server instance must stream the data from wherever it is stored, which is likely not local to itself. Allowing redirects on these endpoints would make it possible for the media repo to tell clients/servers to pull data direct from the source, eg. a CDN. From 916e1b38bfaa7a66688303352d2172c044031f94 Mon Sep 17 00:00:00 2001 From: Nick Barrett Date: Wed, 17 Aug 2022 13:20:38 +0100 Subject: [PATCH 04/16] Wrap content --- proposals/3860-media-download-redirect.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/proposals/3860-media-download-redirect.md b/proposals/3860-media-download-redirect.md index c07914761d..1d5a4e3883 100644 --- a/proposals/3860-media-download-redirect.md +++ b/proposals/3860-media-download-redirect.md @@ -1,11 +1,15 @@ # MSC3860: Media Download Redirects -Currently the media download endpoints must return either a 200 with content or error responses. This means the media server instance must stream the data from wherever it is stored, which is likely not local to itself. Allowing redirects on these endpoints would -make it possible for the media repo to tell clients/servers to pull data direct from the source, eg. a CDN. +Currently the media download endpoints must return either a 200 with content or error responses. This +means the media server instance must stream the data from wherever it is stored, which is likely not +local to itself. Allowing redirects on these endpoints would make it possible for the media repo to +tell clients/servers to pull data direct from the source, eg. a CDN. ## Proposal -This MSC simply proposes that a 307 redirect code is allowed and followed according to the `Location` header. It is possible some clients would already follow these which needs to be confirmed. Specifc endpoints in question: +This MSC simply proposes that a 307 redirect code is allowed and followed according to the `Location` +header. It is possible some clients would already follow these which needs to be confirmed. Specifc +endpoints in question: + `/_matrix/media/v3/download/{serverName}/{mediaId}` + `/_matrix/media/v3/download/{serverName}/{mediaId}/{fileName}` From 9807bf420c52ea32993d89da6ac1d813a6d4729a Mon Sep 17 00:00:00 2001 From: Nick Barrett Date: Wed, 17 Aug 2022 13:20:50 +0100 Subject: [PATCH 05/16] Add note about MSC proposal and implementation work --- proposals/3860-media-download-redirect.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/proposals/3860-media-download-redirect.md b/proposals/3860-media-download-redirect.md index 1d5a4e3883..d217d101b7 100644 --- a/proposals/3860-media-download-redirect.md +++ b/proposals/3860-media-download-redirect.md @@ -14,3 +14,7 @@ endpoints in question: + `/_matrix/media/v3/download/{serverName}/{mediaId}` + `/_matrix/media/v3/download/{serverName}/{mediaId}/{fileName}` + `/_matrix/media/v3/thumbnail/{serverName}/{mediaId}` + +The media repo already conforms to standard HTTP practices so this may already work as expected. The +MSC is proposing to add redirects to the list of possible HTTP responses for the above endpoints in +the Matrix Specifiction. Implementation would be confirming that this works in all the major clients. From 6746c86c5e1fbd3eff638f6b3bcae3bde76c2018 Mon Sep 17 00:00:00 2001 From: Nick Barrett Date: Tue, 23 Aug 2022 09:51:27 +0100 Subject: [PATCH 06/16] Add missing sections to MSC --- proposals/3860-media-download-redirect.md | 24 +++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/proposals/3860-media-download-redirect.md b/proposals/3860-media-download-redirect.md index d217d101b7..ed1a25d67e 100644 --- a/proposals/3860-media-download-redirect.md +++ b/proposals/3860-media-download-redirect.md @@ -18,3 +18,27 @@ endpoints in question: The media repo already conforms to standard HTTP practices so this may already work as expected. The MSC is proposing to add redirects to the list of possible HTTP responses for the above endpoints in the Matrix Specifiction. Implementation would be confirming that this works in all the major clients. + +## Potential Issues + +There may be clients that don't follow redirect responses properly, in which case they would fail +to retrieve the media. One possible workaround for this is utilising an opt-in query string parameter +to allow redirects, e.g `?allow-redirect=true`. + +## Alternatives + +None at this time. + +## Security Considerations + +A media repo could redirect requests to a bad actor, although this would make the primary media +repo itself a bad actor, this does present any increased security issues. + +## Unstable Prefix + +No need for an unstable prefix for redirects as it stands. If a query string was to be used to +enable this functionality this could use the following unstable prefix: + +``` +?com.beeper.msc3860.allow-redirects=true +``` From 6767a5175a5fe4a67dc051e5a84f08065770897d Mon Sep 17 00:00:00 2001 From: Nick Barrett Date: Fri, 21 Oct 2022 15:54:19 +0100 Subject: [PATCH 07/16] Rework to be opt-in via query param --- proposals/3860-media-download-redirect.md | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/proposals/3860-media-download-redirect.md b/proposals/3860-media-download-redirect.md index ed1a25d67e..92f05c73b9 100644 --- a/proposals/3860-media-download-redirect.md +++ b/proposals/3860-media-download-redirect.md @@ -7,7 +7,7 @@ tell clients/servers to pull data direct from the source, eg. a CDN. ## Proposal -This MSC simply proposes that a 307 redirect code is allowed and followed according to the `Location` +This MSC proposes that a 307 redirect code is allowed and followed according to the `Location` header. It is possible some clients would already follow these which needs to be confirmed. Specifc endpoints in question: @@ -15,15 +15,17 @@ endpoints in question: + `/_matrix/media/v3/download/{serverName}/{mediaId}/{fileName}` + `/_matrix/media/v3/thumbnail/{serverName}/{mediaId}` -The media repo already conforms to standard HTTP practices so this may already work as expected. The -MSC is proposing to add redirects to the list of possible HTTP responses for the above endpoints in -the Matrix Specifiction. Implementation would be confirming that this works in all the major clients. +To prevent breaking clients that don't properly follow the redirect response this functionality will +be enabled by a query string flag `allow_redirect=true`. So specifically in the above cases if a +client respects redirect responses it can make requests like so to the media endpoints: + ++ `/_matrix/media/v3/download/{serverName}/{mediaId}?allow_redirect=true` ++ `/_matrix/media/v3/download/{serverName}/{mediaId}/{fileName}?allow_redirect=true` ++ `/_matrix/media/v3/thumbnail/{serverName}/{mediaId}?allow_redirect=true` ## Potential Issues -There may be clients that don't follow redirect responses properly, in which case they would fail -to retrieve the media. One possible workaround for this is utilising an opt-in query string parameter -to allow redirects, e.g `?allow-redirect=true`. +None, as opt-in functionality this change is 100% backwards compatible. ## Alternatives @@ -40,5 +42,5 @@ No need for an unstable prefix for redirects as it stands. If a query string was enable this functionality this could use the following unstable prefix: ``` -?com.beeper.msc3860.allow-redirects=true +?com.beeper.msc3860.allow_redirect=true ``` From aa144017a5968482d78b2eaa5300aa893687c2f4 Mon Sep 17 00:00:00 2001 From: Sumner Evans Date: Wed, 29 Mar 2023 08:48:14 +0200 Subject: [PATCH 08/16] Fix typos Signed-off-by: Sumner Evans --- proposals/3860-media-download-redirect.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/3860-media-download-redirect.md b/proposals/3860-media-download-redirect.md index 92f05c73b9..72c67c97c6 100644 --- a/proposals/3860-media-download-redirect.md +++ b/proposals/3860-media-download-redirect.md @@ -3,12 +3,12 @@ Currently the media download endpoints must return either a 200 with content or error responses. This means the media server instance must stream the data from wherever it is stored, which is likely not local to itself. Allowing redirects on these endpoints would make it possible for the media repo to -tell clients/servers to pull data direct from the source, eg. a CDN. +tell clients/servers to pull data direct from the source, e.g. a CDN. ## Proposal This MSC proposes that a 307 redirect code is allowed and followed according to the `Location` -header. It is possible some clients would already follow these which needs to be confirmed. Specifc +header. It is possible some clients would already follow these which needs to be confirmed. Specific endpoints in question: + `/_matrix/media/v3/download/{serverName}/{mediaId}` From c9896a9652c22eb15e826b903e1e36b5d8012e1c Mon Sep 17 00:00:00 2001 From: Sumner Evans Date: Wed, 29 Mar 2023 08:58:06 +0200 Subject: [PATCH 09/16] Clarify security considerations Signed-off-by: Sumner Evans --- proposals/3860-media-download-redirect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/3860-media-download-redirect.md b/proposals/3860-media-download-redirect.md index 72c67c97c6..e31f23502f 100644 --- a/proposals/3860-media-download-redirect.md +++ b/proposals/3860-media-download-redirect.md @@ -34,7 +34,7 @@ None at this time. ## Security Considerations A media repo could redirect requests to a bad actor, although this would make the primary media -repo itself a bad actor, this does present any increased security issues. +repo itself a bad actor, thus this does not present any increased security issues. ## Unstable Prefix From 4b61481a1b57ed67e778df6db59d6ad35be9f8ae Mon Sep 17 00:00:00 2001 From: Sumner Evans Date: Thu, 30 Mar 2023 07:25:50 +0200 Subject: [PATCH 10/16] Clarify that unstable prefix should be used Signed-off-by: Sumner Evans --- proposals/3860-media-download-redirect.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/3860-media-download-redirect.md b/proposals/3860-media-download-redirect.md index e31f23502f..04250ecc00 100644 --- a/proposals/3860-media-download-redirect.md +++ b/proposals/3860-media-download-redirect.md @@ -38,8 +38,8 @@ repo itself a bad actor, thus this does not present any increased security issue ## Unstable Prefix -No need for an unstable prefix for redirects as it stands. If a query string was to be used to -enable this functionality this could use the following unstable prefix: +Until this functionality has landed in the spec, the `allow_redirect` query +parameter should be prefixed with `com.beeper.msc3860.`: ``` ?com.beeper.msc3860.allow_redirect=true From c2e9d45d0d52966dfeaddc2c8584a0d57787d34e Mon Sep 17 00:00:00 2001 From: Nick Mills-Barrett Date: Sat, 22 Apr 2023 16:47:21 +0100 Subject: [PATCH 11/16] Add 308 status code for redirects Co-authored-by: Travis Ralston --- proposals/3860-media-download-redirect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/3860-media-download-redirect.md b/proposals/3860-media-download-redirect.md index 04250ecc00..fae514ffe6 100644 --- a/proposals/3860-media-download-redirect.md +++ b/proposals/3860-media-download-redirect.md @@ -7,7 +7,7 @@ tell clients/servers to pull data direct from the source, e.g. a CDN. ## Proposal -This MSC proposes that a 307 redirect code is allowed and followed according to the `Location` +This MSC proposes that a 307 or 308 redirect code is allowed and followed according to the `Location` header. It is possible some clients would already follow these which needs to be confirmed. Specific endpoints in question: From 117f90c01441ff6a333c2a78e290ce41d4de08f8 Mon Sep 17 00:00:00 2001 From: Nick Mills-Barrett Date: Tue, 2 May 2023 13:55:35 +0100 Subject: [PATCH 12/16] Add clarification about no/false redirect param --- proposals/3860-media-download-redirect.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/proposals/3860-media-download-redirect.md b/proposals/3860-media-download-redirect.md index fae514ffe6..82fac5814d 100644 --- a/proposals/3860-media-download-redirect.md +++ b/proposals/3860-media-download-redirect.md @@ -23,6 +23,9 @@ client respects redirect responses it can make requests like so to the media end + `/_matrix/media/v3/download/{serverName}/{mediaId}/{fileName}?allow_redirect=true` + `/_matrix/media/v3/thumbnail/{serverName}/{mediaId}?allow_redirect=true` +In the case where a client wishes not to redirect (either implictly with no parameter or explicitly +providing `allow_redirect=false`) the server must continue to serve media directly with no redirect. + ## Potential Issues None, as opt-in functionality this change is 100% backwards compatible. From bf2d22cf51584fb484882e74c689639f7f91aaf1 Mon Sep 17 00:00:00 2001 From: Nick Mills-Barrett Date: Tue, 2 May 2023 13:57:36 +0100 Subject: [PATCH 13/16] Add link to current spec media endpoints --- proposals/3860-media-download-redirect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/3860-media-download-redirect.md b/proposals/3860-media-download-redirect.md index 82fac5814d..51c3063d28 100644 --- a/proposals/3860-media-download-redirect.md +++ b/proposals/3860-media-download-redirect.md @@ -9,7 +9,7 @@ tell clients/servers to pull data direct from the source, e.g. a CDN. This MSC proposes that a 307 or 308 redirect code is allowed and followed according to the `Location` header. It is possible some clients would already follow these which needs to be confirmed. Specific -endpoints in question: +endpoints in question ([current spec link for these](https://spec.matrix.org/v1.6/client-server-api/#get_matrixmediav3downloadservernamemediaid)): + `/_matrix/media/v3/download/{serverName}/{mediaId}` + `/_matrix/media/v3/download/{serverName}/{mediaId}/{fileName}` From a7733d91770d517ee40cc640259cc970ef02d8e9 Mon Sep 17 00:00:00 2001 From: Nick Mills-Barrett Date: Tue, 2 May 2023 14:03:04 +0100 Subject: [PATCH 14/16] Add note about additional benefit of not proxying untrusted homeserver media --- proposals/3860-media-download-redirect.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/proposals/3860-media-download-redirect.md b/proposals/3860-media-download-redirect.md index 51c3063d28..fba4f431fe 100644 --- a/proposals/3860-media-download-redirect.md +++ b/proposals/3860-media-download-redirect.md @@ -5,6 +5,11 @@ means the media server instance must stream the data from wherever it is stored, local to itself. Allowing redirects on these endpoints would make it possible for the media repo to tell clients/servers to pull data direct from the source, e.g. a CDN. +Additionally redirects could be used to avoid downloading media from untrusted homeservers. Currently +users can make their homeserver download, cache and proxy any matrix mid that I want, including +bad/illegal content. Allowing for a 307 redirect would permit cautious server operators to not +store and provide any media that floats in the matrixverse, but just refer to the "original" media. + ## Proposal This MSC proposes that a 307 or 308 redirect code is allowed and followed according to the `Location` From 9312cb3b9d568ded7ca1ee932bdf92d2fa1bdb57 Mon Sep 17 00:00:00 2001 From: Nick Mills-Barrett Date: Tue, 2 May 2023 14:03:16 +0100 Subject: [PATCH 15/16] Expand issues section to cover possible homeserver abuse --- proposals/3860-media-download-redirect.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/proposals/3860-media-download-redirect.md b/proposals/3860-media-download-redirect.md index fba4f431fe..ae642538f2 100644 --- a/proposals/3860-media-download-redirect.md +++ b/proposals/3860-media-download-redirect.md @@ -33,7 +33,14 @@ providing `allow_redirect=false`) the server must continue to serve media direct ## Potential Issues -None, as opt-in functionality this change is 100% backwards compatible. +None for clients, as opt-in functionality this change is 100% backwards compatible. + +With redirects it becomes easier to force another homeserver to be your CDN, which isn't great +(clients could already do that, but not without recompiling). + +Redirects also potentially allow changing of media underneath the users as different redirects could +be returned over time. It is worth noting that a badly behaving media server can already just return +different content as well. ## Alternatives From 2b93616f5260014f256db96692fd0c46a31b2548 Mon Sep 17 00:00:00 2001 From: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> Date: Thu, 4 May 2023 19:49:42 +0100 Subject: [PATCH 16/16] fix a typo --- proposals/3860-media-download-redirect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/3860-media-download-redirect.md b/proposals/3860-media-download-redirect.md index ae642538f2..90092aee9f 100644 --- a/proposals/3860-media-download-redirect.md +++ b/proposals/3860-media-download-redirect.md @@ -28,7 +28,7 @@ client respects redirect responses it can make requests like so to the media end + `/_matrix/media/v3/download/{serverName}/{mediaId}/{fileName}?allow_redirect=true` + `/_matrix/media/v3/thumbnail/{serverName}/{mediaId}?allow_redirect=true` -In the case where a client wishes not to redirect (either implictly with no parameter or explicitly +In the case where a client wishes not to redirect (either implicitly with no parameter or explicitly providing `allow_redirect=false`) the server must continue to serve media directly with no redirect. ## Potential Issues