From f496046703f72fcaea50b539a7a34ff3c2ffedc4 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 18 Oct 2019 11:11:55 +0100 Subject: [PATCH 1/9] What if we allowed people to ship sooner? --- proposals/0000-when-to-ship.md | 66 ++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) create mode 100644 proposals/0000-when-to-ship.md diff --git a/proposals/0000-when-to-ship.md b/proposals/0000-when-to-ship.md new file mode 100644 index 00000000000..31bda1ad3b4 --- /dev/null +++ b/proposals/0000-when-to-ship.md @@ -0,0 +1,66 @@ +# MSC0000: Facilitating early releases of software dependent on spec + +*Note*: This is a process change MSC, not a change to the spec itself. + +There's currently an unanswered question by the spec process: when is it +safe to start using stable endpoints or to present a feature as "stable"? +Historically this question would receive very different answers depending +on who you asked, so in an effort to come up with a concise answer the +following process change is proposed. + +## Proposal + +The new process, from start to finish, is proposed as: + +1. Have an idea for a feature. +2. Optionally: implement the feature using unstable endpoints, vendor prefixes, + and unstable feature flags as appropriate. + * When using unstable endpoints, they MUST include a vendor prefix. For + example: `/_matrix/client/unstable/com.example/login`. Vendor prefixes + throughout this proposal always use the Java package naming convention. + * Unstable endpoints **do not** inherit from stable (`/r0`) APIs. + * When using an unstable feature flag, they MUST include a vendor prefix. + This kind of flag shows up in `/versions`. Eg: `com.example.new_login`. + * You can ship the feature at *any* time, so long as you are able to accept + the technical debt. The implementation MUST support the flag disappearing + and be generally safe for users. Note that implementations early in the MSC + review process may also be required to provide backwards compatibility with + earlier editions of the proposal. + * If you don't/can't support the technical debt, do not implement the feature + and wait for Step 7. + * If at any point the idea changes, the feature flag should also change so + that implementations can adapt as needed. +3. In parallel, or ahead of implementation, open an MSC and solicit review. +4. Before an FCP (Final Comment Period) can be called, the Spec Core Team will + require that evidence to prove the MSC works be presented. A typical example + of this is an implementation of the MSC (ie: Step 2). +5. FCP is gone through, and assuming nothing is flagged the MSC lands. +6. A spec PR is written to incorporate the changes into Matrix. +7. A spec release happens. +8. Implementations switch to using stable prefixes (ie: `/r0`) if the server + supports the specification version released. If the server doesn't advertise + the specification version, but does have the feature flag, unstable prefixes + should still be used. +9. A transition period of about 2 months since the spec release happens before + implementations start to encourage other implementations to switch to stable + endpoints. For example, the Synapse team should start asking the Riot team to + support the stable endpoints (as per Step 8) 2 months after the spec release. + +It's worth repeating that this process generally only applies if the implementation +wants to ship the feature ahead of the spec being available. By doing so, it takes +on the risk that the spec/MSC may change and it must adapt. If the implementation +is unable to take on that risk, or simply doesn't mind waiting, it should go through +the normal spec process. + +To help MSCs get incorporated by implementations as stable features, the spec core +team plans to release the specification more often. How often is undefined and is +largely a case-by-case basis. + +To reiterate: + +* Implementations MUST NOT use stable endpoints before the MSC is in the spec. This + includes NOT using stable endpoints post-FCP. +* Implementations CAN ship features that are exposed by default to users before an + MSC has been merged to the spec, provided they follow the process above. +* Implementations SHOULD be wary of the technical debt they are incuring by moving + faster than the spec. From 330d94acd41d22f95b8105b6cb5016503fd41a7b Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 18 Oct 2019 11:18:08 +0100 Subject: [PATCH 2/9] Assign number --- proposals/{0000-when-to-ship.md => 2324-when-to-ship.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename proposals/{0000-when-to-ship.md => 2324-when-to-ship.md} (98%) diff --git a/proposals/0000-when-to-ship.md b/proposals/2324-when-to-ship.md similarity index 98% rename from proposals/0000-when-to-ship.md rename to proposals/2324-when-to-ship.md index 31bda1ad3b4..5e03f2dd683 100644 --- a/proposals/0000-when-to-ship.md +++ b/proposals/2324-when-to-ship.md @@ -1,4 +1,4 @@ -# MSC0000: Facilitating early releases of software dependent on spec +# MSC2324: Facilitating early releases of software dependent on spec *Note*: This is a process change MSC, not a change to the spec itself. From bc44d31fda8a5c602a6795b81c15526cc22f502a Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 18 Oct 2019 11:31:51 +0100 Subject: [PATCH 3/9] Clarifications for developers --- proposals/2324-when-to-ship.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/proposals/2324-when-to-ship.md b/proposals/2324-when-to-ship.md index 5e03f2dd683..8135888403c 100644 --- a/proposals/2324-when-to-ship.md +++ b/proposals/2324-when-to-ship.md @@ -64,3 +64,11 @@ To reiterate: MSC has been merged to the spec, provided they follow the process above. * Implementations SHOULD be wary of the technical debt they are incuring by moving faster than the spec. + +To clarify: + +* The vendor prefix is chosen by the developer of the feature, using the Java package + naming convention. For example, `org.matrix` is the foundation's vendor prefix. +* The vendor prefixes, unstable feature flags, and unstable endpoints should be included + in the MSC so other developers can benefit. The MSC MUST still say what the stable + endpoints are to look like. From 0052a2b5134494c2a020f5c9915244299f4750d8 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 23 Oct 2019 19:32:01 -0600 Subject: [PATCH 4/9] Apply suggestions from code review Co-Authored-By: Matthew Hodgson --- proposals/2324-when-to-ship.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/proposals/2324-when-to-ship.md b/proposals/2324-when-to-ship.md index 8135888403c..eb4f1b835ef 100644 --- a/proposals/2324-when-to-ship.md +++ b/proposals/2324-when-to-ship.md @@ -22,35 +22,35 @@ The new process, from start to finish, is proposed as: * When using an unstable feature flag, they MUST include a vendor prefix. This kind of flag shows up in `/versions`. Eg: `com.example.new_login`. * You can ship the feature at *any* time, so long as you are able to accept - the technical debt. The implementation MUST support the flag disappearing + the technical debt that results from needing to provide adequate backwards and forwards compatibility for the vendor prefixed implementation. The implementation MUST support the flag disappearing and be generally safe for users. Note that implementations early in the MSC review process may also be required to provide backwards compatibility with earlier editions of the proposal. - * If you don't/can't support the technical debt, do not implement the feature + * If you don't want to support the technical debt (or if it's impossible to provide adequate backwards/forwards compatibility - e.g. a user authentication change which can't be safely rolled back), do not implement the feature and wait for Step 7. * If at any point the idea changes, the feature flag should also change so that implementations can adapt as needed. 3. In parallel, or ahead of implementation, open an MSC and solicit review. -4. Before an FCP (Final Comment Period) can be called, the Spec Core Team will +4. Before a FCP (Final Comment Period) can be called, the Spec Core Team will require that evidence to prove the MSC works be presented. A typical example of this is an implementation of the MSC (ie: Step 2). 5. FCP is gone through, and assuming nothing is flagged the MSC lands. 6. A spec PR is written to incorporate the changes into Matrix. 7. A spec release happens. -8. Implementations switch to using stable prefixes (ie: `/r0`) if the server +8. Implementations switch to using stable prefixes (e.g.: `/r0`) if the server supports the specification version released. If the server doesn't advertise the specification version, but does have the feature flag, unstable prefixes should still be used. -9. A transition period of about 2 months since the spec release happens before - implementations start to encourage other implementations to switch to stable +9. A transition period of about 2 months starts immediately after the spec release, before + implementations start to loudly encourage other implementations to switch to stable endpoints. For example, the Synapse team should start asking the Riot team to - support the stable endpoints (as per Step 8) 2 months after the spec release. + support the stable endpoints (as per Step 8) 2 months after the spec release if they haven't already. It's worth repeating that this process generally only applies if the implementation wants to ship the feature ahead of the spec being available. By doing so, it takes on the risk that the spec/MSC may change and it must adapt. If the implementation is unable to take on that risk, or simply doesn't mind waiting, it should go through -the normal spec process. +the spec process without shipping an unstable implementation. To help MSCs get incorporated by implementations as stable features, the spec core team plans to release the specification more often. How often is undefined and is @@ -62,7 +62,7 @@ To reiterate: includes NOT using stable endpoints post-FCP. * Implementations CAN ship features that are exposed by default to users before an MSC has been merged to the spec, provided they follow the process above. -* Implementations SHOULD be wary of the technical debt they are incuring by moving +* Implementations SHOULD be wary of the technical debt they are incurring by moving faster than the spec. To clarify: From 3b10cc8d4d4641e3669eecbd563a441646b3ced8 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 23 Oct 2019 19:33:40 -0600 Subject: [PATCH 5/9] Fix line length --- proposals/2324-when-to-ship.md | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/proposals/2324-when-to-ship.md b/proposals/2324-when-to-ship.md index eb4f1b835ef..044edba6008 100644 --- a/proposals/2324-when-to-ship.md +++ b/proposals/2324-when-to-ship.md @@ -22,12 +22,16 @@ The new process, from start to finish, is proposed as: * When using an unstable feature flag, they MUST include a vendor prefix. This kind of flag shows up in `/versions`. Eg: `com.example.new_login`. * You can ship the feature at *any* time, so long as you are able to accept - the technical debt that results from needing to provide adequate backwards and forwards compatibility for the vendor prefixed implementation. The implementation MUST support the flag disappearing - and be generally safe for users. Note that implementations early in the MSC - review process may also be required to provide backwards compatibility with - earlier editions of the proposal. - * If you don't want to support the technical debt (or if it's impossible to provide adequate backwards/forwards compatibility - e.g. a user authentication change which can't be safely rolled back), do not implement the feature - and wait for Step 7. + the technical debt that results from needing to provide adequate backwards + and forwards compatibility for the vendor prefixed implementation. The + implementation MUST support the flag disappearing and be generally safe for + users. Note that implementations early in the MSC review process may also be + required to provide backwards compatibility with earlier editions of the + proposal. + * If you don't want to support the technical debt (or if it's impossible to + provide adequate backwards/forwards compatibility - e.g. a user authentication + change which can't be safely rolled back), do not implement the feature and + wait for Step 7. * If at any point the idea changes, the feature flag should also change so that implementations can adapt as needed. 3. In parallel, or ahead of implementation, open an MSC and solicit review. @@ -44,7 +48,8 @@ The new process, from start to finish, is proposed as: 9. A transition period of about 2 months starts immediately after the spec release, before implementations start to loudly encourage other implementations to switch to stable endpoints. For example, the Synapse team should start asking the Riot team to - support the stable endpoints (as per Step 8) 2 months after the spec release if they haven't already. + support the stable endpoints (as per Step 8) 2 months after the spec release if they + haven't already. It's worth repeating that this process generally only applies if the implementation wants to ship the feature ahead of the spec being available. By doing so, it takes From 55d2cc418fe447ac5c34958d3c1aa761cb2ea37c Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 23 Oct 2019 19:35:26 -0600 Subject: [PATCH 6/9] Expand on unstable endpoint inheritance --- proposals/2324-when-to-ship.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/proposals/2324-when-to-ship.md b/proposals/2324-when-to-ship.md index 044edba6008..a8ccda30d37 100644 --- a/proposals/2324-when-to-ship.md +++ b/proposals/2324-when-to-ship.md @@ -18,7 +18,11 @@ The new process, from start to finish, is proposed as: * When using unstable endpoints, they MUST include a vendor prefix. For example: `/_matrix/client/unstable/com.example/login`. Vendor prefixes throughout this proposal always use the Java package naming convention. - * Unstable endpoints **do not** inherit from stable (`/r0`) APIs. + * Unstable endpoints **do not** inherit from stable (`/r0`) APIs. Previously, + one could access the entirety of the Matrix API through `/unstable` however + this is generally considered a bad practice. Therefore an implementation + can no longer assume that because its feature-specific endpoint exists that + any other endpoint will exist in the same unstable namespace. * When using an unstable feature flag, they MUST include a vendor prefix. This kind of flag shows up in `/versions`. Eg: `com.example.new_login`. * You can ship the feature at *any* time, so long as you are able to accept From 46c7d15d7b70978ce17f6bc1b33ab3fd4297a70d Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 23 Oct 2019 19:38:03 -0600 Subject: [PATCH 7/9] Clarify that feature flags are not required for everything --- proposals/2324-when-to-ship.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/proposals/2324-when-to-ship.md b/proposals/2324-when-to-ship.md index a8ccda30d37..2e05913137b 100644 --- a/proposals/2324-when-to-ship.md +++ b/proposals/2324-when-to-ship.md @@ -24,7 +24,9 @@ The new process, from start to finish, is proposed as: can no longer assume that because its feature-specific endpoint exists that any other endpoint will exist in the same unstable namespace. * When using an unstable feature flag, they MUST include a vendor prefix. - This kind of flag shows up in `/versions`. Eg: `com.example.new_login`. + This kind of flag shows up in `/versions` as, for example, `com.example.new_login`. + This flag should only be created if the client needs to be sure that the + server supports the feature. * You can ship the feature at *any* time, so long as you are able to accept the technical debt that results from needing to provide adequate backwards and forwards compatibility for the vendor prefixed implementation. The From cd80cac7af41aa093185dbed07d92ba0178ba38b Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 15 Nov 2019 15:13:36 -0700 Subject: [PATCH 8/9] Add a bunch of procedural words to the proposal Mostly stuff that has been discussed in comments, a bit of common sense, and a hint of clarity. --- proposals/2324-when-to-ship.md | 47 +++++++++++++++++++++++++++++++--- 1 file changed, 43 insertions(+), 4 deletions(-) diff --git a/proposals/2324-when-to-ship.md b/proposals/2324-when-to-ship.md index 2e05913137b..e302de8756c 100644 --- a/proposals/2324-when-to-ship.md +++ b/proposals/2324-when-to-ship.md @@ -23,10 +23,10 @@ The new process, from start to finish, is proposed as: this is generally considered a bad practice. Therefore an implementation can no longer assume that because its feature-specific endpoint exists that any other endpoint will exist in the same unstable namespace. - * When using an unstable feature flag, they MUST include a vendor prefix. - This kind of flag shows up in `/versions` as, for example, `com.example.new_login`. - This flag should only be created if the client needs to be sure that the - server supports the feature. + * If the client needs to be sure the server supports the feature, an unstable + feature flag that MUST be vendor prefixed is to be used. This kind of flag + shows up in the `unstable_features` field of `/versions` as, for example, + `com.example.new_login`. * You can ship the feature at *any* time, so long as you are able to accept the technical debt that results from needing to provide adequate backwards and forwards compatibility for the vendor prefixed implementation. The @@ -83,3 +83,42 @@ To clarify: * The vendor prefixes, unstable feature flags, and unstable endpoints should be included in the MSC so other developers can benefit. The MSC MUST still say what the stable endpoints are to look like. + +### Specific examples outside of the client-server API + +There are some instances where a change might be made outside of the client-server API, +which is where much of this proposal is targetted. The general spirit of the process +should be followed where possible, if implementations decide to work ahead of spec releases. + +#### Room versions + +When a new room version is needed, implementations MUST use vendor-prefixed versions +before using the namespace reserved for Matrix (see https://matrix.org/docs/spec/#room-versions). +A room version is considered released once it is listed as an "available room version" in +the spec. Often a new room version is accompanied with a server-server API release, but +doesn't have to be. + +#### Server-server / Identity / Push / Appservice API + +These APIs don't yet have a `/versions` endpoint or similar. Typically behaviour changes in +these APIs are introduced with backwards compatibility in mind (try X and if that fails fall +back to Y) and therefore don't always need a flag to indicate support. If a flag were to +be required, an `unstable_features` or similar array would need to be exposed somewhere. + +#### Changes to request/response parameters + +Parameters being added to request/response bodies and query strings MUST be vendor-prefixed +per the proposed process. For example, a new JSON field might be `{"org.matrix.example": true}` +with the proposal being for `example` being added. A query string parameter would be prefixed +the same way: `?org.matrix.example=true`. + +If the MSC is simply adding fields to already-versioned endpoints, it does not need to put +the whole endpoint into the `/unstable` namespace provided the new parameters are prefixed +appropriately. + +#### .well-known and other APIs that can't be versioned + +Best effort is appreciated. Typically these endpoints will be receiving minor behavioural +changes or new fields. New fields should be appropriately prefixed, and behaviour changes +should be rolled out cautiously by implementations (waiting until after FCP has concluded +is probably best to ensure there's no major problems with the new behaviour). From c98922b259cbe2066adf1b199d8c667a11f3d3df Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 20 Nov 2019 18:08:41 -0700 Subject: [PATCH 9/9] Update proposals/2324-when-to-ship.md Co-Authored-By: Matthew Hodgson --- proposals/2324-when-to-ship.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2324-when-to-ship.md b/proposals/2324-when-to-ship.md index e302de8756c..1335bfab989 100644 --- a/proposals/2324-when-to-ship.md +++ b/proposals/2324-when-to-ship.md @@ -43,7 +43,7 @@ The new process, from start to finish, is proposed as: 3. In parallel, or ahead of implementation, open an MSC and solicit review. 4. Before a FCP (Final Comment Period) can be called, the Spec Core Team will require that evidence to prove the MSC works be presented. A typical example - of this is an implementation of the MSC (ie: Step 2). + of this is an implementation of the MSC (which does not necessarily need to have been shipped anywhere). 5. FCP is gone through, and assuming nothing is flagged the MSC lands. 6. A spec PR is written to incorporate the changes into Matrix. 7. A spec release happens.