From 9b04f3b126f38148b2d59e41efab562ca98b442e Mon Sep 17 00:00:00 2001 From: Joao Grassi Date: Tue, 26 Jul 2022 13:12:25 +0200 Subject: [PATCH 1/8] Add partial success response guidance --- CHANGELOG.md | 3 + specification/protocol/otlp.md | 139 +++++++++++++++++++++++++++------ 2 files changed, 119 insertions(+), 23 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3b49b8972d0..7898a476668 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -64,6 +64,9 @@ release. ### OpenTelemetry Protocol +- Add support for partial success in an OTLP export response + ([#???](https://github.com/open-telemetry/opentelemetry-specification/pull/???)). + ### SDK Configuration - Mark `OTEL_METRIC_EXPORT_INTERVAL`, `OTEL_METRIC_EXPORT_TIMEOUT` diff --git a/specification/protocol/otlp.md b/specification/protocol/otlp.md index 42766e721c6..77507e8bacf 100644 --- a/specification/protocol/otlp.md +++ b/specification/protocol/otlp.md @@ -16,14 +16,18 @@ nodes such as collectors and telemetry backends. * [OTLP/gRPC](#otlpgrpc) + [OTLP/gRPC Concurrent Requests](#otlpgrpc-concurrent-requests) + [OTLP/gRPC Response](#otlpgrpc-response) + - [Success](#success) + - [Partial Success](#partial-success) + - [Failures](#failures) + [OTLP/gRPC Throttling](#otlpgrpc-throttling) + [OTLP/gRPC Service and Protobuf Definitions](#otlpgrpc-service-and-protobuf-definitions) + [OTLP/gRPC Default Port](#otlpgrpc-default-port) * [OTLP/HTTP](#otlphttp) + [OTLP/HTTP Request](#otlphttp-request) + [OTLP/HTTP Response](#otlphttp-response) - - [Success](#success) - - [Failures](#failures) + - [Success](#success-1) + - [Partial Success](#partial-success-1) + - [Failures](#failures-1) - [Bad Data](#bad-data) - [OTLP/HTTP Throttling](#otlphttp-throttling) - [All Other Responses](#all-other-responses) @@ -35,7 +39,7 @@ nodes such as collectors and telemetry backends. - [Known Limitations](#known-limitations) * [Request Acknowledgements](#request-acknowledgements) + [Duplicate Data](#duplicate-data) - * [Partial Success](#partial-success) + + [Partial Success Retry](#partial-success-retry) - [Future Versions and Interoperability](#future-versions-and-interoperability) - [Glossary](#glossary) - [References](#references) @@ -145,16 +149,59 @@ was not delivered. #### OTLP/gRPC Response -The server may respond with either a success or an error to the requests. +The response MUST be the appropriate serialized Protobuf message (see below for +the specific message to use in the [Success](#success), +[Partial Success](#partial-success) and [Failure](#failures) cases). + +##### Success + +The success response indicates telemetry data is successfully accepted by the +server. -The success response indicates telemetry data is successfully processed by the -server. If the server receives an empty request (a request that does not carry +If the server receives an empty request (a request that does not carry any telemetry data) the server SHOULD respond with success. -Success response is returned via -[Export*ServiceResponse](https://github.com/open-telemetry/opentelemetry-proto) -message (`ExportTraceServiceResponse` for traces, `ExportMetricsServiceResponse` -for metrics, `ExportLogsServiceResponse` for logs). +On success, the server response MUST be a Protobuf-encoded +[ExportServiceResponse](https://github.com/open-telemetry/opentelemetry-proto) +message (`ExportTraceServiceResponse` for traces, +`ExportMetricsServiceResponse` for metrics and +`ExportLogsServiceResponse` for logs). + +The server MUST leave the `partial_success` field unset +in case of a successful response. + +##### Partial Success + +If the request is only partially accepted +(i.e. when the server accepts only parts of the data and rejects the rest), the +server response MUST be a Protobuf-encoded +[ExportServiceResponse](https://github.com/open-telemetry/opentelemetry-proto) +message (`ExportTraceServiceResponse` for traces, +`ExportMetricsServiceResponse` for metrics and +`ExportLogsServiceResponse` for logs). + +Additionally, the server MUST initialize the `partial_success` field +(`ExportTracePartialSuccess` message for traces, +`ExportMetricsPartialSuccess` message for metrics and +`ExportLogsPartialSuccess` message for logs), and it MUST set the respective +`rejected_spans`, `rejected_data_points` or `rejected_log_records` field with +the number of spans/data points/log records it rejected. + +A `partial_success` with the `rejected_` field holding a `0` value +is an invalid result. In such cases, senders MUST ignore the `partial_success` +field and handle such response from a server in the same way as defined in the +[Success](#success) section. + +The server SHOULD populate the `error_message` field with a human-readable +error message in English. The message should explain why the +server rejected parts of the data, and might offer guidance on how users +can address the issues. +The protocol does not attempt to define the structure of the error message. + +The client MUST NOT retry the request when it receives a partial success +response where the `partial_success` is populated. + +##### Failures When an error is returned by the server it falls into 2 broad categories: retryable and not-retryable: @@ -382,8 +429,9 @@ numbers or strings are accepted when decoding. #### OTLP/HTTP Response -Response body MUST be the appropriate serialized Protobuf message (see below for -the specific message to use in the Success and Failure cases). +The response body MUST be the appropriate serialized Protobuf message (see below for +the specific message to use in the [Success](#success-1), +[Partial Success](#partial-success-1) and [Failure](#failures-1) cases). The server MUST set "Content-Type: application/x-protobuf" header if the response body is binary-encoded Protobuf payload. The server MUST set @@ -397,13 +445,51 @@ header. ##### Success -On success the server MUST respond with `HTTP 200 OK`. Response body MUST be -Protobuf-encoded `ExportTraceServiceResponse` message for traces, -`ExportMetricsServiceResponse` message for metrics and -`ExportLogsServiceResponse` message for logs. +The success response indicates telemetry data is successfully accepted by the +server. + +If the server receives an empty request (a request that does not carry +any telemetry data) the server SHOULD respond with success. -The server SHOULD respond with success no sooner than after successfully -decoding and validating the request. +On success, the server MUST respond with `HTTP 200 OK`. The response body MUST be +a Protobuf-encoded [ExportServiceResponse](https://github.com/open-telemetry/opentelemetry-proto) +message (`ExportTraceServiceResponse` for traces, +`ExportMetricsServiceResponse` for metrics and +`ExportLogsServiceResponse` for logs). + +The server MUST leave the `partial_success` field unset +in case of a successful response. + +##### Partial Success + +If the request is only partially accepted +(i.e. when the server accepts only parts of the data and rejects the rest), the +server MUST respond with `HTTP 200 OK`. The response body MUST be +a Protobuf-encoded [ExportServiceResponse](https://github.com/open-telemetry/opentelemetry-proto) +message (`ExportTraceServiceResponse` for traces, +`ExportMetricsServiceResponse` for metrics and +`ExportLogsServiceResponse` for logs). + +Additionally, the server MUST initialize the `partial_success` field +(`ExportTracePartialSuccess` message for traces, +`ExportMetricsPartialSuccess` message for metrics and +`ExportLogsPartialSuccess` message for logs), and it MUST set the respective +`rejected_spans`, `rejected_data_points` or `rejected_log_records` field with +the number of spans/data points/log records it rejected. + +A `partial_success` with the `rejected_` field holding a `0` value +is an invalid result. In such cases, senders MUST ignore the `partial_success` +field and handle such response from a server in the same way as defined in the +[Success](#success-1) section. + +The server SHOULD populate the `error_message` field with a human-readable +error message in English. The message should explain why the +server rejected parts of the data, and might offer guidance on how users +can address the issues. +The protocol does not attempt to define the structure of the error message. + +The client MUST NOT retry the request when it receives a partial success +response where the `partial_success` is populated. ##### Failures @@ -520,12 +606,19 @@ received yet. The client will typically choose to re-send such data to guarantee delivery, which may result in duplicate data on the server side. This is a deliberate choice and is considered to be the right tradeoff for telemetry data. -### Partial Success +#### Partial Success Retry + +The partial success defined by the protocol is neither designed nor intended +to be used as a mechanism for clients to automatically retry an export request. + +Servers should return a partial success response when they fully understand that +resending the same bundle of telemetry would lead to the same error again, +thus preventing retry loops. -The protocol does not attempt to communicate partial reception success from the -server to the client (i.e. when part of the data can be received by the server -and part of it cannot). Attempting to do so would complicate the protocol and -implementations significantly and is left out as a possible future area of work. +The protocol does not attempt to define how clients should automatically retry +a partially successful request. +Attempting to do so would complicate the protocol and implementations +significantly and is left out as a possible future area of work. ## Future Versions and Interoperability From aad267cf87a915ca5b7595819c415b18748aef62 Mon Sep 17 00:00:00 2001 From: Joao Grassi Date: Tue, 26 Jul 2022 13:19:34 +0200 Subject: [PATCH 2/8] Update changelog with PR numger --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7898a476668..5dc142537e6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -65,7 +65,7 @@ release. ### OpenTelemetry Protocol - Add support for partial success in an OTLP export response - ([#???](https://github.com/open-telemetry/opentelemetry-specification/pull/???)). + ([#2696](https://github.com/open-telemetry/opentelemetry-specification/pull/2696)). ### SDK Configuration From 55c1b61887f7f9649d1e179c4fe579c0569d2030 Mon Sep 17 00:00:00 2001 From: Joao Grassi Date: Fri, 29 Jul 2022 11:08:22 +0200 Subject: [PATCH 3/8] PR suggestions --- specification/protocol/otlp.md | 52 +++++++++++----------------------- 1 file changed, 17 insertions(+), 35 deletions(-) diff --git a/specification/protocol/otlp.md b/specification/protocol/otlp.md index 77507e8bacf..f0bd5c0cde9 100644 --- a/specification/protocol/otlp.md +++ b/specification/protocol/otlp.md @@ -16,7 +16,7 @@ nodes such as collectors and telemetry backends. * [OTLP/gRPC](#otlpgrpc) + [OTLP/gRPC Concurrent Requests](#otlpgrpc-concurrent-requests) + [OTLP/gRPC Response](#otlpgrpc-response) - - [Success](#success) + - [Full Success](#full-success) - [Partial Success](#partial-success) - [Failures](#failures) + [OTLP/gRPC Throttling](#otlpgrpc-throttling) @@ -25,7 +25,7 @@ nodes such as collectors and telemetry backends. * [OTLP/HTTP](#otlphttp) + [OTLP/HTTP Request](#otlphttp-request) + [OTLP/HTTP Response](#otlphttp-response) - - [Success](#success-1) + - [Full Success](#full-success-1) - [Partial Success](#partial-success-1) - [Failures](#failures-1) - [Bad Data](#bad-data) @@ -39,7 +39,6 @@ nodes such as collectors and telemetry backends. - [Known Limitations](#known-limitations) * [Request Acknowledgements](#request-acknowledgements) + [Duplicate Data](#duplicate-data) - + [Partial Success Retry](#partial-success-retry) - [Future Versions and Interoperability](#future-versions-and-interoperability) - [Glossary](#glossary) - [References](#references) @@ -149,11 +148,11 @@ was not delivered. #### OTLP/gRPC Response -The response MUST be the appropriate serialized Protobuf message (see below for -the specific message to use in the [Success](#success), +The response MUST be the appropriate serialized message (see below for +the specific message to use in the [Full Success](#full-success), [Partial Success](#partial-success) and [Failure](#failures) cases). -##### Success +##### Full Success The success response indicates telemetry data is successfully accepted by the server. @@ -161,8 +160,8 @@ server. If the server receives an empty request (a request that does not carry any telemetry data) the server SHOULD respond with success. -On success, the server response MUST be a Protobuf-encoded -[ExportServiceResponse](https://github.com/open-telemetry/opentelemetry-proto) +On success, the server response MUST be a +[ExportServiceResponse](https://github.com/open-telemetry/opentelemetry-proto/tree/main/opentelemetry/proto/collector) message (`ExportTraceServiceResponse` for traces, `ExportMetricsServiceResponse` for metrics and `ExportLogsServiceResponse` for logs). @@ -174,11 +173,9 @@ in case of a successful response. If the request is only partially accepted (i.e. when the server accepts only parts of the data and rejects the rest), the -server response MUST be a Protobuf-encoded -[ExportServiceResponse](https://github.com/open-telemetry/opentelemetry-proto) -message (`ExportTraceServiceResponse` for traces, -`ExportMetricsServiceResponse` for metrics and -`ExportLogsServiceResponse` for logs). +server response MUST be the same +[ExportServiceResponse](https://github.com/open-telemetry/opentelemetry-proto/tree/main/opentelemetry/proto/collector) +message as in the [Full Success](#full-success) case. Additionally, the server MUST initialize the `partial_success` field (`ExportTracePartialSuccess` message for traces, @@ -430,7 +427,7 @@ numbers or strings are accepted when decoding. #### OTLP/HTTP Response The response body MUST be the appropriate serialized Protobuf message (see below for -the specific message to use in the [Success](#success-1), +the specific message to use in the [Full Success](#full-success-1), [Partial Success](#partial-success-1) and [Failure](#failures-1) cases). The server MUST set "Content-Type: application/x-protobuf" header if the @@ -443,7 +440,7 @@ If the request header "Accept-Encoding: gzip" is present in the request the server MAY gzip-encode the response and set "Content-Encoding: gzip" response header. -##### Success +##### Full Success The success response indicates telemetry data is successfully accepted by the server. @@ -452,7 +449,8 @@ If the server receives an empty request (a request that does not carry any telemetry data) the server SHOULD respond with success. On success, the server MUST respond with `HTTP 200 OK`. The response body MUST be -a Protobuf-encoded [ExportServiceResponse](https://github.com/open-telemetry/opentelemetry-proto) +a Protobuf-encoded +[ExportServiceResponse](https://github.com/open-telemetry/opentelemetry-proto/tree/main/opentelemetry/proto/collector) message (`ExportTraceServiceResponse` for traces, `ExportMetricsServiceResponse` for metrics and `ExportLogsServiceResponse` for logs). @@ -464,11 +462,9 @@ in case of a successful response. If the request is only partially accepted (i.e. when the server accepts only parts of the data and rejects the rest), the -server MUST respond with `HTTP 200 OK`. The response body MUST be -a Protobuf-encoded [ExportServiceResponse](https://github.com/open-telemetry/opentelemetry-proto) -message (`ExportTraceServiceResponse` for traces, -`ExportMetricsServiceResponse` for metrics and -`ExportLogsServiceResponse` for logs). +server MUST respond with `HTTP 200 OK`. The response body MUST be the same +[ExportServiceResponse](https://github.com/open-telemetry/opentelemetry-proto/tree/main/opentelemetry/proto/collector) +message as in the [Full Success](#full-success-1) case. Additionally, the server MUST initialize the `partial_success` field (`ExportTracePartialSuccess` message for traces, @@ -606,20 +602,6 @@ received yet. The client will typically choose to re-send such data to guarantee delivery, which may result in duplicate data on the server side. This is a deliberate choice and is considered to be the right tradeoff for telemetry data. -#### Partial Success Retry - -The partial success defined by the protocol is neither designed nor intended -to be used as a mechanism for clients to automatically retry an export request. - -Servers should return a partial success response when they fully understand that -resending the same bundle of telemetry would lead to the same error again, -thus preventing retry loops. - -The protocol does not attempt to define how clients should automatically retry -a partially successful request. -Attempting to do so would complicate the protocol and implementations -significantly and is left out as a possible future area of work. - ## Future Versions and Interoperability OTLP will evolve and change over time. Future versions of OTLP must be designed From da2c08c7dcbf1a5fafe12fcbb9f81d8a6377422f Mon Sep 17 00:00:00 2001 From: Joao Grassi Date: Fri, 29 Jul 2022 11:23:38 +0200 Subject: [PATCH 4/8] Fix md links --- specification/protocol/otlp.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specification/protocol/otlp.md b/specification/protocol/otlp.md index f0bd5c0cde9..ab8a7e0e20a 100644 --- a/specification/protocol/otlp.md +++ b/specification/protocol/otlp.md @@ -187,7 +187,7 @@ the number of spans/data points/log records it rejected. A `partial_success` with the `rejected_` field holding a `0` value is an invalid result. In such cases, senders MUST ignore the `partial_success` field and handle such response from a server in the same way as defined in the -[Success](#success) section. +[Success](#full-success) section. The server SHOULD populate the `error_message` field with a human-readable error message in English. The message should explain why the @@ -476,7 +476,7 @@ the number of spans/data points/log records it rejected. A `partial_success` with the `rejected_` field holding a `0` value is an invalid result. In such cases, senders MUST ignore the `partial_success` field and handle such response from a server in the same way as defined in the -[Success](#success-1) section. +[Success](#full-success-1) section. The server SHOULD populate the `error_message` field with a human-readable error message in English. The message should explain why the From fef18366bc9a860037147540cb6263318d22224e Mon Sep 17 00:00:00 2001 From: Joao Grassi Date: Mon, 1 Aug 2022 13:08:14 +0200 Subject: [PATCH 5/8] Allow using error message to convey warnings when rejected is 0 --- specification/protocol/otlp.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/specification/protocol/otlp.md b/specification/protocol/otlp.md index ab8a7e0e20a..264a3bc6540 100644 --- a/specification/protocol/otlp.md +++ b/specification/protocol/otlp.md @@ -184,17 +184,17 @@ Additionally, the server MUST initialize the `partial_success` field `rejected_spans`, `rejected_data_points` or `rejected_log_records` field with the number of spans/data points/log records it rejected. -A `partial_success` with the `rejected_` field holding a `0` value -is an invalid result. In such cases, senders MUST ignore the `partial_success` -field and handle such response from a server in the same way as defined in the -[Success](#full-success) section. - The server SHOULD populate the `error_message` field with a human-readable error message in English. The message should explain why the server rejected parts of the data, and might offer guidance on how users can address the issues. The protocol does not attempt to define the structure of the error message. +A `partial_success` with the `rejected_` field holding a `0` value +indicates that the whole request was accepted, as in the +[Full Success](#full-success) case. In such cases, senders MAY still make use of +the `error_message` field, for example, to convey warnings. + The client MUST NOT retry the request when it receives a partial success response where the `partial_success` is populated. @@ -473,17 +473,17 @@ Additionally, the server MUST initialize the `partial_success` field `rejected_spans`, `rejected_data_points` or `rejected_log_records` field with the number of spans/data points/log records it rejected. -A `partial_success` with the `rejected_` field holding a `0` value -is an invalid result. In such cases, senders MUST ignore the `partial_success` -field and handle such response from a server in the same way as defined in the -[Success](#full-success-1) section. - The server SHOULD populate the `error_message` field with a human-readable error message in English. The message should explain why the server rejected parts of the data, and might offer guidance on how users can address the issues. The protocol does not attempt to define the structure of the error message. +A `partial_success` with the `rejected_` field holding a `0` value +indicates that the whole request was accepted, as in the +[Full Success](#full-success-1) case. In such cases, senders MAY still make use of +the `error_message` field, for example, to convey warnings. + The client MUST NOT retry the request when it receives a partial success response where the `partial_success` is populated. From ae09792e2ececc625adba8296ea292b8fcf0b379 Mon Sep 17 00:00:00 2001 From: Joao Grassi Date: Mon, 1 Aug 2022 14:19:55 +0200 Subject: [PATCH 6/8] Sender -> Server in partial success --- specification/protocol/otlp.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/specification/protocol/otlp.md b/specification/protocol/otlp.md index 264a3bc6540..52f6f1c3769 100644 --- a/specification/protocol/otlp.md +++ b/specification/protocol/otlp.md @@ -192,8 +192,9 @@ The protocol does not attempt to define the structure of the error message. A `partial_success` with the `rejected_` field holding a `0` value indicates that the whole request was accepted, as in the -[Full Success](#full-success) case. In such cases, senders MAY still make use of -the `error_message` field, for example, to convey warnings. +[Full Success](#full-success-1) case. In such cases, the server MAY still make +use of the `error_message` field, for example, to convey warnings/suggestions +to clients. The client MUST NOT retry the request when it receives a partial success response where the `partial_success` is populated. @@ -481,8 +482,9 @@ The protocol does not attempt to define the structure of the error message. A `partial_success` with the `rejected_` field holding a `0` value indicates that the whole request was accepted, as in the -[Full Success](#full-success-1) case. In such cases, senders MAY still make use of -the `error_message` field, for example, to convey warnings. +[Full Success](#full-success-1) case. In such cases, the server MAY still make +use of the `error_message` field, for example, to convey warnings/suggestions +to clients. The client MUST NOT retry the request when it receives a partial success response where the `partial_success` is populated. From 37cad7e5992ce6f6fe4b9ad0e5e0876ec70c8ac0 Mon Sep 17 00:00:00 2001 From: Joao Grassi Date: Mon, 1 Aug 2022 14:38:58 +0200 Subject: [PATCH 7/8] Adapt text once again, making it more consistent with the proto --- specification/protocol/otlp.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/specification/protocol/otlp.md b/specification/protocol/otlp.md index 52f6f1c3769..efbbd7354cc 100644 --- a/specification/protocol/otlp.md +++ b/specification/protocol/otlp.md @@ -190,11 +190,10 @@ server rejected parts of the data, and might offer guidance on how users can address the issues. The protocol does not attempt to define the structure of the error message. -A `partial_success` with the `rejected_` field holding a `0` value -indicates that the whole request was accepted, as in the -[Full Success](#full-success-1) case. In such cases, the server MAY still make -use of the `error_message` field, for example, to convey warnings/suggestions -to clients. +Servers MAY also make use of the `partial_success` field to convey +warnings/suggestions to clients when the request was fully accepted. +In such cases, the `rejected_` field MUST have a value of `0` and +the `error_message` field MUST be non-empty. The client MUST NOT retry the request when it receives a partial success response where the `partial_success` is populated. @@ -480,11 +479,10 @@ server rejected parts of the data, and might offer guidance on how users can address the issues. The protocol does not attempt to define the structure of the error message. -A `partial_success` with the `rejected_` field holding a `0` value -indicates that the whole request was accepted, as in the -[Full Success](#full-success-1) case. In such cases, the server MAY still make -use of the `error_message` field, for example, to convey warnings/suggestions -to clients. +Servers MAY also make use of the `partial_success` field to convey +warnings/suggestions to clients when the request was fully accepted. +In such cases, the `rejected_` field MUST have a value of `0` and +the `error_message` field MUST be non-empty. The client MUST NOT retry the request when it receives a partial success response where the `partial_success` is populated. From 21266d0cd68aa00e53e9070df293e05528fc9558 Mon Sep 17 00:00:00 2001 From: Joao Grassi Date: Tue, 2 Aug 2022 17:36:21 +0200 Subject: [PATCH 8/8] PR suggestions --- specification/protocol/otlp.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specification/protocol/otlp.md b/specification/protocol/otlp.md index efbbd7354cc..e8d548a4994 100644 --- a/specification/protocol/otlp.md +++ b/specification/protocol/otlp.md @@ -148,7 +148,7 @@ was not delivered. #### OTLP/gRPC Response -The response MUST be the appropriate serialized message (see below for +The response MUST be the appropriate message (see below for the specific message to use in the [Full Success](#full-success), [Partial Success](#partial-success) and [Failure](#failures) cases). @@ -191,7 +191,7 @@ can address the issues. The protocol does not attempt to define the structure of the error message. Servers MAY also make use of the `partial_success` field to convey -warnings/suggestions to clients when the request was fully accepted. +warnings/suggestions to clients even when the request was fully accepted. In such cases, the `rejected_` field MUST have a value of `0` and the `error_message` field MUST be non-empty. @@ -480,7 +480,7 @@ can address the issues. The protocol does not attempt to define the structure of the error message. Servers MAY also make use of the `partial_success` field to convey -warnings/suggestions to clients when the request was fully accepted. +warnings/suggestions to clients even when the request was fully accepted. In such cases, the `rejected_` field MUST have a value of `0` and the `error_message` field MUST be non-empty.