-
Notifications
You must be signed in to change notification settings - Fork 224
Document OTLP logs and traces endpoints, move page out of TSDS subtree #6219
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
3289404
Document OTLP logs and traces endpoints, move page out of TSDS subtree
felixbarny 799ad0a
Merge branch 'main' into otlp-docs
felixbarny c9c3cf6
Address review comments and CI checks
felixbarny 4033372
Merge branch 'main' into otlp-docs
theletterf 73d9742
Apply suggestions from code review
felixbarny 62fca66
Clarify Elasticsearch OTLP endpoint docs
felixbarny 2c85185
Normalize OTLP endpoint doc line endings
felixbarny bd6213f
Clarify OTLP Collector configuration example
felixbarny 05f885b
Add TSDS-focused OTLP ingest page
felixbarny 3dbdca6
Merge branch 'main' into otlp-docs
felixbarny 290bae7
Tighten TSDS OTLP page wording
felixbarny a676839
Clarify OTLP metrics advantages
felixbarny ed8bd5d
Merge branch 'main' into otlp-docs
felixbarny File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
133 changes: 9 additions & 124 deletions
133
manage-data/data-store/data-streams/tsds-ingest-otlp.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,135 +1,20 @@ | ||
| --- | ||
| navigation_title: "OTLP/HTTP endpoint" | ||
| description: "Ingest OpenTelemetry metrics into time series data streams through the Elasticsearch OTLP/HTTP endpoint." | ||
| applies_to: | ||
| deployment: | ||
| self: | ||
| ece: | ||
| eck: | ||
| self: ga 9.2 | ||
| ece: ga | ||
| eck: ga | ||
| products: | ||
| - id: elasticsearch | ||
| --- | ||
|
|
||
| # OTLP/HTTP endpoint | ||
| # Ingest metrics into a TSDS using the OTLP/HTTP endpoint | ||
|
|
||
| In addition to the ingestion of metrics data through the bulk API, | ||
| {{es}} offers an alternative way to ingest data through the [OpenTelemetry Protocol (OTLP)](https://opentelemetry.io/docs/specs/otlp). | ||
| {{es}} supports metrics ingestion through the [OpenTelemetry Protocol (OTLP)](https://opentelemetry.io/docs/specs/otlp). | ||
|
|
||
| The endpoint is available under `/_otlp/v1/metrics`. | ||
| For OpenTelemetry metrics, prefer the {{es}} OTLP/HTTP endpoint over the Bulk API because it's optimized for OTLP ingest performance. | ||
| It also simplifies setup by automatically creating [TSDS](/manage-data/data-store/data-streams/time-series-data-stream-tsds.md) through built-in index templates, and deriving dimensions and metric mappings from OTLP metadata. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| An {{es}} cluster running {{stack}} 9.2 or later. The OTLP endpoint is available on ECE, ECK, and self-managed deployments that run this stack version. | ||
|
|
||
| ## Overview and deployment options | ||
|
|
||
| :::{important} | ||
| The recommended approach for sending OpenTelemetry Protocol (OTLP) data depends on your deployment: | ||
|
|
||
| - **ECH and {{serverless-short}}:** Use the [{{motlp}}](opentelemetry://reference/motlp.md) directly. | ||
| - **ECE, ECK, and self-managed:** Use the {{es}} OTLP endpoint described on this page, ideally through an OpenTelemetry Collector in [Gateway mode](elastic-agent://reference/edot-collector/config/default-config-standalone.md#gateway-mode). | ||
|
|
||
| For details on the recommended way to set up OpenTelemetry-based data ingestion, refer to the [EDOT reference architecture](opentelemetry://reference/architecture/index.md). | ||
| ::: | ||
|
|
||
| Ingesting metrics data using the OTLP endpoint has the following advantages: | ||
|
|
||
| * Improved ingestion performance, especially if the data contains many resource attributes. | ||
| * Simplified index mapping: | ||
| there's no need to manually create data streams, index templates, or define dimensions and metrics. | ||
| Metrics are dynamically mapped using the metadata included in the OTLP requests. | ||
|
|
||
| :::{note} | ||
| {{es}} only supports [OTLP/HTTP](https://opentelemetry.io/docs/specs/otlp/#otlphttp), | ||
| not [OTLP/gRPC](https://opentelemetry.io/docs/specs/otlp/#otlpgrpc). | ||
| ::: | ||
|
|
||
| Don't send metrics from applications directly to the {{es}} OTLP endpoint, especially if there are many individual applications that periodically send a small amount of metrics. Instead, send data to an OpenTelemetry Collector first. This helps with handling many connections, and with creating bigger batches to improve ingestion performance. | ||
|
|
||
| ## How to send data to the OTLP endpoint | ||
|
|
||
| To send data from an OpenTelemetry Collector to the {{es}} OTLP endpoint, | ||
| use the [`OTLP/HTTP` exporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlphttpexporter). | ||
| This is an example configuration: | ||
|
|
||
| ```yaml | ||
| extensions: | ||
| basicauth/elasticsearch: | ||
| client_auth: | ||
| username: <user> | ||
| password: <password> | ||
| exporters: | ||
| otlphttp/elasticsearch-metrics: | ||
| endpoint: <es_endpoint>/_otlp | ||
| sending_queue: | ||
| enabled: true | ||
| sizer: bytes | ||
| queue_size: 50_000_000 # 50MB uncompressed | ||
| block_on_overflow: true | ||
| batch: | ||
| flush_timeout: 1s | ||
| min_size: 1_000_000 # 1MB uncompressed | ||
| max_size: 4_000_000 # 4MB uncompressed | ||
| auth: | ||
| authenticator: basicauth/elasticsearch | ||
| service: | ||
| extensions: [basicauth/elasticsearch] | ||
| pipelines: | ||
| metrics: | ||
| exporters: [otlphttp/elasticsearch-metrics] | ||
| receivers: ... | ||
| ``` | ||
|
|
||
| The supported options for `compression` are `gzip` (default value of the `OTLP/HTTP` exporter) and `none`. | ||
|
|
||
| % TODO we might actually also support snappy and zstd, test and update accordingly) | ||
|
|
||
| To track metrics in your custom application, | ||
| use the [OpenTelemetry language SDK](https://opentelemetry.io/docs/getting-started/dev/) of your choice. | ||
|
|
||
| :::{note} | ||
| Only `encoding: proto` is supported, which the `OTLP/HTTP` exporter uses by default. | ||
| ::: | ||
|
|
||
| ## Send data to different data streams | ||
|
|
||
| By default, metrics are ingested into the `metrics-generic.otel-default` data stream. You can influence the target data stream by setting specific attributes on your data: | ||
|
|
||
| - `data_stream.dataset` or `data_stream.namespace` in attributes, with the following order of precedence: data point attribute -> scope attribute -> resource attribute | ||
| - Otherwise, if the scope name contains `/receiver/<somereceiver>`, `data_stream.dataset` is set to the receiver name. | ||
| - Otherwise, `data_stream.dataset` falls back to `generic` and `data_stream.namespace` falls back to `default`. | ||
|
|
||
| The target data stream name is constructed as `metrics-${data_stream.dataset}.otel-${data_stream.namespace}`. | ||
|
|
||
| ## Configure histogram handling | ||
| ```{applies_to} | ||
| stack: preview =9.3, ga 9.4+ | ||
| ``` | ||
|
|
||
| You can configure how OTLP histograms are mapped using the `xpack.otel_data.histogram_field_type` cluster setting. Valid values are: | ||
|
|
||
| - `histogram` (default on {applies_to}`stack: preview =9.3`): Map histograms as T-Digests using the `histogram` field type | ||
| - `exponential_histogram` (default on {applies_to}`stack: ga 9.4+` {applies_to}`serverless: ga`): Map histograms as exponential histograms using the `exponential_histogram` field type | ||
|
|
||
| The setting is dynamic and can be updated at runtime: | ||
|
|
||
| ```console | ||
| PUT /_cluster/settings | ||
| { | ||
| "persistent" : { | ||
| "xpack.otel_data.histogram_field_type" : "exponential_histogram" | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| Because both `histogram` and `exponential_histogram` support [coerce](elasticsearch://reference/elasticsearch/mapping-reference/coerce.md), changing this setting dynamically does not risk mapping conflicts or ingestion failures. | ||
|
|
||
| This setting only applies to metrics ingested using the [{{es}} OTLP endpoint](./tsds-ingest-otlp.md). | ||
| Documents ingested with the _bulk API (for example using the {{es}} exporter for the OpenTelemetry Collector) are not affected. | ||
|
|
||
| ## Limitations | ||
|
|
||
| * Only the OTLP metrics endpoint (`/_otlp/v1/metrics`) is supported. | ||
| To ingest logs, traces, and profiles, use a distribution of the OpenTelemetry Collector that includes the [{{es}} exporter](opentelemetry://reference/edot-collector/components/elasticsearchexporter.md), | ||
| such as the [{{edot}} (EDOT) Collector](opentelemetry://reference/edot-collector/index.md). | ||
| * Histograms are only supported in delta temporality. Set the temporality preference to delta in your SDKs, or use the [`cumulativetodelta` processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/cumulativetodeltaprocessor) to avoid cumulative histograms to be dropped. | ||
| * Exemplars are not supported. | ||
| For more details, refer to the [{{es}} OTLP/HTTP endpoint](/manage-data/ingest/otlp-endpoint.md) reference. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,235 @@ | ||
| --- | ||
| navigation_title: "OTLP/HTTP endpoint" | ||
| description: "Send metrics, logs, and traces directly to Elasticsearch through its native OTLP/HTTP endpoints, without running an OpenTelemetry Collector." | ||
| applies_to: | ||
| deployment: | ||
| self: ga 9.2 | ||
| ece: ga | ||
| eck: ga | ||
| products: | ||
| - id: elasticsearch | ||
| --- | ||
|
|
||
| # {{es}} OTLP/HTTP endpoint | ||
|
|
||
| In addition to ingesting data through the Bulk API, {{es}} accepts data through the [OpenTelemetry Protocol (OTLP)](https://opentelemetry.io/docs/specs/otlp). | ||
| The {{es}} OTLP/HTTP endpoint exposes three signal-specific paths: | ||
|
|
||
| | Signal | Path | Availability | | ||
| | --- | --- | --- | | ||
| | Metrics | `/_otlp/v1/metrics` | {applies_to}`stack: ga 9.2+` | | ||
| | Logs | `/_otlp/v1/logs` | {applies_to}`stack: preview 9.5` | | ||
| | Traces | `/_otlp/v1/traces` | {applies_to}`stack: preview 9.5` | | ||
|
|
||
| :::{important} | ||
| {{es}} only supports [OTLP/HTTP](https://opentelemetry.io/docs/specs/otlp/#otlphttp), not [OTLP/gRPC](https://opentelemetry.io/docs/specs/otlp/#otlpgrpc). | ||
| ::: | ||
|
|
||
| ## When to use the {{es}} OTLP endpoint | ||
|
|
||
| For most users, one of the following higher-level ingestion paths is recommended: | ||
|
|
||
| | Deployment | Recommended ingestion path | | ||
| | --- | --- | | ||
| | {{ech}} and {{serverless-short}} | [{{motlp}}](opentelemetry://reference/motlp.md) | | ||
| | {{ece}}, {{eck}}, and self-managed | OpenTelemetry Collector in [Gateway mode](elastic-agent://reference/edot-collector/config/default-config-standalone.md#gateway-mode), using the [{{es}} exporter](opentelemetry://reference/edot-collector/components/elasticsearchexporter.md) | | ||
|
|
||
| Use {{motlp}} if it's available in your deployment, even when an application can target the {{es}} OTLP endpoint directly. | ||
|
|
||
| For an overview of the recommended OpenTelemetry-based ingestion architecture, refer to the [EDOT reference architecture](opentelemetry://reference/architecture/index.md). | ||
|
|
||
| Use the {{es}} OTLP endpoint directly when one of the following applies: | ||
|
|
||
| * You have an application that exports OTLP natively and you want it to send data to {{es}} without running an OpenTelemetry Collector. | ||
| For example, a lightweight development setup (SDK to {{es}}). | ||
| * You operate a self-managed gateway Collector and prefer the `OTLP/HTTP` exporter over the [{{es}} exporter](opentelemetry://reference/edot-collector/components/elasticsearchexporter.md). | ||
|
|
||
| :::{warning} | ||
| Don't send telemetry from many individual applications directly to the {{es}} OTLP endpoint at the same time. | ||
| Send to an OpenTelemetry Collector first so it can absorb connection churn and batch records to improve ingestion performance. | ||
| ::: | ||
|
|
||
| ## Advantages of OTLP ingest over Bulk API | ||
|
|
||
| Compared to the Bulk API, ingesting through OTLP offers: | ||
|
|
||
| * Improved ingestion performance, especially for payloads with many resource attributes. | ||
| * Simplified mapping: data streams, index templates, dimensions, and metrics are derived dynamically from OTLP metadata. | ||
| There's no need to set them up manually. | ||
|
|
||
| ## How to send data to the {{es}} OTLP endpoint | ||
|
|
||
| ### Create an API key | ||
|
|
||
| Authenticate to the {{es}} OTLP endpoint with an API key. | ||
| Refer to the API key documentation for your deployment type for instructions on how to create one: | ||
|
|
||
| * [{{es}} API keys](/deploy-manage/api-keys/elasticsearch-api-keys.md) (self-managed, {{ece}}, {{eck}}) | ||
| * [{{ech}} API keys](/deploy-manage/api-keys/elastic-cloud-api-keys.md) | ||
| * [{{ece}} API keys](/deploy-manage/api-keys/elastic-cloud-enterprise-api-keys.md) | ||
| * [{{serverless-short}} project API keys](/deploy-manage/api-keys/serverless-project-api-keys.md) | ||
|
|
||
| The API key needs `create_doc` and `auto_configure` privileges on the data stream patterns it writes to. | ||
| `create_doc` allows writing documents without overwriting existing ones. | ||
| `auto_configure` allows the endpoint to create the target data streams on first write. | ||
|
|
||
| The minimum index patterns depend on which signals you ingest: | ||
|
|
||
| | Signals ingested | Required `names` patterns | | ||
| | --- | --- | | ||
| | Metrics | `metrics-*` | | ||
| | Logs | `logs-*` | | ||
| | Traces | `traces-*`, `logs-*` | | ||
| | All three | `metrics-*`, `logs-*`, `traces-*` | | ||
|
|
||
| Traces ingestion also writes span events to `logs-*` data streams, so it requires both patterns. | ||
|
|
||
| For example, an API key role descriptor that allows ingesting all three signals: | ||
|
|
||
| ```json | ||
| { | ||
| "indices": [ | ||
| { | ||
| "names": ["logs-*", "metrics-*", "traces-*"], | ||
| "privileges": ["create_doc", "auto_configure"] | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| ### Configure an OpenTelemetry Collector | ||
|
|
||
| To send data from an OpenTelemetry Collector to an {{es}} OTLP endpoint, configure the [`OTLP/HTTP` exporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlphttpexporter): | ||
|
|
||
| ```yaml | ||
| exporters: | ||
| otlphttp/elasticsearch: | ||
| endpoint: <es_endpoint>/_otlp | ||
| headers: | ||
| Authorization: "ApiKey <api_key>" | ||
| sending_queue: | ||
| enabled: true | ||
| sizer: bytes <1> | ||
| queue_size: 50_000_000 <2> | ||
| block_on_overflow: true | ||
| batch: <3> | ||
| flush_timeout: 1s | ||
| min_size: 1_000_000 | ||
| max_size: 4_000_000 | ||
| service: | ||
| pipelines: | ||
| logs: | ||
| exporters: [otlphttp/elasticsearch] | ||
| receivers: ... | ||
| traces: | ||
| exporters: [otlphttp/elasticsearch] | ||
| receivers: ... | ||
| metrics: | ||
| exporters: [otlphttp/elasticsearch] | ||
| receivers: ... | ||
| ``` | ||
|
|
||
| 1. Sizes the queue and batches by uncompressed bytes. | ||
| 2. Limits the queue to 50 MB of uncompressed data. | ||
| Increasing this value can absorb longer {{es}} outages or traffic bursts, but also increases Collector memory usage. | ||
| 3. Controls the uncompressed batch size sent to {{es}}. | ||
| In this example, batches are sent at 1 MB and capped at 4 MB. | ||
| Larger batches reduce request overhead, but increase peak memory usage and the amount of data retried after a failed request. | ||
|
|
||
| The exporter appends the signal-specific path (`/v1/logs`, `/v1/traces`, `/v1/metrics`) to the configured `endpoint`. | ||
|
|
||
| These values are starting points for a gateway Collector. | ||
| Tune them for your workload and Collector resources. | ||
| They are local to each Collector instance and don't increase {{es}} ingest capacity. | ||
| If many applications need to send telemetry, scale out the gateway Collector instead of sending directly from each application. | ||
|
|
||
| Supported `compression` values are `gzip` (the `OTLP/HTTP` exporter default) and `none`. | ||
|
|
||
| To send data from a custom application, use the [OpenTelemetry language SDK](https://opentelemetry.io/docs/getting-started/dev/) of your choice and point its OTLP/HTTP exporter at the corresponding {{es}} OTLP endpoint path. | ||
|
|
||
| :::{note} | ||
| Only `encoding: proto` is supported, which the `OTLP/HTTP` exporter uses by default. | ||
| ::: | ||
|
|
||
| ## Routing to data streams | ||
|
|
||
| By default, records are written to the following data streams: | ||
|
|
||
| | Signal | Default data stream | | ||
| | --- | --- | | ||
| | Logs | `logs-generic.otel-default` | | ||
| | Traces | `traces-generic.otel-default` | | ||
| | Metrics | `metrics-generic.otel-default` | | ||
|
|
||
| For more about how OTLP metrics are stored as time series data streams, refer to [Ingest metrics into a TSDS using the OTLP/HTTP endpoint](/manage-data/data-store/data-streams/tsds-ingest-otlp.md). | ||
|
|
||
| The target data stream name follows the pattern `<type>-<dataset>.otel-<namespace>`. | ||
| You can influence `dataset` and `namespace` by setting attributes on your data: | ||
|
|
||
| * Set `data_stream.dataset` and/or `data_stream.namespace` as attributes. | ||
| Precedence: data point or log record attribute, then scope attribute, then resource attribute. | ||
| * Otherwise, if the scope name contains `/receiver/<somereceiver>`, `data_stream.dataset` is set to the receiver name. | ||
| * Otherwise, `data_stream.dataset` falls back to `generic` and `data_stream.namespace` falls back to `default`. | ||
|
kkrik-es marked this conversation as resolved.
|
||
|
|
||
| Examples: | ||
|
|
||
| | Signal | Attributes or scope name | Target data stream | | ||
| | --- | --- | --- | | ||
| | Logs | `data_stream.dataset: nginx.access`, `data_stream.namespace: prod` | `logs-nginx.access.otel-prod` | | ||
| | Traces | `data_stream.dataset: checkout`, `data_stream.namespace: staging` | `traces-checkout.otel-staging` | | ||
| | Metrics | Scope name contains `/receiver/hostmetrics`, no `data_stream.*` attributes | `metrics-hostmetrics.otel-default` | | ||
| | Metrics | No matching attributes or receiver scope name | `metrics-generic.otel-default` | | ||
|
|
||
| ## Body map mode for logs | ||
| ```{applies_to} | ||
| stack: preview 9.5 | ||
| ``` | ||
|
|
||
| By default, OTLP log records are mapped into {{es}}'s standard OTel document structure, preserving resource, scope, and record metadata. | ||
|
|
||
| If an upstream component has already shaped the log body to match the desired document structure, you can opt into the body map mode. | ||
| In this mode, the log record's body map is used as the complete document, without copying the surrounding OTLP metadata. | ||
|
|
||
| Enable body map mode in either of two ways: | ||
|
|
||
| * Per request, by setting the `X-Elastic-Mapping-Mode` HTTP header to `bodymap`. | ||
| * Per instrumentation scope, by setting the `elastic.mapping.mode` scope attribute to `bodymap`. | ||
| The scope attribute takes precedence over the header. | ||
|
|
||
| ## Configure histogram handling for metrics | ||
| ```{applies_to} | ||
| stack: preview =9.3, ga 9.4+ | ||
| ``` | ||
|
|
||
| You can configure how OTLP histogram metrics are mapped using the `xpack.otel_data.histogram_field_type` cluster setting. | ||
| Valid values are: | ||
|
|
||
| - `histogram` (default on {applies_to}`stack: preview =9.3`): Map histograms as T-Digests using the `histogram` field type | ||
| - `exponential_histogram` (default on {applies_to}`stack: ga 9.4+`): Map histograms as exponential histograms using the `exponential_histogram` field type | ||
|
|
||
| The setting is dynamic and can be updated at runtime: | ||
|
|
||
| ```console | ||
| PUT /_cluster/settings | ||
| { | ||
| "persistent" : { | ||
| "xpack.otel_data.histogram_field_type" : "exponential_histogram" | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| Because both `histogram` and `exponential_histogram` support [coerce](elasticsearch://reference/elasticsearch/mapping-reference/coerce.md), changing this setting dynamically does not risk mapping conflicts or ingestion failures. | ||
|
|
||
| This setting only applies to metrics ingested through the {{es}} OTLP endpoint. | ||
| Documents ingested using the Bulk API (for example through the {{es}} exporter for the OpenTelemetry Collector) are not affected. | ||
|
|
||
| ## Limitations | ||
|
|
||
| * **Delivery guarantees:** {{es}} can only acknowledge an OTLP request as a whole, not on a per-record basis. | ||
| If part of a request fails, the client retries the entire batch, which can produce duplicate logs or trace spans. | ||
| Metrics are not affected because metric points written to time series data streams are [deduplicated based on their dimensions and timestamp](/manage-data/data-store/data-streams/time-series-data-stream-tsds.md#time-series-dimension). | ||
| * **Profiles:** Profiles are not supported. | ||
| To ingest profiles, use a distribution of the OpenTelemetry Collector that includes the [{{es}} exporter](opentelemetry://reference/edot-collector/components/elasticsearchexporter.md), such as the [{{edot}} (EDOT) Collector](opentelemetry://reference/edot-collector/index.md). | ||
| * **Histogram temporality:** Histograms are only supported in delta temporality. | ||
| Set the temporality preference to delta in your SDKs, or use the [`cumulativetodelta` processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/cumulativetodeltaprocessor) so cumulative histograms aren't dropped. | ||
| * **Exemplars:** Exemplars are not supported yet. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.