diff --git a/blog-service/2025-10-30-collection.md b/blog-service/2025-10-30-collection.md new file mode 100644 index 0000000000..e0184df926 --- /dev/null +++ b/blog-service/2025-10-30-collection.md @@ -0,0 +1,17 @@ +--- +title: Token-Based Authentication for Secure HTTPS Log and Metric Sources (Collection) +image: https://assets-www.sumologic.com/company-logos/_800x418_crop_center-center_82_none/SumoLogic_Preview_600x600.jpg?mtime=1617040082 +keywords: + - http source + - authentication +hide_table_of_contents: true +--- + +import useBaseUrl from '@docusaurus/useBaseUrl'; + +We're excited to introduce secure token-based authentication for HTTPS and OTLP sources. This new capability allows you to authenticate using a unique token in the request header, maintaining the existing HTTPS endpoint behavior while adding token validation per source. + +Obtain the token to use in an auth header when you configure an HTTP source or regenerate the URL. To learn more, see: +* [Configure an HTTP Logs and Metrics Source](/docs/send-data/hosted-collectors/http-source/logs-metrics/#configure-an-httplogs-and-metrics-source) +* [Create an OTLP/HTTP Source](/docs/send-data/hosted-collectors/http-source/otlp/#create-an-otlphttpsource) +* [Generate a New URL for an HTTP Source](/docs/send-data/hosted-collectors/http-source/generate-new-url/) \ No newline at end of file diff --git a/docs/send-data/hosted-collectors/http-source/generate-new-url.md b/docs/send-data/hosted-collectors/http-source/generate-new-url.md index f586de1102..7a3b1b0e0d 100644 --- a/docs/send-data/hosted-collectors/http-source/generate-new-url.md +++ b/docs/send-data/hosted-collectors/http-source/generate-new-url.md @@ -11,10 +11,14 @@ You can generate a new URL for an HTTP Source at any time. Generating a new UR To generate a new URL: 1. [**New UI**](/docs/get-started/sumo-logic-ui). In the Sumo Logic main menu select **Data Management**, and then under **Data Collection** select **Collection**. You can also click the **Go To...** menu at the top of the screen and select **Collection**.  
[**Classic UI**](/docs/get-started/sumo-logic-ui-classic). In the main Sumo Logic menu, select **Manage Data > Collection > Collection**. -1. On the **Manage Collection** page, click **Regenerate URL** next to the HTTP source.
HTTPregenURL -1. In the **HTTP Source Address** dialog box, click **Generate**.
http_source_dialog_generate.png +1. On the **Manage Collection** page, click **Regenerate URL** next to the HTTP source.
Regenerate URL link +1. In the **HTTP Source Address** dialog box, select one of the following to regenerate the URL where the source data will be stored: + * **Presigned URL**. Select to copy a presigned URL with embedded authentication.
Generate New URL + * **Auth Header**. Select to copy the URL, as well as a separate authorization header that contains an authentication token. This option provides greater security than a presigned URL because placing the authentication token in the authorization header of a request prevents the token from being exposed in the URL.
Generate New URL and token +1. Click **Generate**. 1. When asked to confirm the generation, click **OK**. -1. In the **HTTP Source Address** dialog box, the new URL is displayed. Copy and paste the URL, then click **OK**.
http_source_dialog_generate.png +1. In the resulting dialog box, the newly-generated URL is displayed, as well as the authorization header if you selected **Auth Header**. Copy the URL (and header if applicable) and keep in a safe place. +1. Use the copied URL (and header if appropriate) when you [upload data to your HTTP Logs and Metrics source](/docs/send-data/hosted-collectors/http-source/logs-metrics/#upload-data-to-the-httplogs-and-metrics-source). :::note If you see a 401 (failed to authenticate) error message right after generating a new URL, wait a few minutes, then try the new URL again. diff --git a/docs/send-data/hosted-collectors/http-source/logs-metrics/index.md b/docs/send-data/hosted-collectors/http-source/logs-metrics/index.md index a30c2bd22c..5fd69ae853 100644 --- a/docs/send-data/hosted-collectors/http-source/logs-metrics/index.md +++ b/docs/send-data/hosted-collectors/http-source/logs-metrics/index.md @@ -45,7 +45,11 @@ To configure an HTTP Logs and Metrics Source: * **One Message Per Request.** Select this option if you'll be sending a single message with each HTTP request. For more information, see [Multiline options in HTTP sources](#multiline-options-in-http-sources).  1. **Processing Rules.** Configure any desired filters, such as allowlist, denylist, hash, or mask, as described in Create a Processing Rule. Processing rules are applied to log data, but not to metric data. 1. When you are finished configuring the Source, click **Save**. -1. When the URL associated with the source is displayed, copy the URL so you can use it to upload data.
![A screenshot showing the 'HTTP Source Address' dialog in Sumo Logic. The dialog instructs to use the given address to send data to the Collector and warns to keep this address private since anyone can use it to send data. The URL provided starts with 'https://collectors.sumologic.com/receiver/v1/http/...'. There is an 'OK' button at the bottom right.](/img/send-data/http-source-address.png) +1. In the **HTTP Source Address** dialog box, select one of the following to copy the URL where the source data will be stored: + * **Presigned URL**. Select to copy a presigned URL with embedded authentication.
HTTP Source Address with presigned URL + * **Auth Header**. Select to copy the URL, as well as a separate authorization header that contains an authentication token. This option provides greater security than a presigned URL because placing the authentication token in the authorization header of a request prevents the token from being exposed in the URL.
HTTP Source Address with authorization header +1. Copy the URL (and header if applicable) and keep in a safe place. You will use the URL in the next step: [Upload data to the HTTP Logs and Metrics Source](#upload-data-to-the-httplogs-and-metrics-source). +1. Click **Done**. :::note * Metrics reported with a timestamp older than 24 hours ago or newer than 24 hours in the future from the time they are reported are dropped. Make sure that the Metrics sent to HTTP Endpoint have appropriate timestamps. diff --git a/docs/send-data/hosted-collectors/http-source/logs-metrics/upload-logs.md b/docs/send-data/hosted-collectors/http-source/logs-metrics/upload-logs.md index b91e163308..621e475ed4 100644 --- a/docs/send-data/hosted-collectors/http-source/logs-metrics/upload-logs.md +++ b/docs/send-data/hosted-collectors/http-source/logs-metrics/upload-logs.md @@ -29,9 +29,11 @@ We recommend that the POST data payload have a size, before compression, of 100K * Data line 2 * Data line 3 * Method: POST -* URL: `https://[SumoEndpoint]/receiver/v1/http/[UniqueHTTPCollectorCode]` where - * [SumoEndpoint] is your Sumo collection endpoint - * [UniqueHTTPCollectorCode] is the string that follows the last forward slash (`/`) in the upload URL for the HTTP source +* URL:
Enter the URL obtained when you [configure the HTTP Logs and Metrics Source](/docs/send-data/hosted-collectors/http-source/logs-metrics/#configure-an-httplogs-and-metrics-source) or when you [regenerate the URL](/docs/send-data/hosted-collectors/http-source/generate-new-url/). Enter either a presigned URL or a URL to be used with an auth header: + * Presigned URL: `https://[SumoEndpoint]/receiver/v1/http/[UniqueHTTPCollectorCode]`
where + * [SumoEndpoint] is your Sumo collection endpoint + * [UniqueHTTPCollectorCode] is the string that follows the last forward slash (`/`) in the upload URL for the HTTP source + * URL used with auth header: `https://[SumoEndpoint]/receiver/v1/http`
where [SumoEndpoint] is your Sumo collection endpoint * Body * Data line 1 * Data line 2 @@ -54,8 +56,9 @@ Overridden metadata field values are not returned with [Search Autocomplete](/do | Custom Source Host | `X-Sumo-Host` | Desired host name.
Useful if you want to override the source host configured for the source. | | Custom Source Category | `X-Sumo-Category` | Desired source category.
Useful if you want to override the source category configured for the source. | | Fields as custom metadata | `X-Sumo-Fields` | [Fields](/docs/manage/fields) need to be in a comma separated list of key-value pairs. | +| Token authentication | `x-sumo-token` | Token to be used for authentication in an authorization header. Obtain the token when you select **Auth Header** when you [configure the HTTP Logs and Metrics Source](/docs/send-data/hosted-collectors/http-source/logs-metrics/#configure-an-httplogs-and-metrics-source), or when you [regenerate the URL](/docs/send-data/hosted-collectors/http-source/generate-new-url/). | -## Command Line Examples +## Command line examples :::note Data is ingested according to the configured [processing rules](/docs/send-data/collection/processing-rules). Messages blocked by filters will not be ingested. @@ -67,53 +70,109 @@ When using cURL to POST data from a file:  * Make sure to use the -T parameter to specify the file path, not -d. The -d parameter causes new lines to be removed from the content, which will interfere with message boundary detection. * Make sure that each line in the file follows the format specified by the Content-Type header for the HTTP request. +* Enter the URL (and auth header if applicable) obtained when you [configured the HTTP Logs and Metrics Source](/docs/send-data/hosted-collectors/http-source/logs-metrics/#configure-an-httplogs-and-metrics-source) or when you [regenerate the URL](/docs/send-data/hosted-collectors/http-source/generate-new-url/). If you use an auth header, enter it in this format:
`-H "x-sumo-token: [TokenString]"` **POST upload** +Presigned URL: ```bash -curl -v -X POST -T [local_file_name] https://collectors.sumologic.com/receiver/v1/http/[UniqueHTTPCollectorCode] +curl -v -X POST -T [local_file_name] https://collectors.sumologic.com/receiver/v1/http/[UniqueHTTPCollectorCode] +``` +URL with auth header: +```bash +curl -v -X POST -H "x-sumo-token: [TokenString]" -T [local_file_name] https://collectors.sumologic.com/receiver/v1/http ``` **POST upload (gzip compressed data)**  +Presigned URL: ```bash curl -v -X POST -H 'Content-Encoding:gzip' -T [local_file_name.gz] https://collectors.sumologic.com/receiver/v1/http/[UniqueHTTPCollectorCode] ``` +URL with auth header: +```bash +curl -v -X POST -H 'Content-Encoding:gzip' -H "x-sumo-token: [TokenString]" -T [local_file_name.gz] https://collectors.sumologic.com/receiver/v1/http +``` + **POST upload with custom Source Category** +Presigned URL: ```bash curl -v -X POST -H 'X-Sumo-Category:myNewCategory' -T [local_file_name] https://collectors.sumologic.com/receiver/v1/http/[UniqueHTTPCollectorCode] ``` +URL with auth header: +```bash +curl -v -X POST -H 'X-Sumo-Category:myNewCategory' -H "x-sumo-token: [TokenString]" -T [local_file_name] https://collectors.sumologic.com/receiver/v1/http +``` + **POST upload with custom Fields** +Presigned URL: ```bash curl -v -X POST -H 'X-Sumo-Fields:environment=dev,cluster=k8s' -T [local_file_name] https://collectors.sumologic.com/receiver/v1/http/[UniqueHTTPCollectorCode] ``` +URL with auth header: +```bash +curl -v -X POST -H 'X-Sumo-Fields:environment=dev,cluster=k8s' -H "x-sumo-token: [TokenString]" -T [local_file_name] https://collectors.sumologic.com/receiver/v1/http +``` ### PowerShell **POST upload** +In the following examples when a URL is used with an auth header, `$headers` is defined as follows: + +```bash +$headers = @{ + Authorization="x-sumo-token: [TokenString]" + Content='application/json' +} +``` + +Presigned URL: ```bash Invoke-WebRequest -Method POST -InFile [local_file_name] 'https://collectors.sumologic.com/receiver/v1/http/[UniqueHTTPCollectorCode]' ``` +URL with auth header: +```bash +Invoke-WebRequest -Method POST -InFile [local_file_name] 'https://collectors.sumologic.com/receiver/v1/http' -Headers $headers +``` + **POST upload (gzip compressed data)**  +Presigned URL: ```bash Invoke-WebRequest -Method POST -Headers @{'Content-Encoding' = 'gzip'} -InFile [local_file_name.gz] 'https://collectors.sumologic.com/receiver/v1/http/[UniqueHTTPCollectorCode]' ``` +URL with auth header: +```bash +Invoke-WebRequest -Method POST -Headers @{'Content-Encoding' = 'gzip'} -InFile [local_file_name.gz] 'https://collectors.sumologic.com/receiver/v1/http' -Headers $headers +``` + **POST upload with custom Source Category** +Presigned URL: ```bash Invoke-WebRequest -Method POST -Headers @{'X-Sumo-Category' = 'myCustomCategory'} -InFile [local_file_name] 'https://collectors.sumologic.com/receiver/v1/http/[UniqueHTTPCollectorCode]' ``` +URL with auth header: +```bash +Invoke-WebRequest -Method POST -Headers @{'X-Sumo-Category' = 'myCustomCategory'} -InFile [local_file_name] 'https://collectors.sumologic.com/receiver/v1/http' -Headers $headers +``` + **POST upload with custom Field** +Presigned URL: ```bash Invoke-WebRequest -Method POST -Headers @{'X-Sumo-Fields' = 'environment=dev'} -InFile [local_file_name] 'https://collectors.sumologic.com/receiver/v1/http/[UniqueHTTPCollectorCode]' ``` + +URL with auth header: +```bash +Invoke-WebRequest -Method POST -Headers @{'X-Sumo-Fields' = 'environment=dev'} -InFile [local_file_name] 'https://collectors.sumologic.com/receiver/v1/http' -Headers $headers +``` \ No newline at end of file diff --git a/docs/send-data/hosted-collectors/http-source/logs-metrics/upload-metrics.md b/docs/send-data/hosted-collectors/http-source/logs-metrics/upload-metrics.md index 0114a30f73..1ebf63c590 100644 --- a/docs/send-data/hosted-collectors/http-source/logs-metrics/upload-metrics.md +++ b/docs/send-data/hosted-collectors/http-source/logs-metrics/upload-metrics.md @@ -93,6 +93,7 @@ Overridden metadata field values are not returned with [Search Autocomplete](/do | Custom Source Category | `X-Sumo-Category` | Desired source category.
Useful if you want to override the source category configured for the source. | | Custom Metric Dimensions | `X-Sumo-Dimensions` | Comma-separated key=value list of dimensions to apply to every metric.
For metrics only. Custom dimensions will allow you to query your metrics at a more granular level. | | Custom Metric Metadata | `X-Sumo-Metadata` | Comma-separated, key=value list of metadata to apply to every metric.
For metrics only. Custom metadata will allow you to query your metrics at a more granular level. | +| Token authentication | `x-sumo-token` | Token to be used for authentication in an authorization header. Obtain the token when you select **Auth Header** when you [configure the HTTP Logs and Metrics Source](/docs/send-data/hosted-collectors/http-source/logs-metrics/#configure-an-httplogs-and-metrics-source), or when you [regenerate the URL](/docs/send-data/hosted-collectors/http-source/generate-new-url/). | ## Data volume and metadata limits for metrics @@ -104,31 +105,56 @@ When using cURL to POST data from a file:  * Make sure to use the `-T` parameter to specify the file path, not `-d`. The `-d` parameter causes new lines to be removed from the content, which will interfere with message boundary detection. * Make sure that each line in the file follows the format specified by the `Content-Type` header for the HTTP request. + * Enter the URL (and auth header if applicable) obtained when you [configured the HTTP Logs and Metrics Source](/docs/send-data/hosted-collectors/http-source/logs-metrics/#configure-an-httplogs-and-metrics-source) or when you [regenerate the URL](/docs/send-data/hosted-collectors/http-source/generate-new-url/). If you use an auth header, enter it in this format:
`-H "x-sumo-token: [TokenString]"` **POST upload ([Graphite](http://metrics20.org/implementations/)-formatted metrics)** +Presigned URL: ```bash curl -v -X POST -H 'Content-Type:application/vnd.sumologic.graphite' -T [local_file_name] https://collectors.sumologic.com/receiver/v1/http/[UniqueHTTPCollectorCode] ``` +URL with auth header: +```bash +curl -v -X POST -H 'Content-Type:application/vnd.sumologic.graphite' -H "x-sumo-token: [TokenString]" -T [local_file_name] https://collectors.sumologic.com/receiver/v1/http +``` + **POST upload ([Carbon 2.0](http://graphite.readthedocs.io/en/latest/feeding-carbon.html#the-plaintext-protocol)-formatted metrics)** +Presigned URL: ```bash curl -v -X POST -H 'Content-Type:application/vnd.sumologic.carbon2' -T [local_file_name] https://collectors.sumologic.com/receiver/v1/http/[UniqueHTTPCollectorCode] ``` +URL with auth header: +```bash +curl -v -X POST -H 'Content-Type:application/vnd.sumologic.carbon2' -H "x-sumo-token: [TokenString]" -T [local_file_name] https://collectors.sumologic.com/receiver/v1/http +``` + **POST upload (gzip compressed [Graphite](http://metrics20.org/implementations/)-formatted metrics)**  +Presigned URL: ```bash curl -v -X POST -H 'Content-Encoding:gzip' -H 'Content-Type:application/vnd.sumologic.graphite' -T [local_file_name.gz] https://collectors.sumologic.com/receiver/v1/http/[UniqueHTTPCollectorCode] ``` +URL with auth header: +```bash +curl -v -X POST -H 'Content-Encoding:gzip' -H 'Content-Type:application/vnd.sumologic.graphite' -H "x-sumo-token: [TokenString]" -T [local_file_name.gz] https://collectors.sumologic.com/receiver/v1/http +``` + **POST upload ([Prometheus](https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md)-formatted metrics)** +Presigned URL: ```bash curl -v -X POST -H 'Content-Type:application/vnd.sumologic.prometheus' -T [local_file_name] http://collectors.sumologic.com/receiver/v1/http/[UniqueHTTPCollectorCode] ``` +URL with auth header: +```bash +curl -v -X POST -H 'Content-Type:application/vnd.sumologic.prometheus' -H "x-sumo-token: [TokenString]" -T [local_file_name] http://collectors.sumologic.com/receiver/v1/http +``` + ## Prometheus Metrics Not Accepted by Sumo By design, Sumo does not ingest Prometheus comments. Sumo also rejects Prometheus metrics that do not conform to the Prometheus metric format. This page lists the conditions that will cause Sumo to reject Prometheus metrics diff --git a/docs/send-data/hosted-collectors/http-source/otlp.md b/docs/send-data/hosted-collectors/http-source/otlp.md index 208edc1d17..d0dced8e38 100644 --- a/docs/send-data/hosted-collectors/http-source/otlp.md +++ b/docs/send-data/hosted-collectors/http-source/otlp.md @@ -36,6 +36,11 @@ To configure an OTLP/HTTP Source: * **Timestamp Format**. By default, Sumo Logic will automatically detect the timestamp format of your logs. However, you can manually specify a timestamp format for a Source. See [Timestamps, Time Zones, Time Ranges, and Date Formats](/docs/send-data/reference-information/time-reference for more information. 1. [Create any Processing Rules](/docs/send-data/collection/processing-rules/create-processing-rule) you'd like for the OTLP/HTTP Source. 
![OTLP processing rules.png](/img/send-data/OTLP-processing-rules.png) 1. When you are finished configuring the Source, click **Save**. +1. In the **HTTP Source Address** dialog box, select one of the following to copy the URL where the source data will be stored: + * **Presigned URL**. Select to copy a presigned URL with embedded authentication.
HTTP Source Address with presigned URL + * **Auth Header**. Select to copy the URL, as well as a separate authorization header that contains an authentication token. This option provides greater security than a presigned URL because placing the authentication token in the authorization header of a request prevents the token from being exposed in the URL.
HTTP Source Address with authorization header +1. Copy the URL (and header if applicable) and keep in a safe place. You will use the URL in the next step: [Using the OTLP/HTTP Source](#using-the-otlphttp-source). +1. Click **Done**. :::note * Metrics reported with a timestamp older than 24 hours ago or newer than 24 hours in the future from the time they are reported are dropped. Make sure that the Metrics sent to OTLP Endpoint have appropriate timestamps. @@ -53,6 +58,7 @@ If you need to access the Source's URL again, click **Show URL**. ### Data differentiation When you set up an OTLP/HTTP Source, a unique URL is assigned to the Source. The exporter automatically appends the specific signal type when sending the data, as described by [OpenTelemetry Protocol Exporter specification](https://github.com/open-telemetry/opentelemetry-specification/blob/v1.10.0/specification/protocol/exporter.md). + The following table shows the URL format based on data types. | Data Type | URL Format | @@ -64,12 +70,14 @@ The following table shows the URL format based on data types. ### OTLP/HTTP Exporter -To use the URL, include the url in otlphttp exporter of OpenTelemetry Collector and refer it in the pipelines, as needed. The exporter will automatically attach the signal type suffix: +To use the URL, include the URL in otlphttp exporter of OpenTelemetry Collector and refer it in the pipelines, as needed. The exporter will automatically attach the signal type suffix: ``` exporters: otlphttp: endpoint: + headers: + Authorization: "x-sumo-token: [TokenString]" ... @@ -86,6 +94,10 @@ service: exporters: [otlphttp] ``` +:::note +In the above example, the authorization header value is obtained when you select **Auth Header** when you [Create an OTLP/HTTP Source](#create-an-otlphttpsource), or when you [regenerate the URL](/docs/send-data/hosted-collectors/http-source/generate-new-url/). +::: + ### Alternative It is also possible to optionally configure per-signal URL, e.g. @@ -96,6 +108,8 @@ exporters: traces_endpoint: /v1/traces metrics_endpoint: /v1/metrics logs_endpoint: /v1/logs + headers: + Authorization: "x-sumo-token: [TokenString]" ... diff --git a/static/img/send-data/generate-new-url-and-token.png b/static/img/send-data/generate-new-url-and-token.png new file mode 100644 index 0000000000..7befb5aef1 Binary files /dev/null and b/static/img/send-data/generate-new-url-and-token.png differ diff --git a/static/img/send-data/generate-new-url.png b/static/img/send-data/generate-new-url.png index 1457ebb904..1cb8f49c81 100644 Binary files a/static/img/send-data/generate-new-url.png and b/static/img/send-data/generate-new-url.png differ diff --git a/static/img/send-data/http-source-address-and-auth-header.png b/static/img/send-data/http-source-address-and-auth-header.png new file mode 100644 index 0000000000..2808305ece Binary files /dev/null and b/static/img/send-data/http-source-address-and-auth-header.png differ diff --git a/static/img/send-data/http-source-address-otlp-auth.png b/static/img/send-data/http-source-address-otlp-auth.png new file mode 100644 index 0000000000..83837456e9 Binary files /dev/null and b/static/img/send-data/http-source-address-otlp-auth.png differ diff --git a/static/img/send-data/http-source-address-otlp.png b/static/img/send-data/http-source-address-otlp.png new file mode 100644 index 0000000000..0c050cc26c Binary files /dev/null and b/static/img/send-data/http-source-address-otlp.png differ diff --git a/static/img/send-data/http-source-address.png b/static/img/send-data/http-source-address.png index 9774c8ed84..f203e457d7 100644 Binary files a/static/img/send-data/http-source-address.png and b/static/img/send-data/http-source-address.png differ