Skip to content

docs: pipeline: outputs: logdna: document hostname promotion #2678

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
eschabell wants to merge 1 commit into
base: master
Choose a base branch
from
Open

docs: pipeline: outputs: logdna: document hostname promotion #2678

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 9 additions & 6 deletions pipeline/outputs/logdna.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ This plugin uses the following configuration parameters:
| :--- | :--- | :--- |
| `api_key` | Required. The API key to get access to the service. | _none_ |
| `app` | Name of the application. This value is automatically discovered on each record. If no value is found, the default value is used. | `Fluent Bit` |
| `exclude_promoted_keys` | When enabled, keys promoted to the top-level line object (`meta`, `level`, `severity`, `app`, `file`) are excluded from the `line` body to avoid duplication. | `false` |
| `exclude_promoted_keys` | When enabled, keys promoted to the top-level line object (`meta`, `level`, `severity`, `app`, `file`, `hostname`) are excluded from the `line` body to avoid duplication. | `false` |
Comment thread
coderabbitai[bot] marked this conversation as resolved.
| `file` | Optional name of a file being monitored. This value is only set if the record doesn't contain a reference to it. | _none_ |
| `hostname` | Name of the local machine or device where Fluent Bit is running. If no value is specified, Fluent Bit will look up the hostname and auto-populate its value. If Fluent Bit is unable to find a value, it will set the value `unknown` instead. | _none_ |
| `ip` | The IP address of the local hostname. This value is optional. | _none_ |
Expand All @@ -36,10 +36,13 @@ When the plugin processes a record or log, it searches for specific key names th
| `level` | If the record contains a key called `level` or `severity`, Fluent Bit will populate the context `level` key with that value. If not found, Fluent Bit won't set the context key. |
| `file` | If the record contains a key called `file`, it will populate the context `file` with the value found. Otherwise, if the plugin configuration provided a `file` property, that value will be used instead. |
| `app` | If the record contains a key called `app`, it will populate the context `app` with the value found, otherwise it will use the value set for `app` in the configuration property. |
| `hostname` | If the record contains a key called `hostname`, it will populate the context `hostname` with the value found, otherwise it will use the hostname resolved when the plugin started. Available in Fluent Bit version 5.1.1 and greater. |
| `meta` | If the record contains a key called `meta`, it will populate the context `meta` with the value found. |

These keys are referred to as _promoted keys_ because their values are lifted to the top-level line object sent to LogDNA. By default, promoted keys also remain present in the `line` JSON body. To remove them from the `line` body and avoid duplication, set `exclude_promoted_keys` to `true`.

Promoting `hostname` on each line lets a single output instance report a different host per record, for example when Fluent Bit forwards logs on behalf of other machines. The hostname resolved at startup, either from the `hostname` configuration property or from the local system, is always sent as a query parameter with the batch. A `hostname` value promoted from a record applies only to that line.

## Example configuration

The following example configuration uses a `dummy` input and `logdna` output:
Expand Down Expand Up @@ -126,7 +129,7 @@ In your LogDNA dashboard, go to the top filters and mark the Tags `aa` and `bb`,

## Exclude promoted keys

When a record contains promoted keys (`level`, `severity`, `app`, `file`, or `meta`), those values are lifted to the top-level line object. By default, they also appear inside the `line` JSON body, which can result in duplicate fields.
When a record contains promoted keys (`level`, `severity`, `app`, `file`, `hostname`, or `meta`), those values are lifted to the top-level line object. By default, they also appear inside the `line` JSON body, which can result in duplicate fields.

Setting `exclude_promoted_keys` to `true` removes the promoted keys from the `line` body, keeping each field in only one place.

Expand All @@ -135,22 +138,22 @@ Setting `exclude_promoted_keys` to `true` removes the promoted keys from the `li
Given this input record:

```json
{"log": "starting service", "severity": "INFO", "app": "my-service", "host": "web-01"}
{"log": "starting service", "severity": "INFO", "app": "my-service", "hostname": "web-01"}
```

With the default behavior (`exclude_promoted_keys: false`), the `line` body sent to LogDNA contains all fields:

```json
{"log": "starting service", "severity": "INFO", "app": "my-service", "host": "web-01"}
{"log": "starting service", "severity": "INFO", "app": "my-service", "hostname": "web-01"}
```

With `exclude_promoted_keys: true`, the `line` body contains only the non-promoted fields:

```json
{"log": "starting service", "host": "web-01"}
{"log": "starting service"}
```

The `severity` and `app` values are still sent, but only as top-level fields in the LogDNA line object, not duplicated inside `line`.
The `severity`, `app`, and `hostname` values are still sent, but only as top-level fields in the LogDNA line object, not duplicated inside `line`.

### Configuration

Expand Down
Loading