From 3ee0c71bf674635ea4395db16a67bfda56a676f3 Mon Sep 17 00:00:00 2001 From: Pranjal-Gupta2 Date: Tue, 26 Apr 2022 01:32:52 +0530 Subject: [PATCH] Update `in_tail` with group directive Signed-off-by: Pranjal-Gupta2 --- input/tail.md | 106 ++++++++++++++ input/tail_with_throttle.md | 281 ------------------------------------ 2 files changed, 106 insertions(+), 281 deletions(-) delete mode 100644 input/tail_with_throttle.md diff --git a/input/tail.md b/input/tail.md index a6c71b61..11a265ec 100644 --- a/input/tail.md +++ b/input/tail.md @@ -17,6 +17,16 @@ It is included in Fluentd's core. @type apache2 + + rate_period 30s + + match { + namespace: /space1|space2|space2/, + podname: /app.*/, + } + limit 200 + + ``` @@ -402,6 +412,80 @@ The `@log_level` option allows the user to set different levels of logging for e Refer to the [Logging](../deployment/logging.md) for more details. + +### `` Section \(highly recommended\) + +The `in_tail` plugin can assign each log file to a group, based on user defined rules. The `limit` parameter controls the total number of lines collected for a group within a `rate_period` time interval. + +Example: + +```text +# group rules -- 1 + + rate_period 5s + + + match { + namespace: /shopping/, + podname: /frontend/, + } + limit 1000 + + + +# group rules -- 2 + + + match { + directoy: /payment/ + } + limit 2000 + + +``` + +#### `pattern` + +| type | default | version | +| :--- | :--- | :--- | +| regexp | `/^\/var\/log\/containers\/(?[a-z0-9]([-a-z0-9]*[a-z0-9])?(\/[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?[^_]+)_(?.+)-(?[a-z0-9]{64})\.log$/`| 1.15 | + +Specifies the regular expression for extracting metadata (namespace, podname) from log file path. Default value of the pattern regexp extracts information about `namespace`, `podname`, `docker_id`, `container` of the log (K8s specific). + +You can also add custom named captures in `pattern` for custom grouping of log files. For example, +```text + pattern /^\/home\/logs\/(?.+)\.log$/ +``` +In this example, filename will be extracted and used to form groups. + +#### `rate_period` + +| type | default | version | +| :--- | :--- | :--- | +| time | 60 \(seconds\) | 1.15 | + +Time period in which the group line limit is applied. `in_tail` resets the counter after every `rate_period` interval. + +#### `` Section \(required\) + +Grouping rules for log files. + +##### match + +| type | default | version | +| :--- | :--- | :--- | +| hash | {"namespace": /./, "podname": /./} | 1.15 | + +`match` parameter is used to check if a file belongs to a particular group based on hash keys (named captures from `pattern`) and hash values (regexp) + +##### limit + +| type | default | version | +| :--- | :--- | :--- | +| integer | -1 | 1.15 | + +Maximum number of lines allowed from a group in `rate_period` time interval. The default value of `-1` doesn't throttle log files of that group. + ## Learn More * [Input Plugin Overview](./) @@ -469,3 +553,25 @@ path C:\\path\\to\\*\\foo.log If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. +### What happens when a file can be assigned to more than one group? + +Example, + +```text + ## Rule1 + match { + namespace: /monitoring/ + } + limit 100 + + + ## Rule2 + match { + namespace: /monitoring/, + podname: /logger/, + } + limit 2000 + +``` + +In this case, rules with more constraints, i.e., greater number of `match` hash keys will be given a higher priority. So a file will be assigned to `Rule2` if it can be assigned to both `Rule1` and `Rule2`. \ No newline at end of file diff --git a/input/tail_with_throttle.md b/input/tail_with_throttle.md deleted file mode 100644 index c8ebe075..00000000 --- a/input/tail_with_throttle.md +++ /dev/null @@ -1,281 +0,0 @@ -# tail_with_throttle - -![](../.gitbook/assets/tail.png) - -The `in_tail_with_throttle` Input plugin allows Fluentd to read events from the tail of text files. Its behavior is similar to that of `in_tail` Input Plugin with the only difference being that this plugin rate limits the number of events read on the basis of groups. - -## Example Configuration - -```text - - @type tail_with_throttle - path /var/log/*.log - pos_file /var/log/temp.log.pos - tag applogs - - @type apache2 - - - pattern /(?[a-z0-9]([-a-z0-9]*[a-z0-9])?(\/[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?[^_]+)_(?.+)-(?[a-z0-9]{64})\.log$ - rate_period 30s - - namespace logging - appname frontend - limit 500 - - - namespace logging - appname backend - limit 1000 - - - limit 20 - - - -``` - -Refer to the [Configuration File](../configuration/config-file.md) article for the basic structure and syntax of the configuration file. - -For ``, see [Parse Section](../configuration/parse-section.md). - -### How It Works - -When Fluentd is configured with `in_tail_with_throttle`, it will form groups based on the rules provided in the configuration file. Based on group formation, log files will be assigned to a group and reading will start from the **tail** of that log, not the beginning. Group limits are equally distributed among the log files in a group. If the group line limit of the log is reached within rate_period time interval, Fluentd will stop reading logs. Once, **rate_period** (time) is over, Fluentd will reset its group counters and restart the reading process. - - -## Plugin Helpers - -* [`timer`](../plugin-helper-overview/api-plugin-helper-timer.md) -* [`event_loop`](../plugin-helper-overview/api-plugin-helper-event_loop.md) -* [`parser`](../plugin-helper-overview/api-plugin-helper-parser.md) -* [`compat_parameters`](../plugin-helper-overview/api-plugin-helper-compat_parameters.md) - -See also: [Linux capability](../deployment/linux-capability.md) - -## Parameters - -See [Common Parameters](../configuration/plugin-common-parameters.md). - -`in_tail_with_throttle` Input Plugin utilizes `in_tail` plugin's configuration. Refer to `in_tail` plugin's parameters [in_tail Parameters](./tail.md) - -The additional parameter which `in_tail_with_throttle` uses is the `` Directive. - -### `` Directive \(required\) - -Grouping rules for the log. - -`in_tail_with_throttle` assigns each log file to a group based on the rules provided in **``** directive. The `limit` parameter controls the total number of lines collected for a group by `tail_with_throttle` Input Plugin. - -Example: - -```text -# group rules - - rate_period 5s - - - namespace shopping - appname frontend - limit 1000 - - - - namespace payment - appname backend - limit 2000 - - -``` - -#### `pattern` - -| type | default | version | -| :--- | :--- | :--- | -| regexp | `/(?[a-z0-9]([-a-z0-9]*[a-z0-9])?(\/[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?[^_]+)_(?.+)-(?[a-z0-9]{64})\.log$`| 0.14.1 | - -Specifies the regular expression for extracting metadata (namespace, appname) from log file path. Default value of the pattern regexp follows CRIO path format which gives information about `namespace`, `pod` (appname), `docker_id`, `container` of the log. - -Please add `namespace` and `appname` regex-groups in your custom regex pattern to extract metadata from filenames. If the pattern specified in the configuration fails to identify `namespace` and `appname` information, it will assign the log to the default group. - - -#### `rate_period` - -| type | default | version | -| :--- | :--- | :--- | -| time | 60 \(seconds\) | 0.14.1 | - -Time period in which the group line limit is applied. `in_tail_with_throttle` resets the line collected counter after every `rate_period` interval. - - -### `` Directive \(required\) - -Specifies the rules and limits for a group. - - -#### `namespace` - -| type | default | version | -| :--- | :--- | :--- | -| array | allow all files [\/.\/] | 0.14.1 | - -Namespace criteria to form rules. This parameter internally converts each string element in the array to a regexp so it can be used to generalize a list of namespaces. - -Examples: -- This rule will apply to all files which have **monitoring** as a substring in their namespace. - -```text - - namespace monitoring - -``` - -- You can also specify multiple namespaces in your configuration file. - -```text - - namespace monitoring, logging - -``` -#### `appname` - -| type | default | version | -| :--- | :--- | :--- | -| array | allow all files [\/.\/] | 0.14.1 | - -Appname criteria to form rules. This parameter, like `namespace`, internally converts each string element in the array to a regexp so it can be used to generalize a list of appnames. - -Examples: -- This rule will apply to all files which have **heavy-stress** as a substring in their appname. -```text - - appname heavy-stress - -``` - -- You can also specify multiple appnames in your configuration files. - -```text - - appname heavy-stress, backend-payment - -``` - - -#### `limit` - -| type | default | version | -| :--- | :--- | :--- | -| integer | -1 | 0.14.1 | - -Maximum number of lines allowed from a group in `rate_period` time interval. The default value of `-1` doesn't throttle log files of that group. - - -## Group Rules Example - -### Generic Rules -This rule will assign files (that do not satisfy any rule) to a group limit of 100 lines. - -Example - - -```text - - limit 100 - -``` - -### Context-Specifc Rules - -#### Namespace based grouping - -This configuration will limit all files that have substring as **monitoring** present in their namespace, and will not check for appname in the filename. - -```text - - namespace monitoring - limit 100 - -``` - -#### Appname based grouping - -Similarly, this configuration will limit all files containing 'frontend` substring present in their appname, and will not check for the namespace in the filename. - -```text - - appname frontend - limit 200 - -``` - -#### Namespace and Appname based grouping - -Examples: - -- This rule will limit all files which have `monitoring` substring and `frontend` substring in namespace and appname fields, respectively. - -```text - - namespace monitoring - appname frontend - limit 50 - -``` -- If we omit the `limit` parameter in a rule, then there will be no rate limiting applied to the files falling in that group. - -```text - - namespace monitoring - appname frontend - -``` - -- When using multiple namespaces and appnames in a single rule, the group line limits are distributed uniformly. - - -```text - - namespace monitoring, logging - appname frontend, backend - limit 4000 - -``` - -The list of groups formed from this rule are: - -| Namespace | Appname | Group Limit | -|:------------: |:----------: |:-----------: | -| /monitoring/ | /frontend/ | 1000 | -| /monitoring/ | /backend/ | 1000 | -| /logging/ | /frontend/ | 1000 | -| /logging/ | /backend/ | 1000 | - -## FAQ - -### What happens when a file can be assigned to both, context-specific and generic rules? - -By default, context-specific rules always precede generic rule applied to default group. Hence the file will be assigned to context-specific rule. - -### What happens when we have the same rule repeated in the configuration file? - -Example, - -```text - - namespace monitoring - appname logger - limit 100 - - - - namespace monitoring - appname logger - limit 2000 - -``` - -In this case, limits are overridden by `in_tail_with_throttle` Input Plugin. So, the final limit of the group for namespace `monitoring` and appname `logger` will be 2000. - -### Can we run `tail` and `tail_with_throttle` simultaneously ? - -Yes, we can run both input plugins simultaneously. \ No newline at end of file