Skip to content

[chore][processor/memorylimiter] Add hard limit recommendations to readme #14356

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bogdandrutu merged 1 commit into from
Jan 6, 2026
Merged

[chore][processor/memorylimiter] Add hard limit recommendations to readme #14356

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 31 additions & 6 deletions processor/memorylimiterprocessor/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,13 @@ The processor will enter memory limited mode and will start refusing the data wh
memory usage exceeds the soft limit. This is done by returning errors to the preceding component
in the pipeline that made the ConsumeLogs/Trace/Metrics function call.

> Warning: Incoming data can consume additional memory in a Collector before the
> memory limiter processor is able to reject it. Be sure to consider this when
> setting your limits, particularly for non-OTLP receivers.
>
> See <https://github.com/open-telemetry/opentelemetry-collector/issues/8632> for
> more.

In memory limited mode the error returned by ConsumeLogs/Trace/Metrics function is a
non-permanent error. When receivers see this error they are expected to retry sending
the same data. The receivers may also apply backpressure to their own data sources
Expand Down Expand Up @@ -79,12 +86,33 @@ memory check interval. Otherwise, memory usage may exceed the hard limit, even i
A good starting point for `spike_limit_mib` is 20% of the hard limit. Bigger
`spike_limit_mib` values may be necessary for spiky traffic or for longer check intervals.

It's recommended to coordinate the hard memory limit with the host environment
Copy link
Copy Markdown
Contributor Author

@evan-bradley evan-bradley Jan 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please let me know if any of this needs adjustment, or if we don't feel confident even making these recommendations.

We determined this while sizing Collectors during some internal testing, but it may not be generally applicable.

of the Collector. As always, you know your environment best, so use your own
judgement to determine the best configuration for your situation. However, the
following points are worth considering when configuring the processor:

For containerized environments or other similar environments that support
setting memory restrictions on the Collector, `limit_percentage` should
generally be used, which allows the host environment to determine the
Collector's max memory allocation without tying the Collector's own config to
its deployment config. The percentage should be large enough to use the majority
of the Collector's allocated memory, but not so much that it risks running out
of memory.

For bare metal or virtualized environments where the Collector's memory is not
constrained by the host environment, it is recommended to use `limit_mib` for
environments where you know the rough data throughput and memory consumption you
expect for a given Collector, regardless of the memory capacity of the machine
it runs on. If the Collector's memory consumption is expected to scale with its
host machine's memory capacity, `limit_percentage` may be more appropriate.

## Configuration

Please refer to [memorylimiter.go](../../internal/memorylimiter/memorylimiter.go) for the config spec.
Please refer to [config.go](../../internal/memorylimiter/config.go) for the config spec.

The following configuration options are available. Note that one of `limit_mib`
or `limit_percentage` must be set.

The following configuration options **must be changed**:
- `check_interval` (default = 0s): Time between measurements of memory
usage. The recommended value is 1 second.
If the expected traffic to the Collector is very spiky then decrease the `check_interval`
Expand All @@ -94,7 +122,7 @@ allocated by the process heap. Note that typically the total memory usage of
process will be about 50MiB higher than this value. This defines the hard limit.
- `spike_limit_mib` (default = 20% of `limit_mib`): Maximum spike expected between the
measurements of memory usage. The value must be less than `limit_mib`. The soft limit
value will be equal to (limit_mib - spike_limit_mib).
value will be equal to (`limit_mib - spike_limit_mib`).
The recommended value for `spike_limit_mib` is about 20% `limit_mib`.
- `limit_percentage` (default = 0): Maximum amount of total memory targeted to be
allocated by the process heap. This configuration is supported on Linux systems with cgroups
Expand Down Expand Up @@ -134,6 +162,3 @@ On a machine with 1000 MiB total memory available:

- Hard limit will be set to 1000 * 0.80 = **800 MiB**.
- Soft limit will be set to 1000 * 0.80 - 1000 * 0.15 = 1000 * 0.65 = **650 MiB**.

Refer to [config.yaml](../../internal/memorylimiter/testdata/config.yaml) for detailed
Copy link
Copy Markdown
Contributor Author

@evan-bradley evan-bradley Jan 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed this after checking the file and finding it only contained information already available in the snippets above.

examples on using the processor.
Loading