input_chunk: in_storage_backlog: Display evicted logs on dropping chunks

[cosmo0920]

Implemented the storage-eviction observability patch.

Changes:

• Live and retry-backed chunks now emit a warning before eviction in flb_input_chunk.c.
• Unloaded backlog chunks emit the same terminal warning in sb.c.
• Logs include chunk name, task ID when assigned, input/output route, bytes, and configured limit.
• Integration coverage verifies both never-flushed eviction and correlation with a prior flush failure in test_in_forward_001.py

Example:

[warn] [input chunk] chunk '...flb' evicted from output queue to make room under storage.total_limit_size: task_id=0, input=forward.0 > output=non_working_endpoint (out_id=1), bytes=16384, limit=32768

Verification passed:

• cmake --build build -j8
• ctest --test-dir build -R '^flb-it-input_chunk$' --output-on-failure
• Focused in_forward storage-limit pytest: 3 passed
• Same tests with LEAKS=1 LEAKS_STRICT=1: 3 passed
• in_storage_backlog pytest: 3 passed
• Same suite with Leaks strict mode: 3 passed
• ctest --test-dir build -R '^(flb-rt-in_forward|flb-rt-in_storage_backlog)$' --output-on-failure: 2 passed

Closes #12244.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

• Bug Fixes

  • Improved storage eviction warnings with chunk details, including input/output names, output ID, size, configured limit, and task ID when available.
  • Preserved clear warning messages for chunks that are not associated with a task.
  • Improved eviction reporting for filesystem-backed backlog chunks by retaining their original input identity.

• Tests

  • Enhanced storage-limit integration coverage to verify eviction logging and confirm failed chunks are evicted when appropriate.
  • Added coverage for persisted backlog eviction after configuration reloads.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: ed4d6712-ba60-4eb4-bf19-f4ceecc0f0cb

📥 Commits

Reviewing files that changed from the base of the PR and between 9e5997f and 099d039.

📒 Files selected for processing (3)

• src/flb_input_chunk.c
• tests/integration/scenarios/in_storage_backlog/config/reload_with_storage_limit_eviction.yaml
• tests/integration/scenarios/in_storage_backlog/tests/test_in_storage_backlog_001.py

🚧 Files skipped from review as they are similar to previous changes (1)

• src/flb_input_chunk.c

---

📝 Walkthrough

Walkthrough

The change adds warning logs for output queue chunk eviction. The logs include chunk, source, output, size, limit, and task context. Integration tests validate eviction logs, source identity, and removed chunks.

Changes

Storage eviction logging

Layer / File(s)	Summary
Eviction warning context plugins/in_storage_backlog/sb.c, src/flb_input_chunk.c	Eviction warnings now include chunk and output details. Input chunk eviction resolves filesystem-backlog source names and preserves task_id before route removal.
Storage backlog eviction validation tests/integration/scenarios/in_storage_backlog/config/reload_with_storage_limit_eviction.yaml, tests/integration/scenarios/in_storage_backlog/tests/test_in_storage_backlog_001.py	The scenario configures filesystem storage limits and verifies that eviction preserves the original input identity and removes the persisted chunk.
Eviction log validation tests/integration/scenarios/in_forward/tests/test_in_forward_001.py	Forward input tests wait for storage-limit eviction logs and verify that failed chunks are evicted.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to 099d0

The change adds eviction warnings and focused integration coverage without any identified current-head issue; no actionable merge-blocking risk remains after normal checks and review.

Possibly related PRs

• fluent/fluent-bit#11632: Changes related storage-limit eviction and chunk-release behavior.
• fluent/fluent-bit#12235: Modifies storage-limit eviction handling in src/flb_input_chunk.c.
• fluent/fluent-bit#12239: Extends storage-backlog integration coverage for persisted filesystem chunks.

Suggested reviewers: edsiper

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the main change: logging evicted chunks in the input chunk and storage backlog paths.
Linked Issues check	✅ Passed	The changes add terminal eviction warnings with chunk, route, task, size, and storage-limit details, plus tests for the reported lifecycle gaps in issue #12244.
Out of Scope Changes check	✅ Passed	The source, configuration, and integration-test changes directly support eviction logging and validation for issue #12244.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch cosmo0920-display-evicted-logs-on-dropping-chunks

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9e5997fe79

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".