input_chunk: fix storage limit eviction with 32 or more outputs - #12235
Conversation
The storage limit path used a signed int as a secondary output selection mask. Shifting by output IDs at or above 32 is undefined and can alias a high-ID output to a lower output on common architectures. Check each routed output against its configured limit directly before releasing queue space. This keeps limit enforcement aligned with the dynamically sized routing masks. Signed-off-by: Michael Renner <terrorobe@github.com>
Create 33 outputs and place the constrained output at ID 32. Verify that eviction removes only its route while preserving the unrelated route at ID 0. Signed-off-by: Michael Renner <terrorobe@github.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughFilesystem space reclamation now evaluates routed output limits directly. Over-limit detection returns a boolean. A new integration test validates behavior with 33 outputs and filesystem storage. ChangesInput chunk limit handling
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/internal/input_chunk.c`:
- Around line 1252-1254: In the test setup around flb_input_chunk_get_real_size,
store the result in a signed ssize_t chunk_real_size, assert that it is positive
before converting it to first_chunk_size, and route assertion failure through
the function’s cleanup path. Add the cleanup label before the task teardown loop
so resources are released while failures prevent deriving total_limit_size from
a negative value.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 070dd534-fb16-4c8d-b60a-50e07dcbfca4
📒 Files selected for processing (2)
src/flb_input_chunk.ctests/internal/input_chunk.c
Keep the real chunk size signed until its error result is checked. Avoid deriving the test limit from a failed size lookup. Signed-off-by: Michael Renner <terrorobe@github.com>
cosmo0920
left a comment
cosmo0920
left a comment
There was a problem hiding this comment.
This change is reasonable to me because:
- Eliminating the fixed-width per-output mask is required.
- Changing has_overlimit_routes() to return a simple Boolean is reasonable.
- Removing the Boolean argument from find_space_new_data() is not independently required, but is the correct cleanup because the caller already performs that check and the callee needs live, per-output information rather than a global flag.
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3355097b98
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| flb_free(root_path); | ||
| } | ||
|
|
||
| void flb_test_input_chunk_limit_with_many_outputs(void) |
There was a problem hiding this comment.
This commit combines the core src/flb_input_chunk.c behavior change with a new tests/internal regression test, but the repository requires changes to different interfaces—including implementation and test areas—to be committed separately rather than bundled merely because they support the same fix. Split these into input_chunk: and tests: internal: commits.
AGENTS.md reference: AGENTS.md:L169-L172
Useful? React with 👍 / 👎.
| int flb_input_chunk_find_space_new_data(struct flb_input_chunk *ic, | ||
| size_t chunk_size, int overlimit) | ||
| size_t chunk_size) |
There was a problem hiding this comment.
The reviewed commit message has no Signed-off-by: trailer, so it does not satisfy the repository's mandatory DCO signing requirement and may be rejected by contribution checks; recreate the commit with an appropriate DCO sign-off.
AGENTS.md reference: AGENTS.md:L140-L140
Useful? React with 👍 / 👎.
3 participants
In configurations with many outputs,
storage.total_limit_sizeeviction can drop records queued for the wrong destination. In the reproduced case, output ID 32 reached its limit, but Fluent Bit also cleared the route for output ID 0. It then deleted the oldest chunk even though that chunk was still queued for output 0.The storage-limit path represented over-limit outputs in a signed 32-bit integer, even though chunk routing masks support more than 32 outputs. Shifting
1by output ID 32 is undefined; on the tested platform, it reused the bit for output ID 0.This change checks each routed output directly against its configured limit. It keeps the existing two linear output scans and adds neither allocations nor list traversals. The regression test creates 33 outputs and verifies that eviction for ID 32 preserves the route for ID 0.
Before the fix:
After the fix:
The affected code is present in v5.0.9, so this should be backported to the
5.0branch.Summary by CodeRabbit
Bug Fixes
Tests