utils: fix SIMD boundary for JSON string encoding

[cosmo0920]

Summary

Fix JSON string corruption caused by SIMD reads crossing the declared end of long strings after processing multibyte UTF-8 characters.

Affected strings could receive 1–15 bytes of adjacent memory on 16-byte SIMD platforms, or fewer bytes with narrower SIMD implementations. Loki JSON output with remove_keys made the issue especially reproducible and could expose fragments of other log records.

Root cause

The JSON string writers rounded str_len down to a SIMD-aligned boundary and used i < vlen to decide whether another vector could be processed.

After scalar processing of a 2–4-byte UTF-8 sequence, i may no longer be SIMD-aligned. Consequently, i < vlen did not guarantee that the complete vector remained within str_len, allowing bytes past the logical end of the string to be read and copied.

Changes

• Require the entire SIMD vector to fit within the remaining input:

  i + inst_len <= str_len

• Apply the correction to both escaped and raw JSON string writers.

• Add an internal regression test with poisoned bytes after the declared input boundary.

• Cover SIMD-width, 24 KiB, and 36 KiB payload sizes.

• Add Loki runtime coverage using JSON line formatting and remove_keys.

• Add an end-to-end integration test that sends 200 approximately 40 KiB Unicode records through tail → Loki and compares every decoded message with its original value.

Compatibility

No configuration or API changes. The only behavioral change is that bytes beyond the declared string length are no longer included in encoded output.

Verification

Recorded in the referenced task, “Fix JSON string corruption”:

• cmake --build build --target flb-it-utils -j8 — passed
• ./build/bin/flb-it-utils write_str_simd_boundary — passed
• ctest --test-dir build -R '^flb-it-utils$' --output-on-failure — passed
• macOS Leaks reported no leaks, although macOS restricted full process inspection

The latest Loki runtime and new Python integration tests still need final functional and macOS Leaks runs before submitting the PR.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

• Bug Fixes

  • Improved handling of long escaped and raw strings, including multibyte Unicode content.
  • Prevented data corruption when string lengths fall near SIMD processing boundaries.
  • Preserved complete JSON messages and formatting for large Loki payloads.

• Tests

  • Added coverage for long Unicode messages, boundary-length payloads, and Loki integration output.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: c0b68f21-32f0-4481-b1ed-fc48370cc48f

📥 Commits

Reviewing files that changed from the base of the PR and between 6c071bf and 3710135.

📒 Files selected for processing (7)

• src/flb_utils.c
• tests/integration/scenarios/out_loki/config/out_loki_long_unicode.yaml
• tests/integration/scenarios/out_loki/config/parsers.conf
• tests/integration/scenarios/out_loki/tests/test_out_loki_001.py
• tests/integration/src/server/http_server.py
• tests/internal/utils.c
• tests/runtime/out_loki.c

---

📝 Walkthrough

Walkthrough

The SIMD string writers now stop vector processing at complete vectors. Internal, runtime, and integration tests cover long Unicode JSON payloads through Loki output formatting and HTTP ingestion.

Changes

Unicode and Loki boundary handling

Layer / File(s)	Summary
SIMD string boundary handling src/flb_utils.c, tests/internal/utils.c	The escaped and raw string writers process complete SIMD vectors before scalar handling. Internal tests cover multibyte alignment and boundary lengths.
Runtime Loki payload validation tests/runtime/out_loki.c	Runtime tests generate 24 KiB and 36 KiB Unicode JSON payloads. They verify preserved message endings and removal of the seq field.
Integration Loki flow tests/integration/scenarios/out_loki/..., tests/integration/src/server/http_server.py	The integration scenario configures Loki parsing and output, exposes the Loki push route, and validates 200 long Unicode records.
Estimated code review effort: 3 (Moderate)	~25 minutes

Merge Risk: ⚪ Minimal · up to 37101

This change prevents JSON encoders from including bytes beyond the declared string boundary and adds targeted regression and Loki coverage. No actionable merge-blocking risk remains beyond normal checks and review.

Suggested reviewers: edsiper

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: fixing the SIMD boundary for JSON string encoding.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch cosmo0920-plug-garbage-data-on-json-encoding

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3710135f56

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".