feat: Expose Functionality to Handle syslogs with Empty MSG Field by blewis12 · Pull Request #5687 · grafana/alloy

blewis12 · 2026-02-27T18:51:14Z

This relates to this escalation: https://github.com/grafana/support-escalations/issues/20753

Currently, in our loki.source.syslog component, we will drop logs with empty messages and increase our loki_source_syslog_empty_messages_total counter to indicate this. This has different ramifications for the different formats we support

RAW

Our raw format basically parses the entire log line as the message, and so an empty message in this case should be treated as an anomaly and be dropped

RFC3164

In this case, the log format is <PRI> <HEADER> <MSG>

For example <13>Mar 13 12:01:33 firewall01 sshd: I am a log. This would be parsed like:

PRI: <13>
HEADER.timestamp: Mar 13 12:01:33
HEADER.hostname: firewall01
MSG.appname: sshd
MSG.message: I am a log -> previously if this was empty, we would drop the log

In our parser (leodido/go-syslog/v4), omitting the "I am a log" part would result in a message set to sshd. That means that an empty message in this case would mean a log like <13>Mar 13 12:01:33 firewall01. This log line doesn't provide a lot of use, therefore I think it would make sense to continue dropping this, though ensure that it is clearly documented

It's also worth noting the the RFC doesn't provide any message-less examples

RFC5424

In this case, the log format is more granular <PRI>VERSION TIMESTAMP HOSTNAME APP-NAME PROCID MSGID STRUCTURED-DATA [MSG]

For example <13>1 2025-03-13T12:01:33.000Z firewall01 sshd 12345 msg-001 [example@12345 key="value"] I am a log. This would be parsed like

PRI: <13>
VERSION: 1
TIMESTAMP: Mar 13 12:01:33
HOSTNAME: firewall01
APP-NAME: sshd
PROCID: 12345
MSGID: msg-001
STRUCTURED-DATA: map[example@12345:map[key:value]]
MSG: I am a log -> previously if this was empty, we would drop the log

In this case, an empty message does not mean an entirely useless log - the system can still be sending over structured data with contextual information, which can also be applied in relabelling/processing rules in later pipeline steps to output a useful log line. This is the same scenario the customer found themselves in, and was confused to see log lines "silently" being dropped

My proposal is that we introduce opt-in functionality to allow for empty message fields in RFC5424 formatted logs. It's worth noting that the RFC explicitly states that an empty MSG field is a valid log, see example 4 in https://www.rfc-editor.org/rfc/rfc5424#section-6.5

github-actions · 2026-03-11T16:28:40Z

TruffleHog Scan Results

Summary: Found 3 potential secrets (0 verified, 3 unverified)

Possible secret (PrivateKey) at internal/component/loki/source/syslog/internal/syslogtarget/syslogtarget_test.go:55 → ----***---
Possible secret (PrivateKey) at internal/component/loki/source/syslog/internal/syslogtarget/syslogtarget_test.go:74 → ----***---
Possible secret (PrivateKey) at internal/component/loki/source/syslog/internal/syslogtarget/syslogtarget_test.go:162 → ----***---

Review: Check if unverified secrets are false positives.

Ignoring False Positives:
To mark a false positive, add # trufflehog:ignore as an inline comment on the same line as the detected secret:

my_fake_secret = "AKIAIOSFODNN7EXAMPLE"  # trufflehog:ignore

This works for files that support line numbers (most source files). After adding the comment, push your changes and the scan will re-run.

github-actions · 2026-03-11T16:29:46Z

💻 Deploy preview available (Handle syslogs with empty message field):

https://deploy-preview-alloy-5687-zb444pucvq-vp.a.run.app/docs/alloy/latest/

github-actions · 2026-03-13T08:53:35Z

💻 Deploy preview deleted (feat: Expose Functionality to Handle syslogs with Empty MSG Field).

Copilot

Pull request overview

Adds an opt-in configuration to allow forwarding RFC5424 syslog entries that omit the MSG field (so structured data can still be ingested/relabeled), while keeping existing drop behavior for other formats.

Changes:

Introduces allow_empty_rfc5424_msg config/argument and validates it only applies to syslog_format = "rfc5424".
Updates RFC5424 handling to optionally forward messages when msg.Message == nil (and preserves the empty-message metric increment).
Extends docs and adds RFC5424 empty-MSG tests; tightens raw handling to drop PRI-only/empty-body raw messages.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
internal/component/loki/source/syslog/types.go	Adds new listener option, validates applicability, and converts into target config.
internal/component/loki/source/syslog/types_test.go	Extends raw-only validation test mappings to include the new option.
internal/component/loki/source/syslog/internal/syslogtarget/syslogtarget.go	Implements the new RFC5424 forwarding behavior and drops empty raw bodies.
internal/component/loki/source/syslog/internal/syslogtarget/syslogtarget_test.go	Adds tests for RFC5424 empty-MSG allowed vs not allowed.
internal/component/loki/source/syslog/config/config.go	Adds YAML field to target config struct.
docs/sources/reference/components/loki/loki.source.syslog.md	Documents the new option and clarifies empty-message behavior per format.

clayton-cornell

Some doc suggestions to put things into active voice and remove the semicolon etc.

Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com>

Co-authored-by: Denys Sedchenko <9203548+x1unix@users.noreply.github.com>

🤖 I have created a release *beep* *boop* --- ## [1.15.0](v1.14.0...v1.15.0) (2026-03-26) ### ⚠ BREAKING CHANGES * **otelcol:** Upgrade to OTel Collector v0.147.0 ([#5784](#5784)) * Renamed undocumented metrics that was previously prefixed with <component_id>_<metric_name> to loki_source_awsfirehose_<metric_name> ### Features 🌟 * **alloy-mixin:** Add filters, groupBy, and multi-select dashboard variables ([#5611](#5611)) ([3ef714e](3ef714e)) * **beyla.ebpf:** Add support for Prometheus native histograms ([#5812](#5812)) ([7d806fb](7d806fb)) * **beyla.ebpf:** Bump Beyla to v3.6 ([#5833](#5833)) ([cd878d5](cd878d5)) * **converters:** Support converting Promtail limits_config ([#5777](#5777)) ([9491385](9491385)) * **database_observability.mysql:** Add filtering of query samples and wait events by minimum duration ([#5678](#5678)) ([5a4d03b](5a4d03b)) * **database_observability.mysql:** Embed prometheus exporter within db-o11y component ([#5711](#5711)) ([88bffb0](88bffb0)) * **database_observability.postgres:** Add configurable limit to `pg_stat_statements` query ([#5639](#5639)) ([0de0a3f](0de0a3f)) * **database_observability.postgres:** Embed prometheus exporter within db-o11y component ([#5714](#5714)) ([9dc2e83](9dc2e83)) * **database_observability:** Add scaffolding for db-o11y integration tests ([#5575](#5575)) ([ca637d8](ca637d8)) * **database_observability:** Promote components to stable ([#5736](#5736)) ([21a9af6](21a9af6)) * Expose Functionality to Handle syslogs with Empty MSG Field ([#5687](#5687)) ([178b1e6](178b1e6)) * **helm:** Allow setting `revisionHistoryLimit` in the helm chart ([#5847](#5847)) ([9713ad4](9713ad4)) * **loki.process:** Support structured metadata as source type of stage.labels for loki.process ([#5055](#5055)) ([eda3152](eda3152)) * **loki.secretfilter:** Add sampling for secretfilter entries ([#5663](#5663)) ([9997802](9997802)) * **loki.source.gcplog:** Add alloy config for MaxOutstandingBytes and MaxOutstandingMessages ([#5760](#5760)) ([c2b9f0b](c2b9f0b)) * **loki.write:** Add loki pipeline latency metric ([#5702](#5702)) ([cc744a1](cc744a1)) * **mixin:** Update loki dashboard ([#5848](#5848)) ([b616d58](b616d58)) * **otelcol.receiver.datadog:** Expose intake proxy and trace_id_cache_size settings ([#5776](#5776)) ([0384ad4](0384ad4)) * **otelcol:** Upgrade to OTel Collector v0.147.0 ([#5784](#5784)) ([a9b5396](a9b5396)) * **prometheus.exporter.cloudwatch:** Use aws-sdk-go-v2 by default ([#5768](#5768)) ([a2f3489](a2f3489)) * **pyroscope.ebpf:** Add comm, pid labels and kernel frame options ([#5769](#5769)) ([4fa7068](4fa7068)) * **pyroscope.ebpf:** Expose OTel eBPF profiler internal metrics to Prometheus ([#5774](#5774)) ([e713392](e713392)) * **pyroscope:** Copy prometheus common/config HTTP client into promhttp2 package ([#5810](#5810)) ([0b31aaa](0b31aaa)) ### Bug Fixes 🐛 * **beyla:** Inject Beyla version into binary via ldflags ([#5735](#5735)) ([71c03ec](71c03ec)) * Correctly handle the deprecated topic field in otelcol.receiver.kafka configuration ([#5726](#5726)) ([538ac75](538ac75)) * **database_observability.mysql:** Ensure result sets are properly closed ([#5893](#5893)) ([f28f91c](f28f91c)) * **database_observability:** Ensure all collectors are properly stopped ([#5796](#5796)) ([6bfa2a7](6bfa2a7)) * **database_observability:** Ensure that `connection_info` metric is only emitted for a given DB instance when it is available ([#5707](#5707)) ([bf0c3dc](bf0c3dc)) * **database_observability:** Solve test flakiness in MySQL and Postgres sample collectors ([#5130](#5130)) ([a7590d1](a7590d1)) * **deps:** Update module github.com/buger/jsonparser to v1.1.2 [SECURITY] ([#5834](#5834)) ([b2fee8a](b2fee8a)) * **deps:** Update module github.com/buger/jsonparser to v1.1.2 [SECURITY] ([#5870](#5870)) ([698b4e7](698b4e7)) * **deps:** Update module google.golang.org/grpc to v1.79.3 [SECURITY] ([#5825](#5825)) ([5cfbcc4](5cfbcc4)) * **deps:** Update module google.golang.org/grpc to v1.79.3 [SECURITY] ([#5871](#5871)) ([259152d](259152d)) * **deps:** Update npm dependencies ([#5876](#5876)) ([f0f6a11](f0f6a11)) * **deps:** Update npm deps across repo to address CVE-2026-26996 and CVE-2026-22029 ([#5872](#5872)) ([df518dd](df518dd)) * **go:** Update build image to go v1.25.8 ([#5832](#5832)) ([f9b3043](f9b3043)) * **go:** Update go to 1.25.8 ([#5844](#5844)) ([534e7db](534e7db)) * Helm: alloy.extraPorts not working with service.type=NodePort [COPY] ([#5892](#5892)) ([162c6f7](162c6f7)) * **loki.enrich:** Use shared loki functions and fix locking ([#5821](#5821)) ([f916c72](f916c72)) * **loki.process:** Multiline no longer pass empty entry if start was flushed ([#5746](#5746)) ([7bdedf1](7bdedf1)) * **loki.process:** Protect against json that does not look like docker json format ([#5761](#5761)) ([0af6eaa](0af6eaa)) * **loki.secretfilter:** Fix bug where entries were being shadow dropped ([#5786](#5786)) ([90243f9](90243f9)) * **loki.source.file:** Fix position tracking when component stops ([#5800](#5800)) ([9762946](9762946)) * **loki.source.file:** Keep positions for compressed files when reading is finished ([#5723](#5723)) ([fb41d0a](fb41d0a)) * **loki.source.gcplog:** Update to pubsub v2 and fix shutdown semantics ([#5713](#5713)) ([e9d9b69](e9d9b69)) * **loki.source.heroku:** Fix shutdown semantics and consume logs in batches ([#5804](#5804)) ([deda452](deda452)) * **loki.write:** Remove noisy log ([#5837](#5837)) ([8e28f35](8e28f35)) * **loki:** Make drain forward entries with fallback timeout ([#5830](#5830)) ([cfbca90](cfbca90)) * **prometheus.scrape:** Update arguments and targets even if `scrape_native_histograms` and `extra_metrics` are updated ([#5787](#5787)) ([dc4cb0a](dc4cb0a)) * **pyroscope.ebpf:** Update opentelemetry-ebpf-profiler ([#5904](#5904)) ([dfaec47](dfaec47)) * Stop components in a deterministic order ([#5613](#5613)) ([00cd371](00cd371)) ### Chores * Use shared source structures for aws firehose ([#5739](#5739)) ([aef19dc](aef19dc)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: grafana-alloybot[bot] <167359181+grafana-alloybot[bot]@users.noreply.github.com>

blewis12 added the publish-dev:linux builds and deploys an image to grafana/alloy-dev container repository label Feb 27, 2026

blewis12 force-pushed the handle-syslogs-with-empty-message-field branch 3 times, most recently from 0c44e65 to efabe45 Compare March 12, 2026 18:09

blewis12 changed the title ~~Handle syslogs with empty message field~~ feat: Expose Functionality to Handle syslogs with Empty MSG Field Mar 12, 2026

blewis12 force-pushed the handle-syslogs-with-empty-message-field branch from efabe45 to e24acce Compare March 13, 2026 08:51

blewis12 force-pushed the handle-syslogs-with-empty-message-field branch from 74c37b9 to 154eb28 Compare March 13, 2026 09:49

blewis12 marked this pull request as ready for review March 13, 2026 09:50

blewis12 requested review from a team and clayton-cornell as code owners March 13, 2026 09:50

blewis12 requested a review from Copilot March 13, 2026 09:52

Copilot started reviewing on behalf of blewis12 March 13, 2026 09:53 View session

blewis12 added 2 commits March 13, 2026 10:55

Expose functionality to drop empty MSG field for RFC5424 syslogs

bfad53c

update docs

0ed65cb

blewis12 force-pushed the handle-syslogs-with-empty-message-field branch from 154eb28 to 0ed65cb Compare March 13, 2026 09:55

Copilot AI reviewed Mar 13, 2026

View reviewed changes

Comment thread internal/component/loki/source/syslog/internal/syslogtarget/syslogtarget.go

Comment thread docs/sources/reference/components/loki/loki.source.syslog.md Outdated

Comment thread internal/component/loki/source/syslog/internal/syslogtarget/syslogtarget_test.go Outdated

blewis12 force-pushed the handle-syslogs-with-empty-message-field branch 2 times, most recently from e7b0b1b to b88ff09 Compare March 13, 2026 11:26

add explicit tests for empty messages for raw and 3164 formats

f8bcff7

blewis12 force-pushed the handle-syslogs-with-empty-message-field branch from b88ff09 to f8bcff7 Compare March 13, 2026 11:29

clayton-cornell reviewed Mar 13, 2026

View reviewed changes

Comment thread docs/sources/reference/components/loki/loki.source.syslog.md Outdated

Comment thread docs/sources/reference/components/loki/loki.source.syslog.md Outdated

Comment thread docs/sources/reference/components/loki/loki.source.syslog.md Outdated

clayton-cornell added the type/docs Docs Squad label across all Grafana Labs repos label Mar 13, 2026

blewis12 and others added 3 commits March 13, 2026 13:28

Update docs/sources/reference/components/loki/loki.source.syslog.md

754c1e8

Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com>

Update docs/sources/reference/components/loki/loki.source.syslog.md

2cb026b

Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com>

Update docs/sources/reference/components/loki/loki.source.syslog.md

c335933

Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com>

x1unix reviewed Mar 13, 2026

View reviewed changes

Comment thread docs/sources/reference/components/loki/loki.source.syslog.md Outdated

x1unix reviewed Mar 13, 2026

View reviewed changes

Comment thread docs/sources/reference/components/loki/loki.source.syslog.md Outdated

x1unix reviewed Mar 13, 2026

View reviewed changes

Comment thread internal/component/loki/source/syslog/config/config.go Outdated

blewis12 and others added 3 commits March 13, 2026 19:00

AllowEmptyRFC5424Msg -> RFC5424AllowEmptyMsg

12894ee

Update docs/sources/reference/components/loki/loki.source.syslog.md

7037cda

Co-authored-by: Denys Sedchenko <9203548+x1unix@users.noreply.github.com>

Update docs/sources/reference/components/loki/loki.source.syslog.md

e8a6d4a

Co-authored-by: Denys Sedchenko <9203548+x1unix@users.noreply.github.com>

blewis12 requested a review from x1unix March 13, 2026 18:01

x1unix approved these changes Mar 13, 2026

View reviewed changes

blewis12 merged commit 178b1e6 into main Mar 16, 2026
49 checks passed

blewis12 deleted the handle-syslogs-with-empty-message-field branch March 16, 2026 07:11

grafana-alloybot Bot mentioned this pull request Mar 16, 2026

chore(main): Release 1.15.0 #5170

Merged

BrewTestBot mentioned this pull request Mar 30, 2026

grafana-alloy 1.15.0 Homebrew/homebrew-core#275095

Merged

github-actions Bot mentioned this pull request Mar 31, 2026

Alloy 1.15.0 sysext build triggered noahwhite/alloy-sysext-build#56

Open

github-actions Bot added the frozen-due-to-age label Mar 31, 2026

github-actions Bot locked as resolved and limited conversation to collaborators Mar 31, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Expose Functionality to Handle syslogs with Empty MSG Field#5687

feat: Expose Functionality to Handle syslogs with Empty MSG Field#5687
blewis12 merged 9 commits into
mainfrom
handle-syslogs-with-empty-message-field

blewis12 commented Feb 27, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Mar 11, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Mar 11, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Mar 13, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

clayton-cornell left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

blewis12 commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

RAW

RFC3164

RFC5424

Uh oh!

github-actions Bot commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

clayton-cornell left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

blewis12 commented Feb 27, 2026 •

edited

Loading

github-actions Bot commented Mar 11, 2026 •

edited

Loading

github-actions Bot commented Mar 11, 2026 •

edited

Loading

github-actions Bot commented Mar 13, 2026 •

edited

Loading