Update log-collection to v0.29.0 (final independent release)

[djaglowski]

Several changes have been made that affect configuration for the filelog, syslog, tcplog, udplog, and journald receivers. The following is a user guide for upgrading from Collector v0.48.0 to v0.49.0.

Updating Receiver Configuration

Update all usages of field syntax

Field syntax no longer uses the $ character. Instead each field must begin with body, attributes, or resource.

Deprecated Example	Updated Equivalent
$$attributes["log.file.name"]	attributes["log.file.name"]
$$resource["host.name"]	resource["host.name"]
$$body.foo	body.foo
$$.foo	body.foo
foo	body.foo

Tip for updating sub-parsers

To update the parse_from field in a "sub-parser", such as timestamp or severity, consider where the value would reside if the sub-parser was excluded.

Deprecated Example	Updated Equivalent
operators: - type: regex_parser regex: '^Time=(?P<time>\d{4}-\d{2}-\d{2})...' parse_to: body # default timestamp: parse_from: time ...	operators: - type: regex_parser regex: '^Time=(?P<time>\d{4}-\d{2}-\d{2})...' parse_to: attributes # default timestamp: parse_from: attributes.time ...

attributes and resource now support arbitrary structure

Field syntax can now refer to nested fields in both attributes and resource. Nested fields were previously only supported in body. For example, attributes.foo.bar refers to the value "hello" in the following entry:

{
  "attributes": {
    "foo": {
      "bar": "hello"
    }
  }
}

Update references to parsed values

The default value of parse_to has been changed from body to attributes. As a general rule of thumb, any parsers that previously specified the parse_to field are unaffected. If the parse_to field was not previously specified, then subsequent operators may need to be updated.

Deprecated Example	Updated Equivalent
operators: - type: regex_parser regex: '^Foo=(?P<foo>\s.*)...' parse_from: body # parse_to: body (old default) - type: move from: foo to: attributes.bar ...	operators: - type: regex_parser regex: '^Foo=(?P<foo>\s.*)...' parse_from: body # parse_to: attributes (new default) - type: move from: attributes.foo to: attributes.bar ...

Remove or replace usages of preserve_to

Parsers no longer support a preserve_to setting. Instead, the parse_from field is preserved by default. To preserve and move the field, use the move operator after the parse operator. To remove the field after parsing, use the remove operator.

Deprecated Example	Updated Equivalent
operators: - type: regex_parser regex: '^Foo=(?P<foo>\s.*)...' parse_from: body preserve_to: attributes.backup ...	operators: - type: regex_parser regex: '^Foo=(?P<foo>\s.*)...' parse_from: body - type: move from: body to: attributes.backup ...
operators: - type: regex_parser regex: '^Foo=(?P<foo>\s.*)...' parse_from: body # implicitly deleted ...	operators: - type: regex_parser regex: '^Foo=(?P<foo>\s.*)...' parse_from: body - type: remove field: body ...

Replace usages of restructure operator

The restructure operator has been removed. Use add, copy, flatten, move, remove, and retain operators instead.

Deprecated Example	Updated Equivalent
operators: - type: restructure ops: - add: field: set_me value: foo - add: field: overwrite_me value: bar - move: from: details.env to: env - remove: field: delete_me	operators: - type: add field: body.set_me value: foo - type: add field: body.overwrite_me value: bar - type: move from: body.details.env to: body.env - type: remove field: body.delete_me

Replace usages of metadata operator

The metadata operator has been removed. Use add, copy, or move operators instead.

Deprecated Example	Updated Equivalent
operators: - type: metadata attributes: environment: production file: 'EXPR( $$body.file )' resource: cluster: blue	operators: - type: add field: attributes.environment value: production - type: copy from: body.file to: attributes.file - type: add field: resource.cluster value: blue - type: move from: body.foo to: attributes.bar

Update filelog attribute references

The filelog receiver has adopted slightly different attribute names in order to match newly established semantic conventions. Configurations that previously refered to the file.* attributes should be updated.

Deprecated Attribute	Updated Equivalent
file.name	log.file.name
file.path	log.file.path
file.name.resolved	log.file.name_resolved
file.path.resolved	log.file.path_resolved

Note to Vendors: A log record's Timestamp field may now be nil

A recent change to the Log Data Model has redefined the usage of the Timestamp field. Correspondingly, this field is no longer initialized by default. All Log Exporters should be evaluated to ensure this change is handled accordingly.

Log exporters can use the following logic to mimic the previous functionality (psuedocode):

timestamp := log.ObservedTimestamp
if log.Timestamp != nil {
  timestamp = log.Timestamp
}

---

log-collection v0.29.0 Release Notes

Breaking Changes

• The default value of parse_to field in all parsers has been changed to attributes. (PR463)
• Parsers that contain a parse_to setting will no longer delete the parse_from field. (PR464)
• The preserve_to setting has been removed from parsers. (PR464)

Added

• key_value_parser (PR459)
• severity parsing can now use preset: otel to recognize both numeric and text representations of OpenTelemetry's log data model. (PR460)
• regex_parser can now cache parsing parsing results using the cache setting. This can dramatically increase performance in cases where the same string is parsed repeatedly. (PR440)

Fixed

• Issue where scope name parser would fail to initialize. (PR465)

[andrzej-stencel]

Update all usages of field syntax

The correct link for "field" is https://github.com/open-telemetry/opentelemetry-log-collection/blob/v0.29.0/docs/types/field.md for anyone wondering. 🙂

[djaglowski]

Thanks @astencel-sumo, I've updated the guide.