feat(new transform): Add new window transform

[ilinas]

Summary

A variant of ring buffer / backtrace logging. Keeps events in a buffer until a trigger is encountered and the buffer is flushed. When the buffer is full, the oldest events are being dropped, and it works pretty much like the filter transform.

This is rewrite of my previous PR #21071 with the complexity and the scope reduced.

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

How did you test this PR?

Sample configuration:

sources:
  log_generator:
    type: demo_logs
    format: json
    interval: 0.1

  trigger:
    type: demo_logs
    format: shuffle
    lines:
      - '{"message": "flush"}'
      - '{"message": "pass"}'
      - '{"message": "hello"}'
    interval: 1

transforms:
  parse_json:
    type: remap
    inputs:
      - log_generator
      - trigger
    source: ". = parse_json!(string!(.message))"

  window_transform:
    type: window
    inputs:
      - parse_json
    pass_when: '.message == "pass"'
    flush_when: '.message == "flush"'
    events_before: 3
    events_after: 2

sinks:
  output:
    type: console
    inputs:
      - window_transform
    encoding:
      codec: json

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the "no-changelog" label to this PR.

Checklist

• Please read our Vector contributor resources.
  • make check-all is a good command to run locally. This check is
    defined here. Some of these
    checks might not be relevant to your PR. For Rust changes, at the very least you should run:
    • cargo fmt --all
    • cargo clippy --workspace --all-targets -- -D warnings
    • cargo nextest run --workspace (alternatively, you can run cargo test --all)
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run dd-rust-license-tool write to regenerate the license inventory and commit the changes (if any). More details here.

References

• Closes: Ring buffer / backtrace event handling #15263
• Old PR: feat(new transform): Add buffered gate transform #21071

[joepeeples]

A couple small line edit suggestions but otherwise LGTM, thanks!

[jorgehermo9]

I think I liked more the gate name you proposed before! #15263 (comment)

[ilinas]

I think I liked more the gate name you proposed before! #15263 (comment)

I chose a new name because I felt that gate could be a separate transform that implements open_when and close_when and does not use a buffer. Like a dynamic filter. In the end this proved not really useful in my infrastructure, so it never came to be. Well, not yet.

The window name is a reference to the sliding window technique, which I believe makes it easier to explain what it actually does.

[ilinas]

Thank you for the review and edits @joepeeples.

I thought the original description of the transform was not great, so I decided to update it. Would you mind taking a look a the changes I made in commit 92ae4af?

[ilinas]

This is a real-life example of what the transform does.

The service is logging at debug level, and the transform is configured with pass_when: .level == "info" and flush_when: .level == "error".

[pront]

This is a real-life example of what the transform does. The service is logging at debug level, and the transform is configured with pass_when: .level == "info" and flush_when: .level == "error".

Thank you @ilinas. This PR is on my radar. I will do a proper review soon!

[pront]

This looks great, thanks @ilinas. I left some comments, mostly nits.

[ilinas]

I think this is the final version of the code.

[pront]

I think this is the final version of the code.

Nice! I will do one more review but we can easily include this in the next release.

[pront]

@ilinas this looks like a good first version!

[pront]

Please git merge origin master (I tried to do it but your remote rejected the push)

[ilinas]

@pront I could use some help figuring out what is it failing on this time?

[pront]

@pront I could use some help figuring out what is it failing on this time?

Not sure, you can try merging the latest origin/master just to re-trigger the CI checks. It would be easier to help if I could push to the remote.