Skip to content

Fix/log4j routing to avoid create spurious file#12965

Merged
andsel merged 5 commits intoelastic:masterfrom
andsel:fix/log4j_routing_to_avoid_create_spurious_file
Jun 16, 2021
Merged

Fix/log4j routing to avoid create spurious file#12965
andsel merged 5 commits intoelastic:masterfrom
andsel:fix/log4j_routing_to_avoid_create_spurious_file

Conversation

@andsel
Copy link
Contributor

@andsel andsel commented Jun 4, 2021

Release notes

What does this PR do?

This PR avoid to create pipeline log appender when the log event doesn't contain the pipeline.id

Why is it important/What is the impact to the user?

Avoid that the user see files name like 'pipeline_${ctx:pipeline.id}.log' which are empty

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files (and/or docker env variables)
  • I have added tests that prove my fix is effective or that my feature works

Author's Checklist

  • [ ]

How to test this PR locally

  • in config/log4j2.properties find the line
    appender.routing.pipeline.policy.size = 100MB
  • change the size from 100MB to 1KB
  • enable pipeline.separate_logs in config/logstash.yml
  • run Logstash with following pipeline.conf bin/logstash -f pipeline.conf --debug
  • check in logs/ directory the not existence of a file named pipeline_${ctx:pipeline.id}.log. On master it's present before this PR, in the branch of this PR that file is not created.

pipeline.conf

input {
  java_generator {
    lines => [
      "line 1",
      "line 2",
      "line 3"
    ]
    eps => 1
  }
}

output {
  stdout {
    codec => rubydebug
  }

Related issues

Use cases

Screenshots

Logs

@andsel andsel force-pushed the fix/log4j_routing_to_avoid_create_spurious_file branch from 8dfa45f to 863069c Compare June 15, 2021 12:18
@andsel andsel marked this pull request as ready for review June 15, 2021 14:14
@elasticsearch-bot elasticsearch-bot self-assigned this Jun 15, 2021
@kaisecheng kaisecheng self-requested a review June 15, 2021 18:57
Copy link
Contributor

@kaisecheng kaisecheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have tested the branch, confirmed there is no pipeline_${ctx:pipeline.id}.log and rotation work properly. The master branch and the testing branch is throwing

org.reflections.ReflectionsException: could not create class object from file org/logstash/plugins/plugin_aliases.yml

which is not in the scope of this issue.
Nice integration test 👍 . LGTM

@andsel andsel merged commit 1d6a3e4 into elastic:master Jun 16, 2021
andsel added a commit to andsel/logstash that referenced this pull request Jun 17, 2021
Avoid the creation of log4j routing appender for log events without the `pipeline.id` fishtag.
In this way no spurious log file named "pipeline_${ctx:pipeline.id}.log" and logs are not duplicated with main Logstash log file.

(cherry picked from commit 1d6a3e4)
andsel added a commit that referenced this pull request Jun 18, 2021
Avoid the creation of log4j routing appender for log events without the `pipeline.id` fishtag.
In this way no spurious log file named "pipeline_${ctx:pipeline.id}.log" and logs are not duplicated with main Logstash log file.

(cherry picked from commit 1d6a3e4)
kares added a commit to kares/logstash that referenced this pull request Jul 1, 2021
* master: (41 commits)
  Test: resolve integration failure due ECS mode (elastic#13044)
  Feat: event factory support (elastic#13017)
  Doc: Add geoip database API to node stats (elastic#13019)
  Add geoip database metrics to /node/stats API (elastic#13004)
  ecs: on-by-default plus docs (elastic#12830)
  ispec: fix cross-spec leak from fatal error integration specs (elastic#13002)
  Fix UBI source URL (elastic#13008)
  update fpm to allow pkg creation on jdk11+jruby 9.2 (elastic#13005)
  Add unit test to grant that production aliases correspond to a published RubyGem (elastic#12993)
  Fix logstash.bat not setting exit code (elastic#12948)
  Use the OS separator to invoke gradlew from Rake script (elastic#13000)
  Allow per-pipeline config of ECS Compatibility mode via Central Management (elastic#12861)
  Update jinja2 dependency in docker build (elastic#12994)
  fix database manager with multiple pipelines (elastic#12862)
  Fix Reflections stack traces when process yml files in classpath and debug is enabled (elastic#12991)
  Fix/log4j routing to avoid create spurious file (elastic#12965)
  Deps: update JRuby to 9.2.19.0 (elastic#12989)
  Doc: Add tip for checking for existing field (elastic#12899)
  Added test to cover the installation of aliased plugins (elastic#12967)
  CI: Update logstash_release.json after 7.3.12 (elastic#12986)
  ...
@andsel andsel added the v7.14.0 label Jul 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants