[7.x](backport #25998) Add more ECS fields to logs #26266

mergify · 2021-06-11T20:46:47Z

This is an automatic backport of pull request #25998 done by Mergify.

Mergify commands and options

More conditions and actions can be found in the documentation.

You can also trigger Mergify actions by commenting on this pull request:

@Mergifyio refresh will re-evaluate the rules
@Mergifyio rebase will rebase this PR on its base branch
@Mergifyio update will merge the base branch into this PR
@Mergifyio backport <destination> will backport this PR on <destination> branch

Additionally, on Mergify dashboard you can:

look at your merge queues
generate the Mergify configuration with the config editor.

Finally, you can contact us on https://mergify.io/

(cherry picked from commit bfbadaa)

botelastic · 2021-06-11T20:46:52Z

This pull request doesn't have a Team:<team> label.

elasticmachine · 2021-06-11T22:21:50Z

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #26266 opened
Start Time: 2021-06-11T20:47:03.951+0000
Duration: 94 min 56 sec
Commit: 96dbe30

Test stats 🧪

Test	Results
Failed	1
Passed	47463
Skipped	5207
Total	52671

Trends 🧪

Test errors

Expand to view the tests failures

`Extended / filebeat-windows-2016-windows-2016 / test_rotating_file – filebeat.tests.system.test_registrar.Test`

Expand to view the error details

 beat.beat.TimeoutError: Timeout waiting for 'cond' to be true. Waited 10 seconds.

Expand to view the stacktrace

 self = <test_registrar.Test testMethod=test_rotating_file>

    def test_rotating_file(self):
        """
        Checks that the registry is properly updated after a file is rotated
        """
        self.render_config_template(
            path=os.path.abspath(self.working_dir) + "/log/*",
            close_inactive="1s"
        )
    
        os.mkdir(self.working_dir + "/log/")
        testfile_path = self.working_dir + "/log/test.log"
    
        filebeat = self.start_beat()
    
        with open(testfile_path, 'w') as testfile:
            testfile.write("offset 9\n")
    
        self.wait_until(lambda: self.output_has(lines=1),
                        max_timeout=10)
    
        testfilerenamed = self.working_dir + "/log/test.1.log"
        os.rename(testfile_path, testfilerenamed)
    
        with open(testfile_path, 'w') as testfile:
            testfile.write("offset 10\n")
    
        self.wait_until(lambda: self.output_has(lines=2),
                        max_timeout=10)
    
        # Wait until rotation is detected
>       self.wait_until(
            lambda: self.log_contains(
                "Updating state for renamed file"),
            max_timeout=10)

tests\system\test_registrar.py:308: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <test_registrar.Test testMethod=test_rotating_file>
cond = <function Test.test_rotating_file.<locals>.<lambda> at 0x000002227253B430>
max_timeout = 10, poll_interval = 0.1, name = 'cond'

    def wait_until(self, cond, max_timeout=10, poll_interval=0.1, name="cond"):
        """
        Waits until the cond function returns true,
        or until the max_timeout is reached. Calls the cond
        function every poll_interval seconds.
    
        If the max_timeout is reached before cond() returns
        true, an exception is raised.
        """
        start = datetime.now()
        while not cond():
            if datetime.now() - start > timedelta(seconds=max_timeout):
>               raise TimeoutError("Timeout waiting for '{}' to be true. ".format(name) +
                                   "Waited {} seconds.".format(max_timeout))
E               beat.beat.TimeoutError: Timeout waiting for 'cond' to be true. Waited 10 seconds.

..\libbeat\tests\system\beat\beat.py:362: TimeoutError

Steps errors

Expand to view the steps failures

`filebeat-windows-2016-windows-2016 - mage build unitTest`

Took 5 min 27 sec . View more details on here
Description: mage build unitTest

`gsutil -m -q cp -a public-read test-build-artifacts-filebeat-windows-2016-windows-2016-tgz gs://beat`

Took 0 min 5 sec . View more details on here
Description: @echo off gsutil -m -q cp -a public-read test-build-artifacts-filebeat-windows-2016-windows-2016-tgz gs://beats-ci-temp/Beats/beats/PR-26266-1

`Error signal`

Took 0 min 0 sec . View more details on here
Description: Error 'hudson.AbortException: script returned exit code 1'

Log output

Expand to view the last 100 lines of log output

[2021-06-11T22:18:03.358Z] Copy .\packetbeat\null\build into build\packetbeat\null\build
[2021-06-11T22:18:03.403Z] Running in C:\Users\jenkins\workspace\PR-26266-1-4270169e-dba2-4b86-bdf2-64c91c8f21ed\src\github.com\elastic\beats\build
[2021-06-11T22:18:03.466Z] Recording test results
[2021-06-11T22:18:04.112Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-06-11T22:18:04.486Z] + gcloud auth activate-service-account --key-file ****
[2021-06-11T22:18:05.054Z] Activated service account credentials for: [[email protected]]
[2021-06-11T22:18:06.089Z] [Checks API] No suitable checks publisher found.
[2021-06-11T22:18:06.469Z] 
[2021-06-11T22:18:06.469Z] C:\Users\jenkins\workspace\PR-26266-1-a9999d6f-d1d4-4433-8a40-e922ecf3ed5b\src\github.com\elastic\beats>python .ci/scripts/pre_archive_test.py 
[2021-06-11T22:18:06.518Z] + gsutil -m -q cp -a public-read ZmlsZWJlYXQtd2luZG93cy0xMC13aW5kb3dzLTEwOTZkYmUzMGEwNjEwMjM2NmY0ZTBmMjRjMzlmNWRmY2ZjNzUyNzQ3Ng gs://beats-ci-temp/ci/cache/
[2021-06-11T22:18:06.828Z] 
[2021-06-11T22:18:06.828Z] C:\Users\jenkins\workspace\PR-26266-1-4270169e-dba2-4b86-bdf2-64c91c8f21ed\src\github.com\elastic\beats>go clean -modcache 
[2021-06-11T22:18:09.769Z] Copy .\filebeat\build into build\filebeat\build
[2021-06-11T22:18:09.769Z] Copy .\filebeat\null\build into build\filebeat\null\build
[2021-06-11T22:18:09.813Z] Running in C:\Users\jenkins\workspace\PR-26266-1-a9999d6f-d1d4-4433-8a40-e922ecf3ed5b\src\github.com\elastic\beats\build
[2021-06-11T22:18:09.878Z] Recording test results
[2021-06-11T22:18:12.461Z] [Checks API] No suitable checks publisher found.
[2021-06-11T22:18:13.010Z] 
[2021-06-11T22:18:13.010Z] C:\Users\jenkins\workspace\PR-26266-1-a9999d6f-d1d4-4433-8a40-e922ecf3ed5b\src\github.com\elastic\beats>go clean -modcache 
[2021-06-11T22:18:38.098Z] + gsutil --version
[2021-06-11T22:18:39.616Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-06-11T22:18:39.970Z] + gcloud auth activate-service-account --key-file ****
[2021-06-11T22:18:40.543Z] Activated service account credentials for: [[email protected]]
[2021-06-11T22:18:40.973Z] + gsutil --version
[2021-06-11T22:18:41.212Z] + gsutil -m -q cp -a public-read cGFja2V0YmVhdC13aW5kb3dzLTEwLXdpbmRvd3MtMTA5NmRiZTMwYTA2MTAyMzY2ZjRlMGYyNGMzOWY1ZGZjZmM3NTI3NDc2 gs://beats-ci-temp/ci/cache/
[2021-06-11T22:18:42.933Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-06-11T22:18:43.299Z] + gcloud auth activate-service-account --key-file ****
[2021-06-11T22:18:43.868Z] Activated service account credentials for: [[email protected]]
[2021-06-11T22:18:44.577Z] + gsutil -m -q cp -a public-read cGFja2V0YmVhdC13aW5kb3dzLTIwMTYtd2luZG93cy0yMDE2OTZkYmUzMGEwNjEwMjM2NmY0ZTBmMjRjMzlmNWRmY2ZjNzUyNzQ3Ng gs://beats-ci-temp/ci/cache/
[2021-06-11T22:19:18.804Z] module\system\test_system.py .....s..s...s.s.                            [ 96%]
[2021-06-11T22:19:18.804Z] module\traefik\test_traefik.py s                                         [ 97%]
[2021-06-11T22:19:18.804Z] module\uwsgi\test_uwsgi.py ss                                            [ 97%]
[2021-06-11T22:19:18.804Z] module\vsphere\test_vsphere.py sss                                       [ 99%]
[2021-06-11T22:19:18.804Z] module\zookeeper\test_zookeeper.py ss                                    [100%]
[2021-06-11T22:19:18.804Z] 
[2021-06-11T22:19:18.804Z] ============================== warnings summary ===============================
[2021-06-11T22:19:18.804Z] tests\system\test_lightmodules.py:57
[2021-06-11T22:19:18.804Z]   C:\Users\jenkins\workspace\PR-26266-1-27bc42cb-71f4-4fd8-8e11-c1124b7e1cc3\src\github.com\elastic\beats\metricbeat\tests\system\test_lightmodules.py:57: PytestCollectionWarning: cannot collect test class 'TestHTTPHandler' because it has a __init__ constructor (from: metricbeat/tests/system/test_lightmodules.py)
[2021-06-11T22:19:18.804Z]     class TestHTTPHandler(http.server.BaseHTTPRequestHandler):
[2021-06-11T22:19:18.804Z] 
[2021-06-11T22:19:18.804Z] -- Docs: https://docs.pytest.org/en/stable/warnings.html
[2021-06-11T22:19:18.804Z] - generated xml file: C:\Users\jenkins\workspace\PR-26266-1-27bc42cb-71f4-4fd8-8e11-c1124b7e1cc3\src\github.com\elastic\beats\metricbeat\build\TEST-python-unit.xml -
[2021-06-11T22:19:18.804Z] ============================ slowest 20 durations =============================
[2021-06-11T22:19:18.804Z] 16.92s call     metricbeat/tests/system/test_cmd.py::TestCommands::test_modules_list
[2021-06-11T22:19:18.805Z] 16.89s call     metricbeat/tests/system/test_cmd.py::TestCommands::test_modules_disable
[2021-06-11T22:19:18.805Z] 16.88s call     metricbeat/tests/system/test_cmd.py::TestCommands::test_modules_enable
[2021-06-11T22:19:18.805Z] 13.04s call     metricbeat/tests/system/test_config.py::ConfigTest::test_service_name
[2021-06-11T22:19:18.805Z] 12.56s call     metricbeat/tests/system/test_reload.py::Test::test_start_stop
[2021-06-11T22:19:18.805Z] 10.81s call     metricbeat/tests/system/test_cmd.py::TestCommands::test_modules_test_error
[2021-06-11T22:19:18.805Z] 10.20s call     metricbeat/module/system/test_system.py::Test::test_process
[2021-06-11T22:19:18.805Z] 9.80s call     metricbeat/tests/system/test_processors.py::Test::test_dropevent_with_condition
[2021-06-11T22:19:18.805Z] 9.57s call     metricbeat/tests/system/test_reload.py::Test::test_reload
[2021-06-11T22:19:18.805Z] 8.90s call     metricbeat/tests/system/test_base.py::Test::test_export_index_pattern_migration
[2021-06-11T22:19:18.805Z] 8.90s call     metricbeat/tests/system/test_timeseries.py::TestTimeseries::test_enable_timeseries
[2021-06-11T22:19:18.805Z] 8.87s call     metricbeat/module/http/test_http.py::Test::test_server
[2021-06-11T22:19:18.805Z] 8.84s call     metricbeat/tests/system/test_base.py::Test::test_export_index_pattern
[2021-06-11T22:19:18.805Z] 8.77s call     metricbeat/tests/system/test_base.py::Test::test_export_template
[2021-06-11T22:19:18.805Z] 8.77s call     metricbeat/module/system/test_system.py::Test::test_memory
[2021-06-11T22:19:18.805Z] 8.73s call     metricbeat/module/system/test_system.py::Test::test_cpu
[2021-06-11T22:19:18.805Z] 8.73s call     metricbeat/module/system/test_system.py::Test::test_process_summary
[2021-06-11T22:19:18.805Z] 8.73s call     metricbeat/module/system/test_system.py::Test::test_network
[2021-06-11T22:19:18.805Z] 8.73s call     metricbeat/module/system/test_system.py::Test::test_process_metricbeat
[2021-06-11T22:19:18.805Z] 8.72s call     metricbeat/module/system/test_system.py::Test::test_core_with_cpu_ticks
[2021-06-11T22:19:18.805Z] =========== 40 passed, 206 skipped, 1 warning in 397.22s (0:06:37) ============
[2021-06-11T22:19:18.805Z] >> python test: Unit Testing Complete
[2021-06-11T22:19:19.331Z] 
[2021-06-11T22:19:19.331Z] C:\Users\jenkins\workspace\PR-26266-1-27bc42cb-71f4-4fd8-8e11-c1124b7e1cc3\src\github.com\elastic\beats>FOR / %d IN ("ve") DO @IF EXIST "%d" rmdir /s /q "%d" 
[2021-06-11T22:19:22.196Z] 
[2021-06-11T22:19:22.197Z] C:\Users\jenkins\workspace\PR-26266-1-27bc42cb-71f4-4fd8-8e11-c1124b7e1cc3\src\github.com\elastic\beats>python .ci/scripts/pre_archive_test.py 
[2021-06-11T22:19:22.812Z] + gsutil --version
[2021-06-11T22:19:24.101Z] Copy .\metricbeat\build into build\metricbeat\build
[2021-06-11T22:19:24.101Z] Copy .\metricbeat\null\build into build\metricbeat\null\build
[2021-06-11T22:19:24.147Z] Running in C:\Users\jenkins\workspace\PR-26266-1-27bc42cb-71f4-4fd8-8e11-c1124b7e1cc3\src\github.com\elastic\beats\build
[2021-06-11T22:19:24.206Z] Recording test results
[2021-06-11T22:19:24.289Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-06-11T22:19:24.662Z] + gcloud auth activate-service-account --key-file ****
[2021-06-11T22:19:25.232Z] Activated service account credentials for: [[email protected]]
[2021-06-11T22:19:26.140Z] [Checks API] No suitable checks publisher found.
[2021-06-11T22:19:26.517Z] + gsutil -m -q cp -a public-read ZmlsZWJlYXQtd2luZG93cy03LTMyLXdpbmRvd3MtNy0zMi1iaXQ5NmRiZTMwYTA2MTAyMzY2ZjRlMGYyNGMzOWY1ZGZjZmM3NTI3NDc2 gs://beats-ci-temp/ci/cache/
[2021-06-11T22:19:26.794Z] 
[2021-06-11T22:19:26.794Z] C:\Users\jenkins\workspace\PR-26266-1-27bc42cb-71f4-4fd8-8e11-c1124b7e1cc3\src\github.com\elastic\beats>go clean -modcache 
[2021-06-11T22:19:30.608Z] + gsutil --version
[2021-06-11T22:19:32.054Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-06-11T22:19:32.457Z] + gcloud auth activate-service-account --key-file ****
[2021-06-11T22:19:33.031Z] Activated service account credentials for: [[email protected]]
[2021-06-11T22:19:33.721Z] + gsutil -m -q cp -a public-read cGFja2V0YmVhdC13aW5kb3dzLTctMzItd2luZG93cy03LTMyLWJpdDk2ZGJlMzBhMDYxMDIzNjZmNGUwZjI0YzM5ZjVkZmNmYzc1Mjc0NzY gs://beats-ci-temp/ci/cache/
[2021-06-11T22:20:50.227Z] + gsutil --version
[2021-06-11T22:20:51.662Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-06-11T22:20:52.028Z] + gcloud auth activate-service-account --key-file ****
[2021-06-11T22:20:52.600Z] Activated service account credentials for: [[email protected]]
[2021-06-11T22:20:53.323Z] + gsutil -m -q cp -a public-read bWV0cmljYmVhdC13aW5kb3dzLTEwLXdpbmRvd3MtMTA5NmRiZTMwYTA2MTAyMzY2ZjRlMGYyNGMzOWY1ZGZjZmM3NTI3NDc2 gs://beats-ci-temp/ci/cache/
[2021-06-11T22:20:57.570Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-06-11T22:20:57.674Z] Stage "Packaging-Pipeline" skipped due to earlier failure(s)
[2021-06-11T22:20:57.815Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-26266/src/github.com/elastic/beats
[2021-06-11T22:20:59.062Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-26266
[2021-06-11T22:20:59.176Z] [INFO] getVaultSecret: Getting secrets
[2021-06-11T22:20:59.233Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-06-11T22:21:00.259Z] + chmod 755 generate-build-data.sh
[2021-06-11T22:21:00.259Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26266/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26266/runs/1 FAILURE 5636034
[2021-06-11T22:21:00.260Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26266/runs/1/steps/?limit=10000 -o steps-info.json

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	1
Passed	47463
Skipped	5207
Total	52671

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

Name: Extended / filebeat-windows-2016-windows-2016 / test_rotating_file – filebeat.tests.system.test_registrar.Test

jalvz · 2021-06-14T07:13:08Z

jenkins retest this please

jalvz · 2021-06-16T07:56:03Z

@urso same thing here, can you merge this for me, please?

Add more ECS fields to logs (#25998)

96dbe30

(cherry picked from commit bfbadaa)

mergify bot added the backport label Jun 11, 2021

mergify bot assigned jalvz Jun 11, 2021

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 11, 2021

urso merged commit c67e0d4 into 7.x Jun 16, 2021

urso deleted the mergify/bp/7.x/pr-25998 branch June 16, 2021 14:09

sayden mentioned this pull request Jul 1, 2021

Cherry-pick #26266 to 7.14: [7.x](backport #25998) Add more ECS fields to logs #26650

Closed

sayden added the v7.14.0 label Jul 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[7.x](backport #25998) Add more ECS fields to logs #26266

[7.x](backport #25998) Add more ECS fields to logs #26266

Uh oh!

mergify bot commented Jun 11, 2021 •

edited by zube bot

Loading

Uh oh!

botelastic bot commented Jun 11, 2021

Uh oh!

elasticmachine commented Jun 11, 2021

Build stats

Test stats 🧪

Trends 🧪

`Extended / filebeat-windows-2016-windows-2016 / test_rotating_file – filebeat.tests.system.test_registrar.Test`

`filebeat-windows-2016-windows-2016 - mage build unitTest`

`gsutil -m -q cp -a public-read test-build-artifacts-filebeat-windows-2016-windows-2016-tgz gs://beat`

`Error signal`

Test stats 🧪

Genuine test errors

Uh oh!

jalvz commented Jun 14, 2021

Uh oh!

jalvz commented Jun 16, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[7.x](backport #25998) Add more ECS fields to logs #26266

[7.x](backport #25998) Add more ECS fields to logs #26266

Uh oh!

Conversation

mergify bot commented Jun 11, 2021 • edited by zube bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

botelastic bot commented Jun 11, 2021

Uh oh!

elasticmachine commented Jun 11, 2021

💔 Tests Failed

Build stats

Test stats 🧪

Trends 🧪

Test errors

Extended / filebeat-windows-2016-windows-2016 / test_rotating_file – filebeat.tests.system.test_registrar.Test

Steps errors

filebeat-windows-2016-windows-2016 - mage build unitTest

gsutil -m -q cp -a public-read test-build-artifacts-filebeat-windows-2016-windows-2016-tgz gs://beat

Error signal

Log output

🐛 Flaky test report

Test stats 🧪

Genuine test errors

Uh oh!

jalvz commented Jun 14, 2021

Uh oh!

jalvz commented Jun 16, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mergify bot commented Jun 11, 2021 •

edited by zube bot

Loading

`Extended / filebeat-windows-2016-windows-2016 / test_rotating_file – filebeat.tests.system.test_registrar.Test`

`filebeat-windows-2016-windows-2016 - mage build unitTest`

`gsutil -m -q cp -a public-read test-build-artifacts-filebeat-windows-2016-windows-2016-tgz gs://beat`

`Error signal`