[updatecli] update elastic stack version for testing 8.7.1-1bebafa7

[ghost]

Bump elastic-stack to latest snapshot version

Update snapshot.yml

Bump elastic-stack to latest snapshot version

Update snapshot.yml

---

Updatecli options

Most of Updatecli configuration is done via Updatecli manifest.

• If you close this pullrequest, Updatecli will automatically reopen it, the next time it runs.
• If you close this pullrequest, and delete the base branch, Updatecli will automatically recreate it, erasing all previous commits made.

---

Action triggered automatically by Updatecli.

Feel free to report any issues at github.com/updatecli/updatecli.
If you find this tool useful, do not hesitate to star our GitHub repository github.com/updatecli/updatecli as a sign of appreciation.
Or tell us directly on our chat

[elasticmachine]

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-04-07T15:10:02.744+0000

• Duration: 138 min 50 sec

Test stats 🧪

Test	Results
Failed	8
Passed	25961
Skipped	1968
Total	27937

Test errors

Expand to view the tests failures

Build&Test / filebeat-pythonIntegTest / test_fileset_file_070_elasticsearch – filebeat.tests.system.test_modules.Test

Expand to view the error details

 AssertionError: not error expected but got: {'input': {'type': 'log'}, 'agent': {'name': 'beats-ci-immutable-ubuntu-2204-1680880858224167267', 'id': '03e9fb2d-832c-4c20-a4c6-f716985d7e29', 'ephemeral_id': '45e28e0a-e937-4cb5-abeb-af4a92e55b5d', 'type': 'filebeat', 'version': '8.7.1'}, '@timestamp': '2023-04-07T16:08:07.720786159Z', 'ecs': {'version': '1.12.0'}, 'log': {'file': {'path': '/var/lib/jenkins/workspace/PR-34589-25-b1ccb118-95b2-4bce-8276-1a5d2b3ac616/src/github.com/elastic/beats/filebeat/module/elasticsearch/slowlog/test/auditlog_index_indexing_slowlog.log'}, 'offset': 2028, 'flags': ['multiline']}, 'service': {'type': 'elasticsearch'}, 'first_char': '{', 'host': {'name': 'beats-ci-immutable-ubuntu-2204-1680880858224167267'}, 'event': {'ingested': '2023-04-07T16:08:08.748572434Z', 'created': '2023-04-07T16:08:07.720786159Z', 'module': 'elasticsearch', 'dataset': 'elasticsearch.slowlog'}, 'message': '{\n  "@timestamp":"2018-07-04T21:27:30.730Z",\n  "metricset":{\n    "name":"network",\n    "module":"system",\n    "rtt":7264},\n    "system":{\n      "network":{\n        "name":"lo0",\n        "in":{\n          "errors":0,\n          "dropped":0,\n          "bytes":77666873,\n          "packets":244595},\n          "out":{\n            "packets":244595,\n            "bytes":77666873,\n            "errors":0,\n            "dropped":0\n          }\n        }\n      },\n      "beat":{\n        "name":"Rados-MacBook-Pro.local",\n        "hostname":"Rados-MacBook-Pro.local",\n        "version":"6.3.0"\n      },\n      "host":{\n        "name":"Rados-MacBook-Pro.local"\n      }\n    }]', 'fileset': {'name': 'slowlog'}, 'error': {'message': 'cannot access method/field [slowlog] from a null def reference'}}.
   The related error message is: cannot access method/field [slowlog] from a null def reference
assert 'error' not in {'@timestamp': '2023-04-07T16:08:07.720786159Z', 'agent': {'ephemeral_id': '45e28e0a-e937-4cb5-abeb-af4a92e55b5d', 'id...s': {'version': '1.12.0'}, 'error': {'message': 'cannot access method/field [slowlog] from a null def reference'}, ...} 

Expand to view the stacktrace

 a = (<test_modules.Test testMethod=test_fileset_file_070_elasticsearch>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../../../../../python-env/build/ve/linux/lib/python3.10/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:150: in test_fileset_file
    self.run_on_file(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_070_elasticsearch>
module = 'elasticsearch', fileset = 'slowlog'
test_file = '/var/lib/jenkins/workspace/PR-34589-25-b1ccb118-95b2-4bce-8276-1a5d2b3ac616/src/github.com/elastic/beats/filebeat/module/elasticsearch/slowlog/test/auditlog_index_indexing_slowlog.log'
cfgfile = '/var/lib/jenkins/workspace/PR-34589-25-b1ccb118-95b2-4bce-8276-1a5d2b3ac616/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_modules.Test.test_fileset_file_070_elasticsearch962/filebeat.yml'

    def run_on_file(self, module, fileset, test_file, cfgfile):
        print("Testing {}/{} on {}".format(module, fileset, test_file))
    
        self.assert_explicit_ecs_version_set(module, fileset)
    
        try:
            self.es.indices.delete_data_stream(self.index_name)
        except BaseException:
            pass
        self.wait_until(lambda: not self.es.indices.exists(self.index_name))
    
        cmd = [
            self.filebeat, "-systemTest",
            "-e", "-d", "*", "-once",
            "-c", cfgfile,
            "-E", "setup.ilm.enabled=false",
            "-modules={}".format(module),
            "-M", "{module}.*.enabled=false".format(module=module),
            "-M", "{module}.{fileset}.enabled=true".format(
                module=module, fileset=fileset),
            "-M", "{module}.{fileset}.var.input=file".format(
                module=module, fileset=fileset),
            "-M", "{module}.{fileset}.var.paths=[{test_file}]".format(
                module=module, fileset=fileset, test_file=test_file),
            "-M", "*.*.input.close_eof=true",
        ]
        # allow connecting older versions of Elasticsearch
        if os.getenv("TESTING_FILEBEAT_ALLOW_OLDER"):
            cmd.extend(["-E", "output.elasticsearch.allow_older_versions=true"])
    
        # Based on the convention that if a name contains -json the json format is needed. Currently used for LS.
        if "-json" in test_file:
            cmd.append("-M")
            cmd.append("{module}.{fileset}.var.format=json".format(
                module=module, fileset=fileset))
    
        output_path = os.path.join(self.working_dir)
        # Runs inside a with block to ensure file is closed afterwards
        with open(os.path.join(output_path, "output.log"), "ab") as output:
            output.write(bytes(" ".join(cmd) + "\n", "utf-8"))
    
            # Use a fixed timezone so results don't vary depending on the environment
            # Don't use UTC to avoid hiding that non-UTC timezones are not being converted as needed,
            # this can happen because UTC uses to be the default timezone in date parsers when no other
            # timezone is specified.
            local_env = os.environ.copy()
            local_env["TZ"] = 'Etc/GMT+2'
    
            subprocess.Popen(cmd,
                             env=local_env,
                             stdin=None,
                             stdout=output,
                             stderr=subprocess.STDOUT,
                             bufsize=0).wait()
    
        # List of errors to check in filebeat output logs
        errors = ["error loading pipeline for fileset"]
        # Checks if the output of filebeat includes errors
        contains_error, error_line = file_contains(
            os.path.join(output_path, "output.log"), errors)
        assert contains_error is False, "Error found in log:{}".format(
            error_line)
    
        # Make sure index exists
        self.wait_until(lambda: self.es.indices.exists(self.index_name),
                        name="indices present for {}".format(test_file))
    
        self.es.indices.refresh(index=self.index_name)
        # Loads the first 100 events to be checked
        res = self.es.search(index=self.index_name,
                             body={"query": {"match_all": {}}, "size": 100, "sort": {"log.offset": {"order": "asc"}}})
        objects = [o["_source"] for o in res["hits"]["hits"]]
        assert len(objects) > 0
        for obj in objects:
            assert obj["event"]["module"] == module, "expected event.module={} but got {}".format(
                module, obj["event"]["module"])
    
            # All modules must include a set processor that adds the time that
            # the event was ingested to Elasticsearch
            assert "ingested" in obj["event"], "missing event.ingested timestamp"
    
>           assert "error" not in obj, "not error expected but got: {}.\n The related error message is: {}".format(
                obj, obj["error"].get("message"))
E           AssertionError: not error expected but got: {'input': {'type': 'log'}, 'agent': {'name': 'beats-ci-immutable-ubuntu-2204-1680880858224167267', 'id': '03e9fb2d-832c-4c20-a4c6-f716985d7e29', 'ephemeral_id': '45e28e0a-e937-4cb5-abeb-af4a92e55b5d', 'type': 'filebeat', 'version': '8.7.1'}, '@timestamp': '2023-04-07T16:08:07.720786159Z', 'ecs': {'version': '1.12.0'}, 'log': {'file': {'path': '/var/lib/jenkins/workspace/PR-34589-25-b1ccb118-95b2-4bce-8276-1a5d2b3ac616/src/github.com/elastic/beats/filebeat/module/elasticsearch/slowlog/test/auditlog_index_indexing_slowlog.log'}, 'offset': 2028, 'flags': ['multiline']}, 'service': {'type': 'elasticsearch'}, 'first_char': '{', 'host': {'name': 'beats-ci-immutable-ubuntu-2204-1680880858224167267'}, 'event': {'ingested': '2023-04-07T16:08:08.748572434Z', 'created': '2023-04-07T16:08:07.720786159Z', 'module': 'elasticsearch', 'dataset': 'elasticsearch.slowlog'}, 'message': '{\n  "@timestamp":"2018-07-04T21:27:30.730Z",\n  "metricset":{\n    "name":"network",\n    "module":"system",\n    "rtt":7264},\n    "system":{\n      "network":{\n        "name":"lo0",\n        "in":{\n          "errors":0,\n          "dropped":0,\n          "bytes":77666873,\n          "packets":244595},\n          "out":{\n            "packets":244595,\n            "bytes":77666873,\n            "errors":0,\n            "dropped":0\n          }\n        }\n      },\n      "beat":{\n        "name":"Rados-MacBook-Pro.local",\n        "hostname":"Rados-MacBook-Pro.local",\n        "version":"6.3.0"\n      },\n      "host":{\n        "name":"Rados-MacBook-Pro.local"\n      }\n    }]', 'fileset': {'name': 'slowlog'}, 'error': {'message': 'cannot access method/field [slowlog] from a null def reference'}}.
E              The related error message is: cannot access method/field [slowlog] from a null def reference
E           assert 'error' not in {'@timestamp': '2023-04-07T16:08:07.720786159Z', 'agent': {'ephemeral_id': '45e28e0a-e937-4cb5-abeb-af4a92e55b5d', 'id...s': {'version': '1.12.0'}, 'error': {'message': 'cannot access method/field [slowlog] from a null def reference'}, ...}

tests/system/test_modules.py:237: AssertionError 

Build&Test / filebeat-pythonIntegTest / test_fileset_file_074_elasticsearch – filebeat.tests.system.test_modules.Test

Expand to view the error details

 AssertionError: not error expected but got: {'input': {'type': 'log'}, 'agent': {'name': 'beats-ci-immutable-ubuntu-2204-1680880858224167267', 'id': '03e9fb2d-832c-4c20-a4c6-f716985d7e29', 'type': 'filebeat', 'ephemeral_id': '968b3909-ab40-4dee-a8e0-2cc62cfe826e', 'version': '8.7.1'}, '@timestamp': '2023-04-07T16:08:18.435272367Z', 'ecs': {'version': '1.12.0'}, 'log': {'file': {'path': '/var/lib/jenkins/workspace/PR-34589-25-b1ccb118-95b2-4bce-8276-1a5d2b3ac616/src/github.com/elastic/beats/filebeat/module/elasticsearch/slowlog/test/test.log'}, 'offset': 4977, 'flags': ['multiline']}, 'service': {'type': 'elasticsearch'}, 'first_char': '{', 'host': {'name': 'beats-ci-immutable-ubuntu-2204-1680880858224167267'}, 'fileset': {'name': 'slowlog'}, 'event': {'ingested': '2023-04-07T16:08:19.465148732Z', 'created': '2023-04-07T16:08:18.435272367Z', 'module': 'elasticsearch', 'dataset': 'elasticsearch.slowlog'}, 'message': '{\n  "@timestamp":"2018-07-04T21:27:30.730Z",\n  "metricset":{\n    "name":"network",\n    "module":"system",\n    "rtt":7264},\n    "system":{\n      "network":{\n        "name":"lo0",\n        "in":{\n          "errors":0,\n          "dropped":0,\n          "bytes":77666873,\n          "packets":244595},\n          "out":{\n            "packets":244595,\n            "bytes":77666873,\n            "errors":0,\n            "dropped":0\n          }\n        }\n      },\n      "beat":{\n        "name":"Rados-MacBook-Pro.local",\n        "hostname":"Rados-MacBook-Pro.local",\n        "version":"6.3.0"\n      },\n      "host":{\n        "name":"Rados-MacBook-Pro.local"\n      }\n    }]', 'error': {'message': 'cannot access method/field [slowlog] from a null def reference'}}.
   The related error message is: cannot access method/field [slowlog] from a null def reference
assert 'error' not in {'@timestamp': '2023-04-07T16:08:18.435272367Z', 'agent': {'ephemeral_id': '968b3909-ab40-4dee-a8e0-2cc62cfe826e', 'id...s': {'version': '1.12.0'}, 'error': {'message': 'cannot access method/field [slowlog] from a null def reference'}, ...} 

Expand to view the stacktrace

 a = (<test_modules.Test testMethod=test_fileset_file_074_elasticsearch>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../../../../../python-env/build/ve/linux/lib/python3.10/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:150: in test_fileset_file
    self.run_on_file(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_074_elasticsearch>
module = 'elasticsearch', fileset = 'slowlog'
test_file = '/var/lib/jenkins/workspace/PR-34589-25-b1ccb118-95b2-4bce-8276-1a5d2b3ac616/src/github.com/elastic/beats/filebeat/module/elasticsearch/slowlog/test/test.log'
cfgfile = '/var/lib/jenkins/workspace/PR-34589-25-b1ccb118-95b2-4bce-8276-1a5d2b3ac616/src/github.com/elastic/beats/filebeat/build/system-tests/run/test_modules.Test.test_fileset_file_074_elasticsearch446/filebeat.yml'

    def run_on_file(self, module, fileset, test_file, cfgfile):
        print("Testing {}/{} on {}".format(module, fileset, test_file))
    
        self.assert_explicit_ecs_version_set(module, fileset)
    
        try:
            self.es.indices.delete_data_stream(self.index_name)
        except BaseException:
            pass
        self.wait_until(lambda: not self.es.indices.exists(self.index_name))
    
        cmd = [
            self.filebeat, "-systemTest",
            "-e", "-d", "*", "-once",
            "-c", cfgfile,
            "-E", "setup.ilm.enabled=false",
            "-modules={}".format(module),
            "-M", "{module}.*.enabled=false".format(module=module),
            "-M", "{module}.{fileset}.enabled=true".format(
                module=module, fileset=fileset),
            "-M", "{module}.{fileset}.var.input=file".format(
                module=module, fileset=fileset),
            "-M", "{module}.{fileset}.var.paths=[{test_file}]".format(
                module=module, fileset=fileset, test_file=test_file),
            "-M", "*.*.input.close_eof=true",
        ]
        # allow connecting older versions of Elasticsearch
        if os.getenv("TESTING_FILEBEAT_ALLOW_OLDER"):
            cmd.extend(["-E", "output.elasticsearch.allow_older_versions=true"])
    
        # Based on the convention that if a name contains -json the json format is needed. Currently used for LS.
        if "-json" in test_file:
            cmd.append("-M")
            cmd.append("{module}.{fileset}.var.format=json".format(
                module=module, fileset=fileset))
    
        output_path = os.path.join(self.working_dir)
        # Runs inside a with block to ensure file is closed afterwards
        with open(os.path.join(output_path, "output.log"), "ab") as output:
            output.write(bytes(" ".join(cmd) + "\n", "utf-8"))
    
            # Use a fixed timezone so results don't vary depending on the environment
            # Don't use UTC to avoid hiding that non-UTC timezones are not being converted as needed,
            # this can happen because UTC uses to be the default timezone in date parsers when no other
            # timezone is specified.
            local_env = os.environ.copy()
            local_env["TZ"] = 'Etc/GMT+2'
    
            subprocess.Popen(cmd,
                             env=local_env,
                             stdin=None,
                             stdout=output,
                             stderr=subprocess.STDOUT,
                             bufsize=0).wait()
    
        # List of errors to check in filebeat output logs
        errors = ["error loading pipeline for fileset"]
        # Checks if the output of filebeat includes errors
        contains_error, error_line = file_contains(
            os.path.join(output_path, "output.log"), errors)
        assert contains_error is False, "Error found in log:{}".format(
            error_line)
    
        # Make sure index exists
        self.wait_until(lambda: self.es.indices.exists(self.index_name),
                        name="indices present for {}".format(test_file))
    
        self.es.indices.refresh(index=self.index_name)
        # Loads the first 100 events to be checked
        res = self.es.search(index=self.index_name,
                             body={"query": {"match_all": {}}, "size": 100, "sort": {"log.offset": {"order": "asc"}}})
        objects = [o["_source"] for o in res["hits"]["hits"]]
        assert len(objects) > 0
        for obj in objects:
            assert obj["event"]["module"] == module, "expected event.module={} but got {}".format(
                module, obj["event"]["module"])
    
            # All modules must include a set processor that adds the time that
            # the event was ingested to Elasticsearch
            assert "ingested" in obj["event"], "missing event.ingested timestamp"
    
>           assert "error" not in obj, "not error expected but got: {}.\n The related error message is: {}".format(
                obj, obj["error"].get("message"))
E           AssertionError: not error expected but got: {'input': {'type': 'log'}, 'agent': {'name': 'beats-ci-immutable-ubuntu-2204-1680880858224167267', 'id': '03e9fb2d-832c-4c20-a4c6-f716985d7e29', 'type': 'filebeat', 'ephemeral_id': '968b3909-ab40-4dee-a8e0-2cc62cfe826e', 'version': '8.7.1'}, '@timestamp': '2023-04-07T16:08:18.435272367Z', 'ecs': {'version': '1.12.0'}, 'log': {'file': {'path': '/var/lib/jenkins/workspace/PR-34589-25-b1ccb118-95b2-4bce-8276-1a5d2b3ac616/src/github.com/elastic/beats/filebeat/module/elasticsearch/slowlog/test/test.log'}, 'offset': 4977, 'flags': ['multiline']}, 'service': {'type': 'elasticsearch'}, 'first_char': '{', 'host': {'name': 'beats-ci-immutable-ubuntu-2204-1680880858224167267'}, 'fileset': {'name': 'slowlog'}, 'event': {'ingested': '2023-04-07T16:08:19.465148732Z', 'created': '2023-04-07T16:08:18.435272367Z', 'module': 'elasticsearch', 'dataset': 'elasticsearch.slowlog'}, 'message': '{\n  "@timestamp":"2018-07-04T21:27:30.730Z",\n  "metricset":{\n    "name":"network",\n    "module":"system",\n    "rtt":7264},\n    "system":{\n      "network":{\n        "name":"lo0",\n        "in":{\n          "errors":0,\n          "dropped":0,\n          "bytes":77666873,\n          "packets":244595},\n          "out":{\n            "packets":244595,\n            "bytes":77666873,\n            "errors":0,\n            "dropped":0\n          }\n        }\n      },\n      "beat":{\n        "name":"Rados-MacBook-Pro.local",\n        "hostname":"Rados-MacBook-Pro.local",\n        "version":"6.3.0"\n      },\n      "host":{\n        "name":"Rados-MacBook-Pro.local"\n      }\n    }]', 'error': {'message': 'cannot access method/field [slowlog] from a null def reference'}}.
E              The related error message is: cannot access method/field [slowlog] from a null def reference
E           assert 'error' not in {'@timestamp': '2023-04-07T16:08:18.435272367Z', 'agent': {'ephemeral_id': '968b3909-ab40-4dee-a8e0-2cc62cfe826e', 'id...s': {'version': '1.12.0'}, 'error': {'message': 'cannot access method/field [slowlog] from a null def reference'}, ...}

tests/system/test_modules.py:237: AssertionError 

Build&Test / filebeat-pythonIntegTest / test_fileset_file_094_mysql – filebeat.tests.system.test_modules.Test

Expand to view the error details

 AssertionError: The following expected object doesn't match:
   Diff:
  {'type_changes': {"root['mysql.thread_id']": {'old_type': <class 'str'>, 'new_type': <class 'list'>, 'old_value': '16', 'new_value': [16, '16']}}}, full object: 
  {'log.offset': 589, 'log.flags': ['multiline'], 'source.domain': 'localhost', 'fileset.name': 'slowlog', 'input.type': 'log', '@timestamp': '2019-03-24T14:04:51.000Z', 'service.type': 'mysql', 'mysql.thread_id': [16, '16'], 'mysql.slowlog.schema': 'employees', 'mysql.slowlog.read_last': 0, 'mysql.slowlog.read_rnd_next': 301663, 'mysql.slowlog.rows_examined': 3145718, 'mysql.slowlog.read_next': 2844047, 'mysql.slowlog.sort_scan_count': 1, 'mysql.slowlog.read_first': 1, 'mysql.slowlog.query': 'SELECT last_name, MAX(salary) AS salary FROM employees\n    INNER JOIN salaries ON employees.emp_no = salaries.emp_no\n    GROUP BY last_name\n    ORDER BY salary DESC\n    LIMIT 10;', 'mysql.slowlog.sort_merge_passes': 0, 'mysql.slowlog.bytes_received': 0, 'mysql.slowlog.bytes_sent': 312, 'mysql.slowlog.tmp_disk_tables': 0, 'mysql.slowlog.killed': 0, 'mysql.slowlog.lock_time.sec': 0.000145, 'mysql.slowlog.read_rnd': 10, 'mysql.slowlog.rows_sent': 10, 'mysql.slowlog.sort_range_count': 0, 'mysql.slowlog.sort_rows': 10, 'mysql.slowlog.tmp_tables': 1, 'mysql.slowlog.last_errno': 0, 'mysql.slowlog.read_key': 3144072, 'mysql.slowlog.read_prev': 0, 'mysql.slowlog.current_user': 'root', 'event.duration': 2631844000, 'event.kind': 'event', 'event.module': 'mysql', 'event.start': '2019-03-24T14:04:51.082107Z', 'event.end': '2019-03-24T14:04:53.713951Z', 'event.category': ['database'], 'event.type': ['info'], 'event.dataset': 'mysql.slowlog', 'user.name': 'root'}
assert 1 == 0
 +  where 1 = len({'type_changes': {"root['mysql.thread_id']": {'old_type': <class 'str'>, 'new_type': <class 'list'>, 'old_value': '16', 'new_value': [16, '16']}}}) 

Expand to view the stacktrace

 a = (<test_modules.Test testMethod=test_fileset_file_094_mysql>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../../../../../python-env/build/ve/linux/lib/python3.10/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:150: in test_fileset_file
    self.run_on_file(
tests/system/test_modules.py:247: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_094_mysql>
test_file = '/var/lib/jenkins/workspace/PR-34589-25-b1ccb118-95b2-4bce-8276-1a5d2b3ac616/src/github.com/elastic/beats/filebeat/module/mysql/slowlog/test/mysql-ubuntu-8.0.15.log'
objects = [{'@timestamp': '2019-03-24T14:01:45.000Z', 'agent': {'ephemeral_id': '13c9962a-c491-4ea2-81bd-5d57dc4dffd0', 'id': '0...y': ['database'], 'dataset': 'mysql.slowlog', 'duration': 2631844000, 'end': '2019-03-24T14:04:53.713951Z', ...}, ...}]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
                    for key in objects[k].keys():
                        if isinstance(objects[k][key], list) and key not in array_fields_dont_sort:
                            objects[k][key].sort(key=str)
    
                json.dump(objects, f, indent=4, separators=(
                    ',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        # Do not perform a comparison between the resulting and expected documents
        # if the TESTING_FILEBEAT_SKIP_DIFF flag is set.
        #
        # This allows to run a basic check with older versions of ES that can lead
        # to slightly different documents without maintaining multiple sets of
        # golden files.
        if os.getenv("TESTING_FILEBEAT_SKIP_DIFF"):
            return
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(
                d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'type_changes': {"root['mysql.thread_id']": {'old_type': <class 'str'>, 'new_type': <class 'list'>, 'old_value': '16', 'new_value': [16, '16']}}}, full object: 
E             {'log.offset': 589, 'log.flags': ['multiline'], 'source.domain': 'localhost', 'fileset.name': 'slowlog', 'input.type': 'log', '@timestamp': '2019-03-24T14:04:51.000Z', 'service.type': 'mysql', 'mysql.thread_id': [16, '16'], 'mysql.slowlog.schema': 'employees', 'mysql.slowlog.read_last': 0, 'mysql.slowlog.read_rnd_next': 301663, 'mysql.slowlog.rows_examined': 3145718, 'mysql.slowlog.read_next': 2844047, 'mysql.slowlog.sort_scan_count': 1, 'mysql.slowlog.read_first': 1, 'mysql.slowlog.query': 'SELECT last_name, MAX(salary) AS salary FROM employees\n    INNER JOIN salaries ON employees.emp_no = salaries.emp_no\n    GROUP BY last_name\n    ORDER BY salary DESC\n    LIMIT 10;', 'mysql.slowlog.sort_merge_passes': 0, 'mysql.slowlog.bytes_received': 0, 'mysql.slowlog.bytes_sent': 312, 'mysql.slowlog.tmp_disk_tables': 0, 'mysql.slowlog.killed': 0, 'mysql.slowlog.lock_time.sec': 0.000145, 'mysql.slowlog.read_rnd': 10, 'mysql.slowlog.rows_sent': 10, 'mysql.slowlog.sort_range_count': 0, 'mysql.slowlog.sort_rows': 10, 'mysql.slowlog.tmp_tables': 1, 'mysql.slowlog.last_errno': 0, 'mysql.slowlog.read_key': 3144072, 'mysql.slowlog.read_prev': 0, 'mysql.slowlog.current_user': 'root', 'event.duration': 2631844000, 'event.kind': 'event', 'event.module': 'mysql', 'event.start': '2019-03-24T14:04:51.082107Z', 'event.end': '2019-03-24T14:04:53.713951Z', 'event.category': ['database'], 'event.type': ['info'], 'event.dataset': 'mysql.slowlog', 'user.name': 'root'}
E           assert 1 == 0
E            +  where 1 = len({'type_changes': {"root['mysql.thread_id']": {'old_type': <class 'str'>, 'new_type': <class 'list'>, 'old_value': '16', 'new_value': [16, '16']}}})

tests/system/test_modules.py:292: AssertionError 

Build&Test / filebeat-pythonIntegTest / test_fileset_file_098_mysql – filebeat.tests.system.test_modules.Test

Expand to view the error details

 AssertionError: The following expected object doesn't match:
   Diff:
  {'type_changes': {"root['mysql.slowlog.schema']": {'old_type': <class 'str'>, 'new_type': <class 'list'>, 'old_value': 'employees-test', 'new_value': ['employees-test', 'employees-test']}}}, full object: 
  {'log.offset': 230, 'log.flags': ['multiline'], 'source.domain': 'localhost', 'fileset.name': 'slowlog', 'input.type': 'log', '@timestamp': '2019-03-24T16:03:00.000Z', 'service.type': 'mysql', 'mysql.thread_id': '37', 'mysql.slowlog.schema': ['employees-test', 'employees-test'], 'mysql.slowlog.tmp_table_sizes': 4026528, 'mysql.slowlog.rows_examined': 3145718, 'mysql.slowlog.priority_queue': True, 'mysql.slowlog.full_scan': True, 'mysql.slowlog.query': 'SELECT last_name, MAX(salary) AS salary FROM employees\n    INNER JOIN salaries ON employees.emp_no = salaries.emp_no\n    GROUP BY last_name\n    ORDER BY salary DESC\n    LIMIT 10;', 'mysql.slowlog.merge_passes': 0, 'mysql.slowlog.filesort': True, 'mysql.slowlog.bytes_sent': 319, 'mysql.slowlog.tmp_disk_tables': '0', 'mysql.slowlog.tmp_table': True, 'mysql.slowlog.lock_time.sec': 0.000196, 'mysql.slowlog.rows_sent': 10, 'mysql.slowlog.rows_affected': 0, 'mysql.slowlog.filesort_on_disk': False, 'mysql.slowlog.tmp_tables': 1, 'mysql.slowlog.query_cache_hit': False, 'mysql.slowlog.full_join': False, 'mysql.slowlog.tmp_table_on_disk': False, 'mysql.slowlog.current_user': 'root', 'event.duration': 2461578000, 'event.kind': 'event', 'event.module': 'mysql', 'event.category': ['database'], 'event.type': ['info'], 'event.dataset': 'mysql.slowlog', 'user.name': 'root'}
assert 1 == 0
 +  where 1 = len({'type_changes': {"root['mysql.slowlog.schema']": {'old_type': <class 'str'>, 'new_type': <class 'list'>, 'old_value': 'employees-test', 'new_value': ['employees-test', 'employees-test']}}}) 

Expand to view the stacktrace

 a = (<test_modules.Test testMethod=test_fileset_file_098_mysql>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../../../../../python-env/build/ve/linux/lib/python3.10/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:150: in test_fileset_file
    self.run_on_file(
tests/system/test_modules.py:247: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_098_mysql>
test_file = '/var/lib/jenkins/workspace/PR-34589-25-b1ccb118-95b2-4bce-8276-1a5d2b3ac616/src/github.com/elastic/beats/filebeat/module/mysql/slowlog/test/mariadb-10.3.13.log'
objects = [{'@timestamp': '2019-03-24T16:03:00.000Z', 'agent': {'ephemeral_id': '9a6ab88b-7cb4-4b6b-a74f-3bb76954a093', 'id': '0...tabase'], 'dataset': 'mysql.slowlog', 'duration': 2461578000, 'ingested': '2023-04-07T16:09:27.337536255Z', ...}, ...}]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
                    for key in objects[k].keys():
                        if isinstance(objects[k][key], list) and key not in array_fields_dont_sort:
                            objects[k][key].sort(key=str)
    
                json.dump(objects, f, indent=4, separators=(
                    ',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        # Do not perform a comparison between the resulting and expected documents
        # if the TESTING_FILEBEAT_SKIP_DIFF flag is set.
        #
        # This allows to run a basic check with older versions of ES that can lead
        # to slightly different documents without maintaining multiple sets of
        # golden files.
        if os.getenv("TESTING_FILEBEAT_SKIP_DIFF"):
            return
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(
                d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'type_changes': {"root['mysql.slowlog.schema']": {'old_type': <class 'str'>, 'new_type': <class 'list'>, 'old_value': 'employees-test', 'new_value': ['employees-test', 'employees-test']}}}, full object: 
E             {'log.offset': 230, 'log.flags': ['multiline'], 'source.domain': 'localhost', 'fileset.name': 'slowlog', 'input.type': 'log', '@timestamp': '2019-03-24T16:03:00.000Z', 'service.type': 'mysql', 'mysql.thread_id': '37', 'mysql.slowlog.schema': ['employees-test', 'employees-test'], 'mysql.slowlog.tmp_table_sizes': 4026528, 'mysql.slowlog.rows_examined': 3145718, 'mysql.slowlog.priority_queue': True, 'mysql.slowlog.full_scan': True, 'mysql.slowlog.query': 'SELECT last_name, MAX(salary) AS salary FROM employees\n    INNER JOIN salaries ON employees.emp_no = salaries.emp_no\n    GROUP BY last_name\n    ORDER BY salary DESC\n    LIMIT 10;', 'mysql.slowlog.merge_passes': 0, 'mysql.slowlog.filesort': True, 'mysql.slowlog.bytes_sent': 319, 'mysql.slowlog.tmp_disk_tables': '0', 'mysql.slowlog.tmp_table': True, 'mysql.slowlog.lock_time.sec': 0.000196, 'mysql.slowlog.rows_sent': 10, 'mysql.slowlog.rows_affected': 0, 'mysql.slowlog.filesort_on_disk': False, 'mysql.slowlog.tmp_tables': 1, 'mysql.slowlog.query_cache_hit': False, 'mysql.slowlog.full_join': False, 'mysql.slowlog.tmp_table_on_disk': False, 'mysql.slowlog.current_user': 'root', 'event.duration': 2461578000, 'event.kind': 'event', 'event.module': 'mysql', 'event.category': ['database'], 'event.type': ['info'], 'event.dataset': 'mysql.slowlog', 'user.name': 'root'}
E           assert 1 == 0
E            +  where 1 = len({'type_changes': {"root['mysql.slowlog.schema']": {'old_type': <class 'str'>, 'new_type': <class 'list'>, 'old_value': 'employees-test', 'new_value': ['employees-test', 'employees-test']}}})

tests/system/test_modules.py:292: AssertionError 

Build&Test / filebeat-pythonIntegTest / test_fileset_file_104_mysql – filebeat.tests.system.test_modules.Test

Expand to view the error details

 AssertionError: The following expected object doesn't match:
   Diff:
  {'type_changes': {"root['mysql.slowlog.schema']": {'old_type': <class 'str'>, 'new_type': <class 'list'>, 'old_value': 'employees', 'new_value': ['employees', 'employees']}}}, full object: 
  {'log.offset': 244, 'log.flags': ['multiline'], 'source.domain': 'localhost', 'fileset.name': 'slowlog', 'input.type': 'log', '@timestamp': '2019-03-24T16:22:41.000Z', 'service.type': 'mysql', 'mysql.thread_id': 182, 'mysql.slowlog.schema': ['employees', 'employees'], 'mysql.slowlog.lock_time.sec': 0.000138, 'mysql.slowlog.rows_affected': 0, 'mysql.slowlog.rows_sent': 10, 'mysql.slowlog.rows_examined': 3145718, 'mysql.slowlog.query': "SELECT last_name, MAX(salary) AS salary FROM employees     INNER JOIN salaries ON employees.emp_no = salaries.emp_no     GROUP BY last_name     ORDER BY salary DESC     LIMIT 10;\n/usr/sbin/mysqld, Version: 8.0.15-5 (Percona Server (GPL), Release '5', Revision 'f8a9e99'). started with:\nTcp port: 0  Unix socket: /var/run/mysqld/mysqld.sock\nTime                 Id Command    Argument", 'mysql.slowlog.last_errno': 0, 'mysql.slowlog.killed': 0, 'mysql.slowlog.bytes_sent': 312, 'mysql.slowlog.current_user': 'root', 'event.duration': 2746607000, 'event.kind': 'event', 'event.module': 'mysql', 'event.category': ['database'], 'event.type': ['info'], 'event.dataset': 'mysql.slowlog', 'user.name': 'root'}
assert 1 == 0
 +  where 1 = len({'type_changes': {"root['mysql.slowlog.schema']": {'old_type': <class 'str'>, 'new_type': <class 'list'>, 'old_value': 'employees', 'new_value': ['employees', 'employees']}}}) 

Expand to view the stacktrace

 a = (<test_modules.Test testMethod=test_fileset_file_104_mysql>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../../../../../python-env/build/ve/linux/lib/python3.10/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:150: in test_fileset_file
    self.run_on_file(
tests/system/test_modules.py:247: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_104_mysql>
test_file = '/var/lib/jenkins/workspace/PR-34589-25-b1ccb118-95b2-4bce-8276-1a5d2b3ac616/src/github.com/elastic/beats/filebeat/module/mysql/slowlog/test/percona-ubuntu-8.0.15.log'
objects = [{'@timestamp': '2019-03-24T16:22:41.000Z', 'agent': {'ephemeral_id': '7d6fe736-0213-452d-9e0b-1d503f14cb2b', 'id': '0...tabase'], 'dataset': 'mysql.slowlog', 'duration': 3133066000, 'ingested': '2023-04-07T16:09:43.525904793Z', ...}, ...}]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
                    for key in objects[k].keys():
                        if isinstance(objects[k][key], list) and key not in array_fields_dont_sort:
                            objects[k][key].sort(key=str)
    
                json.dump(objects, f, indent=4, separators=(
                    ',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        # Do not perform a comparison between the resulting and expected documents
        # if the TESTING_FILEBEAT_SKIP_DIFF flag is set.
        #
        # This allows to run a basic check with older versions of ES that can lead
        # to slightly different documents without maintaining multiple sets of
        # golden files.
        if os.getenv("TESTING_FILEBEAT_SKIP_DIFF"):
            return
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(
                d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'type_changes': {"root['mysql.slowlog.schema']": {'old_type': <class 'str'>, 'new_type': <class 'list'>, 'old_value': 'employees', 'new_value': ['employees', 'employees']}}}, full object: 
E             {'log.offset': 244, 'log.flags': ['multiline'], 'source.domain': 'localhost', 'fileset.name': 'slowlog', 'input.type': 'log', '@timestamp': '2019-03-24T16:22:41.000Z', 'service.type': 'mysql', 'mysql.thread_id': 182, 'mysql.slowlog.schema': ['employees', 'employees'], 'mysql.slowlog.lock_time.sec': 0.000138, 'mysql.slowlog.rows_affected': 0, 'mysql.slowlog.rows_sent': 10, 'mysql.slowlog.rows_examined': 3145718, 'mysql.slowlog.query': "SELECT last_name, MAX(salary) AS salary FROM employees     INNER JOIN salaries ON employees.emp_no = salaries.emp_no     GROUP BY last_name     ORDER BY salary DESC     LIMIT 10;\n/usr/sbin/mysqld, Version: 8.0.15-5 (Percona Server (GPL), Release '5', Revision 'f8a9e99'). started with:\nTcp port: 0  Unix socket: /var/run/mysqld/mysqld.sock\nTime                 Id Command    Argument", 'mysql.slowlog.last_errno': 0, 'mysql.slowlog.killed': 0, 'mysql.slowlog.bytes_sent': 312, 'mysql.slowlog.current_user': 'root', 'event.duration': 2746607000, 'event.kind': 'event', 'event.module': 'mysql', 'event.category': ['database'], 'event.type': ['info'], 'event.dataset': 'mysql.slowlog', 'user.name': 'root'}
E           assert 1 == 0
E            +  where 1 = len({'type_changes': {"root['mysql.slowlog.schema']": {'old_type': <class 'str'>, 'new_type': <class 'list'>, 'old_value': 'employees', 'new_value': ['employees', 'employees']}}})

tests/system/test_modules.py:292: AssertionError 

Build&Test / filebeat-pythonIntegTest / test_fileset_file_108_iis – filebeat.tests.system.test_modules.Test

Expand to view the error details

 AssertionError: The following expected object doesn't match:
   Diff:
  {'type_changes': {"root['source.address']": {'old_type': <class 'str'>, 'new_type': <class 'list'>, 'old_value': '10.100.118.31', 'new_value': ['10.100.118.31', '10.100.118.31']}, "root['destination.address']": {'old_type': <class 'str'>, 'new_type': <class 'list'>, 'old_value': '10.100.220.70', 'new_value': ['10.100.220.70', '10.100.220.70']}}, 'dictionary_item_removed': [root['destination.ip'], root['event.type'], root['source.ip'], root['related.ip']], 'iterable_item_removed': {"root['event.category'][0]": 'network'}}, full object: 
  {'log.offset': 244, 'destination.address': ['10.100.220.70', '10.100.220.70'], 'destination.port': 80, 'source.address': ['10.100.118.31', '10.100.118.31'], 'fileset.name': 'access', 'url.path': '/', 'url.original': '/', 'input.type': 'log', 'iis.access.sub_status': 4, 'iis.access.win32_status': 2, '@timestamp': '2018-08-28T18:24:25.000Z', 'service.type': 'iis', 'http.request.method': 'GET', 'http.response.status_code': 404, 'event.duration': 792000000, 'event.original': '2018-08-28 18:24:25 [10.100.220.70](http://10.100.220.70) GET / - 80 - [10.100.118.31](http://10.100.118.31) Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.3;+WOW64;+Trident/7.0;+.NET4.0E;+.NET4.0C;+.NET+CLR+3.5.30729;+.NET+CLR[+2.0.50727](tel:+2050727);+.NET+CLR+3.0.30729) 404 4 2 792', 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web'], 'event.dataset': 'iis.access', 'event.outcome': 'failure', 'user_agent.original': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.3; WOW64; Trident/7.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR[ 2.0.50727](tel: 2050727); .NET CLR 3.0.30729)', 'user_agent.os.name': 'Windows', 'user_agent.os.version': '8.1', 'user_agent.os.full': 'Windows 8.1', 'user_agent.name': 'IE', 'user_agent.device.name': 'Other', 'user_agent.version': '11.0'}
assert 3 == 0
 +  where 3 = len({'type_changes': {"root['source.address']": {'old_type': <class 'str'>, 'new_type': <class 'list'>, 'old_value': '10.1...vent.type'], root['source.ip'], root['related.ip']], 'iterable_item_removed': {"root['event.category'][0]": 'network'}}) 

Expand to view the stacktrace

 a = (<test_modules.Test testMethod=test_fileset_file_108_iis>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../../../../../python-env/build/ve/linux/lib/python3.10/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:150: in test_fileset_file
    self.run_on_file(
tests/system/test_modules.py:247: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_108_iis>
test_file = '/var/lib/jenkins/workspace/PR-34589-25-b1ccb118-95b2-4bce-8276-1a5d2b3ac616/src/github.com/elastic/beats/filebeat/module/iis/access/test/test-iis-7.5.log'
objects = [{'@timestamp': '2018-08-28T18:24:25.000Z', 'agent': {'ephemeral_id': '3fa25f81-8674-4ff8-ab2d-e080614defc8', 'id': '0...0000:0000:3257:9652', 'ip': '2001:cdba:0000:0000:0000:0000:3257:9652', 'port': 80}, 'ecs': {'version': '1.12.0'}, ...}]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
                    for key in objects[k].keys():
                        if isinstance(objects[k][key], list) and key not in array_fields_dont_sort:
                            objects[k][key].sort(key=str)
    
                json.dump(objects, f, indent=4, separators=(
                    ',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        # Do not perform a comparison between the resulting and expected documents
        # if the TESTING_FILEBEAT_SKIP_DIFF flag is set.
        #
        # This allows to run a basic check with older versions of ES that can lead
        # to slightly different documents without maintaining multiple sets of
        # golden files.
        if os.getenv("TESTING_FILEBEAT_SKIP_DIFF"):
            return
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(
                d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'type_changes': {"root['source.address']": {'old_type': <class 'str'>, 'new_type': <class 'list'>, 'old_value': '10.100.118.31', 'new_value': ['10.100.118.31', '10.100.118.31']}, "root['destination.address']": {'old_type': <class 'str'>, 'new_type': <class 'list'>, 'old_value': '10.100.220.70', 'new_value': ['10.100.220.70', '10.100.220.70']}}, 'dictionary_item_removed': [root['destination.ip'], root['event.type'], root['source.ip'], root['related.ip']], 'iterable_item_removed': {"root['event.category'][0]": 'network'}}, full object: 
E             {'log.offset': 244, 'destination.address': ['10.100.220.70', '10.100.220.70'], 'destination.port': 80, 'source.address': ['10.100.118.31', '10.100.118.31'], 'fileset.name': 'access', 'url.path': '/', 'url.original': '/', 'input.type': 'log', 'iis.access.sub_status': 4, 'iis.access.win32_status': 2, '@timestamp': '2018-08-28T18:24:25.000Z', 'service.type': 'iis', 'http.request.method': 'GET', 'http.response.status_code': 404, 'event.duration': 792000000, 'event.original': '2018-08-28 18:24:25 [10.100.220.70](http://10.100.220.70) GET / - 80 - [10.100.118.31](http://10.100.118.31) Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.3;+WOW64;+Trident/7.0;+.NET4.0E;+.NET4.0C;+.NET+CLR+3.5.30729;+.NET+CLR[+2.0.50727](tel:+2050727);+.NET+CLR+3.0.30729) 404 4 2 792', 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web'], 'event.dataset': 'iis.access', 'event.outcome': 'failure', 'user_agent.original': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.3; WOW64; Trident/7.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR[ 2.0.50727](tel: 2050727); .NET CLR 3.0.30729)', 'user_agent.os.name': 'Windows', 'user_agent.os.version': '8.1', 'user_agent.os.full': 'Windows 8.1', 'user_agent.name': 'IE', 'user_agent.device.name': 'Other', 'user_agent.version': '11.0'}
E           assert 3 == 0
E            +  where 3 = len({'type_changes': {"root['source.address']": {'old_type': <class 'str'>, 'new_type': <class 'list'>, 'old_value': '10.1...vent.type'], root['source.ip'], root['related.ip']], 'iterable_item_removed': {"root['event.category'][0]": 'network'}})

tests/system/test_modules.py:292: AssertionError 

Build&Test / x-pack/filebeat-pythonIntegTest / test_fileset_file_012_envoyproxy – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest

Expand to view the error details

 AssertionError: not error expected but got: {'input': {'type': 'log'}, 'agent': {'name': 'beats-ci-immutable-ubuntu-2204-1680880858408932028', 'id': 'e8e9c659-0cd4-47e0-b7a5-6a78f5e21b82', 'type': 'filebeat', 'ephemeral_id': '07c9f9da-af82-4fa0-bd38-e3f202813dae', 'version': '8.7.1'}, '@timestamp': '2023-04-07T16:49:43.025487619Z', 'ecs': {'version': '1.12.0'}, 'log': {'file': {'path': '/var/lib/jenkins/workspace/PR-34589-25-d8af0c8a-84a0-49e7-b4fe-baca69949804/src/github.com/elastic/beats/x-pack/filebeat/module/envoyproxy/log/test/envoy-json.log'}, 'offset': 518}, 'service': {'type': 'envoyproxy'}, 'host': {'name': 'beats-ci-immutable-ubuntu-2204-1680880858408932028'}, 'message': '{"message":"ACCESS [2019-04-06T06:20:05.972Z] \\"- - -\\" 0 UF,URX 0 0 0 - \\"-\\" \\"-\\" \\"-\\" \\"-\\" \\"127.0.0.1:9200\\"","stream":"stdout","time":"2019-02-12T18:37:43.139620629Z", "kubernetes": { "container": { "name": "ambassador" }, "node": { "name": "minikube" }, "pod": { "uid": "e57d545e-2a9d-11e9-995f-08002730e0dc", "name": "ambassador-76c58d9df4-jwhsg" }, "namespace": "default", "labels": { "service": "ambassador" }}}}', 'event': {'ingested': '2023-04-07T16:49:44.403508952Z', 'created': '2023-04-07T16:49:43.025487619Z', 'kind': 'event', 'module': 'envoyproxy', 'category': ['network'], 'dataset': 'envoyproxy.log', 'outcome': ['success']}, 'fileset': {'name': 'log'}, 'error': {'message': "pipeline-json: Unexpected close marker '}': expected ']' (for root starting at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 0])\\n at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 425]"}, 'tags': ['envoyproxy']}.
   The related error message is: pipeline-json: Unexpected close marker '}': expected ']' (for root starting at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 0])\n at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 425]
assert 'error' not in {'@timestamp': '2023-04-07T16:49:43.025487619Z', 'agent': {'ephemeral_id': '07c9f9da-af82-4fa0-bd38-e3f202813dae', 'id...1, column: 0])\\n at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 425]"}, ...} 

Expand to view the stacktrace

 a = (<test_xpack_modules.XPackTest testMethod=test_fileset_file_012_envoyproxy>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../../../../../../python-env/build/ve/linux/lib/python3.10/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../filebeat/tests/system/test_modules.py:150: in test_fileset_file
    self.run_on_file(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_xpack_modules.XPackTest testMethod=test_fileset_file_012_envoyproxy>
module = 'envoyproxy', fileset = 'log'
test_file = '/var/lib/jenkins/workspace/PR-34589-25-d8af0c8a-84a0-49e7-b4fe-baca69949804/src/github.com/elastic/beats/x-pack/filebeat/module/envoyproxy/log/test/envoy-json.log'
cfgfile = '/var/lib/jenkins/workspace/PR-34589-25-d8af0c8a-84a0-49e7-b4fe-baca69949804/src/github.com/elastic/beats/x-pack/filebeat/build/system-tests/run/test_xpack_modules.XPackTest.test_fileset_file_012_envoyproxy545/filebeat.yml'

    def run_on_file(self, module, fileset, test_file, cfgfile):
        print("Testing {}/{} on {}".format(module, fileset, test_file))
    
        self.assert_explicit_ecs_version_set(module, fileset)
    
        try:
            self.es.indices.delete_data_stream(self.index_name)
        except BaseException:
            pass
        self.wait_until(lambda: not self.es.indices.exists(self.index_name))
    
        cmd = [
            self.filebeat, "-systemTest",
            "-e", "-d", "*", "-once",
            "-c", cfgfile,
            "-E", "setup.ilm.enabled=false",
            "-modules={}".format(module),
            "-M", "{module}.*.enabled=false".format(module=module),
            "-M", "{module}.{fileset}.enabled=true".format(
                module=module, fileset=fileset),
            "-M", "{module}.{fileset}.var.input=file".format(
                module=module, fileset=fileset),
            "-M", "{module}.{fileset}.var.paths=[{test_file}]".format(
                module=module, fileset=fileset, test_file=test_file),
            "-M", "*.*.input.close_eof=true",
        ]
        # allow connecting older versions of Elasticsearch
        if os.getenv("TESTING_FILEBEAT_ALLOW_OLDER"):
            cmd.extend(["-E", "output.elasticsearch.allow_older_versions=true"])
    
        # Based on the convention that if a name contains -json the json format is needed. Currently used for LS.
        if "-json" in test_file:
            cmd.append("-M")
            cmd.append("{module}.{fileset}.var.format=json".format(
                module=module, fileset=fileset))
    
        output_path = os.path.join(self.working_dir)
        # Runs inside a with block to ensure file is closed afterwards
        with open(os.path.join(output_path, "output.log"), "ab") as output:
            output.write(bytes(" ".join(cmd) + "\n", "utf-8"))
    
            # Use a fixed timezone so results don't vary depending on the environment
            # Don't use UTC to avoid hiding that non-UTC timezones are not being converted as needed,
            # this can happen because UTC uses to be the default timezone in date parsers when no other
            # timezone is specified.
            local_env = os.environ.copy()
            local_env["TZ"] = 'Etc/GMT+2'
    
            subprocess.Popen(cmd,
                             env=local_env,
                             stdin=None,
                             stdout=output,
                             stderr=subprocess.STDOUT,
                             bufsize=0).wait()
    
        # List of errors to check in filebeat output logs
        errors = ["error loading pipeline for fileset"]
        # Checks if the output of filebeat includes errors
        contains_error, error_line = file_contains(
            os.path.join(output_path, "output.log"), errors)
        assert contains_error is False, "Error found in log:{}".format(
            error_line)
    
        # Make sure index exists
        self.wait_until(lambda: self.es.indices.exists(self.index_name),
                        name="indices present for {}".format(test_file))
    
        self.es.indices.refresh(index=self.index_name)
        # Loads the first 100 events to be checked
        res = self.es.search(index=self.index_name,
                             body={"query": {"match_all": {}}, "size": 100, "sort": {"log.offset": {"order": "asc"}}})
        objects = [o["_source"] for o in res["hits"]["hits"]]
        assert len(objects) > 0
        for obj in objects:
            assert obj["event"]["module"] == module, "expected event.module={} but got {}".format(
                module, obj["event"]["module"])
    
            # All modules must include a set processor that adds the time that
            # the event was ingested to Elasticsearch
            assert "ingested" in obj["event"], "missing event.ingested timestamp"
    
>           assert "error" not in obj, "not error expected but got: {}.\n The related error message is: {}".format(
                obj, obj["error"].get("message"))
E           AssertionError: not error expected but got: {'input': {'type': 'log'}, 'agent': {'name': 'beats-ci-immutable-ubuntu-2204-1680880858408932028', 'id': 'e8e9c659-0cd4-47e0-b7a5-6a78f5e21b82', 'type': 'filebeat', 'ephemeral_id': '07c9f9da-af82-4fa0-bd38-e3f202813dae', 'version': '8.7.1'}, '@timestamp': '2023-04-07T16:49:43.025487619Z', 'ecs': {'version': '1.12.0'}, 'log': {'file': {'path': '/var/lib/jenkins/workspace/PR-34589-25-d8af0c8a-84a0-49e7-b4fe-baca69949804/src/github.com/elastic/beats/x-pack/filebeat/module/envoyproxy/log/test/envoy-json.log'}, 'offset': 518}, 'service': {'type': 'envoyproxy'}, 'host': {'name': 'beats-ci-immutable-ubuntu-2204-1680880858408932028'}, 'message': '{"message":"ACCESS [2019-04-06T06:20:05.972Z] \\"- - -\\" 0 UF,URX 0 0 0 - \\"-\\" \\"-\\" \\"-\\" \\"-\\" \\"127.0.0.1:9200\\"","stream":"stdout","time":"2019-02-12T18:37:43.139620629Z", "kubernetes": { "container": { "name": "ambassador" }, "node": { "name": "minikube" }, "pod": { "uid": "e57d545e-2a9d-11e9-995f-08002730e0dc", "name": "ambassador-76c58d9df4-jwhsg" }, "namespace": "default", "labels": { "service": "ambassador" }}}}', 'event': {'ingested': '2023-04-07T16:49:44.403508952Z', 'created': '2023-04-07T16:49:43.025487619Z', 'kind': 'event', 'module': 'envoyproxy', 'category': ['network'], 'dataset': 'envoyproxy.log', 'outcome': ['success']}, 'fileset': {'name': 'log'}, 'error': {'message': "pipeline-json: Unexpected close marker '}': expected ']' (for root starting at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 0])\\n at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 425]"}, 'tags': ['envoyproxy']}.
E              The related error message is: pipeline-json: Unexpected close marker '}': expected ']' (for root starting at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 0])\n at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 425]
E           assert 'error' not in {'@timestamp': '2023-04-07T16:49:43.025487619Z', 'agent': {'ephemeral_id': '07c9f9da-af82-4fa0-bd38-e3f202813dae', 'id...1, column: 0])\\n at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 425]"}, ...}

../../filebeat/tests/system/test_modules.py:237: AssertionError 

Build&Test / x-pack/filebeat-pythonIntegTest / test_fileset_file_146_mysqlenterprise – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest

Expand to view the error details

 AssertionError: not error expected but got: {'input': {'type': 'log'}, 'agent': {'name': 'beats-ci-immutable-ubuntu-2204-1680880858408932028', 'id': 'e8e9c659-0cd4-47e0-b7a5-6a78f5e21b82', 'ephemeral_id': 'aef6124d-1cc3-4c76-9d35-1b4582ee5c03', 'type': 'filebeat', 'version': '8.7.1'}, '@timestamp': '2023-04-07T16:59:22.808936609Z', 'ecs': {'version': '1.12.0'}, 'log': {'file': {'path': '/var/lib/jenkins/workspace/PR-34589-25-d8af0c8a-84a0-49e7-b4fe-baca69949804/src/github.com/elastic/beats/x-pack/filebeat/module/mysqlenterprise/audit/test/mysql_audit_test.log'}, 'offset': 0}, 'service': {'type': 'mysqlenterprise'}, 'host': {'name': 'beats-ci-immutable-ubuntu-2204-1680880858408932028'}, 'event': {'ingested': '2023-04-07T16:59:23.936903163Z', 'timezone': '-02:00', 'module': 'mysqlenterprise', 'dataset': 'mysqlenterprise.audit'}, 'fileset': {'name': 'audit'}, 'message': '{ "timestamp": "2020-10-19 19:21:33", "id": 0, "class": "audit", "event": "startup", "connection_id": 0, "account": { "user": "skip-grants user", "host": "" }, "login": { "user": "", "os": "", "ip": "", "proxy": "" }, "startup_data": { "server_id": 1, "os_version": "x86_64-Linux", "mysql_version": "8.0.22-commercial", "args": ["/usr/local/mysql/bin/mysqld", "--loose-audit-log-format=JSON", "--log-error=log.err", "--pid-file=mysqld.pid", "--port=3306" ] } },', 'error': {'message': "Unexpected character (',' (code 44)): expected a value\\n at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 462]"}, 'tags': ['mysqlenterprise-audit']}.
   The related error message is: Unexpected character (',' (code 44)): expected a value\n at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 462]
assert 'error' not in {'@timestamp': '2023-04-07T16:59:22.808936609Z', 'agent': {'ephemeral_id': 'aef6124d-1cc3-4c76-9d35-1b4582ee5c03', 'id...pected a value\\n at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 462]"}, ...} 

Expand to view the stacktrace

 a = (<test_xpack_modules.XPackTest testMethod=test_fileset_file_146_mysqlenterprise>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../../../../../../python-env/build/ve/linux/lib/python3.10/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../filebeat/tests/system/test_modules.py:150: in test_fileset_file
    self.run_on_file(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_xpack_modules.XPackTest testMethod=test_fileset_file_146_mysqlenterprise>
module = 'mysqlenterprise', fileset = 'audit'
test_file = '/var/lib/jenkins/workspace/PR-34589-25-d8af0c8a-84a0-49e7-b4fe-baca69949804/src/github.com/elastic/beats/x-pack/filebeat/module/mysqlenterprise/audit/test/mysql_audit_test.log'
cfgfile = '/var/lib/jenkins/workspace/PR-34589-25-d8af0c8a-84a0-49e7-b4fe-baca69949804/src/github.com/elastic/beats/x-pack/filebeat/build/system-tests/run/test_xpack_modules.XPackTest.test_fileset_file_146_mysqlenterprise940/filebeat.yml'

    def run_on_file(self, module, fileset, test_file, cfgfile):
        print("Testing {}/{} on {}".format(module, fileset, test_file))
    
        self.assert_explicit_ecs_version_set(module, fileset)
    
        try:
            self.es.indices.delete_data_stream(self.index_name)
        except BaseException:
            pass
        self.wait_until(lambda: not self.es.indices.exists(self.index_name))
    
        cmd = [
            self.filebeat, "-systemTest",
            "-e", "-d", "*", "-once",
            "-c", cfgfile,
            "-E", "setup.ilm.enabled=false",
            "-modules={}".format(module),
            "-M", "{module}.*.enabled=false".format(module=module),
            "-M", "{module}.{fileset}.enabled=true".format(
                module=module, fileset=fileset),
            "-M", "{module}.{fileset}.var.input=file".format(
                module=module, fileset=fileset),
            "-M", "{module}.{fileset}.var.paths=[{test_file}]".format(
                module=module, fileset=fileset, test_file=test_file),
            "-M", "*.*.input.close_eof=true",
        ]
        # allow connecting older versions of Elasticsearch
        if os.getenv("TESTING_FILEBEAT_ALLOW_OLDER"):
            cmd.extend(["-E", "output.elasticsearch.allow_older_versions=true"])
    
        # Based on the convention that if a name contains -json the json format is needed. Currently used for LS.
        if "-json" in test_file:
            cmd.append("-M")
            cmd.append("{module}.{fileset}.var.format=json".format(
                module=module, fileset=fileset))
    
        output_path = os.path.join(self.working_dir)
        # Runs inside a with block to ensure file is closed afterwards
        with open(os.path.join(output_path, "output.log"), "ab") as output:
            output.write(bytes(" ".join(cmd) + "\n", "utf-8"))
    
            # Use a fixed timezone so results don't vary depending on the environment
            # Don't use UTC to avoid hiding that non-UTC timezones are not being converted as needed,
            # this can happen because UTC uses to be the default timezone in date parsers when no other
            # timezone is specified.
            local_env = os.environ.copy()
            local_env["TZ"] = 'Etc/GMT+2'
    
            subprocess.Popen(cmd,
                             env=local_env,
                             stdin=None,
                             stdout=output,
                             stderr=subprocess.STDOUT,
                             bufsize=0).wait()
    
        # List of errors to check in filebeat output logs
        errors = ["error loading pipeline for fileset"]
        # Checks if the output of filebeat includes errors
        contains_error, error_line = file_contains(
            os.path.join(output_path, "output.log"), errors)
        assert contains_error is False, "Error found in log:{}".format(
            error_line)
    
        # Make sure index exists
        self.wait_until(lambda: self.es.indices.exists(self.index_name),
                        name="indices present for {}".format(test_file))
    
        self.es.indices.refresh(index=self.index_name)
        # Loads the first 100 events to be checked
        res = self.es.search(index=self.index_name,
                             body={"query": {"match_all": {}}, "size": 100, "sort": {"log.offset": {"order": "asc"}}})
        objects = [o["_source"] for o in res["hits"]["hits"]]
        assert len(objects) > 0
        for obj in objects:
            assert obj["event"]["module"] == module, "expected event.module={} but got {}".format(
                module, obj["event"]["module"])
    
            # All modules must include a set processor that adds the time that
            # the event was ingested to Elasticsearch
            assert "ingested" in obj["event"], "missing event.ingested timestamp"
    
>           assert "error" not in obj, "not error expected but got: {}.\n The related error message is: {}".format(
                obj, obj["error"].get("message"))
E           AssertionError: not error expected but got: {'input': {'type': 'log'}, 'agent': {'name': 'beats-ci-immutable-ubuntu-2204-1680880858408932028', 'id': 'e8e9c659-0cd4-47e0-b7a5-6a78f5e21b82', 'ephemeral_id': 'aef6124d-1cc3-4c76-9d35-1b4582ee5c03', 'type': 'filebeat', 'version': '8.7.1'}, '@timestamp': '2023-04-07T16:59:22.808936609Z', 'ecs': {'version': '1.12.0'}, 'log': {'file': {'path': '/var/lib/jenkins/workspace/PR-34589-25-d8af0c8a-84a0-49e7-b4fe-baca69949804/src/github.com/elastic/beats/x-pack/filebeat/module/mysqlenterprise/audit/test/mysql_audit_test.log'}, 'offset': 0}, 'service': {'type': 'mysqlenterprise'}, 'host': {'name': 'beats-ci-immutable-ubuntu-2204-1680880858408932028'}, 'event': {'ingested': '2023-04-07T16:59:23.936903163Z', 'timezone': '-02:00', 'module': 'mysqlenterprise', 'dataset': 'mysqlenterprise.audit'}, 'fileset': {'name': 'audit'}, 'message': '{ "timestamp": "2020-10-19 19:21:33", "id": 0, "class": "audit", "event": "startup", "connection_id": 0, "account": { "user": "skip-grants user", "host": "" }, "login": { "user": "", "os": "", "ip": "", "proxy": "" }, "startup_data": { "server_id": 1, "os_version": "x86_64-Linux", "mysql_version": "8.0.22-commercial", "args": ["/usr/local/mysql/bin/mysqld", "--loose-audit-log-format=JSON", "--log-error=log.err", "--pid-file=mysqld.pid", "--port=3306" ] } },', 'error': {'message': "Unexpected character (',' (code 44)): expected a value\\n at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 462]"}, 'tags': ['mysqlenterprise-audit']}.
E              The related error message is: Unexpected character (',' (code 44)): expected a value\n at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 462]
E           assert 'error' not in {'@timestamp': '2023-04-07T16:59:22.808936609Z', 'agent': {'ephemeral_id': 'aef6124d-1cc3-4c76-9d35-1b4582ee5c03', 'id...pected a value\\n at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 462]"}, ...}

../../filebeat/tests/system/test_modules.py:237: AssertionError 

Steps errors

Expand to view the steps failures

filebeat-pythonIntegTest - mage pythonIntegTest

• Took 16 min 53 sec . View more details here
• Description: mage pythonIntegTest

filebeat-pythonIntegTest - mage pythonIntegTest

• Took 13 min 7 sec . View more details here
• Description: mage pythonIntegTest

filebeat-pythonIntegTest - mage pythonIntegTest

• Took 13 min 5 sec . View more details here
• Description: mage pythonIntegTest

libbeat-unitTest - mage build unitTest

• Took 3 min 43 sec . View more details here
• Description: mage build unitTest

x-pack/filebeat-pythonIntegTest - mage pythonIntegTest

• Took 38 min 12 sec . View more details here
• Description: mage pythonIntegTest

x-pack/filebeat-pythonIntegTest - mage pythonIntegTest

• Took 36 min 53 sec . View more details here
• Description: mage pythonIntegTest

x-pack/filebeat-pythonIntegTest - mage pythonIntegTest

• Took 36 min 58 sec . View more details here
• Description: mage pythonIntegTest

Error signal

• Took 0 min 0 sec . View more details here
• Description: Error 'hudson.AbortException: script returned exit code 1'

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

• Name: Build&Test / filebeat-pythonIntegTest / test_fileset_file_070_elasticsearch – filebeat.tests.system.test_modules.Test
• Name: Build&Test / filebeat-pythonIntegTest / test_fileset_file_074_elasticsearch – filebeat.tests.system.test_modules.Test
• Name: Build&Test / filebeat-pythonIntegTest / test_fileset_file_094_mysql – filebeat.tests.system.test_modules.Test
• Name: Build&Test / filebeat-pythonIntegTest / test_fileset_file_098_mysql – filebeat.tests.system.test_modules.Test
• Name: Build&Test / filebeat-pythonIntegTest / test_fileset_file_104_mysql – filebeat.tests.system.test_modules.Test
• Name: Build&Test / filebeat-pythonIntegTest / test_fileset_file_108_iis – filebeat.tests.system.test_modules.Test
• Name: Build&Test / x-pack/filebeat-pythonIntegTest / test_fileset_file_012_envoyproxy – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
• Name: Build&Test / x-pack/filebeat-pythonIntegTest / test_fileset_file_146_mysqlenterprise – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[rdner]

I'm still investigating but it seems like these test failures revealed actual bugs in the modules, so it's not simply fixing the tests anymore.

[rdner]

This snapshot version was applied to #35221 to make it pass the CI.