Skip to content

Conversation

@faec
Copy link
Contributor

@faec faec commented Sep 28, 2020

Cherry-pick of PR #21176 to 7.x branch. Original message:

What does this PR do?

This PR implements a new disk-based queue for libbeat.

This is a draft PR: it is ready to start receiving feedback and rolling reviews, but some implementation details are still pending and it isn't ready for checkin.

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Add to the beat configuration:

queue.disk:
  max_size: 1GB

The beat should operate as usual, and the event data should be stored in data/diskqueue while waiting to be ingested.

Initial implementation of the new libbeat disk queue

(cherry picked from commit 2b8fd7c)
@faec faec added [zube]: In Review backport Team:Integrations Label for the Integrations team labels Sep 28, 2020
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 28, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations (Team:Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Sep 28, 2020
@faec faec added libbeat and removed backport labels Sep 28, 2020
@elasticmachine
Copy link
Collaborator

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #21349 opened]

  • Start Time: 2020-09-28T14:56:32.674+0000

  • Duration: 66 min 45 sec

Test stats 🧪

Test Results
Failed 1
Passed 16110
Skipped 1355
Total 17466

Test errors

Expand to view the tests failures

  • Name: Build&Test / x-pack/metricbeat-build / TestFetch – ec2

    • Age: 1
    • Duration: 18.39
    • Error Details: Failed

Steps errors

Expand to view the steps failures

  • Name: mage build test

    • Description: mage build test

    • Duration: 25 min 32 sec

    • Start Time: 2020-09-28T15:29:41.040+0000

    • log

  • Name: Notifies GitHub of the status of a Pull Request

    • Description: script returned exit code 1

    • Duration: 0 min 1 sec

    • Start Time: 2020-09-28T15:54:33.762+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/metricbeat-goIntegTest/metricbeat/build/TEST-go-integration-munin.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/metricbeat-goIntegTest/metricbeat/build/TEST-go-integration-uwsgi.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/metricbeat-goIntegTest/metricbeat/build/TEST-go-integration-etcd.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/metricbeat-goIntegTest/metricbeat/build/TEST-go-integration-docker.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/metricbeat-goIntegTest/metricbeat/build/TEST-go-integration-beat.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/metricbeat-goIntegTest/metricbeat/build/TEST-go-integration-mongodb.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/metricbeat-goIntegTest/metricbeat/build/TEST-go-integration-golang.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/metricbeat-goIntegTest/metricbeat/build/TEST-go-integration-rabbitmq.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/metricbeat-goIntegTest/metricbeat/build/TEST-go-integration-jolokia.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/x-pack-auditbeat-build/x-pack/auditbeat/build/TEST-go-unit.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/x-pack-auditbeat-build/x-pack/auditbeat/build/TEST-python-integration.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/x-pack-auditbeat-build/x-pack/auditbeat/build/TEST-python-unit.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/x-pack-auditbeat-build/x-pack/auditbeat/build/TEST-go-integration.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/x-pack-filebeat-windows-windows-2019/x-pack/filebeat/build/TEST-go-unit.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/x-pack-filebeat-windows-windows-2019/x-pack/filebeat/build/TEST-python-unit.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/filebeat-windows-windows-2019/filebeat/build/TEST-go-unit.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/filebeat-windows-windows-2019/filebeat/build/TEST-python-unit.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/auditbeat-windows-windows-2019/auditbeat/build/TEST-go-unit.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/auditbeat-windows-windows-2019/auditbeat/build/TEST-python-unit.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/auditbeat-build/auditbeat/build/TEST-go-unit.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/auditbeat-build/auditbeat/build/TEST-python-integration.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/auditbeat-build/auditbeat/build/TEST-python-unit.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/auditbeat-build/auditbeat/build/TEST-go-integration.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/packetbeat-build/packetbeat/build/TEST-go-unit.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/packetbeat-build/packetbeat/build/TEST-python-unit.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-cockroachdb.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-unit.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-coredns.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-cloudfoundry.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-sql.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-prometheus.xml
[2020-09-28T16:01:51.946Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-azure.xml
[2020-09-28T16:01:51.947Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-ibmmq.xml
[2020-09-28T16:01:51.947Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-mssql.xml
[2020-09-28T16:01:51.947Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-09-28T16:01:51.947Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-python-unit.xml
[2020-09-28T16:01:51.947Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-aws.xml
[2020-09-28T16:01:51.947Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-09-28T16:01:51.947Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-redisenterprise.xml
[2020-09-28T16:01:51.947Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-09-28T16:01:51.947Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-oracle.xml
[2020-09-28T16:01:51.947Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-stan.xml
[2020-09-28T16:01:51.947Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-googlecloud.xml
[2020-09-28T16:01:51.947Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-09-28T16:01:51.947Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-statsd.xml
[2020-09-28T16:01:51.947Z] ./src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-appsearch.xml
[2020-09-28T16:01:51.947Z] ./src/github.com/elastic/beats/x-pack-functionbeat-build/x-pack/functionbeat/build/TEST-go-unit.xml
[2020-09-28T16:01:51.947Z] ./src/github.com/elastic/beats/x-pack-functionbeat-build/x-pack/functionbeat/build/TEST-python-unit.xml
[2020-09-28T16:01:51.947Z] ./src/github.com/elastic/beats/metricbeat-pythonIntegTest/metricbeat/build/TEST-python-integration.xml
[2020-09-28T16:01:51.947Z] + cat
[2020-09-28T16:01:51.947Z] + /usr/local/bin/runbld ./runbld-script --job-name elastic+beats+pull-request
[2020-09-28T16:01:51.947Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-09-28T16:02:00.118Z] runbld>>> runbld started
[2020-09-28T16:02:00.118Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-09-28T16:02:01.073Z] runbld>>> The following profiles matched the job 'elastic+beats+pull-request' in order of occurrence in the config (last value wins).
[2020-09-28T16:02:01.073Z] runbld>>> Matches in the system config:
[2020-09-28T16:02:01.073Z] runbld>>> - Matched ^elastic\+beats
[2020-09-28T16:02:01.073Z] runbld>>> - Matched ^elastic\+beats\+pull-request
[2020-09-28T16:02:02.468Z] runbld>>> Debug logging enabled.
[2020-09-28T16:02:02.468Z] runbld>>> Storing result
[2020-09-28T16:02:02.730Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-09-28T16:02:02.730Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20200928160202-F5426F86
[2020-09-28T16:02:02.730Z] runbld>>> Adding system facts.
[2020-09-28T16:02:04.124Z] runbld>>> Adding vcs info for the latest commit:  365ccdd7cf04931f843d98b336eba89d1aeaa6f5
[2020-09-28T16:02:04.124Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-09-28T16:02:04.124Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-09-28T16:02:04.124Z] + echo 'Processing JUnit reports with runbld...'
[2020-09-28T16:02:04.124Z] Processing JUnit reports with runbld...
[2020-09-28T16:02:04.388Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-09-28T16:02:04.388Z] runbld>>> DURATION: 62ms
[2020-09-28T16:02:04.388Z] runbld>>> STDOUT: 40 bytes
[2020-09-28T16:02:04.388Z] runbld>>> STDERR: 49 bytes
[2020-09-28T16:02:04.388Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-09-28T16:02:04.388Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-21349
[2020-09-28T16:02:05.336Z] runbld>>> Storing build metadata: 
[2020-09-28T16:02:05.336Z] runbld>>> Adding test report.
[2020-09-28T16:02:05.336Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-21349/src/github.com/elastic/beats
[2020-09-28T16:02:06.284Z] runbld>>> Found 122 test output files
[2020-09-28T16:02:08.848Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21349/src/github.com/elastic/beats/metricbeat-goIntegTest/metricbeat/build/TEST-go-integration-graphite.xml
[2020-09-28T16:02:08.848Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21349/src/github.com/elastic/beats/metricbeat-goIntegTest/metricbeat/build/TEST-go-integration-windows.xml
[2020-09-28T16:02:09.112Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21349/src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-09-28T16:02:09.112Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21349/src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-09-28T16:02:09.112Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21349/src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-09-28T16:02:09.112Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21349/src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-09-28T16:02:09.112Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21349/src/github.com/elastic/beats/x-pack-metricbeat-build/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-09-28T16:02:09.376Z] runbld>>> Test output logs contained: Errors: 0 Failures: 1 Tests: 17466 Skipped: 1088
[2020-09-28T16:02:09.640Z] runbld>>> Storing result
[2020-09-28T16:02:09.640Z] runbld>>> FAILURES: 1
[2020-09-28T16:02:09.906Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-09-28T16:02:09.906Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20200928160202-F5426F86
[2020-09-28T16:02:10.172Z] runbld>>> Email notification disabled by environment variable.
[2020-09-28T16:02:10.172Z] runbld>>> Slack notification disabled by environment variable.
[2020-09-28T16:02:16.450Z] Running on worker-395930 in /var/lib/jenkins/workspace/Beats_beats_PR-21349
[2020-09-28T16:02:16.516Z] [INFO] getVaultSecret: Getting secrets
[2020-09-28T16:02:16.604Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-09-28T16:02:18.695Z] + chmod 755 generate-build-data.sh
[2020-09-28T16:02:18.696Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21349/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21349/runs/1 FAILURE 3944609
[2020-09-28T16:02:18.696Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21349/runs/1/steps/?limit=10000 -o steps-info.json
[2020-09-28T16:02:23.878Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21349/runs/1/tests/?status=FAILED -o tests-errors.json

@faec faec added the backport label Sep 29, 2020
@faec faec merged commit e9d1ec0 into elastic:7.x Sep 29, 2020
@zube zube bot removed the [zube]: Done label Dec 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport libbeat Team:Integrations Label for the Integrations team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants