Skip to content

[Testing] 7.x Disable geoip downloader#25272

Merged
andrewkroh merged 1 commit intoelastic:7.xfrom
andrewkroh:feature/7x/disable-geoip-downloader
Apr 23, 2021
Merged

[Testing] 7.x Disable geoip downloader#25272
andrewkroh merged 1 commit intoelastic:7.xfrom
andrewkroh:feature/7x/disable-geoip-downloader

Conversation

@andrewkroh
Copy link
Copy Markdown
Member

@andrewkroh andrewkroh commented Apr 23, 2021

This is the same change as #25248 but not automated b/c the branches differ in that golden files don't need updated in 7.x. Once 7.14.0-SNAPSHOT is ready #25192 will update them, but for now use 7.13.0-SNAPSHOT of the stack.

What does this PR do?

Disable database updates via an Elasticsearch property ingest.geoip.downloader.enabled: false. This will cause Elasticsearch to use the circa 2019 maxmind database version that is embedded. This would work for now, but the embedded database will be removed in 8.x IIUC.

Closes: #25159

Why is it important?

This prevents our golden file tests from failing everytime the database changes geo or asn values. It also prevents race conditions in tests that might run before the database has been updated or in cases where the downloader service is unavailable.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation
  • [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Disable database updates via an Elasticsearch property ingest.geoip.downloader.enabled: false. This will cause Elasticsearch to use the circa 2019 maxmind database version that is embedded. This would work for now, but the embedded database will be removed in 8.x IIUC.

This prevents our golden file tests from failing everytime the database changes geo or asn values. It also prevents race conditions in tests that might run before the database has been updated or in cases where the downloader service is unavailable.

Closes: elastic#25159
@botelastic botelastic Bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 23, 2021
@botelastic
Copy link
Copy Markdown

botelastic Bot commented Apr 23, 2021

This pull request doesn't have a Team:<team> label.

@andrewkroh andrewkroh added the backport-v7.13.0 Automated backport with mergify label Apr 23, 2021
@elasticmachine
Copy link
Copy Markdown
Contributor

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #25272 opened

  • Start Time: 2021-04-23T12:14:09.336+0000

  • Duration: 108 min 44 sec

  • Commit: 7b25094

Test stats 🧪

Test Results
Failed 1
Passed 47163
Skipped 5184
Total 52348

Trends 🧪

Image of Build Times

Image of Tests

Test errors 1

Expand to view the tests failures

Build&Test / auditbeat-windows-10-windows-10 / TestNonRecursive – github.com/elastic/beats/v7/auditbeat/module/file_integrity/monitor
    Expand to view the error details

     Failed 
    

    Expand to view the stacktrace

     === RUN   TestNonRecursive
        monitor_test.go:73: 
            	Error Trace:	monitor_test.go:73
            	Error:      	An error is expected but got nil.
            	Test:       	TestNonRecursive
        monitor_test.go:74: 
            	Error Trace:	monitor_test.go:74
            	Error:      	Not equal: 
            	            	expected: *errors.errorString(&errors.errorString{s:"read timeout"})
            	            	actual  : <nil>(<nil>)
            	Test:       	TestNonRecursive
    --- FAIL: TestNonRecursive (3.17s)
     
    

Steps errors 5

Expand to view the steps failures

auditbeat-windows-10-windows-10 - mage build unitTest
  • Took 10 min 8 sec . View more details on here
  • Description: mage build unitTest
generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package
  • Took 6 min 31 sec . View more details on here
  • Description: make -C generator/_templates/metricbeat test test-package
generator-beat-test - make -C generator/_templates/beat test test-package
  • Took 6 min 5 sec . View more details on here
  • Description: make -C generator/_templates/beat test test-package
Print Message
  • Took 0 min 1 sec . View more details on here
  • Description: �[39;49m[INFO] getVaultSecret: Getting secrets�[0m
Error signal
  • Took 0 min 0 sec . View more details on here
  • Description: Error 'hudson.AbortException: script returned exit code 1'

Log output

Expand to view the last 100 lines of log output

[2021-04-23T14:01:17.197Z] 23.04s setup    metricbeat/module/aerospike/test_aerospike.py::Test::test_namespace
[2021-04-23T14:01:17.197Z] 22.72s setup    metricbeat/module/mongodb/test_mongodb.py::Test::test_status
[2021-04-23T14:01:17.197Z] 22.04s setup    metricbeat/module/elasticsearch/test_elasticsearch.py::Test::test_metricsets_0_ccr
[2021-04-23T14:01:17.197Z] 21.70s setup    metricbeat/module/mysql/test_mysql.py::Test_1::test_status
[2021-04-23T14:01:17.197Z] =========== 231 passed, 15 skipped, 3 warnings in 1651.80s (0:27:31) ===========
[2021-04-23T14:01:17.456Z] >> python test: Integration Testing Complete
[2021-04-23T14:01:19.891Z] Cleaning up /var/lib/jenkins/workspace/PR-25272-1-b58851b2-86bb-46b6-a78f-646985d11909
[2021-04-23T14:01:19.891Z] Client: Docker Engine - Community
[2021-04-23T14:01:19.891Z]  Version:           20.10.3
[2021-04-23T14:01:19.891Z]  API version:       1.41
[2021-04-23T14:01:19.891Z]  Go version:        go1.13.15
[2021-04-23T14:01:19.891Z]  Git commit:        48d30b5
[2021-04-23T14:01:19.891Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-04-23T14:01:19.891Z]  OS/Arch:           linux/amd64
[2021-04-23T14:01:19.891Z]  Context:           default
[2021-04-23T14:01:19.891Z]  Experimental:      true
[2021-04-23T14:01:19.891Z] 
[2021-04-23T14:01:19.891Z] Server: Docker Engine - Community
[2021-04-23T14:01:19.891Z]  Engine:
[2021-04-23T14:01:19.891Z]   Version:          20.10.3
[2021-04-23T14:01:19.891Z]   API version:      1.41 (minimum version 1.12)
[2021-04-23T14:01:19.891Z]   Go version:       go1.13.15
[2021-04-23T14:01:19.891Z]   Git commit:       46229ca
[2021-04-23T14:01:19.891Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-04-23T14:01:19.891Z]   OS/Arch:          linux/amd64
[2021-04-23T14:01:19.891Z]   Experimental:     false
[2021-04-23T14:01:19.891Z]  containerd:
[2021-04-23T14:01:19.891Z]   Version:          1.4.4
[2021-04-23T14:01:19.891Z]   GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
[2021-04-23T14:01:19.891Z]  runc:
[2021-04-23T14:01:19.891Z]   Version:          1.0.0-rc93
[2021-04-23T14:01:19.891Z]   GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
[2021-04-23T14:01:19.891Z]  docker-init:
[2021-04-23T14:01:19.891Z]   Version:          0.19.0
[2021-04-23T14:01:19.891Z]   GitCommit:        de40ad0
[2021-04-23T14:01:19.891Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-04-23T14:01:19.891Z] Unable to find image 'alpine:3.4' locally
[2021-04-23T14:01:20.828Z] 3.4: Pulling from library/alpine
[2021-04-23T14:01:20.828Z] c1e54eec4b57: Pulling fs layer
[2021-04-23T14:01:21.087Z] c1e54eec4b57: Verifying Checksum
[2021-04-23T14:01:21.087Z] c1e54eec4b57: Download complete
[2021-04-23T14:01:21.346Z] c1e54eec4b57: Pull complete
[2021-04-23T14:01:21.346Z] Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
[2021-04-23T14:01:21.346Z] Status: Downloaded newer image for alpine:3.4
[2021-04-23T14:01:23.253Z] Change permissions with write access of all files inside the specific folder
[2021-04-23T14:01:24.675Z] Running in /var/lib/jenkins/workspace/PR-25272-1-b58851b2-86bb-46b6-a78f-646985d11909/src/github.com/elastic/beats/build
[2021-04-23T14:01:25.023Z] + rm -rf ve
[2021-04-23T14:01:25.023Z] + find . -type d -name vendor -exec rm -r {} ;
[2021-04-23T14:01:25.484Z] + python .ci/scripts/pre_archive_test.py
[2021-04-23T14:01:26.423Z] Copy ./metricbeat/build into build/metricbeat/build
[2021-04-23T14:01:26.478Z] Running in /var/lib/jenkins/workspace/PR-25272-1-b58851b2-86bb-46b6-a78f-646985d11909/src/github.com/elastic/beats/build
[2021-04-23T14:01:26.546Z] Recording test results
[2021-04-23T14:01:28.140Z] [Checks API] No suitable checks publisher found.
[2021-04-23T14:01:28.851Z] + go clean -modcache
[2021-04-23T14:01:32.581Z] Cleaning up /var/lib/jenkins/workspace/PR-25272-1-b58851b2-86bb-46b6-a78f-646985d11909
[2021-04-23T14:01:32.581Z] Client: Docker Engine - Community
[2021-04-23T14:01:32.581Z]  Version:           20.10.3
[2021-04-23T14:01:32.581Z]  API version:       1.41
[2021-04-23T14:01:32.581Z]  Go version:        go1.13.15
[2021-04-23T14:01:32.581Z]  Git commit:        48d30b5
[2021-04-23T14:01:32.581Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-04-23T14:01:32.581Z]  OS/Arch:           linux/amd64
[2021-04-23T14:01:32.581Z]  Context:           default
[2021-04-23T14:01:32.581Z]  Experimental:      true
[2021-04-23T14:01:32.581Z] 
[2021-04-23T14:01:32.581Z] Server: Docker Engine - Community
[2021-04-23T14:01:32.581Z]  Engine:
[2021-04-23T14:01:32.581Z]   Version:          20.10.3
[2021-04-23T14:01:32.581Z]   API version:      1.41 (minimum version 1.12)
[2021-04-23T14:01:32.581Z]   Go version:       go1.13.15
[2021-04-23T14:01:32.581Z]   Git commit:       46229ca
[2021-04-23T14:01:32.581Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-04-23T14:01:32.581Z]   OS/Arch:          linux/amd64
[2021-04-23T14:01:32.581Z]   Experimental:     false
[2021-04-23T14:01:32.581Z]  containerd:
[2021-04-23T14:01:32.581Z]   Version:          1.4.4
[2021-04-23T14:01:32.581Z]   GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
[2021-04-23T14:01:32.581Z]  runc:
[2021-04-23T14:01:32.581Z]   Version:          1.0.0-rc93
[2021-04-23T14:01:32.581Z]   GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
[2021-04-23T14:01:32.581Z]  docker-init:
[2021-04-23T14:01:32.581Z]   Version:          0.19.0
[2021-04-23T14:01:32.581Z]   GitCommit:        de40ad0
[2021-04-23T14:01:32.581Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-04-23T14:01:35.250Z] Change permissions with write access of all files inside the specific folder
[2021-04-23T14:01:35.914Z] Running in /var/lib/jenkins/workspace/PR-25272-1-b58851b2-86bb-46b6-a78f-646985d11909
[2021-04-23T14:01:41.311Z] + gsutil --version
[2021-04-23T14:01:42.749Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-04-23T14:01:43.152Z] + gcloud auth activate-service-account --key-file ****
[2021-04-23T14:01:43.723Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-04-23T14:01:44.163Z] + gsutil -m -q cp -a public-read bWV0cmljYmVhdC1weXRob25JbnRlZ1Rlc3Q3YjI1MDk0MjFkYjFlZTlhYmU1NTdhY2EwZjk3ZWM3MGUyNmZjNThj gs://beats-ci-temp/ci/cache/
[2021-04-23T14:01:50.915Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-04-23T14:01:51.096Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-25272/src/github.com/elastic/beats
[2021-04-23T14:01:52.004Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-25272
[2021-04-23T14:01:52.145Z] [INFO] getVaultSecret: Getting secrets
[2021-04-23T14:01:52.205Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-04-23T14:01:53.235Z] + chmod 755 generate-build-data.sh
[2021-04-23T14:01:53.235Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25272/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25272/runs/1 FAILURE 6463626
[2021-04-23T14:01:53.235Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25272/runs/1/steps/?limit=10000 -o steps-info.json
[2021-04-23T14:02:05.332Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25272/runs/1/tests/?status=FAILED -o tests-errors.json

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test Results
Failed 1
Passed 47163
Skipped 5184
Total 52348

Genuine test errors 1

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

  • Name: Build&Test / auditbeat-windows-10-windows-10 / TestNonRecursive – github.com/elastic/beats/v7/auditbeat/module/file_integrity/monitor

@andrewkroh
Copy link
Copy Markdown
Member Author

Merging with some red b/c we need to get 7.x and 7.13 branches "more green" than they are. They generator tests that failed should be fixed by another PR that has since been merged.

@andrewkroh andrewkroh merged commit 3b4b10e into elastic:7.x Apr 23, 2021
@andrewkroh andrewkroh added needs_backport PR is waiting to be backported to other branches. and removed backport-v7.13.0 Automated backport with mergify labels Apr 23, 2021
andrewkroh added a commit to andrewkroh/beats that referenced this pull request Apr 23, 2021
Disable database updates via an Elasticsearch property ingest.geoip.downloader.enabled: false. This will cause Elasticsearch to use the circa 2019 maxmind database version that is embedded. This would work for now, but the embedded database will be removed in 8.x IIUC.

This prevents our golden file tests from failing everytime the database changes geo or asn values. It also prevents race conditions in tests that might run before the database has been updated or in cases where the downloader service is unavailable.

Closes: elastic#25159
(cherry picked from commit 3b4b10e)
@andrewkroh andrewkroh added v7.13.0 and removed needs_backport PR is waiting to be backported to other branches. labels Apr 23, 2021
andrewkroh added a commit that referenced this pull request Apr 23, 2021
Disable database updates via an Elasticsearch property ingest.geoip.downloader.enabled: false. This will cause Elasticsearch to use the circa 2019 maxmind database version that is embedded. This would work for now, but the embedded database will be removed in 8.x IIUC.

This prevents our golden file tests from failing everytime the database changes geo or asn values. It also prevents race conditions in tests that might run before the database has been updated or in cases where the downloader service is unavailable.

Closes: #25159
(cherry picked from commit 3b4b10e)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport Filebeat Filebeat needs_team Indicates that the issue/PR needs a Team:* label review :Testing v7.13.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants