Skip to content

[7.x] [ML] Stop reading the ml.max_open_jobs node attribute (#115524)#115580

Merged
kibanamachine merged 1 commit intoelastic:7.xfrom
kibanamachine:backport/7.x/pr-115524
Oct 19, 2021
Merged

[7.x] [ML] Stop reading the ml.max_open_jobs node attribute (#115524)#115580
kibanamachine merged 1 commit intoelastic:7.xfrom
kibanamachine:backport/7.x/pr-115524

Conversation

@kibanamachine
Copy link
Contributor

Backports the following commits to 7.x:

The ml.max_open_jobs node attribute is going away in
version 8, as the maximum number of open jobs has been
defined by a dynamic cluster-wide setting during the 7
series and there is no chance of version 8 needing to
run in a mixed version cluster with version 6.

The ml.machine_memory attribute will still be available,
so this can be checked instead as a way of detecting ML
nodes.
@kibanamachine kibanamachine added the backport This PR is a backport of another PR label Oct 19, 2021
@kibanamachine kibanamachine enabled auto-merge (squash) October 19, 2021 14:45
@kibanamachine
Copy link
Contributor Author

💛 Build succeeded, but was flaky


Test Failures

Kibana Pipeline / general / X-Pack API Integration Tests.x-pack/test/api_integration/apis/security_solution/overview_network·ts.apis SecuritySolution Endpoints Overview Network With packetbeat Make sure that we get OverviewNetwork data

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 1 times on tracked branches: https://github.com/elastic/kibana/issues/106941

[00:00:00]     │
[00:00:00]       └-: apis
[00:00:00]         └-> "before all" hook in "apis"
[00:07:22]         └-: SecuritySolution Endpoints
[00:07:22]           └-> "before all" hook in "SecuritySolution Endpoints"
[00:08:01]           └-: Overview Network
[00:08:01]             └-> "before all" hook in "Overview Network"
[00:08:03]             └-: With packetbeat
[00:08:03]               └-> "before all" hook for "Make sure that we get OverviewNetwork data"
[00:08:03]               └-> "before all" hook for "Make sure that we get OverviewNetwork data"
[00:08:03]                 │ info [x-pack/test/functional/es_archives/packetbeat/overview] Loading "mappings.json"
[00:08:03]                 │ info [x-pack/test/functional/es_archives/packetbeat/overview] Loading "data.json.gz"
[00:08:03]                 │ info [o.e.c.m.MetadataCreateIndexService] [node-01] [packetbeat-8.0.0-2019.02.19-000001] creating index, cause [api], templates [], shards [1]/[1]
[00:08:03]                 │ info [x-pack/test/functional/es_archives/packetbeat/overview] Created index "packetbeat-8.0.0-2019.02.19-000001"
[00:08:03]                 │ debg [x-pack/test/functional/es_archives/packetbeat/overview] "packetbeat-8.0.0-2019.02.19-000001" settings {"index":{"lifecycle":{"name":"packetbeat-8.0.0","rollover_alias":"packetbeat-8.0.0"},"mapping":{"total_fields":{"limit":"10000"}},"number_of_replicas":"1","number_of_shards":"1","query":{"default_field":["tags","message","agent.version","agent.name","agent.type","agent.id","agent.ephemeral_id","client.address","client.mac","client.domain","client.geo.continent_name","client.geo.country_name","client.geo.region_name","client.geo.city_name","client.geo.country_iso_code","client.geo.region_iso_code","client.geo.name","cloud.provider","cloud.availability_zone","cloud.region","cloud.instance.id","cloud.instance.name","cloud.machine.type","cloud.account.id","container.runtime","container.id","container.image.name","container.image.tag","container.name","destination.address","destination.mac","destination.domain","destination.geo.continent_name","destination.geo.country_name","destination.geo.region_name","destination.geo.city_name","destination.geo.country_iso_code","destination.geo.region_iso_code","destination.geo.name","ecs.version","error.id","error.message","error.code","event.id","event.kind","event.category","event.action","event.outcome","event.type","event.module","event.dataset","event.hash","event.timezone","file.path","file.target_path","file.extension","file.type","file.device","file.inode","file.uid","file.owner","file.gid","file.group","file.mode","group.id","group.name","host.hostname","host.name","host.id","host.mac","host.type","host.architecture","host.os.platform","host.os.name","host.os.full","host.os.family","host.os.version","host.os.kernel","host.geo.continent_name","host.geo.country_name","host.geo.region_name","host.geo.city_name","host.geo.country_iso_code","host.geo.region_iso_code","host.geo.name","http.request.method","http.request.body.content","http.request.referrer","http.response.body.content","http.version","log.level","network.name","network.type","network.iana_number","network.transport","network.application","network.protocol","network.direction","network.community_id","observer.mac","observer.hostname","observer.vendor","observer.version","observer.serial_number","observer.type","observer.os.platform","observer.os.name","observer.os.full","observer.os.family","observer.os.version","observer.os.kernel","observer.geo.continent_name","observer.geo.country_name","observer.geo.region_name","observer.geo.city_name","observer.geo.country_iso_code","observer.geo.region_iso_code","observer.geo.name","organization.name","organization.id","os.platform","os.name","os.full","os.family","os.version","os.kernel","process.name","process.args","process.executable","process.title","process.working_directory","server.address","server.mac","server.domain","server.geo.continent_name","server.geo.country_name","server.geo.region_name","server.geo.city_name","server.geo.country_iso_code","server.geo.region_iso_code","server.geo.name","service.id","service.name","service.type","service.state","service.version","service.ephemeral_id","source.address","source.mac","source.domain","source.geo.continent_name","source.geo.country_name","source.geo.region_name","source.geo.city_name","source.geo.country_iso_code","source.geo.region_iso_code","source.geo.name","url.original","url.full","url.scheme","url.domain","url.path","url.query","url.fragment","url.username","url.password","user.id","user.name","user.full_name","user.email","user.hash","user.group.id","user.group.name","user_agent.original","user_agent.name","user_agent.version","user_agent.device.name","user_agent.os.platform","user_agent.os.name","user_agent.os.full","user_agent.os.family","user_agent.os.version","user_agent.os.kernel","agent.hostname","error.type","cloud.project.id","kubernetes.pod.name","kubernetes.pod.uid","kubernetes.namespace","kubernetes.node.name","kubernetes.container.name","kubernetes.container.image","type","server.process.name","server.process.args","server.process.executable","server.process.working_directory","server.process.start","client.process.name","client.process.args","client.process.executable","client.process.working_directory","client.process.start","flow.id","status","method","resource","path","query","params","request","response","amqp.reply-text","amqp.exchange","amqp.exchange-type","amqp.consumer-tag","amqp.routing-key","amqp.queue","amqp.content-type","amqp.content-encoding","amqp.delivery-mode","amqp.correlation-id","amqp.reply-to","amqp.expiration","amqp.message-id","amqp.timestamp","amqp.type","amqp.user-id","amqp.app-id","cassandra.request.headers.flags","cassandra.request.headers.stream","cassandra.request.headers.op","cassandra.request.query","cassandra.response.headers.flags","cassandra.response.headers.stream","cassandra.response.headers.op","cassandra.response.result.type","cassandra.response.result.rows.meta.keyspace","cassandra.response.result.rows.meta.table","cassandra.response.result.rows.meta.flags","cassandra.response.result.rows.meta.paging_state","cassandra.response.result.keyspace","cassandra.response.result.schema_change.change","cassandra.response.result.schema_change.keyspace","cassandra.response.result.schema_change.table","cassandra.response.result.schema_change.object","cassandra.response.result.schema_change.target","cassandra.response.result.schema_change.name","cassandra.response.result.schema_change.args","cassandra.response.result.prepared.prepared_id","cassandra.response.result.prepared.req_meta.keyspace","cassandra.response.result.prepared.req_meta.table","cassandra.response.result.prepared.req_meta.flags","cassandra.response.result.prepared.req_meta.paging_state","cassandra.response.result.prepared.resp_meta.keyspace","cassandra.response.result.prepared.resp_meta.table","cassandra.response.result.prepared.resp_meta.flags","cassandra.response.result.prepared.resp_meta.paging_state","cassandra.response.authentication.class","cassandra.response.warnings","cassandra.response.event.type","cassandra.response.event.change","cassandra.response.event.host","cassandra.response.event.schema_change.change","cassandra.response.event.schema_change.keyspace","cassandra.response.event.schema_change.table","cassandra.response.event.schema_change.object","cassandra.response.event.schema_change.target","cassandra.response.event.schema_change.name","cassandra.response.event.schema_change.args","cassandra.response.error.msg","cassandra.response.error.type","cassandra.response.error.details.read_consistency","cassandra.response.error.details.write_type","cassandra.response.error.details.keyspace","cassandra.response.error.details.table","cassandra.response.error.details.stmt_id","cassandra.response.error.details.num_failures","cassandra.response.error.details.function","cassandra.response.error.details.arg_types","dhcpv4.transaction_id","dhcpv4.flags","dhcpv4.client_mac","dhcpv4.server_name","dhcpv4.op_code","dhcpv4.hardware_type","dhcpv4.option.message_type","dhcpv4.option.parameter_request_list","dhcpv4.option.class_identifier","dhcpv4.option.domain_name","dhcpv4.option.hostname","dhcpv4.option.message","dhcpv4.option.boot_file_name","dns.op_code","dns.response_code","dns.question.name","dns.question.type","dns.question.class","dns.question.registered_domain","dns.answers.name","dns.answers.type","dns.answers.class","dns.answers.data","dns.authorities.name","dns.authorities.type","dns.authorities.class","dns.additionals.name","dns.additionals.type","dns.additionals.class","dns.additionals.data","dns.opt.version","dns.opt.ext_rcode","http.response.status_phrase","icmp.version","icmp.request.message","icmp.response.message","memcache.protocol_type","memcache.request.line","memcache.request.command","memcache.response.command","memcache.request.type","memcache.response.type","memcache.response.error_msg","memcache.request.opcode","memcache.response.opcode","memcache.response.status","memcache.request.raw_args","memcache.request.automove","memcache.response.version","mongodb.error","mongodb.fullCollectionName","mongodb.startingFrom","mongodb.query","mongodb.returnFieldsSelector","mongodb.selector","mongodb.update","mongodb.cursorId","mysql.insert_id","mysql.num_fields","mysql.num_rows","mysql.query","mysql.error_message","nfs.tag","nfs.opcode","nfs.status","rpc.xid","rpc.status","rpc.auth_flavor","rpc.cred.gids","rpc.cred.machinename","pgsql.error_message","pgsql.error_severity","pgsql.num_fields","pgsql.num_rows","redis.return_value","redis.error","thrift.params","thrift.service","thrift.return_value","thrift.exceptions","tls.version","tls.resumption_method","tls.client_hello.version","tls.client_hello.extensions.server_name_indication","tls.client_hello.extensions.application_layer_protocol_negotiation","tls.client_hello.extensions.session_ticket","tls.client_hello.extensions.supported_versions","tls.client_hello.extensions.supported_groups","tls.client_hello.extensions.signature_algorithms","tls.client_hello.extensions.ec_points_formats","tls.client_hello.extensions._unparsed_","tls.server_hello.version","tls.server_hello.selected_cipher","tls.server_hello.selected_compression_method","tls.server_hello.session_id","tls.server_hello.extensions.session_ticket","tls.server_hello.extensions.supported_versions","tls.server_hello.extensions.ec_points_formats","tls.server_hello.extensions._unparsed_","tls.client_certificate.serial_number","tls.client_certificate.public_key_algorithm","tls.client_certificate.signature_algorithm","tls.client_certificate.raw","tls.client_certificate.subject.country","tls.client_certificate.subject.organization","tls.client_certificate.subject.organizational_unit","tls.client_certificate.subject.province","tls.client_certificate.subject.common_name","tls.client_certificate.issuer.country","tls.client_certificate.issuer.organization","tls.client_certificate.issuer.organizational_unit","tls.client_certificate.issuer.province","tls.client_certificate.issuer.common_name","tls.client_certificate.fingerprint.md5","tls.client_certificate.fingerprint.sha1","tls.client_certificate.fingerprint.sha256","tls.server_certificate.serial_number","tls.server_certificate.public_key_algorithm","tls.server_certificate.signature_algorithm","tls.server_certificate.raw","tls.server_certificate.subject.country","tls.server_certificate.subject.organization","tls.server_certificate.subject.organizational_unit","tls.server_certificate.subject.province","tls.server_certificate.subject.common_name","tls.server_certificate.issuer.country","tls.server_certificate.issuer.organization","tls.server_certificate.issuer.organizational_unit","tls.server_certificate.issuer.province","tls.server_certificate.issuer.common_name","tls.server_certificate.fingerprint.md5","tls.server_certificate.fingerprint.sha1","tls.server_certificate.fingerprint.sha256","tls.alert_types","tls.fingerprints.ja3.hash","tls.fingerprints.ja3.str","fields.*"]},"refresh_interval":"5s"}}
[00:08:04]                 │ info [o.e.c.m.MetadataCreateIndexService] [node-01] [packetbeat-overview-flow] creating index, cause [auto(bulk api)], templates [], shards [1]/[1]
[00:08:04]                 │ info [o.e.c.m.MetadataMappingService] [node-01] [packetbeat-overview-flow/k2NGcwkiSvqeWf3EXtO1Sw] create_mapping [_doc]
[00:08:04]                 │ info [o.e.c.m.MetadataMappingService] [node-01] [packetbeat-overview-flow/k2NGcwkiSvqeWf3EXtO1Sw] update_mapping [_doc]
[00:08:04]                 │ info [o.e.c.m.MetadataMappingService] [node-01] [packetbeat-overview-flow/k2NGcwkiSvqeWf3EXtO1Sw] update_mapping [_doc]
[00:08:04]                 │ info [o.e.c.m.MetadataMappingService] [node-01] [packetbeat-overview-flow/k2NGcwkiSvqeWf3EXtO1Sw] update_mapping [_doc]
[00:08:04]                 │ info [o.e.c.m.MetadataCreateIndexService] [node-01] [packetbeat-overview-flow-1] creating index, cause [auto(bulk api)], templates [], shards [1]/[1]
[00:08:04]                 │ info [o.e.c.m.MetadataMappingService] [node-01] [packetbeat-overview-flow-1/3JrwzavuQpKcIL9UUqXQDg] create_mapping [_doc]
[00:08:04]                 │ info [o.e.c.m.MetadataMappingService] [node-01] [packetbeat-overview-flow-1/3JrwzavuQpKcIL9UUqXQDg] update_mapping [_doc]
[00:08:04]                 │ info [o.e.c.m.MetadataMappingService] [node-01] [packetbeat-overview-flow-1/3JrwzavuQpKcIL9UUqXQDg] update_mapping [_doc]
[00:08:04]                 │ info [o.e.c.m.MetadataMappingService] [node-01] [packetbeat-overview-flow-1/3JrwzavuQpKcIL9UUqXQDg] update_mapping [_doc]
[00:08:04]                 │ info [o.e.c.m.MetadataMappingService] [node-01] [packetbeat-overview-flow-1/3JrwzavuQpKcIL9UUqXQDg] update_mapping [_doc]
[00:08:04]                 │ info [o.e.c.m.MetadataMappingService] [node-01] [packetbeat-overview-flow-1/3JrwzavuQpKcIL9UUqXQDg] update_mapping [_doc]
[00:08:04]                 │ info [x-pack/test/functional/es_archives/packetbeat/overview] Indexed 332 docs into "packetbeat-overview-flow"
[00:08:04]                 │ info [x-pack/test/functional/es_archives/packetbeat/overview] Indexed 336 docs into "packetbeat-overview-flow-1"
[00:08:04]               └-> Make sure that we get OverviewNetwork data
[00:08:04]                 └-> "before each" hook: global before each for "Make sure that we get OverviewNetwork data"
[00:08:04]                 └- ✖ fail: apis SecuritySolution Endpoints Overview Network With packetbeat Make sure that we get OverviewNetwork data
[00:08:04]                 │       Error: expected { auditbeatSocket: null,
[00:08:04]                 │   filebeatCisco: null,
[00:08:04]                 │   filebeatNetflow: null,
[00:08:04]                 │   filebeatPanw: null,
[00:08:04]                 │   filebeatSuricata: null,
[00:08:04]                 │   filebeatZeek: null,
[00:08:04]                 │   packetbeatDNS: null,
[00:08:04]                 │   packetbeatFlow: null,
[00:08:04]                 │   packetbeatTLS: null } to sort of equal { auditbeatSocket: 0,
[00:08:04]                 │   filebeatCisco: 0,
[00:08:04]                 │   filebeatNetflow: 0,
[00:08:04]                 │   filebeatPanw: 0,
[00:08:04]                 │   filebeatSuricata: 0,
[00:08:04]                 │   filebeatZeek: 0,
[00:08:04]                 │   packetbeatDNS: 44,
[00:08:04]                 │   packetbeatFlow: 588,
[00:08:04]                 │   packetbeatTLS: 0 }
[00:08:04]                 │       + expected - actual
[00:08:04]                 │ 
[00:08:04]                 │        {
[00:08:04]                 │       -  "auditbeatSocket": [null]
[00:08:04]                 │       -  "filebeatCisco": [null]
[00:08:04]                 │       -  "filebeatNetflow": [null]
[00:08:04]                 │       -  "filebeatPanw": [null]
[00:08:04]                 │       -  "filebeatSuricata": [null]
[00:08:04]                 │       -  "filebeatZeek": [null]
[00:08:04]                 │       -  "packetbeatDNS": [null]
[00:08:04]                 │       -  "packetbeatFlow": [null]
[00:08:04]                 │       -  "packetbeatTLS": [null]
[00:08:04]                 │       +  "auditbeatSocket": 0
[00:08:04]                 │       +  "filebeatCisco": 0
[00:08:04]                 │       +  "filebeatNetflow": 0
[00:08:04]                 │       +  "filebeatPanw": 0
[00:08:04]                 │       +  "filebeatSuricata": 0
[00:08:04]                 │       +  "filebeatZeek": 0
[00:08:04]                 │       +  "packetbeatDNS": 44
[00:08:04]                 │       +  "packetbeatFlow": 588
[00:08:04]                 │       +  "packetbeatTLS": 0
[00:08:04]                 │        }
[00:08:04]                 │       
[00:08:04]                 │       at Assertion.assert (/dev/shm/workspace/parallel/10/kibana/node_modules/@kbn/expect/expect.js:100:11)
[00:08:04]                 │       at Assertion.eql (/dev/shm/workspace/parallel/10/kibana/node_modules/@kbn/expect/expect.js:244:8)
[00:08:04]                 │       at Context.<anonymous> (test/api_integration/apis/security_solution/overview_network.ts:97:36)
[00:08:04]                 │       at runMicrotasks (<anonymous>)
[00:08:04]                 │       at processTicksAndRejections (node:internal/process/task_queues:96:5)
[00:08:04]                 │       at Object.apply (/dev/shm/workspace/parallel/10/kibana/node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:87:16)
[00:08:04]                 │ 
[00:08:04]                 │ 

Stack Trace

Error: expected { auditbeatSocket: null,
  filebeatCisco: null,
  filebeatNetflow: null,
  filebeatPanw: null,
  filebeatSuricata: null,
  filebeatZeek: null,
  packetbeatDNS: null,
  packetbeatFlow: null,
  packetbeatTLS: null } to sort of equal { auditbeatSocket: 0,
  filebeatCisco: 0,
  filebeatNetflow: 0,
  filebeatPanw: 0,
  filebeatSuricata: 0,
  filebeatZeek: 0,
  packetbeatDNS: 44,
  packetbeatFlow: 588,
  packetbeatTLS: 0 }
    at Assertion.assert (/dev/shm/workspace/parallel/10/kibana/node_modules/@kbn/expect/expect.js:100:11)
    at Assertion.eql (/dev/shm/workspace/parallel/10/kibana/node_modules/@kbn/expect/expect.js:244:8)
    at Context.<anonymous> (test/api_integration/apis/security_solution/overview_network.ts:97:36)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Object.apply (/dev/shm/workspace/parallel/10/kibana/node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:87:16) {
  actual: '{\n' +
    '  "auditbeatSocket": [null]\n' +
    '  "filebeatCisco": [null]\n' +
    '  "filebeatNetflow": [null]\n' +
    '  "filebeatPanw": [null]\n' +
    '  "filebeatSuricata": [null]\n' +
    '  "filebeatZeek": [null]\n' +
    '  "packetbeatDNS": [null]\n' +
    '  "packetbeatFlow": [null]\n' +
    '  "packetbeatTLS": [null]\n' +
    '}',
  expected: '{\n' +
    '  "auditbeatSocket": 0\n' +
    '  "filebeatCisco": 0\n' +
    '  "filebeatNetflow": 0\n' +
    '  "filebeatPanw": 0\n' +
    '  "filebeatSuricata": 0\n' +
    '  "filebeatZeek": 0\n' +
    '  "packetbeatDNS": 44\n' +
    '  "packetbeatFlow": 588\n' +
    '  "packetbeatTLS": 0\n' +
    '}',
  showDiff: true
}

Metrics [docs]

✅ unchanged

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @droberts195

@kibanamachine kibanamachine merged commit 660ed36 into elastic:7.x Oct 19, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport This PR is a backport of another PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants