[MySQL] Add condition setting support

[BenB196]

Enhancement

Proposed commit message

This MR adds the ability to set Elastic Agent condition to both the Log and Metrics datasets.

The ability to set this value allows for having multiple MySQL integrations in the same Agent policy while only having them run on specific hosts. (Most likely to be used with the host metadata; ex; host.name or host.platform.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• I added condition as "advanced" fields, similar to other integrations that have this setting.

Before Upgrade

inputs:
  - id: logfile-mysql-1d521dab-3d64-48c7-ac99-99d6985fe9de
    name: mysql-1
    revision: 1
    type: logfile
    use_output: default
    meta:
      package:
        name: mysql
        version: 1.19.0
    data_stream:
      namespace: default
    package_policy_id: 1d521dab-3d64-48c7-ac99-99d6985fe9de
    streams:
      - id: logfile-mysql.error-1d521dab-3d64-48c7-ac99-99d6985fe9de
        data_stream:
          dataset: mysql.error
          type: logs
        paths:
          - /var/log/mysql/error.log*
          - /var/log/mysqld.log*
        exclude_files:
          - .gz$
        multiline:
          pattern: '^([0-9]{4}-[0-9]{2}-[0-9]{2}|[0-9]{6})'
          negate: true
          match: after
        tags:
          - mysql-error
        processors:
          - add_locale: null
      - id: logfile-mysql.slowlog-1d521dab-3d64-48c7-ac99-99d6985fe9de
        data_stream:
          dataset: mysql.slowlog
          type: logs
        paths:
          - /var/log/mysql/*-slow.log*
          - /var/lib/mysql/*-slow.log*
        exclude_files:
          - .gz$
        multiline:
          pattern: '^(# User@Host: |# Time: )'
          negate: true
          match: after
        exclude_lines:
          - '^[\/\w\.]+, Version: .* started with:.*'
          - '^# Time:.*'
        tags:
          - mysql-slowlog
        processors:
          - add_locale: null
  - id: mysql/metrics-mysql-1d521dab-3d64-48c7-ac99-99d6985fe9de
    name: mysql-1
    revision: 1
    type: mysql/metrics
    use_output: default
    meta:
      package:
        name: mysql
        version: 1.19.0
    data_stream:
      namespace: default
    package_policy_id: 1d521dab-3d64-48c7-ac99-99d6985fe9de
    streams:
      - id: mysql/metrics-mysql.performance-1d521dab-3d64-48c7-ac99-99d6985fe9de
        data_stream:
          dataset: mysql.performance
          type: metrics
        metricsets:
          - performance
        hosts:
          - 'tcp(127.0.0.1:3306)/'
        password: '${SECRET_0}'
        period: 10s
        username: root
      - id: mysql/metrics-mysql.status-1d521dab-3d64-48c7-ac99-99d6985fe9de
        data_stream:
          dataset: mysql.status
          type: metrics
        metricsets:
          - status
        hosts:
          - 'tcp(127.0.0.1:3306)/'
        password: '${SECRET_0}'
        period: 10s
        username: root

After Upgrade (no changes)

inputs:
  - id: logfile-mysql-1d521dab-3d64-48c7-ac99-99d6985fe9de
    name: mysql-1
    revision: 2
    type: logfile
    use_output: default
    meta:
      package:
        name: mysql
        version: 1.20.0
    data_stream:
      namespace: default
    package_policy_id: 1d521dab-3d64-48c7-ac99-99d6985fe9de
    streams:
      - id: logfile-mysql.error-1d521dab-3d64-48c7-ac99-99d6985fe9de
        data_stream:
          dataset: mysql.error
          type: logs
        paths:
          - /var/log/mysql/error.log*
          - /var/log/mysqld.log*
        exclude_files:
          - .gz$
        multiline:
          pattern: '^([0-9]{4}-[0-9]{2}-[0-9]{2}|[0-9]{6})'
          negate: true
          match: after
        tags:
          - mysql-error
        processors:
          - add_locale: null
      - id: logfile-mysql.slowlog-1d521dab-3d64-48c7-ac99-99d6985fe9de
        data_stream:
          dataset: mysql.slowlog
          type: logs
        paths:
          - /var/log/mysql/*-slow.log*
          - /var/lib/mysql/*-slow.log*
        exclude_files:
          - .gz$
        multiline:
          pattern: '^(# User@Host: |# Time: )'
          negate: true
          match: after
        exclude_lines:
          - '^[\/\w\.]+, Version: .* started with:.*'
          - '^# Time:.*'
        tags:
          - mysql-slowlog
        processors:
          - add_locale: null
  - id: mysql/metrics-mysql-1d521dab-3d64-48c7-ac99-99d6985fe9de
    name: mysql-1
    revision: 2
    type: mysql/metrics
    use_output: default
    meta:
      package:
        name: mysql
        version: 1.20.0
    data_stream:
      namespace: default
    package_policy_id: 1d521dab-3d64-48c7-ac99-99d6985fe9de
    streams:
      - id: mysql/metrics-mysql.performance-1d521dab-3d64-48c7-ac99-99d6985fe9de
        data_stream:
          dataset: mysql.performance
          type: metrics
        metricsets:
          - performance
        hosts:
          - 'tcp(127.0.0.1:3306)/'
        password: '${SECRET_0}'
        period: 10s
        username: root
      - id: mysql/metrics-mysql.status-1d521dab-3d64-48c7-ac99-99d6985fe9de
        data_stream:
          dataset: mysql.status
          type: metrics
        metricsets:
          - status
        hosts:
          - 'tcp(127.0.0.1:3306)/'
        password: '${SECRET_0}'
        period: 10s
        username: root

After Upgrade (with `condition` set)

inputs:
  - id: logfile-mysql-1d521dab-3d64-48c7-ac99-99d6985fe9de
    name: mysql-1
    revision: 3
    type: logfile
    use_output: default
    meta:
      package:
        name: mysql
        version: 1.20.0
    data_stream:
      namespace: default
    package_policy_id: 1d521dab-3d64-48c7-ac99-99d6985fe9de
    streams:
      - id: logfile-mysql.error-1d521dab-3d64-48c7-ac99-99d6985fe9de
        data_stream:
          dataset: mysql.error
          type: logs
        paths:
          - /var/log/mysql/error.log*
          - /var/log/mysqld.log*
        exclude_files:
          - .gz$
        multiline:
          pattern: '^([0-9]{4}-[0-9]{2}-[0-9]{2}|[0-9]{6})'
          negate: true
          match: after
        tags:
          - mysql-error
        processors:
          - add_locale: null
        condition: '${host.name} == "mydb-server"'
      - id: logfile-mysql.slowlog-1d521dab-3d64-48c7-ac99-99d6985fe9de
        data_stream:
          dataset: mysql.slowlog
          type: logs
        paths:
          - /var/log/mysql/*-slow.log*
          - /var/lib/mysql/*-slow.log*
        exclude_files:
          - .gz$
        multiline:
          pattern: '^(# User@Host: |# Time: )'
          negate: true
          match: after
        exclude_lines:
          - '^[\/\w\.]+, Version: .* started with:.*'
          - '^# Time:.*'
        tags:
          - mysql-slowlog
        processors:
          - add_locale: null
        condition: '${host.name} == "mydb-server"'
  - id: mysql/metrics-mysql-1d521dab-3d64-48c7-ac99-99d6985fe9de
    name: mysql-1
    revision: 3
    type: mysql/metrics
    use_output: default
    meta:
      package:
        name: mysql
        version: 1.20.0
    data_stream:
      namespace: default
    package_policy_id: 1d521dab-3d64-48c7-ac99-99d6985fe9de
    streams:
      - id: mysql/metrics-mysql.galera_status-1d521dab-3d64-48c7-ac99-99d6985fe9de
        data_stream:
          dataset: mysql.galera_status
          type: metrics
        metricsets:
          - galera_status
        hosts:
          - 'tcp(127.0.0.1:3306)/'
        password: '${SECRET_0}'
        period: 10s
        username: root
        condition: '${host.name} == "mydb-server"'
      - id: mysql/metrics-mysql.performance-1d521dab-3d64-48c7-ac99-99d6985fe9de
        data_stream:
          dataset: mysql.performance
          type: metrics
        metricsets:
          - performance
        hosts:
          - 'tcp(127.0.0.1:3306)/'
        password: '${SECRET_0}'
        period: 10s
        username: root
        condition: '${host.name} == "mydb-server"'
      - id: mysql/metrics-mysql.status-1d521dab-3d64-48c7-ac99-99d6985fe9de
        data_stream:
          dataset: mysql.status
          type: metrics
        metricsets:
          - status
        hosts:
          - 'tcp(127.0.0.1:3306)/'
        password: '${SECRET_0}'
        period: 10s
        username: root
        condition: '${host.name} == "mydb-server"'

How to test this PR locally

1. Install current integration version
2. Upgrade integration to this PR
3. Add/Remove condition values.

Related issues

• Related Add condition and processors settings to SQL Input #6358

Screenshots

[ishleenk17]

Changes look fine.
Retriggering the CI.

[ishleenk17]

/test

[kush-elastic]

/test

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[ishleenk17]

/test

[ishleenk17]

/test

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 307ba4f

Failed CI Steps

• Check integrations mysql

History

• 💔 Build #14802 failed 2dab7d9
• 💔 Build #12084 failed 42dd9be
• 💔 Build #11848 failed 42dd9be

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[botelastic]

Hi! This PR has been stale for a while and we're going to close it as part of our cleanup procedure. We appreciate your contribution and would like to apologize if we have not been able to review it, due to the current heavy load of the team. Feel free to re-open this PR if you think it should stay open and is worth rebasing. Thank you for your contribution!