microsoft_sqlserver/data_stream/transaction_log: Add support for fetch_from_all_databases for mssql

[shmsr]

What does this PR do?

To make the Microsoft SQL Server integration setup easier, there's no simple way of getting the predefined queries run against all databases of a server to get the desired metrics. Currently, the only way to add, modify or remove databases (both system and user-defined) is to do them manually while configuring the integration. However, if there are a large number of databases on the server, this manual process can be tedious and time-consuming. To address this issue, there is an option to enable fetching the required transaction_log metrics from all databases on the server, including system and user-defined databases.

This option is controlled by a boolean field called fetch_from_all_databases. By default, this feature is set to false. To use it, simply set the field to true. This will allow the integration to automatically find and fetch metrics from all databases on the server, without requiring users to manually enter each individual database.

Also, fixed some issues in the documentation, etc.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

• Tested on Mac OS Ventura 13.3.1
• Tested against mcr.microsoft.com/azure-sql-edge
• Add a user-defined database called hello_world. Already four system databases are present: master, model, tempdb and msdb
• Metrics should now be available for all 5 databases without the need to specify the database names in the configuration manually

docker run --cap-add SYS_PTRACE -e 'ACCEPT_EULA=Y' -e 'MSSQL_SA_PASSWORD=<password>' -p 1433:1433 --name sqledge -d mcr.microsoft.com/azure-sql-edge

User: SA
Password: *****

Related issues

• Closes MSSQL enhancements: support user databases #4108
• Closes Add support for fetch_from_all_databases in integrations for mssql #7499
• Relates x-pack/metricbeat/module/sql: Add option to execute SQL queries for all databases beats#35688

Screenshots

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-09-08T11:13:22.221+0000

• Duration: 18 min 44 sec

Test stats 🧪

Test	Results
Failed	0
Passed	19
Skipped	0
Total	19

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[lalit-satapathy]

Let's add a documentation update section in the integration README describing the capability from the user perspective.

[shmsr]

Let's add a documentation update section in the integration README describing the capability from the user perspective.

Sure.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (2/2)	💚
Files	100.0% (2/2)	💚
Classes	100.0% (2/2)	💚
Methods	93.75% (30/32)	👍 6.25
Lines	100.0% (1251/1251)	💚 8.02
Conditionals	100.0% (0/0)	💚

[shmsr]

Let's add a documentation update section in the integration README describing the capability from the user perspective.

@lalit-satapathy I have updated the docs.

[muthu-mps]

When we enable the fetch_all_databases toggle in the UI, Is it possible to disable the databases input section in the UI?

[shmsr]

When we enable the fetch_all_databases toggle in the UI, Is it possible to disable the databases input section in the UI?

Such interactivity in the UI is not there AFAIK.

[shmsr]

Linking https://github.com/orgs/elastic/projects/899/views/11?filterQuery=iteration%3A%40current+aliab&pane=issue&itemId=37096632 as per suggestion.

[ali786XI]

LGTM

[muthu-mps]

LGTM!

[elasticmachine]

Package microsoft_sqlserver - 2.2.0 containing this change is available at https://epr.elastic.co/search?package=microsoft_sqlserver