[microsoft_sqlserver] Add transaction log DataStream

[muthu-mps]

What does this PR do?

Collects the transaction log metrics for Microsoft Sql server.

Please refer this PR for performance datastream.

• Added a data stream (transaction_log) under microsoft_sqlserver integration package
• Added support to collect metrics from Named Instance.
• Added data collection logic for the data streams.
• Added fields metadata in the appropriate yaml files.
• Added dashboards and visualisations.
• Updated documentation for the datastreams

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

MSSQL server integration using generic sql, as per the design doc

How to test this PR locally

Install the MSSQL

• Go to Integration UI in Kibana
• Add the Microsoft SQL Server Integration
• Enroll the new elastic-agent
• Test named Instance by providing the named instance port in the Microsoft Sqlserver integration config.
• In the asset -> Transaction Log dashboard you should be able to see the default visualisation.

Screenshots

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-06-14T11:50:47.035+0000

• Duration: 15 min 20 sec

Test stats 🧪

Test	Results
Failed	0
Passed	12
Skipped	0
Total	12

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (1/1)	💚
Files	100.0% (1/1)	💚 3.432
Classes	100.0% (1/1)	💚 3.432
Methods	90.476% (19/21)	👍 1.587
Lines	100.0% (1198/1198)	💚 10.104
Conditionals	100.0% (0/0)	💚

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[ishleenk17]

Add ecs.yml file

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[lalit-satapathy]

@ManojS-shetty @muthu-mps,
Do we have any changes pending or all the feedbacks are resolved now?

@ruflin @jsoriano @r00tu53r, tar-getting the merge of this PR, need your approval for the same.

[muthu-mps]

The changes are done!

[jsoriano]

It looks good in general, but I am concerned about some changes in versions.

[jsoriano]

Actually, do we need the query at all in the final document? This is an implementation detail of the integration, I think that it could be removed.

But, if we are storing metrics coming from different queries in different documents we may need some dimension to distinguish between queries. So we may need to index the query to use it as dimension, or find some other field that could cover this.

We can also wait to have some way to join events (elastic/beats#31806 (comment)), then database_id will be probably enough as dimension.

[muthu-mps]

/test

[jsoriano]

Looks good, thanks for addressing all comments.

The main concern I would have now is about mssql.query. It is not clear to me if we are keeping it at the end or not.

Also I think that this data stream would benefit of elastic/beats#31806, to store all the metrics in the same document. And if we end up storing the results of the queries in the same document, what would we store in mssql.query? Would we remove it? This would be a breaking change.

The more I think about storing the query, the more I think that we shouldn't store it. And it is easier in any case to add it later if I am wrong, than to remove it if needed later.

Main concerns addressed, unblocking PR.

[jsoriano]

LGTM, please remember to add pipeline tests at some moment