Skip to content

Support user databases for transaction log #4869

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ritalwar merged 6 commits into from
Dec 22, 2022
Merged

Support user databases for transaction log #4869

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
6e49623
Support user databases for transaction log
ritalwar Dec 19, 2022
04cd8a3
Update changelog.yml
ritalwar Dec 19, 2022
61afa21
Update sample_event.json
ritalwar Dec 20, 2022
386e0ce
Update README.md
ritalwar Dec 20, 2022
50ce900
Address review comments
ritalwar Dec 21, 2022
3d9368a
Chages for short term solution
ritalwar Dec 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion packages/microsoft_sqlserver/_dev/build/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,8 @@ See: [Instructions about each performance counter metrics](https://docs.microsof

### transaction_log metrics

Collects system level `transaction_log` metrics information for SQL Server instance.
Collects `transaction_log` metrics information for all the system and user Dbs of SQL Server instance.

@lalit-satapathy lalit-satapathy Dec 20, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: let's be consistent with Dbs or dbs

The metrics from sys.dm_db_log_space_usage table for user dbs can be collected by providing list of user databases, by default it will collect metrics for system dbs.

See: [Instructions and the operations supported by transaction log](https://docs.microsoft.com/en-us/sql/relational-databases/logs/the-transaction-log-sql-server?view=sql-server-ver15)

Expand Down
5 changes: 5 additions & 0 deletions packages/microsoft_sqlserver/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.11.0"
changes:
- description: Support user databases for transaction log.
type: enhancement
link: https://github.com/elastic/integrations/pull/4869
- version: "1.10.0"
changes:
- description: Ingest instance name and server names.
Expand Down
18 changes: 11 additions & 7 deletions packages/microsoft_sqlserver/data_stream/transaction_log/agent/stream/stream.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,19 @@ hosts:
period: {{period}}
driver: mssql
raw_data.enabled: true
databases:
{{#each databases}}

@muthu-mps muthu-mps Dec 20, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this database iteration here ?

@ritalwar ritalwar Dec 21, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is needed to get the list of databases to fetch user dbs metrics, upon which the query iterator will work.

- {{this}}
{{/each}}
# Collect the transaction logs from the system database
sql_queries:
- query: "SELECT @@servername AS server_name, @@servicename AS instance_name, name As 'database_name', database_id FROM sys.databases;"
response_format: table
- query: "SELECT @@servername AS server_name, @@servicename AS instance_name, name As 'database_name', s.database_id, total_log_size_mb, active_log_size_mb,log_backup_time,log_since_last_log_backup_mb,log_since_last_checkpoint_mb,log_recovery_size_mb from sys.databases As s CROSS APPLY sys.dm_db_log_stats(s.database_id);"

@muthu-mps muthu-mps Dec 20, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When you show all db data for log_stats and only customer provided db data for log_space_usage, there is a possibility that customer will not provided the custom database names then Kibana will only represent data for log_stats and not the log_space_usage for the selected custom database. I would recommend keeping both the stats rendered in similar way.

@ritalwar ritalwar Dec 21, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As per discussion with Lalit, we are providing blend of long and short term solution, where we are giving all databases metrics for log_stats and for log_space_usage we are providing short term solution i.e user can input dbs and will get the metrics for same.
This has also been updated in docs.
Additional benefit is that user will have list of all the dbs as result of 1st query and they can give the names from there to get metrics of log_space_usage.

response_format: table
{{#if databases}}
{{#each databases as |database_name i|}}
- query: "SELECT @@servername AS server_name, @@servicename AS instance_name, name As 'database_name', database_id FROM sys.databases WHERE name='{{database_name}}';"
response_format: table
- query: "SELECT @@servername AS server_name, @@servicename AS instance_name, name As 'database_name', l.database_id, l.total_log_size_mb, l.active_log_size_mb,l.log_backup_time,l.log_since_last_log_backup_mb,l.log_since_last_checkpoint_mb,l.log_recovery_size_mb from sys.dm_db_log_stats(DB_ID('{{database_name}}')) l INNER JOIN sys.databases s ON l.database_id = s.database_id WHERE s.database_id = DB_ID('{{database_name}}') ;"
response_format: table
- query: "SELECT @@servername AS server_name, @@servicename AS instance_name, name As 'database_name', l.database_id, l.total_log_size_in_bytes As total_log_size_bytes, l.used_log_space_in_bytes As used_log_space_bytes, l.used_log_space_in_percent As used_log_space_pct, l.log_space_in_bytes_since_last_backup from sys.dm_db_log_space_usage l INNER JOIN sys.databases s ON l.database_id = s.database_id WHERE s.database_id = DB_ID('{{database_name}}') ;"
response_format: table
- query: "USE {{database_name}} ; SELECT @@servername AS server_name, @@servicename AS instance_name, name As 'database_name', l.database_id, l.total_log_size_in_bytes As total_log_size_bytes, l.used_log_space_in_bytes As used_log_space_bytes, l.used_log_space_in_percent As used_log_space_pct, l.log_space_in_bytes_since_last_backup from sys.dm_db_log_space_usage l INNER JOIN sys.databases s ON l.database_id = s.database_id WHERE s.database_id = DB_ID('{{database_name}}') ;"
response_format: table
{{/each}}
{{/if}}
{{/if}}
2 changes: 1 addition & 1 deletion packages/microsoft_sqlserver/data_stream/transaction_log/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ streams:
title: Databases
multi: true
required: true
show_user: false
show_user: true
description: Default system databases are preloaded. For custom database please add additional rows and enter the database name.

@lalit-satapathy lalit-satapathy Dec 20, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change from "custom database" to "user defined databases"?

default: ["master", "model", "tempdb", "msdb"]
title: Microsoft SQL Server transaction_log metrics
Expand Down
73 changes: 29 additions & 44 deletions packages/microsoft_sqlserver/data_stream/transaction_log/sample_event.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,67 +1,52 @@
{
"@timestamp": "2022-11-23T10:53:05.080Z",
"@timestamp": "2022-12-20T07:34:29.687Z",
"agent": {
"ephemeral_id": "70f5c0c1-37b1-486b-9806-8105b2cdcd20",
"id": "6d444a4a-2158-445e-8953-dc6eef720a34",
"name": "docker-fleet-agent",
"ephemeral_id": "8d528ff8-5e90-4572-89f6-61fb3a6c96f1",
"id": "d44a1c4a-95bf-47e9-afb0-453a2ef43c00",
"name": "192.168.1.2",
"type": "metricbeat",
"version": "8.5.0"
},
"cloud": {
"account": {},
"instance": {
"id": "b30e45e6-7900-4900-8d67-e37cb13374bc",
"name": "obs-int-windows-dev"
},
"machine": {
"type": "Standard_D16ds_v5"
},
"provider": "azure",
"region": "CentralIndia",
"service": {
"name": "Virtual Machines"
}
"version": "8.5.3"
},
"data_stream": {
"dataset": "microsoft_sqlserver.transaction_log",
"namespace": "ep",
"namespace": "default",
"type": "metrics"
},
"ecs": {
"version": "8.0.0"
},
"elastic_agent": {
"id": "6d444a4a-2158-445e-8953-dc6eef720a34",
"id": "d44a1c4a-95bf-47e9-afb0-453a2ef43c00",
"snapshot": false,
"version": "8.5.0"
"version": "8.5.3"
},
"event": {
"agent_id_status": "verified",
"dataset": "microsoft_sqlserver.transaction_log",
"duration": 1449739900,
"ingested": "2022-11-23T10:53:07Z",
"duration": 2147044750,
"ingested": "2022-12-20T07:34:32Z",
"module": "sql"
},
"host": {
"architecture": "x86_64",
"containerized": false,
"hostname": "docker-fleet-agent",
"id": "66392b0697b84641af8006d87aeb89f1",
"hostname": "192.168.1.2",
"id": "627E8AE5-E918-5073-A58E-8A2D9ED96875",
"ip": [
"172.18.0.5"
"192.168.1.2"
],
"mac": [
"02-42-AC-12-00-05"
"36-F7-DC-28-23-80"
],
"name": "docker-fleet-agent",
"name": "192.168.1.2",
"os": {
"codename": "focal",
"family": "debian",
"kernel": "5.10.104-linuxkit",
"name": "Ubuntu",
"platform": "ubuntu",
"type": "linux",
"version": "20.04.5 LTS (Focal Fossa)"
"build": "21D62",
"family": "darwin",
"kernel": "21.3.0",
"name": "macOS",
"platform": "darwin",
"type": "macos",
"version": "12.2.1"
}
},
"metricset": {
Expand All @@ -70,18 +55,18 @@
},
"mssql": {
"metrics": {
"database_id": 1,
"database_name": "master",
"server_name": "obs-int-mssql20",
"instance_name": "MSSQLSERVER",
"log_space_in_bytes_since_last_backup": 139264,
"server_name": "obs-service-int",
"database_name": "master",
"database_id": 1,
"log_space_in_bytes_since_last_backup": 602112,
"total_log_size_bytes": 2088960,
"used_log_space_bytes": 626688,
"used_log_space_pct": 30
"used_log_space_pct": 49.01960754394531,
"used_log_space_bytes": 1024000
}
},
"service": {
"address": "20.121.218.233",
"address": "20.228.135.242",
"type": "sql"
}
}
76 changes: 31 additions & 45 deletions packages/microsoft_sqlserver/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,8 @@ See: [Instructions about each performance counter metrics](https://docs.microsof

### transaction_log metrics

Collects system level `transaction_log` metrics information for SQL Server instance.
Collects `transaction_log` metrics information for all the system and user Dbs of SQL Server instance.
The metrics from sys.dm_db_log_space_usage table for user dbs can be collected by providing list of user databases, by default it will collect metrics for system dbs.

See: [Instructions and the operations supported by transaction log](https://docs.microsoft.com/en-us/sql/relational-databases/logs/the-transaction-log-sql-server?view=sql-server-ver15)

Expand Down Expand Up @@ -494,69 +495,54 @@ An example event for `transaction_log` looks as following:

```json
{
"@timestamp": "2022-11-23T10:53:05.080Z",
"@timestamp": "2022-12-20T07:34:29.687Z",
"agent": {
"ephemeral_id": "70f5c0c1-37b1-486b-9806-8105b2cdcd20",
"id": "6d444a4a-2158-445e-8953-dc6eef720a34",
"name": "docker-fleet-agent",
"ephemeral_id": "8d528ff8-5e90-4572-89f6-61fb3a6c96f1",
"id": "d44a1c4a-95bf-47e9-afb0-453a2ef43c00",
"name": "192.168.1.2",
"type": "metricbeat",
"version": "8.5.0"
},
"cloud": {
"account": {},
"instance": {
"id": "b30e45e6-7900-4900-8d67-e37cb13374bc",
"name": "obs-int-windows-dev"
},
"machine": {
"type": "Standard_D16ds_v5"
},
"provider": "azure",
"region": "CentralIndia",
"service": {
"name": "Virtual Machines"
}
"version": "8.5.3"
},
"data_stream": {
"dataset": "microsoft_sqlserver.transaction_log",
"namespace": "ep",
"namespace": "default",
"type": "metrics"
},
"ecs": {
"version": "8.0.0"
},
"elastic_agent": {
"id": "6d444a4a-2158-445e-8953-dc6eef720a34",
"id": "d44a1c4a-95bf-47e9-afb0-453a2ef43c00",
"snapshot": false,
"version": "8.5.0"
"version": "8.5.3"
},
"event": {
"agent_id_status": "verified",
"dataset": "microsoft_sqlserver.transaction_log",
"duration": 1449739900,
"ingested": "2022-11-23T10:53:07Z",
"duration": 2147044750,
"ingested": "2022-12-20T07:34:32Z",
"module": "sql"
},
"host": {
"architecture": "x86_64",
"containerized": false,
"hostname": "docker-fleet-agent",
"id": "66392b0697b84641af8006d87aeb89f1",
"hostname": "192.168.1.2",
"id": "627E8AE5-E918-5073-A58E-8A2D9ED96875",
"ip": [
"172.18.0.5"
"192.168.1.2"
],
"mac": [
"02-42-AC-12-00-05"
"36-F7-DC-28-23-80"
],
"name": "docker-fleet-agent",
"name": "192.168.1.2",
"os": {
"codename": "focal",
"family": "debian",
"kernel": "5.10.104-linuxkit",
"name": "Ubuntu",
"platform": "ubuntu",
"type": "linux",
"version": "20.04.5 LTS (Focal Fossa)"
"build": "21D62",
"family": "darwin",
"kernel": "21.3.0",
"name": "macOS",
"platform": "darwin",
"type": "macos",
"version": "12.2.1"
}
},
"metricset": {
Expand All @@ -565,18 +551,18 @@ An example event for `transaction_log` looks as following:
},
"mssql": {
"metrics": {
"database_id": 1,
"database_name": "master",
"server_name": "obs-int-mssql20",
"instance_name": "MSSQLSERVER",
"log_space_in_bytes_since_last_backup": 139264,
"server_name": "obs-service-int",
"database_name": "master",
"database_id": 1,
"log_space_in_bytes_since_last_backup": 602112,
"total_log_size_bytes": 2088960,
"used_log_space_bytes": 626688,
"used_log_space_pct": 30
"used_log_space_pct": 49.01960754394531,
"used_log_space_bytes": 1024000
}
},
"service": {
"address": "20.121.218.233",
"address": "20.228.135.242",
"type": "sql"
}
}
Expand Down
2 changes: 1 addition & 1 deletion packages/microsoft_sqlserver/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: 1.0.0
name: microsoft_sqlserver
title: "Microsoft SQL Server"
version: "1.10.0"
version: "1.11.0"
license: basic
description: Collect events from Microsoft SQL Server with Elastic Agent
type: integration
Expand Down