[microsoft_sqlserver] Add transaction log DataStream

.github/CODEOWNERS

@@ -96,7 +96,7 @@
 /packages/microsoft_defender_endpoint @elastic/security-external-integrations
 /packages/microsoft_dhcp @elastic/security-external-integrations
 /packages/microsoft @elastic/security-external-integrations
-/packages/microsoft_sqlserver @elastic/security-external-integrations
+/packages/microsoft_sqlserver @elastic/security-external-integrations @elastic/obs-service-integrations
 /packages/mimecast @elastic/security-external-integrations
 /packages/modsecurity @elastic/security-external-integrations
 /packages/mongodb @elastic/obs-service-integrations

packages/microsoft_sqlserver/_dev/build/build.yml

@@ -1,3 +1,3 @@
 dependencies:
   ecs:
-    reference: git@v8.2.0
+    reference: git@v8.2.1

packages/microsoft_sqlserver/_dev/build/docs/README.md

@@ -1,16 +1,28 @@
 # Microsoft SQL Server Integration
 
-The Microsoft SQL Server integration package allows you to search, observe and visualize the SQL Server audit events through Elasticsearch. 
+The Microsoft SQL Server integration package allows you to search, observe and visualize the SQL Server audit logs and metrics through Elasticsearch. 
+
 Auditing an instance of the SQL Server Database Engine or an individual database involves tracking and logging events that occur on the Database Engine. 
 SQL Server audit lets you create server audits, which can contain server audit specifications for server level events, and database audit specifications for database level events. 
 See: [SQL Server Audit page](https://docs.microsoft.com/en-us/sql/relational-databases/security/auditing/sql-server-audit-database-engine?view=sql-server-ver15) for more information on SQL Server auditing.
 
+The `performance` metrics gathers the list of performance objects available on that server. Each server will have a different list of performance objects depending on the installed software.
+`transaction_log` metrics collects all usage stats and the total space usage.
+
+## Named Instance
+
+The Microsoft SQL Server has a feature that allows to run multiple databases on the same host (or Clustered hosts) with separate settings. Edit the instance port and provide the named instance port to connect to the named instance and collect metrics.
+
+See: [Instruction on how to configure server to listen Named Instance port](https://docs.microsoft.com/en-us/sql/database-engine/configure-windows/configure-a-server-to-listen-on-a-specific-tcp-port?view=sql-server-ver15)
+
 ## Compatibility
 
-The package collects audit events from the event log. Other log sources such as file are not supported.
+The package collects `performance`, `transaction_log` metrics and `audit` events from the event log. Other log sources such as file are not supported.
 
 ## Configuration
 
+### audit
+
 There are several levels of auditing for SQL Server, depending on government or standards requirements for your installation. The SQL Server Audit feature enables you to audit server-level and database-level groups of events and individual events. 
 
 See: [SQL Server Audit Action Groups and Actions](https://docs.microsoft.com/en-us/sql/relational-databases/security/auditing/sql-server-audit-action-groups-and-actions?view=sql-server-ver15) for more information on the different audit levels.
@@ -19,14 +31,41 @@ See: [Instructions on how to enable auditing for SQL Server](https://docs.micros
 
 >Note: For the integration package to be able to read and send audit events the event target must be configured to be Windows event log.
 
-### Audit Events
+### audit events
 
 Enable to collect SQL Server audit events from the specified windows event log channel.
 
+### performance metrics
+
+Collects the `performance` counter metrics. Dynamic counter feature provides flexibility to collect metrics by providing the counter name as an input.
+
+See: [Instructions about each performance counter metrics](https://docs.microsoft.com/en-us/sql/relational-databases/system-dynamic-management-views/sys-dm-os-performance-counters-transact-sql?view=sql-server-ver15
+)
+
+### transaction_log metrics
+
+Collects system level `transaction_log` metrics information for SQL Server instance.
+
+See: [Instructions and the operations supported by transaction log](https://docs.microsoft.com/en-us/sql/relational-databases/logs/the-transaction-log-sql-server?view=sql-server-ver15)
+
 ## Logs
 
-### Audit
+### audit
 
 The SQL Server audit dataset provides events from the configured Windows event log channel. All SQL Server audit specific fields are available in the `sqlserver.audit` field group.
 
 {{fields "audit"}}
+
+## Metrics
+
+### performance
+
+The Microsoft SQL Server `performance` dataset provides events from the performance counter table. All `performance` metrics will be available in `sqlserver.metrics` field group.
+
+### transaction_log
+
+The Microsoft SQL Server `transaction_log` dataset provides events from the log space usage and log stats tables of the system databases. All `transaction_log` metrics will be available in `sqlserver.metrics` field group.
+
+{{event "transaction_log"}}
+
+{{fields "transaction_log"}}

packages/microsoft_sqlserver/_dev/deploy/docker/Dockerfile

@@ -0,0 +1,9 @@
+ARG IMAGE=${IMAGE:-mcr.microsoft.com/mssql/server:2019-latest}
+FROM ${IMAGE}
+
+ENV ACCEPT_EULA='Y'
+ENV SA_PASSWORD='1234_asdf'
+
+# Use the same healthcheck as the Windows version of the image.
+# https://github.com/Microsoft/mssql-docker/blob/a3020afeec9be1eb2d67645ac739438eb8f2c545/windows/mssql-server-windows/dockerfile#L31
+HEALTHCHECK --interval=1s --retries=90 CMD /opt/mssql-tools/bin/sqlcmd -U SA -P ${SA_PASSWORD} -Q "select 1"

packages/microsoft_sqlserver/_dev/deploy/docker/docker-compose.yml

@@ -0,0 +1,10 @@
+version: '2.3'
+services:
+  microsoft_sqlserver:
+    image: mcr.microsoft.com/mssql/server:2019-latest
+    build:
+      context: .
+      args:
+        MSSQL_VERSION: ${MSSQL_VERSION:-2019-latest}
+    ports:
+      - 1433

packages/microsoft_sqlserver/_dev/deploy/variants.yml

@@ -0,0 +1,4 @@
+variants:
+  MSSQL_2019-latest:
+    IMAGE: 2019-latest
+default: 2019-latest

packages/microsoft_sqlserver/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.1.0"
+  changes:
+    - description: Added transaction log datastream
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/3395
 - version: "1.0.0"
   changes:
     - description: Make GA

packages/microsoft_sqlserver/data_stream/transaction_log/_dev/test/system/test-transaction-config.yml

@@ -0,0 +1,5 @@
+vars:
+  hosts:
+    - "{{Hostname}}"
+  username: sa
+  password: 1234_asdf

packages/microsoft_sqlserver/data_stream/transaction_log/agent/stream/stream.yml.hbs

@@ -0,0 +1,33 @@
+metricsets: ["query"]
+# Specify hosts in the below format. TODO:hosts need to be updated to support multiple entries.
+hosts:
+    - sqlserver://{{username}}:{{password}}@{{hosts}}:{{port}}
+period: {{period}}
+driver: mssql
+raw_data.enabled: true
+# Collect the transaction logs from the system database
+sql_queries:
+    - query: "SELECT name As 'database_name', database_id FROM sys.databases WHERE database_id=1;"
+      response_format: table
+    - query: "SELECT 'master' As database_name, database_id,total_log_size_mb,active_log_size_mb,log_backup_time,log_since_last_log_backup_mb,log_since_last_checkpoint_mb,log_recovery_size_mb FROM sys.dm_db_log_stats(1) master"
+      response_format: table
+    - query: "SELECT 'master' As 'database_name', total_log_size_in_bytes As total_log_size_bytes, used_log_space_in_bytes As used_log_space_bytes, used_log_space_in_percent As used_log_space_pct, log_space_in_bytes_since_last_backup FROM sys.dm_db_log_space_usage master"
+      response_format: table
+    - query: "SELECT  name As 'database_name', database_id FROM sys.databases WHERE database_id=2;"
+      response_format: table
+    - query: "SELECT 'tempdb' As 'database_name', database_id,total_log_size_mb,active_log_size_mb As active_log_size,log_backup_time,log_since_last_log_backup_mb, log_since_last_checkpoint_mb,log_recovery_size_mb FROM sys.dm_db_log_stats(2) tempdb"
+      response_format: table
+    - query: "SELECT 'tempdb' As 'database_name', total_log_size_in_bytes As total_log_size_bytes, used_log_space_in_bytes As used_log_space_bytes, used_log_space_in_percent As used_log_space_pct, log_space_in_bytes_since_last_backup FROM sys.dm_db_log_space_usage tempdb"
+      response_format: table
+    - query: "SELECT  name As 'database_name', database_id FROM sys.databases WHERE database_id=3;"
+      response_format: table
+    - query: "SELECT 'model' As 'database_name', database_id,total_log_size_mb,active_log_size_mb As active_log_size,log_backup_time,log_since_last_log_backup_mb, log_since_last_checkpoint_mb,log_recovery_size_mb FROM sys.dm_db_log_stats(3) model"
+      response_format: table
+    - query: "SELECT 'model' As 'database_name', total_log_size_in_bytes As total_log_size_bytes, used_log_space_in_bytes As used_log_space_bytes, used_log_space_in_percent As used_log_space_pct, log_space_in_bytes_since_last_backup FROM sys.dm_db_log_space_usage model"
+      response_format: table
+    - query: "SELECT name As 'database_name', database_id FROM sys.databases WHERE database_id=4;"
+      response_format: table
+    - query: "SELECT 'msdb' As 'database_name', database_id,total_log_size_mb,active_log_size_mb As active_log_size,log_backup_time,log_since_last_log_backup_mb, log_since_last_checkpoint_mb,log_recovery_size_mb FROM sys.dm_db_log_stats(4) msdb"
+      response_format: table
+    - query: "SELECT 'msdb' As 'database_name', total_log_size_in_bytes As total_log_size_bytes, used_log_space_in_bytes As used_log_space_bytes, used_log_space_in_percent As used_log_space_pct, log_space_in_bytes_since_last_backup FROM sys.dm_db_log_space_usage msdb"
+      response_format: table

packages/microsoft_sqlserver/data_stream/transaction_log/elasticsearch/ingest_pipeline/default.yml

@@ -0,0 +1,75 @@
+---
+description: Pipeline for processing mssql transaction_log
+processors:
+- remove:
+    field: sql.driver
+    ignore_failure: true
+    ignore_missing: true
+- remove:
+    field: sql.query
+    ignore_failure: true
+    ignore_missing: true
+- rename:
+    field: sql
+    target_field: mssql
+    ignore_missing: true
+    ignore_failure: true
+- rename:
+    field: mssql.metrics.log_since_last_checkpoint_mb
+    target_field: mssql.metrics.log_since_last_checkpoint
+    ignore_missing: true
+    ignore_failure: true
+- rename:
+    field: mssql.metrics.log_recovery_size_mb
+    target_field: mssql.metrics.log_recovery_size
+    ignore_missing: true
+    ignore_failure: true
+- rename:
+    field: mssql.metrics.total_log_size_mb
+    target_field: mssql.metrics.total_log_size
+    ignore_missing: true
+    ignore_failure: true
+- rename:
+    field: mssql.metrics.active_log_size_mb
+    target_field: mssql.metrics.active_log_size
+    ignore_missing: true
+    ignore_failure: true
+- rename:
+    field: mssql.metrics.log_since_last_log_backup_mb
+    target_field: mssql.metrics.log_since_last_log_backup
+    ignore_missing: true
+    ignore_failure: true
+- script:
+    lang: painless
+    source: ctx.mssql.metrics.log_since_last_checkpoint = Math.round(ctx.mssql.metrics.log_since_last_checkpoint * params.scale)
+    params:
+        scale: 1048576
+    if: ctx.mssql.metrics.log_since_last_checkpoint != null
+- script:
+    lang: painless
+    source: ctx.mssql.metrics.log_recovery_size = Math.round(ctx.mssql.metrics.log_recovery_size * params.scale)
+    params:
+      scale: 1048576
+    if: ctx.mssql.metrics.log_recovery_size != null
+- script:
+    lang: painless
+    source: ctx.mssql.metrics.total_log_size = Math.round(ctx.mssql.metrics.total_log_size * params.scale)
+    params:
+      scale: 1048576
+    if: ctx.mssql.metrics.total_log_size != null
+- script:
+    lang: painless
+    source: ctx.mssql.metrics.active_log_size = Math.round(ctx.mssql.metrics.active_log_size * params.scale)
+    params:
+      scale: 1048576
+    if: ctx.mssql.metrics.active_log_size != null
+- script:
+    lang: painless
+    source: ctx.mssql.metrics.log_since_last_log_backup = Math.round(ctx.mssql.metrics.log_since_last_log_backup * params.scale)
+    params:
+      scale: 1048576
+    if: ctx.mssql.metrics.log_since_last_log_backup != null
+on_failure:
+- set:
+    field: error.message
+    value: "{{ _ingest.on_failure_message }}"

packages/microsoft_sqlserver/data_stream/transaction_log/fields/base-fields.yml

@@ -0,0 +1,8 @@
+- name: data_stream.type
+  external: ecs
+- name: data_stream.dataset
+  external: ecs
+- name: data_stream.namespace
+  external: ecs
+- name: '@timestamp'
+  external: ecs

packages/microsoft_sqlserver/data_stream/transaction_log/fields/ecs.yml

@@ -0,0 +1,6 @@
+- external: ecs
+  name: ecs.version
+- external: ecs
+  name: service.address
+- external: ecs
+  name: service.type

packages/microsoft_sqlserver/data_stream/transaction_log/fields/fields.yml

@@ -0,0 +1,62 @@
+- name: mssql
+  type: group
+  release: beta
+  fields:
+    - name: metrics
+      type: group
+      fields:
+        - name: database_id
+          type: long
+          dimension: true
+          description: Unique ID of the database inside MSSQL.
+        - name: database_name
+          type: keyword
+          description: Name of the database.
+        - name: log_since_last_checkpoint
+          type: long
+          unit: byte
+          metric_type: gauge
+          description: Log size in bytes since last checkpoint log sequence number (LSN).
+        - name: log_recovery_size
+          type: long
+          unit: byte
+          metric_type: gauge
+          description: Log size in bytes since log recovery log sequence number (LSN).
+        - name: total_log_size
+          type: long
+          unit: byte
+          metric_type: counter
+          description: Total log size.
+        - name: log_backup_time
+          type: date
+          description: Last transaction log backup time.
+        - name: active_log_size
+          type: long
+          unit: byte
+          metric_type: counter
+          description: Total active transaction log size in bytes.
+        - name: log_since_last_log_backup
+          type: long
+          unit: byte
+          metric_type: gauge
+          description: Log file size since last backup in bytes.
+        - name: used_log_space_pct
+          type: float
+          unit: percent
+          metric_type: gauge
+          description: A percentage of the occupied size of the log as a percent of the total log size.
+        - name: used_log_space_bytes
+          type: long
+          unit: byte
+          metric_type: gauge
+          description: The occupied size of the log in bytes.
+        - name: log_space_in_bytes_since_last_backup
+          type: long
+          unit: byte
+          metric_type: gauge
+          description: The amount of space used since the last log backup in bytes.
+        - name: total_log_size_bytes
+          type: long
+          unit: byte
+          metric_type: counter
+          description: Total transaction log size in bytes.

packages/microsoft_sqlserver/data_stream/transaction_log/manifest.yml

@@ -0,0 +1,16 @@
+type: metrics
+title: Microsoft SQL Server transaction_log metrics
+release: beta
+streams:
+  - input: sql/metrics
+    enabled: false
+    vars:
+      - name: period
+        type: text
+        title: Period
+        multi: false
+        required: true
+        show_user: true
+        default: 60s
+    title: Microsoft SQL Server transaction_log metrics
+    description: Collect Microsoft SQL Server transaction_log metrics

packages/microsoft_sqlserver/data_stream/transaction_log/sample_event.json

@@ -0,0 +1,47 @@
+{
+    "@timestamp": "2022-06-08T10:20:14.787809Z",
+    "mssql": {
+        "metrics": {
+            "database_name": "msdb",
+            "database_id": 1,
+            "used_log_space_bytes": 41.17647171020508,
+            "log_space_in_bytes_since_last_backup": 397312,
+            "total_log_size_bytes": 2088960,
+            "used_log_space_pct": 860160
+        }
+    },
+    "metricset": {
+        "period": 10000,
+        "name": "query"
+    },
+    "agent": {
+        "id": "e7b17c22-4223-46c3-b982-ff0d570b5fa6",
+        "ephemeral_id": "d1a76cf4-2463-478a-a474-36e771218467",
+        "type": "metricbeat",
+        "version": "8.3.0"
+    },
+    "service": {
+        "address": "54.90.251.237:1433",
+        "type": "sql"
+    },
+    "elastic_agent": {
+        "id": "e7b17c22-4223-46c3-b982-ff0d570b5fa6",
+        "version": "8.3.0",
+        "snapshot": true
+    },
+    "event": {
+        "duration": 5595352584,
+        "agent_id_status": "verified",
+        "ingested": "2022-05-23T10:20:21Z",
+        "module": "sql",
+        "dataset": "microsoft_sqlserver.transaction_log"
+    },
+    "data_stream": {
+        "namespace": "default",
+        "type": "metrics",
+        "dataset": "microsoft_sqlserver.transaction_log"
+    },
+    "ecs": {
+        "version": "8.0.0"
+    }
+}