Add the PostgreSQL database metricset (#2266)

* Interrogates pg_stat_database for stats about each database.
* Adds system tests

Part of #2255.

metricbeat/docs/fields.asciidoc

@@ -1276,7 +1276,7 @@ PostgreSQL metrics.
 [float]
 == activity Fields
 
-One document per server process, showing information related to the current activity of that process, such as state and current query.
+One document per server process, showing information related to the current activity of that process, such as state and current query. Collected by querying pg_stat_activity.
 
 
 
@@ -1406,6 +1406,165 @@ Current overall state of this backend. Possible values are:
 Text of this backend's most recent query. If state is active this field shows the currently executing query. In all other states, it shows the last query that was executed.
 
 
+[float]
+== database Fields
+
+One row per database, showing database-wide statistics. Collected by querying pg_stat_database
+
+
+
+[float]
+=== postgresql.database.oid
+
+type: long
+
+OID of the database this backend is connected to.
+
+
+[float]
+=== postgresql.database.name
+
+type: keyword
+
+Name of the database this backend is connected to.
+
+
+[float]
+=== postgresql.database.number_of_backends
+
+type: long
+
+Number of backends currently connected to this database.
+
+
+[float]
+=== postgresql.database.transactions.commit
+
+type: long
+
+Number of transactions in this database that have been committed.
+
+
+[float]
+=== postgresql.database.transactions.rollback
+
+type: long
+
+Number of transactions in this database that have been rolled back.
+
+
+[float]
+=== postgresql.database.blocks.read
+
+type: long
+
+Number of disk blocks read in this database.
+
+
+[float]
+=== postgresql.database.blocks.hit
+
+type: long
+
+Number of times disk blocks were found already in the buffer cache, so that a read was not necessary (this only includes hits in the PostgreSQL buffer cache, not the operating system's file system cache).
+
+
+[float]
+=== postgresql.database.blocks.time.read.ms
+
+type: long
+
+Time spent reading data file blocks by backends in this database, in milliseconds.
+
+
+[float]
+=== postgresql.database.blocks.time.write.ms
+
+type: long
+
+Time spent writing data file blocks by backends in this database, in milliseconds.
+
+
+[float]
+=== postgresql.database.rows.returned
+
+type: long
+
+Number of rows returned by queries in this database.
+
+
+[float]
+=== postgresql.database.rows.fetched
+
+type: long
+
+Number of rows fetched by queries in this database.
+
+
+[float]
+=== postgresql.database.rows.inserted
+
+type: long
+
+Number of rows inserted by queries in this database.
+
+
+[float]
+=== postgresql.database.rows.updated
+
+type: long
+
+Number of rows updated by queries in this database.
+
+
+[float]
+=== postgresql.database.rows.deleted
+
+type: long
+
+Number of rows deleted by queries in this database.
+
+
+[float]
+=== postgresql.database.conflicts
+
+type: long
+
+Number of queries canceled due to conflicts with recovery in this database.
+
+
+[float]
+=== postgresql.database.temporary.files
+
+type: long
+
+Number of temporary files created by queries in this database. All temporary files are counted, regardless of why the temporary file was created (e.g., sorting or hashing), and regardless of the log_temp_files setting.
+
+
+[float]
+=== postgresql.database.temporary.bytes
+
+type: long
+
+Total amount of data written to temporary files by queries in this database. All temporary files are counted, regardless of why the temporary file was created, and regardless of the log_temp_files setting.
+
+
+[float]
+=== postgresql.database.deadlocks
+
+type: long
+
+Number of deadlocks detected in this database.
+
+
+[float]
+=== postgresql.database.stats_reset
+
+type: date
+
+Time at which these statistics were last reset.
+
+
 [[exported-fields-redis]]
 == Redis Fields
 

metricbeat/docs/modules/postgresql.asciidoc

@@ -19,7 +19,12 @@ in <<configuration-metricbeat>>. Here is an example configuration:
 ----
 metricbeat.modules:
 #- module: postgresql
-  #metricsets: ["activity"]
+  #metricsets:
+    # Stats about every PostgreSQL database
+    #- database
+
+    # Stats about every PostgreSQL process
+    #-activity
   #enabled: true
   #period: 10s
 
@@ -38,5 +43,9 @@ The following metricsets are available:
 
 * <<metricbeat-metricset-postgresql-activity,activity>>
 
+* <<metricbeat-metricset-postgresql-database,database>>
+
 include::postgresql/activity.asciidoc[]
 
+include::postgresql/database.asciidoc[]
+

metricbeat/docs/modules/postgresql/database.asciidoc

@@ -0,0 +1,19 @@
+////
+This file is generated! See scripts/docs_collector.py
+////
+
+[[metricbeat-metricset-postgresql-database]]
+include::../../../module/postgresql/database/_meta/docs.asciidoc[]
+
+
+==== Fields
+
+For a description of each field in the metricset, see the
+<<exported-fields-postgresql,exported fields>> section.
+
+Here is an example document generated by this metricset:
+
+[source,json]
+----
+include::../../../module/postgresql/database/_meta/data.json[]
+----

metricbeat/etc/beat.full.yml

@@ -105,7 +105,12 @@ metricbeat.modules:
 
 #----------------------------- PostgreSQL Module -----------------------------
 #- module: postgresql
-  #metricsets: ["activity"]
+  #metricsets:
+    # Stats about every PostgreSQL database
+    #- database
+
+    # Stats about every PostgreSQL process
+    #-activity
   #enabled: true
   #period: 10s
 

metricbeat/etc/fields.yml

@@ -742,7 +742,8 @@
           type: group
           description: >
             One document per server process, showing information related to the current
-            activity of that process, such as state and current query.
+            activity of that process, such as state and current query. Collected by
+            querying pg_stat_activity.
           fields:
             - name: database.oid
               type: long
@@ -818,6 +819,100 @@
                 shows the currently executing query. In all other states, it shows the
                 last query that was executed.
 
+        - name: database
+          type: group
+          description: >
+            One row per database, showing database-wide statistics. Collected by querying
+            pg_stat_database
+          fields:
+            - name: oid
+              type: long
+              description: >
+                OID of the database this backend is connected to.
+            - name: name
+              type: keyword
+              description: >
+                Name of the database this backend is connected to.
+            - name: number_of_backends
+              type: long
+              description: >
+                Number of backends currently connected to this database.
+            - name: transactions.commit
+              type: long
+              description: >
+                Number of transactions in this database that have been committed.
+            - name: transactions.rollback
+              type: long
+              description: >
+                Number of transactions in this database that have been rolled back.
+            - name: blocks.read
+              type: long
+              description: >
+                Number of disk blocks read in this database.
+            - name: blocks.hit
+              type: long
+              description: >
+                Number of times disk blocks were found already in the buffer cache, so
+                that a read was not necessary (this only includes hits in the PostgreSQL
+                buffer cache, not the operating system's file system cache).
+            - name: blocks.time.read.ms
+              type: long
+              description: >
+                Time spent reading data file blocks by backends in this database, in
+                milliseconds.
+            - name: blocks.time.write.ms
+              type: long
+              description: >
+                Time spent writing data file blocks by backends in this database, in
+                milliseconds.
+            - name: rows.returned
+              type: long
+              description: >
+                Number of rows returned by queries in this database.
+            - name: rows.fetched
+              type: long
+              description: >
+                Number of rows fetched by queries in this database.
+            - name: rows.inserted
+              type: long
+              description: >
+                Number of rows inserted by queries in this database.
+            - name: rows.updated
+              type: long
+              description: >
+                Number of rows updated by queries in this database.
+            - name: rows.deleted
+              type: long
+              description: >
+                Number of rows deleted by queries in this database.
+            - name: conflicts
+              type: long
+              description: >
+                Number of queries canceled due to conflicts with recovery in this
+                database.
+            - name: temporary.files
+              type: long
+              description: >
+                Number of temporary files created by queries in this database. All
+                temporary files are counted, regardless of why the temporary file was
+                created (e.g., sorting or hashing), and regardless of the log_temp_files
+                setting.
+            - name: temporary.bytes
+              type: long
+              description: >
+                Total amount of data written to temporary files by queries in this
+                database. All temporary files are counted, regardless of why the
+                temporary file was created, and regardless of the log_temp_files
+                setting.
+            - name: deadlocks
+              type: long
+              description: >
+                Number of deadlocks detected in this database.
+            - name: stats_reset
+              type: date
+              description: >
+                Time at which these statistics were last reset.
+
 - key: redis
   title: "Redis"
   description: >

metricbeat/include/list.go

@@ -18,6 +18,7 @@ import (
 	_ "github.com/elastic/beats/metricbeat/module/nginx/stubstatus"
 	_ "github.com/elastic/beats/metricbeat/module/postgresql"
 	_ "github.com/elastic/beats/metricbeat/module/postgresql/activity"
+	_ "github.com/elastic/beats/metricbeat/module/postgresql/database"
 	_ "github.com/elastic/beats/metricbeat/module/redis"
 	_ "github.com/elastic/beats/metricbeat/module/redis/info"
 	_ "github.com/elastic/beats/metricbeat/module/redis/keyspace"

metricbeat/metricbeat.full.yml

@@ -105,7 +105,12 @@ metricbeat.modules:
 
 #----------------------------- PostgreSQL Module -----------------------------
 #- module: postgresql
-  #metricsets: ["activity"]
+  #metricsets:
+    # Stats about every PostgreSQL database
+    #- database
+
+    # Stats about every PostgreSQL process
+    #-activity
   #enabled: true
   #period: 10s