Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mongo dbstats #3228

Merged
merged 38 commits into from
Jan 11, 2017
Merged

Mongo dbstats #3228

Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
9274966
initial dbstats implementation and integration tests
Dec 19, 2016
f28f347
implemented fields.yml for dbstats metricset
Dec 19, 2016
c811c3e
generated documentation and templates via make update
Dec 20, 2016
590b9e2
added dbstats documentation
Dec 20, 2016
b6f8da3
new implementation that establishes direct connetions to mongo hosts
Dec 22, 2016
74f9ee0
successful reporting using direct node connections
Dec 28, 2016
dd895b5
debugged field name. integration tests pass
Dec 29, 2016
2c92245
debugged avg_obj_size field for proper mapping
Dec 29, 2016
5a9ae7f
set FailFast = True for direct connections to prevent a nonresponsive…
Dec 29, 2016
8e1934c
provide mongodb host parser to AddMetricSet()
Dec 29, 2016
1f420d7
implementing multi-node direct reporting for mongodb.serverStatus
Dec 29, 2016
65d8d89
changed database response back to map[string]interface{}{}. otherwis…
Dec 29, 2016
a4a6e75
debugging test assertion values so integration test suite passes
Dec 29, 2016
3c6d97c
debugged integration tests to work with metricbeats testsuite. added …
Dec 29, 2016
aafaafe
copied original metricbeat.yml. Accidentally over-wrote it
Dec 29, 2016
663b4da
Merge pull request #1 from scottcrespo/metricbeat-mongodb-persistent-…
Dec 29, 2016
99bc01f
added documentation and additional log entry in mongodb module
Dec 30, 2016
82be711
more info logs to debug travis ci failures
Dec 30, 2016
98b86a0
additional logging to debug CI. removed host parameter to new metricset
Dec 30, 2016
f207fac
updated docstring. add mongo url parser back to new metric set call
Dec 30, 2016
2b3f230
added build integration comment to integration test modules
Dec 30, 2016
abf8973
eliminated multi-node concurrency patterns because it is handled by t…
Dec 30, 2016
5bde2ee
implementing eventfetcher, not eventsfetcher
Dec 30, 2016
976294a
updated module config to include dbstats metricset
Dec 31, 2016
7b0a38f
ran make update
Dec 31, 2016
827348c
updated metricbeat mongo modules template configuration to list metri…
Jan 8, 2017
2de87a8
udated mongobeat dbstats fields to use byte format where appropriate
Jan 8, 2017
54009f4
handle mb field
Jan 8, 2017
594085b
update import list order
Jan 8, 2017
48f9a53
updating mongodb metricset field format
Jan 8, 2017
757dbd0
added experimental flag to mongodb dbstats metricset
Jan 8, 2017
06b7220
removed redundant Printf statement
Jan 8, 2017
e4b627a
added debugf variable back into metricset modules for easy debug prin…
Jan 8, 2017
28e51fd
ran make generate-json for dbstats metricset
Jan 8, 2017
1c80f5e
removed dialinfo from mongodb metricsets because we have the mongoses…
Jan 8, 2017
a8450fb
updated data mapping to make mmapv1-specific fields optional
Jan 8, 2017
35ac943
updated changelog to include latest changes to Metricbeat's MongoDB m…
Jan 10, 2017
a9dd25f
Merge branch 'master' of github.com:elastic/beats into mongo-dbstats
Jan 10, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions metricbeat/_meta/beat.full.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ metricbeat.modules:

#------------------------------- MongoDB Module ------------------------------
#- module: mongodb
#metricsets: ["status"]
#metricsets: ["status", "dbstats"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably sort this alphabetically. But not sure if we are consistent in other modules with that ...

#enabled: true
#period: 10s

Expand All @@ -134,7 +134,6 @@ metricbeat.modules:
# Password to use when connecting to MongoDB. Empty by default.
#password: pass


#-------------------------------- MySQL Module -------------------------------
#- module: mysql
#metricsets: ["status"]
Expand Down
84 changes: 84 additions & 0 deletions metricbeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2059,6 +2059,90 @@ MongoDB metrics.



[float]
== dbstats Fields

dbstats provides an overview of a particular mongo database. This document is most concerned with data volumes of a database.



[float]
=== mongodb.dbstats.avg_obj_size

type: long

[float]
=== mongodb.dbstats.collections

type: integer

[float]
=== mongodb.dbstats.data_size

type: long

[float]
=== mongodb.dbstats.db

type: keyword

[float]
=== mongodb.dbstats.file_size

type: long

[float]
=== mongodb.dbstats.index_size

type: long

[float]
=== mongodb.dbstats.indexes

type: long

[float]
=== mongodb.dbstats.num_extents

type: long

[float]
=== mongodb.dbstats.objects

type: long

[float]
=== mongodb.dbstats.storage_size

type: long

[float]
=== mongodb.dbstats.ns_size_mb

type: long


[float]
=== mongodb.dbstats.data_file_version.major

type: long

[float]
=== mongodb.dbstats.data_file_version.minor

type: long


[float]
=== mongodb.dbstats.extent_free_list.num

type: long

[float]
=== mongodb.dbstats.extent_free_list.size

type: long

[float]
== status Fields

Expand Down
7 changes: 5 additions & 2 deletions metricbeat/docs/modules/mongodb.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ in <<configuration-metricbeat>>. Here is an example configuration:
----
metricbeat.modules:
#- module: mongodb
#metricsets: ["status"]
#metricsets: ["status", "dbstats"]
#enabled: true
#period: 10s

Expand All @@ -80,15 +80,18 @@ metricbeat.modules:

# Password to use when connecting to MongoDB. Empty by default.
#password: pass

----

[float]
=== Metricsets

The following metricsets are available:

* <<metricbeat-metricset-mongodb-dbstats,dbstats>>

* <<metricbeat-metricset-mongodb-status,status>>

include::mongodb/dbstats.asciidoc[]

include::mongodb/status.asciidoc[]

19 changes: 19 additions & 0 deletions metricbeat/docs/modules/mongodb/dbstats.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
////
This file is generated! See scripts/docs_collector.py
////

[[metricbeat-metricset-mongodb-dbstats]]
include::../../../module/mongodb/dbstats/_meta/docs.asciidoc[]


==== Fields

For a description of each field in the metricset, see the
<<exported-fields-mongodb,exported fields>> section.

Here is an example document generated by this metricset:

[source,json]
----
include::../../../module/mongodb/dbstats/_meta/data.json[]
----
1 change: 1 addition & 0 deletions metricbeat/include/list.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ import (
_ "github.com/elastic/beats/metricbeat/module/kafka"
_ "github.com/elastic/beats/metricbeat/module/kafka/partition"
_ "github.com/elastic/beats/metricbeat/module/mongodb"
_ "github.com/elastic/beats/metricbeat/module/mongodb/dbstats"
_ "github.com/elastic/beats/metricbeat/module/mongodb/status"
_ "github.com/elastic/beats/metricbeat/module/mysql"
_ "github.com/elastic/beats/metricbeat/module/mysql/status"
Expand Down
3 changes: 1 addition & 2 deletions metricbeat/metricbeat.full.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ metricbeat.modules:

#------------------------------- MongoDB Module ------------------------------
#- module: mongodb
#metricsets: ["status"]
#metricsets: ["status", "dbstats"]
#enabled: true
#period: 10s

Expand All @@ -134,7 +134,6 @@ metricbeat.modules:
# Password to use when connecting to MongoDB. Empty by default.
#password: pass


#-------------------------------- MySQL Module -------------------------------
#- module: mysql
#metricsets: ["status"]
Expand Down
59 changes: 59 additions & 0 deletions metricbeat/metricbeat.template-es2x.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1054,6 +1054,65 @@
},
"mongodb": {
"properties": {
"dbstats": {
"properties": {
"avg_obj_size": {
"type": "long"
},
"collections": {
"type": "long"
},
"data_file_version": {
"properties": {
"major": {
"type": "long"
},
"minor": {
"type": "long"
}
}
},
"data_size": {
"type": "long"
},
"db": {
"ignore_above": 1024,
"index": "not_analyzed",
"type": "string"
},
"extent_free_list": {
"properties": {
"num": {
"type": "long"
},
"size": {
"type": "long"
}
}
},
"file_size": {
"type": "long"
},
"index_size": {
"type": "long"
},
"indexes": {
"type": "long"
},
"ns_size_mb": {
"type": "long"
},
"num_extents": {
"type": "long"
},
"objects": {
"type": "long"
},
"storage_size": {
"type": "long"
}
}
},
"status": {
"properties": {
"asserts": {
Expand Down
58 changes: 58 additions & 0 deletions metricbeat/metricbeat.template.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1051,6 +1051,64 @@
},
"mongodb": {
"properties": {
"dbstats": {
"properties": {
"avg_obj_size": {
"type": "long"
},
"collections": {
"type": "long"
},
"data_file_version": {
"properties": {
"major": {
"type": "long"
},
"minor": {
"type": "long"
}
}
},
"data_size": {
"type": "long"
},
"db": {
"ignore_above": 1024,
"type": "keyword"
},
"extent_free_list": {
"properties": {
"num": {
"type": "long"
},
"size": {
"type": "long"
}
}
},
"file_size": {
"type": "long"
},
"index_size": {
"type": "long"
},
"indexes": {
"type": "long"
},
"ns_size_mb": {
"type": "long"
},
"num_extents": {
"type": "long"
},
"objects": {
"type": "long"
},
"storage_size": {
"type": "long"
}
}
},
"status": {
"properties": {
"asserts": {
Expand Down
3 changes: 1 addition & 2 deletions metricbeat/module/mongodb/_meta/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#- module: mongodb
#metricsets: ["status"]
#metricsets: ["status", "dbstats"]
#enabled: true
#period: 10s

Expand All @@ -15,4 +15,3 @@

# Password to use when connecting to MongoDB. Empty by default.
#password: pass

19 changes: 19 additions & 0 deletions metricbeat/module/mongodb/dbstats/_meta/data.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{
"@timestamp":"2016-05-23T08:05:34.853Z",
"beat":{
"hostname":"beathost",
"name":"beathost"
},
"metricset":{
"host":"localhost",
"module":"mysql",
"name":"status",
"rtt":44269
},
"mongodb":{
"dbstats":{
"example": "dbstats"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An example doc should be generated here with make generate-json. But that can still be done after merging the PR.

}
},
"type":"metricsets"
}
3 changes: 3 additions & 0 deletions metricbeat/module/mongodb/dbstats/_meta/docs.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
=== mongodb dbstats MetricSet

This is the dbstats metricset of the module mongodb.
Loading