Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support to CEPH input. #3311

Merged
merged 26 commits into from
Jan 24, 2017
Merged

Support to CEPH input. #3311

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
de6ae45
Initial version.
Jan 3, 2017
dd0f784
Merge remote-tracking branch 'upstream/master'
Jan 3, 2017
98715a7
Refactoring of perf.go
Jan 4, 2017
0c8ae90
Formatted data for Tag.
Jan 4, 2017
a1ff1f7
Merge remote-tracking branch 'upstream/master'
Jan 4, 2017
41c445d
First version of status metricset.
Jan 6, 2017
86982ce
Df command.
Jan 6, 2017
8868254
Command osd pool stats finished.
Jan 9, 2017
b082c45
Repo updated.
Jan 9, 2017
5f8dd37
Merge remote-tracking branch 'upstream/master'
Jan 9, 2017
208818c
Make update and config.yml corrected.
Jan 9, 2017
b8ca266
Merge with upstream.
Jan 10, 2017
1011f02
Config reading changed.
Jan 10, 2017
747debd
Config reading changed.
Jan 10, 2017
fbe6ba2
Merge remote-tracking branch 'upstream/master'
Jan 11, 2017
66bd1d4
Initial tests.
Jan 12, 2017
06fd61c
Merge remote-tracking branch 'upstream/master'
Jan 12, 2017
85a3c26
Integration tests and generated jsons.
Jan 13, 2017
9dec734
Perf metricset removed.
Jan 16, 2017
9cec164
Merge branch 'master' of https://github.com/elastic/beats
Jan 16, 2017
596fd29
Tests corrected. Fields completed.
Jan 16, 2017
46ad8db
Docker compose updated.
Jan 16, 2017
9c8e65a
New health metricset using ceph rest api.
Jan 17, 2017
48c0e38
Fields.yml generated and docker-compose.yml corrected.
Jan 18, 2017
d289374
Fields.yml generated and docker-compose.yml corrected.
Jan 18, 2017
431c99f
Fields corrected and unity test created.
Jan 19, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions metricbeat/_meta/beat.full.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,38 @@ metricbeat.modules:
# Password of hosts. Empty by default
#password: test123

#-------------------------------- ceph Module --------------------------------
- module: ceph
metricsets: ["perf"]
enabled: true
period: 1s

ceph:
# location of ceph binary
binary_path = "/usr/bin/ceph"
#
#####
### MetricSet perf gather statistics via the admin socket, requires socket_dir, mon_prefix, osd_prefix and socket_suffix
#####
# directory in which to look for socket files
socket_dir = "/var/run/ceph"
# prefix of MON and OSD socket files, used to determine socket type
mon_prefix = "ceph-mon"
osd_prefix = "ceph-osd"
# suffix used to identify socket files
socket_suffix = "asok"
#
#####
### MetricSets status,df and poolio gather statistics via ceph commands, requires ceph_user and ceph_config
#####
# Ceph user to authenticate as, ceph will search for the corresponding keyring
# e.g. client.admin.keyring in /etc/ceph
# Consult the ceph documentation for more detail on keyring generation.
user = "client.admin"
# Ceph configuration to use to locate the cluster
config_path = "/etc/ceph/ceph.conf"


#------------------------------ couchbase Module -----------------------------
#- module: couchbase
#metricsets: ["cluster", "node", "bucket"]
Expand Down
32 changes: 32 additions & 0 deletions metricbeat/_meta/beat.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,4 +46,36 @@ metricbeat.modules:
period: 10s
processes: ['.*']

#-------------------------------- ceph Module --------------------------------
- module: ceph
metricsets: ["perf"]
enabled: true
period: 1s

ceph:
# location of ceph binary
binary_path = "/usr/bin/ceph"
#
#####
### MetricSet perf gather statistics via the admin socket, requires socket_dir, mon_prefix, osd_prefix and socket_suffix
#####
# directory in which to look for socket files
socket_dir = "/var/run/ceph"
# prefix of MON and OSD socket files, used to determine socket type
mon_prefix = "ceph-mon"
osd_prefix = "ceph-osd"
# suffix used to identify socket files
socket_suffix = "asok"
#
#####
### MetricSets status,df and poolio gather statistics via ceph commands, requires ceph_user and ceph_config
#####
# Ceph user to authenticate as, ceph will search for the corresponding keyring
# e.g. client.admin.keyring in /etc/ceph
# Consult the ceph documentation for more detail on keyring generation.
user = "client.admin"
# Ceph configuration to use to locate the cluster
config_path = "/etc/ceph/ceph.conf"



74 changes: 74 additions & 0 deletions metricbeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ grouped in the following categories:

* <<exported-fields-apache>>
* <<exported-fields-beat>>
* <<exported-fields-ceph>>
* <<exported-fields-cloud>>
* <<exported-fields-common>>
* <<exported-fields-couchbase>>
Expand Down Expand Up @@ -411,6 +412,79 @@ type: dict
Contains user configurable fields.


[[exported-fields-ceph]]
== ceph Fields

ceph Module



[float]
== ceph Fields




[float]
== df Fields

df



[float]
=== ceph.df.example

type: keyword

Example field


[float]
== osdpoolstats Fields

osdpoolstats



[float]
=== ceph.osdpoolstats.example

type: keyword

Example field


[float]
== perf Fields

perf



[float]
=== ceph.perf.example

type: keyword

Example field


[float]
== status Fields

status



[float]
=== ceph.status.example

type: keyword

Example field


[[exported-fields-cloud]]
== Cloud Provider Metadata Fields

Expand Down
78 changes: 78 additions & 0 deletions metricbeat/docs/modules/ceph.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
////
This file is generated! See scripts/docs_collector.py
////

[[metricbeat-module-ceph]]
== ceph Module

This is the ceph Module.

We used code/reference from:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the reference is only used to get inspiration, these should be removed.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since I'll change to use Rest API, this reference will be no longer necessary indeed. I'll remove.

https://github.com/influxdata/telegraf/blob/master/plugins/inputs/ceph/ceph.go
https://github.com/elastic/beats/blob/master/metricbeat/module/docker/

Thanks!


[float]
=== Example Configuration

The ceph module supports the standard configuration options that are described
in <<configuration-metricbeat>>. Here is an example configuration:

[source,yaml]
----
metricbeat.modules:
- module: ceph
metricsets: ["perf"]
enabled: true
period: 1s

ceph:
# location of ceph binary
binary_path = "/usr/bin/ceph"
#
#####
### MetricSet perf gather statistics via the admin socket, requires socket_dir, mon_prefix, osd_prefix and socket_suffix
#####
# directory in which to look for socket files
socket_dir = "/var/run/ceph"
# prefix of MON and OSD socket files, used to determine socket type
mon_prefix = "ceph-mon"
osd_prefix = "ceph-osd"
# suffix used to identify socket files
socket_suffix = "asok"
#
#####
### MetricSets status,df and poolio gather statistics via ceph commands, requires ceph_user and ceph_config
#####
# Ceph user to authenticate as, ceph will search for the corresponding keyring
# e.g. client.admin.keyring in /etc/ceph
# Consult the ceph documentation for more detail on keyring generation.
user = "client.admin"
# Ceph configuration to use to locate the cluster
config_path = "/etc/ceph/ceph.conf"

----

[float]
=== Metricsets

The following metricsets are available:

* <<metricbeat-metricset-ceph-df,df>>

* <<metricbeat-metricset-ceph-osdpoolstats,osdpoolstats>>

* <<metricbeat-metricset-ceph-perf,perf>>

* <<metricbeat-metricset-ceph-status,status>>

include::ceph/df.asciidoc[]

include::ceph/osdpoolstats.asciidoc[]

include::ceph/perf.asciidoc[]

include::ceph/status.asciidoc[]

19 changes: 19 additions & 0 deletions metricbeat/docs/modules/ceph/df.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
////
This file is generated! See scripts/docs_collector.py
////

[[metricbeat-metricset-ceph-df]]
include::../../../module/ceph/df/_meta/docs.asciidoc[]


==== Fields

For a description of each field in the metricset, see the
<<exported-fields-ceph,exported fields>> section.

Here is an example document generated by this metricset:

[source,json]
----
include::../../../module/ceph/df/_meta/data.json[]
----
19 changes: 19 additions & 0 deletions metricbeat/docs/modules/ceph/osdpoolstats.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
////
This file is generated! See scripts/docs_collector.py
////

[[metricbeat-metricset-ceph-osdpoolstats]]
include::../../../module/ceph/osdpoolstats/_meta/docs.asciidoc[]


==== Fields

For a description of each field in the metricset, see the
<<exported-fields-ceph,exported fields>> section.

Here is an example document generated by this metricset:

[source,json]
----
include::../../../module/ceph/osdpoolstats/_meta/data.json[]
----
19 changes: 19 additions & 0 deletions metricbeat/docs/modules/ceph/perf.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
////
This file is generated! See scripts/docs_collector.py
////

[[metricbeat-metricset-ceph-perf]]
include::../../../module/ceph/perf/_meta/docs.asciidoc[]


==== Fields

For a description of each field in the metricset, see the
<<exported-fields-ceph,exported fields>> section.

Here is an example document generated by this metricset:

[source,json]
----
include::../../../module/ceph/perf/_meta/data.json[]
----
19 changes: 19 additions & 0 deletions metricbeat/docs/modules/ceph/status.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
////
This file is generated! See scripts/docs_collector.py
////

[[metricbeat-metricset-ceph-status]]
include::../../../module/ceph/status/_meta/docs.asciidoc[]


==== Fields

For a description of each field in the metricset, see the
<<exported-fields-ceph,exported fields>> section.

Here is an example document generated by this metricset:

[source,json]
----
include::../../../module/ceph/status/_meta/data.json[]
----
2 changes: 2 additions & 0 deletions metricbeat/docs/modules_list.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ This file is generated! See scripts/docs_collector.py
////

* <<metricbeat-module-apache,Apache>>
* <<metricbeat-module-ceph,ceph>>
* <<metricbeat-module-couchbase,couchbase>>
* <<metricbeat-module-docker,Docker>>
* <<metricbeat-module-haproxy,haproxy>>
Expand All @@ -20,6 +21,7 @@ This file is generated! See scripts/docs_collector.py
--

include::modules/apache.asciidoc[]
include::modules/ceph.asciidoc[]
include::modules/couchbase.asciidoc[]
include::modules/docker.asciidoc[]
include::modules/haproxy.asciidoc[]
Expand Down
5 changes: 5 additions & 0 deletions metricbeat/include/list.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,11 @@ import (
// This list is automatically generated by `make imports`
_ "github.com/elastic/beats/metricbeat/module/apache"
_ "github.com/elastic/beats/metricbeat/module/apache/status"
_ "github.com/elastic/beats/metricbeat/module/ceph"
_ "github.com/elastic/beats/metricbeat/module/ceph/df"
_ "github.com/elastic/beats/metricbeat/module/ceph/osdpoolstats"
_ "github.com/elastic/beats/metricbeat/module/ceph/perf"
_ "github.com/elastic/beats/metricbeat/module/ceph/status"
_ "github.com/elastic/beats/metricbeat/module/couchbase"
_ "github.com/elastic/beats/metricbeat/module/couchbase/bucket"
_ "github.com/elastic/beats/metricbeat/module/couchbase/cluster"
Expand Down
32 changes: 32 additions & 0 deletions metricbeat/metricbeat.full.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,38 @@ metricbeat.modules:
# Password of hosts. Empty by default
#password: test123

#-------------------------------- ceph Module --------------------------------
- module: ceph
metricsets: ["perf"]
enabled: true
period: 1s

ceph:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part of the config must be moved under the module. I assume it doesn't work the way it is at the moment.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved.

# location of ceph binary
binary_path = "/usr/bin/ceph"
#
#####
### MetricSet perf gather statistics via the admin socket, requires socket_dir, mon_prefix, osd_prefix and socket_suffix
#####
# directory in which to look for socket files
socket_dir = "/var/run/ceph"
# prefix of MON and OSD socket files, used to determine socket type
mon_prefix = "ceph-mon"
osd_prefix = "ceph-osd"
# suffix used to identify socket files
socket_suffix = "asok"
#
#####
### MetricSets status,df and poolio gather statistics via ceph commands, requires ceph_user and ceph_config
#####
# Ceph user to authenticate as, ceph will search for the corresponding keyring
# e.g. client.admin.keyring in /etc/ceph
# Consult the ceph documentation for more detail on keyring generation.
user = "client.admin"
# Ceph configuration to use to locate the cluster
config_path = "/etc/ceph/ceph.conf"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general we recommend metricbeat to connect to each node and get the specific metrics for each node instead from the cluster. What is inside this file? You should use hosts: ["..."] for that if possible.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used telegraf as reference and that only collects performance metrics from the MON and OSD nodes in a Ceph storage cluster. I can change the description, perhaps.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not aware of the inner workings of CEPH. Can you share some details on what MON and OSD nodes are?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A Ceph Storage Cluster requires at least one MON and at least 2 OSD.

A OSD (Object Storage Daemon) handles data (storage, replication, recovery...) and provide some monitoring information to MON by checking others OSD for a heartbeat.

A MON (Monitor) handles the cluster state (maps, history...).

You can have this architecture on one server/node but just for tests. Real productions environments works in a distributed way.

If you want metrics just for the daemons where metricbeat its collecting, you can use Admin Socket Stats. In our case here, it's the "perf" metricset.

If you want metrics for the whole cluster, you can use ceph commands. In our case here, it's the "status", "df" and "osdpoolstats" metricsets.

That's why I wrote those comments on config.yml. Depending on choosen metricset, you need just some parameters.

Copy link
Member

@ruflin ruflin Jan 12, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So perf is a local stat that should be collect by all metricbeat instances and the other 3 are cluster stats, means only one of the instances connecting needs to collect them? If there are multiple MON, will they provide different data?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"...means only one of the instances connecting needs to collect them?" Yes.
"If there are multiple MON, will they provide different data?" No, at least it's not expected this behavior.



#------------------------------ couchbase Module -----------------------------
#- module: couchbase
#metricsets: ["cluster", "node", "bucket"]
Expand Down
Loading