Add Vsphere module with metrics. WIP #4028

amandahla · 2017-04-17T18:19:45Z

Hi. This is a initial version for Vsphere module, just collecting datastore usage metrics.
We used Govmomi library and the example in:
https://github.com/vmware/govmomi/blob/master/examples/datastores/main.go

TODO:

Collect cpu and memory of each Host.
Collect cpu and memory of each VM.
Unity tests.

elasticmachine · 2017-04-17T18:22:02Z

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run.

elasticmachine · 2017-04-17T18:22:07Z

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run.

ruflin · 2017-04-18T06:57:36Z

@amandahla Thanks a lot for kicking off this contribution. Let me know when we should do a first review. Be aware that we recently switched to govendor to handle the dependencies and we try to keep the dependencies for a module in a vendor directory local to the module.

andrewkroh · 2017-04-19T14:44:52Z

Looks like you are off to a good start. So would Metricbeat run directly on each ESXi host or would it run on the vCenter host?

amandahla · 2017-04-19T15:55:36Z

Hello @andrewkroh, actually Metricbeat can collect from any Vmware SDK URL (ESXi/VCenter).
But...I would not recommend to run inside those hosts because installing 3rd party software is not supported by the vendor.

amandahla · 2017-04-20T19:40:33Z

Hi @ruflin . I made some modifications and I think that now it's ready for a first review. Thanks!

ruflin

Thanks a lot for the addition. I left a few minor comments.

Could you add a CHANGELOG entry?
Could you add a short readme on how to test it? Similar to what I did here for the kubelete module: https://github.com/elastic/beats/tree/master/metricbeat/module/kubelet/_meta/test as vsphere is also not that straight forward to test as the other modules.

ruflin · 2017-04-21T06:29:44Z

metricbeat/module/vsphere/_meta/config.yml

@@ -0,0 +1,10 @@
+#- module: vsphere


Be aware that we change the logic here a bit in #4049

ruflin · 2017-04-21T06:30:49Z

metricbeat/include/list.go

@@ -85,6 +85,10 @@ import (
 	_ "github.com/elastic/beats/metricbeat/module/system/network"
 	_ "github.com/elastic/beats/metricbeat/module/system/process"
 	_ "github.com/elastic/beats/metricbeat/module/system/socket"
+	_ "github.com/elastic/beats/metricbeat/module/vsphere"
+	_ "github.com/elastic/beats/metricbeat/module/vsphere/datastore_usage"


Do we need the appendix usage for the metricset names? Are there potential other metricsets for datastore for example?

Actually thinking about it...No, we don't need. I removed now.
✅

ruflin · 2017-04-21T06:31:21Z

metricbeat/module/vsphere/_meta/docs.asciidoc

@@ -0,0 +1,4 @@
+== vsphere Module
+
+This is the vsphere Module.


Could you add some more details here, for example link to vSphere (installation) page or docs? Also did you test with a specific version?

ruflin · 2017-04-21T06:33:34Z

metricbeat/module/vsphere/datastore_usage/_meta/data.json

+    },
+    "vsphere":{
+        "datastore_usage":{
+            "example": "datastore_usage"


Needs update with an example event. You can run go test -tags=integration github.com/elastic/beats/metricbeat/module/vsphere/... -data when your environment is running.

I'll try to use this in this case:
http://www.virtuallyghetto.com/2017/04/govcsim-neat-incubation-project-vcenter-server-esxi-api-based-simulator.html

ruflin · 2017-04-21T06:37:57Z

metricbeat/module/vsphere/datastore_usage/_meta/fields.yml

+      type: long
+      description: >
+        Used percent of the datastore
+


You can add format: percent here for Kibana.

ruflin · 2017-04-21T06:46:16Z

metricbeat/module/vsphere/host_usage/host_usage_test.go

+	"testing"
+)
+
+func TestFetchEventContents(t *testing.T) {


Test missing

ruflin · 2017-04-21T06:47:48Z

metricbeat/module/vsphere/vendor/vendor.json

@@ -0,0 +1,97 @@
+{
+	"comment": "",
+	"ignore": "test",


I would expect here "ignore": "test github.com/elastic/beats" as otherwise if you remove the libs from the vendor dir and then run govendor sync it will also load libbeat etc. into it.

ruflin · 2017-04-21T06:48:19Z

metricbeat/module/vsphere/virtualmachine_usage/_meta/data.json

+    },
+    "vsphere":{
+        "virtualmachine_usage":{
+            "example": "virtualmachine_usage"


needs update

ruflin · 2017-04-21T06:49:32Z

metricbeat/module/vsphere/virtualmachine_usage/virtualmachine_usage.go

+					"memory": common.MapStr{
+						"used": common.MapStr{
+							"guest": common.MapStr{
+								"bytes": vm.Summary.QuickStats.GuestMemoryUsage * 1024,


What are the raw values of all these * 1024?

vm.Summary.QuickStats.GuestMemoryUsage, vm.Summary.QuickStats.HostMemoryUsage and vm.Summary.Config.MemorySizeMB are values in MB so I had to multiply all for 1024 to get bytes values.

ruflin · 2017-04-21T06:49:47Z

metricbeat/module/vsphere/virtualmachine_usage/virtualmachine_usage_test.go

+)
+
+func TestFetchEventContents(t *testing.T) {
+	assert.EqualValues(t, 1, 1)


Needs update

…. Vendor beats ignored.

amandahla · 2017-05-03T12:09:13Z

Hi @ruflin . I've made all the changes that you asked but now I don't know why the Travis build is failing. It's something related to filebeat testsuit in OSX nothing regardless this module. :-)

ruflin · 2017-05-03T14:23:18Z

@amandahla Great. I restarted the build manually and will have a look at the PR soon.

ruflin

LGTM

Two requests which can also happen after merging:

Create the data.json with some example events
Add Dashboards

ruflin · 2017-05-03T18:20:27Z

metricbeat/module/vsphere/datastore/_meta/data.json

+    },
+    "vsphere":{
+        "datastore":{
+            "example": "datastore"


Would be nice to have here a full example data set. This can happen in a follow up PR.

ruflin · 2017-05-03T18:22:27Z

metricbeat/module/vsphere/host/_meta/data.json

+    },
+    "vsphere":{
+        "host":{
+            "example": "host"


Example doc here would be nice.

ruflin · 2017-05-03T18:25:09Z

jenkins, test it

ruflin · 2017-05-04T11:34:22Z

@amandahla Thanks a lot for the contribution.

amandahla · 2017-05-04T12:08:48Z

Thanks! :-)

ruflin · 2017-05-04T12:31:51Z

@amandahla If you have some time to open a new PR with the data.json that would be great.

amandahla · 2017-05-04T12:33:21Z

@ruflin Sure, I'm already working on it! 👍

rikatz · 2017-05-04T12:38:00Z

🎉 🎉 🎉 🎉

petericebear · 2017-08-08T12:47:25Z

I get a Bad request using this module. I am using the same credentials that I am using with Chef Knife Vsphere module.
Any idea's how that is possible or how to debug the request?

amandahla · 2017-08-08T12:59:44Z

@petericebear I saw that Knife uses "knife[:vsphere_host] = "vcenter-hostname""
Did you set hosts like "hosts: ["https://localhost/sdk"]" ?

petericebear · 2017-08-08T13:06:03Z

I now have this:

metricsets: ["virtualmachine"]
enabled: true
period: 10s
hosts: ["https://hostnameweblocation.com/"]

with insecure to true

petericebear · 2017-08-08T13:08:33Z

And with this setting:
metricsets: ["datastore, host, virtualmachine"]

I get the error:
Exiting: 1 error: metricset 'vsphere/datastore, host, virtualmachine' is not registered, metricset not found

amandahla · 2017-08-08T13:12:06Z

Please try
metricsets: ["datastore", "host", "virtualmachine"]
hosts: ["https://hostnameweblocation.com/sdk"]

petericebear · 2017-08-08T13:14:14Z

- module: vsphere
  metricsets: 
    - datastore
    - host
    - virtualmachine
  enabled: true
  period: 10s
  hosts: ["https://domain.org/sdk"]

  username: "XXX"
  password: "XXX"
  
  insecure: true
  get_custom_fields: false

This was the fix :) Thank you for your swift reply

amandahla · 2017-08-08T13:16:01Z

So it worked fine now? Did you just need to put get_custom_fields: false?

petericebear · 2017-08-08T13:30:45Z

Only some bytes in the data are negative. But I think I saw a PR for that?
It is this field: vsphere.virtualmachine.memory.total.guest.bytes
that gives bogus numbers.

I added the /sdk to the url, and I changed the metricsets from:
metricsets: ["datastore", "host", "virtualmachine"]

to:

metricsets: 
    - datastore
    - host
    - virtualmachine

Running on OSX (Latest)

amandahla · 2017-08-08T13:48:53Z

Yes, this PR solved that
#4462

petericebear · 2017-08-09T08:18:05Z

Are you working on the functionality of disk usage and network usage for virtual machines?
With that information we can realtime monitor everything from the server.

petericebear · 2017-08-17T08:36:00Z

I found out that Performance Manager can retrieve stats by level and key like cpu.usage.average.
Like this example:
https://gist.github.com/toobulkeh/6124975

If this a possibility for a beats module for given the keys to retrieve? That way we can have more data out of vSphere into Elasticsearch. I saw that the data changed with this module is changed every 60 seconds so the 10second interval is to low as default.

If you have another solution for using the Performance Manager for making similar graphs in Kibana as in vCenter I would be really grateful. If you need more information. Please contact me, very happy to help

amandahla · 2017-08-18T12:58:41Z

@petericebear Hi, thanks for your suggestion. I tried to use this but I couldn't find a way to work (https://github.com/vmware/govmomi/blob/master/performance/manager.go). I found it too difficult since we need for now just a few things. And I'm not sure, but I think that using Performance Manager we don't collect realtime metrics, just historical.

rezaa1 · 2017-09-20T00:52:07Z

Hi,
Thanks for great work.

I installed vspher beat from elastic 6 beta. But I can't see any sample dashboard for this beat. Dose it come with any dashboard?

Add Vsphere module with Datastore_usage metric.

876da87

amandahla changed the title ~~Add Vsphere module with Datastore_usage metric.~~ Add Vsphere module with Datastore_usage metric. WIP Apr 17, 2017

ruflin added Metricbeat Metricbeat review needs tests in progress Pull request is currently in progress. labels Apr 18, 2017

amandahla changed the title ~~Add Vsphere module with Datastore_usage metric. WIP~~ Add Vsphere module with metrics. WIP Apr 18, 2017

amandahla added 3 commits April 18, 2017 12:15

Use of govendor and new metric host_usage.

af3c620

Merge remote-tracking branch 'upstream/master'

d1f9ed4

NOTICE changed.

09ae905

Amanda H. L. de Andrade added 4 commits April 19, 2017 12:03

Merge branch 'master' into master

19ca35d

Metric virtualmachine_usage.

4e87f9d

Merge master

874a6f2

Merge branch 'master' of https://github.com/amandahla/beats

1497091

amandahla added 4 commits April 19, 2017 14:31

Meitrcbeat updated.

a5c285d

Fields removed.

880c877

Merge remote-tracking branch 'upstream/master'

7319a55

Definition of MetricSet updated.

757fc0f

ruflin requested changes Apr 21, 2017

View reviewed changes

amandahla added 5 commits April 24, 2017 10:13

Names changed. Better description. Logs showing that its experimental…

dd0dcaf

…. Vendor beats ignored.

Merged with master.

fe720c8

Add how to test with Govcsim and entry in CHANGELOG.

296a30d

Merge remote-tracking branch 'upstream/master'

801feeb

Notice updated.

5367ee2

amandahla added 2 commits May 2, 2017 10:13

Merge remote-tracking branch 'upstream/master'

b2431ec

Heartbeat fields.asciidoc updated.

6a67ecf

ruflin removed the in progress Pull request is currently in progress. label May 3, 2017

ruflin approved these changes May 3, 2017

View reviewed changes

ruflin merged commit f4e38b9 into elastic:master May 4, 2017

ruflin removed the needs tests label May 4, 2017

amandahla mentioned this pull request May 4, 2017

Add data.json files for vsphere module. WIP #4209

Merged

4 tasks

Add Vsphere module with metrics. WIP #4028

Add Vsphere module with metrics. WIP #4028

Conversation

amandahla commented Apr 17, 2017 • edited Loading

elasticmachine commented Apr 17, 2017

elasticmachine commented Apr 17, 2017

ruflin commented Apr 18, 2017

andrewkroh commented Apr 19, 2017

amandahla commented Apr 19, 2017

amandahla commented Apr 20, 2017

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amandahla commented May 3, 2017 • edited Loading

ruflin commented May 3, 2017

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented May 3, 2017

ruflin commented May 4, 2017

amandahla commented May 4, 2017

ruflin commented May 4, 2017

amandahla commented May 4, 2017

rikatz commented May 4, 2017

petericebear commented Aug 8, 2017

amandahla commented Aug 8, 2017

petericebear commented Aug 8, 2017

petericebear commented Aug 8, 2017

amandahla commented Aug 8, 2017

petericebear commented Aug 8, 2017 • edited Loading

amandahla commented Aug 8, 2017

petericebear commented Aug 8, 2017 • edited Loading

amandahla commented Aug 8, 2017

petericebear commented Aug 9, 2017

petericebear commented Aug 17, 2017 • edited Loading

amandahla commented Aug 18, 2017

rezaa1 commented Sep 20, 2017

amandahla commented Apr 17, 2017 •

edited

Loading

amandahla commented May 3, 2017 •

edited

Loading

petericebear commented Aug 8, 2017 •

edited

Loading

petericebear commented Aug 8, 2017 •

edited

Loading

petericebear commented Aug 17, 2017 •

edited

Loading