Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[v1.0.2.rc2] ES role shows the same on all nodes #109

Closed
dannyk81 opened this issue Nov 25, 2017 · 9 comments · Fixed by #110
Closed

[v1.0.2.rc2] ES role shows the same on all nodes #109

dannyk81 opened this issue Nov 25, 2017 · 9 comments · Fixed by #110
Assignees
Labels

Comments

@dannyk81
Copy link
Contributor

dannyk81 commented Nov 25, 2017

Just upgraded to the latest RC, however I noticed that all nodes have almost the same values for the ES role labels, only semi-correct ones are ingest node (though they show that they are also client node, which they are not):

elasticsearch_indices_flush_time_seconds{cluster="EFKcluster",es_client_node="true",es_data_node="true",es_ingest_node="true",es_master_node="true",host="elasticin-02",instance="elasticin-02",job="elasticsearch-exporter",name="elasticin-02"}
elasticsearch_indices_flush_time_seconds{cluster="EFKcluster",es_client_node="true",es_data_node="true",es_ingest_node="true",es_master_node="true",host="elasticin-01",instance="elasticin-01",job="elasticsearch-exporter",name="elasticin-01"}

The rest are all wrong:

elasticsearch_indices_flush_time_seconds{cluster="EFKcluster",es_client_node="true",es_data_node="true",es_ingest_node="false",es_master_node="true",host="elasticmaster-03",instance="elasticmaster-03",job="elasticsearch-exporter",name="elasticmaster-03"}
elasticsearch_indices_flush_time_seconds{cluster="EFKcluster",es_client_node="true",es_data_node="true",es_ingest_node="false",es_master_node="true",host="elasticmaster-02",instance="elasticmaster-02",job="elasticsearch-exporter",name="elasticmaster-02"}
elasticsearch_indices_flush_time_seconds{cluster="EFKcluster",es_client_node="true",es_data_node="true",es_ingest_node="false",es_master_node="true",host="elasticmaster-01",instance="elasticmaster-01",job="elasticsearch-exporter",name="elasticmaster-01"}
elasticsearch_indices_flush_time_seconds{cluster="EFKcluster",es_client_node="true",es_data_node="true",es_ingest_node="false",es_master_node="true",host="elasticdata-06",instance="elasticdata-06",job="elasticsearch-exporter",name="elasticdata-06"}
elasticsearch_indices_flush_time_seconds{cluster="EFKcluster",es_client_node="true",es_data_node="true",es_ingest_node="false",es_master_node="true",host="elasticdata-05",instance="elasticdata-05",job="elasticsearch-exporter",name="elasticdata-05"}
elasticsearch_indices_flush_time_seconds{cluster="EFKcluster",es_client_node="true",es_data_node="true",es_ingest_node="false",es_master_node="true",host="elasticdata-04",instance="elasticdata-04",job="elasticsearch-exporter",name="elasticdata-04"}
elasticsearch_indices_flush_time_seconds{cluster="EFKcluster",es_client_node="true",es_data_node="true",es_ingest_node="false",es_master_node="true",host="elasticdata-03",instance="elasticdata-03",job="elasticsearch-exporter",name="elasticdata-03"}
elasticsearch_indices_flush_time_seconds{cluster="EFKcluster",es_client_node="true",es_data_node="true",es_ingest_node="false",es_master_node="true",host="elasticdata-02",instance="elasticdata-02",job="elasticsearch-exporter",name="elasticdata-02"}
elasticsearch_indices_flush_time_seconds{cluster="EFKcluster",es_client_node="true",es_data_node="true",es_ingest_node="false",es_master_node="true",host="elasticdata-01",instance="elasticdata-01",job="elasticsearch-exporter",name="elasticdata-01"}
elasticsearch_indices_flush_time_seconds{cluster="EFKcluster",es_client_node="true",es_data_node="true",es_ingest_node="false",es_master_node="true",host="elasticcli-02",instance="elasticcli-02",job="elasticsearch-exporter",name="elasticcli-02"}
elasticsearch_indices_flush_time_seconds{cluster="EFKcluster",es_client_node="true",es_data_node="true",es_ingest_node="false",es_master_node="true",host="elasticcli-01",instance="elasticcli-01",job="elasticsearch-exporter",name="elasticcli-01"}

I double checked our configuration and the roles are defined correctly.

We are using Elasticsearch 5.5.0

Any ideas? any addtional info you need?

@dominikschulz
Copy link
Contributor

Could you provide the output of the following commands?

  • curl http://elasticmaster-01:9200/_nodes/stats
  • curl http://elasticdata-01:9200/_nodes/stats
  • curl http://elasticcli-01:9200/_nodes/stats

@dannyk81
Copy link
Contributor Author

Please see attached, though perhaps I should run the query with the nodeId? e.g. curl http://elasticmaster-01:9200/_nodes/elasticmaster-01/stats (otherwise you get metrics of all nodes, unless this is the expected result).

node_stats.zip

@dominikschulz
Copy link
Contributor

I'm sorry, of course the output from one of the nodes would have been enough. My bad!

But this is very helpful in narrowing down the issue. I'm using it to build some test cases to prevent us from having regressions on this in the future and then we'll fix the role detection.

@dominikschulz dominikschulz self-assigned this Nov 27, 2017
dominikschulz pushed a commit to dominikschulz/elasticsearch_exporter that referenced this issue Nov 27, 2017
This PR adds some more test cases for a multi-node 5.5.0 cluster and fixes the role detection.

Fixes prometheus-community#109
@zwopir
Copy link
Member

zwopir commented Nov 27, 2017

@dannyk81 the fix was merged. I update the docker image and create a new release ASAP

@dannyk81
Copy link
Contributor Author

@dominikschulz @zwopir

Thank you!

@haizaar
Copy link
Contributor

haizaar commented Jan 9, 2018

Hi guys,

We've moved to ES5 and are suffering from this as well. Can you please release a new version?
I tested master and it solves it indeed.

@dominikschulz
Copy link
Contributor

Thank for testing the fix, we'll try to release a new version soon. @zwopir

@haizaar
Copy link
Contributor

haizaar commented Jan 9, 2018

Thanks,
Do you have any concrete ETA? The reason I'm asking is that the last time @zwopir said "ASAP" on Nov 27 and it's still not released :)

@dominikschulz
Copy link
Contributor

We've just released the 1.0.2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants