Skip to content

[small] Add build information as a multi labeled metric#5526

Merged
sougou merged 2 commits intovitessio:masterfrom
tinyspeck:sa-build-labels
Dec 9, 2019
Merged

[small] Add build information as a multi labeled metric#5526
sougou merged 2 commits intovitessio:masterfrom
tinyspeck:sa-build-labels

Conversation

@setassociative
Copy link
Copy Markdown

@setassociative setassociative commented Dec 7, 2019

Problem

I'm using the prometheus stats exporter with a grafana frontend to monitor my deployment. I wanted to utilize build metrics to construct a dashboard that allowed us to watch the deployment progress.

For greater versatility Grafana dashboards can be templatized and I'm using that so we graph % of the fleet on a specified build... In order to set that variable there are, roughly, two options

  • either you can extract the possible values via promql
  • or you can specify the value is a freeform input

If the build number / branch name is expressed as a metric value it is not possible (or very expensive) to express "all possible values" as a promql on dashboard load whereas when we expose this information as a label on a const metric it's extremely simple (label_values(vttablet_build_information, build_number)), fast, and efficient.

Bonus points is that this also allows us to use a much cleaner query for a lot of the dashboard (vttablet_build_information{build="$target_version"}).

Solution

Expert a const value with the build information baked in as a label value.

Testing

I built and deployed this to a canary keyspace and observed vttablet_build_information was operating as expected. Additionally I deployed to a vtgate servicing that keyspace and observed some test queries operated as expected.

There is a unit test failing on this PR according to the validation checks here but it seems unrelated as it's associated with vreplication. It seems consistent though. Logs of the failure below, curious if this is a known issue:

Details
E1207 16:45:36.006484   17547 binlog_player.go:202] error (expected error) in selecting vreplication settings select pos, stop_pos, max_tps, max_replication_lag, state from _vt.vreplication where id=1
E1207 16:45:36.006800   17547 controller.go:149] stream 1: error (expected error) in selecting vreplication settings select pos, stop_pos, max_tps, max_replication_lag, state from _vt.vreplication where id=1, retrying after 10ms
E1207 16:45:36.332395   17547 engine.go:596] Error waiting for pos: MariaDB/0-1-1084, last pos: MariaDB/0-1-1083: context canceled, wait time: 14.4µs
--- FAIL: TestPlayerCopyTableContinuation (3.87s)
##[error]    vcopier_test.go:517: query:
        "update _vt.vreplication set pos='MySQL56/fbaefae5-1910-11ea-ad90-000d3a7d61d8:1-160', time_updated=1575737142, transaction_timestamp=1575737140, message='' where id=13", does not match query 0:
        "/update _vt.vreplication set state='Running'"
E1207 16:45:44.702246   17547 controller.go:149] stream 17: filter rules are not supported for SBR replication: [match:"dst1" filter:"select * from src1" ], retrying after 5s
E1207 16:45:45.476491   17547 controller.go:149] stream 28: Duplicate column name 'val2' (errno 1060) (sqlstate 42S21) during query: alter table t1 add column val2 varchar(128), retrying after 5s
E1207 16:45:52.457828   17547 controller.go:149] stream 38: vstream ended, retrying after 1ms
E1207 16:45:52.575419   17547 vstreamer.go:648] stream error @ fbaefae5-1910-11ea-ad90-000d3a7d61d8:1-590: unknown table t1 in schema
FAIL
FAIL	vitess.io/vitess/go/vt/vttablet/tabletmanager/vreplication

@setassociative setassociative force-pushed the sa-build-labels branch 3 times, most recently from 6c14733 to 129bcd6 Compare December 7, 2019 16:37
Signed-off-by: Richard Bailey <rbailey@slack-corp.com>
Copy link
Copy Markdown
Member

@rafael rafael left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This LGTM. After the tests pass, should be good to got from my perspective.

Copy link
Copy Markdown
Contributor

@sougou sougou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this :)

@sougou sougou merged commit f242a24 into vitessio:master Dec 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants