Skip to content

Metrics #374

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lukesteensen merged 7 commits into from
May 16, 2019
Merged

Metrics #374

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
b6128d6
move metrics definition to event module
lukesteensen May 8, 2019
81e466d
add metric event variant
lukesteensen May 14, 2019
fc7f5ad
barely move statsd and prom to metrics type
lukesteensen May 14, 2019
6bb2907
clean up imports
lukesteensen May 14, 2019
b3aa5a0
simplify prometheus sink
lukesteensen May 14, 2019
9f8b47c
don't require prom metrics to be preconfigured
lukesteensen May 15, 2019
3d1ea8c
add initial log to metric transform
lukesteensen May 16, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 40 additions & 2 deletions proto/event.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,52 @@ package event.proto;
message EventWrapper {
oneof event {
Log log = 1;
Metric metric = 2;
}
}

message Log {
map<string, Value> structured = 1;
}

message Value {
bytes data = 1;
bool explicit = 2;
}

message Log {
map<string, Value> structured = 1;
message Metric {
Copy link
Contributor

@LucioFranco LucioFranco May 16, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just gonna drop this thought here, do we want to include tags?

Copy link
Member Author

@lukesteensen lukesteensen May 16, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That'd be a good subject for an RFC! Personally I'm not super familiar with metrics systems that support tags, so I'd have to do some more research and probably implement a sink or two that uses them before I'd be comfortable baking them into the data model.

Copy link
Contributor

@LucioFranco LucioFranco May 16, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it should be something we can add later that is backwards compat with our proto.

oneof metric {
Counter counter = 1;
Timer timer = 2;
Gauge gauge = 3;
Set set = 4;
}
}

message Counter {
string name = 1;
uint32 val = 2;
float sampling = 3;
}

message Timer {
string name = 1;
uint32 val = 2;
float sampling = 3;
}

message Gauge {
string name = 1;
uint32 val = 2;
enum Direction {
None = 0;
Plus = 1;
Minus = 2;
}
Direction direction = 3;
}

message Set {
string name = 1;
string val = 2;
}
135 changes: 126 additions & 9 deletions src/event.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
use self::proto::{event_wrapper::Event as EventProto, Log};
use self::proto::{event_wrapper::Event as EventProto, metric::Metric as MetricProto, Log};
use bytes::Bytes;
use chrono::{DateTime, SecondsFormat, Utc};
use lazy_static::lazy_static;
Expand All @@ -7,6 +7,10 @@ use std::borrow::Cow;
use std::collections::HashMap;
use string_cache::DefaultAtom as Atom;

pub mod metric;

pub use metric::Metric;

pub mod proto {
include!(concat!(env!("OUT_DIR"), "/event.proto.rs"));
}
Expand All @@ -20,6 +24,7 @@ lazy_static! {
#[derive(PartialEq, Debug, Clone)]
pub enum Event {
Log(LogEvent),
Metric(Metric),
}

#[derive(PartialEq, Debug, Clone)]
Expand All @@ -37,18 +42,28 @@ impl Event {
pub fn as_log(&self) -> &LogEvent {
match self {
Event::Log(log) => log,
_ => panic!("failed type coercion, {:?} is not a log event", self),
}
}

pub fn as_mut_log(&mut self) -> &mut LogEvent {
match self {
Event::Log(log) => log,
_ => panic!("failed type coercion, {:?} is not a log event", self),
}
}

pub fn into_log(self) -> LogEvent {
match self {
Event::Log(log) => log,
_ => panic!("failed type coercion, {:?} is not a log event", self),
}
}

pub fn into_metric(self) -> Metric {
Copy link
Contributor

@LucioFranco LucioFranco May 16, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

possible to have a is_metric?

Copy link
Member Author

@lukesteensen lukesteensen May 16, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still think that's an anti pattern and consumers should know what type they're getting and just do the coercion.

match self {
Event::Metric(metric) => metric,
_ => panic!("failed type coercion, {:?} is not a metric", self),
}
}
}
Expand Down Expand Up @@ -233,6 +248,51 @@ impl From<proto::EventWrapper> for Event {

Event::Log(LogEvent { structured })
}
EventProto::Metric(proto) => {
let metric = proto.metric.unwrap();
match metric {
MetricProto::Counter(counter) => {
let sampling = if counter.sampling == 0f32 {
None
} else {
Some(counter.sampling)
};
Event::Metric(Metric::Counter {
name: counter.name,
val: counter.val,
sampling,
})
}
MetricProto::Timer(timer) => {
let sampling = if timer.sampling == 0f32 {
None
} else {
Some(timer.sampling)
};
Event::Metric(Metric::Timer {
name: timer.name,
val: timer.val,
sampling,
})
}
MetricProto::Gauge(gauge) => {
let direction = match gauge.direction() {
proto::gauge::Direction::None => None,
proto::gauge::Direction::Plus => Some(metric::Direction::Plus),
proto::gauge::Direction::Minus => Some(metric::Direction::Minus),
};
Event::Metric(Metric::Gauge {
name: gauge.name,
val: gauge.val,
direction,
})
}
MetricProto::Set(set) => Event::Metric(Metric::Set {
name: set.name,
val: set.val,
}),
}
}
}
}
}
Expand All @@ -256,20 +316,77 @@ impl From<Event> for proto::EventWrapper {

proto::EventWrapper { event: Some(event) }
}
Event::Metric(Metric::Counter {
name,
val,
sampling,
}) => {
let counter = proto::Counter {
name,
val,
sampling: sampling.unwrap_or(0f32),
};
let event = EventProto::Metric(proto::Metric {
metric: Some(MetricProto::Counter(counter)),
});
proto::EventWrapper { event: Some(event) }
}
Event::Metric(Metric::Timer {
name,
val,
sampling,
}) => {
let timer = proto::Timer {
name,
val,
sampling: sampling.unwrap_or(0f32),
};
let event = EventProto::Metric(proto::Metric {
metric: Some(MetricProto::Timer(timer)),
});
proto::EventWrapper { event: Some(event) }
}
Event::Metric(Metric::Gauge {
name,
val,
direction,
}) => {
let direction = match direction {
None => proto::gauge::Direction::None,
Some(metric::Direction::Plus) => proto::gauge::Direction::Plus,
Some(metric::Direction::Minus) => proto::gauge::Direction::Minus,
}
.into();
let gauge = proto::Gauge {
name,
val,
direction,
};
let event = EventProto::Metric(proto::Metric {
metric: Some(MetricProto::Gauge(gauge)),
});
proto::EventWrapper { event: Some(event) }
}
Event::Metric(Metric::Set { name, val }) => {
let set = proto::Set { name, val };
let event = EventProto::Metric(proto::Metric {
metric: Some(MetricProto::Set(set)),
});
proto::EventWrapper { event: Some(event) }
}
}
}
}

// TODO: should probably get rid of this
impl From<Event> for Vec<u8> {
fn from(event: Event) -> Vec<u8> {
match event {
Event::Log(LogEvent { mut structured }) => structured
.remove(&MESSAGE)
.unwrap()
.value
.as_bytes()
.into_owned(),
}
event
.into_log()
.into_value(&MESSAGE)
.unwrap()
.as_bytes()
.into_owned()
}
}

Expand Down
28 changes: 28 additions & 0 deletions src/event/metric.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
#[derive(Debug, Clone, PartialEq)]
pub enum Metric {
Counter {
name: String,
val: u32,
sampling: Option<f32>,
},
Timer {
name: String,
val: u32,
sampling: Option<f32>,
},
Gauge {
name: String,
val: u32,
direction: Option<Direction>,
},
Set {
name: String,
val: String,
},
}

#[derive(Debug, Clone, PartialEq)]
pub enum Direction {
Plus,
Minus,
}
Loading