Log and metric collector recommendations?

[jerrac]

What do you do to collect all the logs and metrics you need and get them into greptimedb?

I've been trying to use Telegraf as my single collector, but I'm running into issues with logs. Specifically if I query too many logs, greptime will use up enough memory that it causes problems. I'm still learning about greptimedb, but I think part of the solution is to configure pipelines that create proper indexes and parse logs out into their fields. My initial configuration has been using the loki telegraf output and greptime input, but it turns out that greptime doesn't support pipelines with the loki ingest. (https://github.com/orgs/GreptimeTeam/discussions/6270)

(Note: metrics via the infludb v1 output/ingest work great. Super fast, and no issues with large queries.)

I just tried the Elasticsearch telegraf output, it didn't work. The http output pointed at the elasticsearch ingest resulted 404's for some reason.

I like Vector.dev, but it doesn't have a docker metrics source (I think I read in their issue queue that they just want to leave that up to their cgroups source..) I've looked at Grafana Alloy, but I haven't wanted to try it due to needing to decipher yet another configuration syntax.

Anyway, what do you use? Do you use one collector, or more?

[sunng87]

I would recommend vector.dev and our team maintains the greptimedb sinks of vector so it has first class support.

For metrics, if you are from Prometheus world I would recommend you to use vector's prometheus remote write sink. For the docker metrics source, do you know if there is any prometheus exporter for that? If it's compatible with Prometheus's openmetric data, you can also configure vector to scrape it.

[jerrac]

I tried Prometheus and didn't find the scraping method of gathering data very appealing. I've tried to get a solid Elastic stack going, but between lack of resources on our side, and feature bloat on theirs, it has never been as useful or maintainable as I'd like.

I also tried influxdbv2 for storage, vector for logs, telegraf for metrics, and grafana for UI, but influxdbv2 is very much not made for logs and I never figured out how to make it work the way I wanted.

I just looked up docker stats in vector's issue queue and found an older comment of mine. vectordotdev/vector#1545 (comment) So it looks like that feature is still a ways out.

Makes me wish I could find something that'd get me over the initial rust learning curve. Every time I've tried, so far, I hit some weird thing in how rust does things that I get stuck on, and that causes me to take a nice long break from it...

Ah well. Anyway, I'll look into Vector again. Maybe there's another way to get the Docker stats if I think about it hard enough.

[sunng87]

Do you have an example of what types of docker stats you would like to get? Is it possible to get that type of data from telegraph directly?

[jerrac]

Container specific CPU, RAM, Network in/out, disk read/write. If you have docker running, just run docker stats.

It will look something like:

CONTAINER ID   NAME                            CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
containerhash   containername           0.00%     32.09MiB / 30.96GiB         0.10%     14MB / 13MB       108MB / 0B        13

Is it possible to get that type of data from telegraph directly?

Er, you mean Docker directly? Yes. docker stats and/or use the docker api.

I was just looking into using the Vector exec source to run try and do that. :)

[jerrac]

Oh, you can also get it in json format from docker stats --format json.

[sunng87]

The exec source may give you a log entry in vector, are you will need to convert it to metrics via a transform.