Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Buffer size for udp is too small #2331

Closed
kbarukhov opened this issue Jul 7, 2020 · 4 comments
Closed

Buffer size for udp is too small #2331

kbarukhov opened this issue Jul 7, 2020 · 4 comments

Comments

@kbarukhov
Copy link
Contributor

Describe the bug
Default buffer size for udp is too small. Because of that some udp-packets are dropped

What troubleshooting steps did you try?
We have quick fix for this issue. May be we should merge it?
https://github.com/ozonru/jaeger/blob/1.18-udp-buffer-size/cmd/agent/app/servers/thriftudp/transport.go#L90

@kbarukhov kbarukhov added the bug label Jul 7, 2020
@ghost ghost added the needs-triage label Jul 7, 2020
@yurishkuro
Copy link
Member

+1, please do a PR. Could you also provide more context - your setup, under what conditions you notice data loss, how did you arrive to the current buffer size? What is the default buffer size? Is it OS-dependent?

It may make sense to expose this parameter as configurable.

@kbarukhov
Copy link
Contributor Author

With something like 1.2 million spans in second and about 5 Gbits total traffic we saw package dropping. Default buffer size is around 4 kB. Current size was found experimently) I'll test if these syscalls work on windows and do a PR

@yurishkuro
Copy link
Member

yurishkuro commented Jul 8, 2020

"1.2 million spans/sec" for one agent?

@mkabischev
Copy link

Noo, 1.2kk spans is total throughput for our setup. We have some instances with "heavy" services that send a lot of telemetry and that caused this problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants