Out_of_memory with Dns_resolver_mirage
#298
Comments
|
Do you know what version of ocaml-dns was used? |
|
The version is |
|
Thanks for your report. I've no insight which |
|
Currently, my unikernel is available here: https://github.com/mirage/dns-resolver I will relaunch it with |
|
That would be good -- though I'd expect more useful data if you add some metrics (or periodic log messages) about memory consumption and cache size (since the LRU of the cache is not measured in actual used bytes, but items (where the weight function is the size of the resource record map (and each resource record may be different in size)). |
|
The reason I asked is because I can't make sense of the line number |
|
I opened #299 that should help to figure out whether the cache is causing (as created) the OOM. Is there a way to have a over-approximation of the memory used by an OCaml value (AFAICT not, esp. not if using bigarrays) -- |
It's probably due to that |
|
This is my last logs just before the |
|
Not sure if that can help as it seems to take a long time to trigger the memory fault, but I tried to print memory usage, my modifications can be seen https://github.com/palainp/dns-resolver/tree/print-mem-usage. It displays every 10" 3 memory sizes: the memory size available above top of the heap (including the stack) this can be used by malloc to growth the ocaml memory, the real free memory (above the heap (including the stack) + free memory inside the heap), and total memory available (the --mem arg minus the kernel size). |
|
Dear all, Unfortunately the caller's address But it's interesting to note that for each query, we allocate several buffers of size 0, 43, 85 (resp. 45 and 87, as the size increases with the DNS query, I assume these buffers come from ocaml-dns/mirage/stub/dns_stub_mirage.ml Line 247 in 83a0f93 I think these buffers lead to memory fragmentation and can be a sensitive cause of OOM. Since the mirage-tcpip interface uses Cstruct, most of them won't be avoidable, but perhaps some can be replaced by strings? |
interesting -- at the same time, is this related to the query? i.e. |
|
Thanks for your comment, is there any way you get a backtrace with more depth on allocation, and not only the caml_ba_alloc? |
Yes, I understand that difference with the same assumption :) Unfortunately that backtrace print was a bit hacky in the C part of the solo5 allocator, and when I try to get more depth I have to define a number of function calls to print, and I fail at some point with a segfault :(, but I can try to get an educated guess where that allocation might be and print it from Ocaml :) |
|
So the 43B allocation (for mirage.io) is due to line 247 in ocaml-dns/mirage/stub/dns_stub_mirage.ml, I just checked it with a logprint before and after. Printing backtraces adds a few extra allocations, but I can follow the trail :) EDIT: and the 85B is allocated in the |
|
And the 12 byte allocation is likely coming from Ipv4_packet.Marshal.pseudoheader called by Udp.write. Certainly moving these to bytes/string shouldn't be too hard... only takes time... also the 12 byte pseudoheader could be cached. |
After ~2 weeks, I systematically get an
Out_of_memorywith this trace:It seems about an internal
Hashtbl. The unikernel is launched with 32MB. I will try to introspect further to understand where is the error.The text was updated successfully, but these errors were encountered: