Memory leak detected in valgrind when running libyang

[Ebulus7899]

Version: Libyang 2.1.80

Hi There,

We have an application and it has been found recently that a process in that application has been increasing in memory. Through some binary searching I was able to narrow it down to a change that was made involving libyang.

We have a typescript project that makes use of napi-rs to call rust code that natively runs libyang

I managed to run valgrind on our application running libyang and a definite memory leak was detected

==23236== LEAK SUMMARY:
==23236== definitely lost: 2,048 bytes in 1 blocks
==23236== indirectly lost: 0 bytes in 0 blocks
==23236== possibly lost: 857,350 bytes in 7,307 blocks
==23236== still reachable: 384 bytes in 2 blocks
==23236== of which reachable via heuristic:
==23236== newarray : 150,032 bytes in 1,044 blocks
==23236== suppressed: 0 bytes in 0 blocks
==23236== Reachable blocks (those to which a pointer was found) are not shown.
==23236== To see them, rerun with: --leak-check=full --show-leak-kinds=all

The leak seems to be coming from here

==23236== 2,048 bytes in 1 blocks are definitely lost in loss record 1,061 of 1,125
==23236== at 0x483AD7B: realloc (vg_replace_malloc.c:834)
==23236== by 0x485F58C: ly_realloc (in /usr/lib/x86_64-linux-gnu/libyang.so.2)
==23236== by 0x48907C7: ly_write_ (in /usr/lib/x86_64-linux-gnu/libyang.so.2)
==23236== by 0x4892B7F: json_print_string.part.0 (in /usr/lib/x86_64-linux-gnu/libyang.so.2)
==23236== by 0x4892C68: json_print_value (in /usr/lib/x86_64-linux-gnu/libyang.so.2)
==23236== by 0x4893C3F: json_print_node.part.0 (in /usr/lib/x86_64-linux-gnu/libyang.so.2)
==23236== by 0x48946FA: json_print_inner (in /usr/lib/x86_64-linux-gnu/libyang.so.2)
==23236== by 0x4894212: json_print_node.part.0 (in /usr/lib/x86_64-linux-gnu/libyang.so.2)
==23236== by 0x48946FA: json_print_inner (in /usr/lib/x86_64-linux-gnu/libyang.so.2)
==23236== by 0x4894212: json_print_node.part.0 (in /usr/lib/x86_64-linux-gnu/libyang.so.2)
==23236== by 0x48946FA: json_print_inner (in /usr/lib/x86_64-linux-gnu/libyang.so.2)
==23236== by 0x4893C72: json_print_node.part.0 (in /usr/lib/x86_64-linux-gnu/libyang.so.2)

If I run the application multiple times then the memory lost will increase by number of runs * 2Kb

Kind regards

[michalvasko]

Please rerun the valgrind command with --num-callers=30 so that the full stack trace can be seen, thanks.

[Ebulus7899]

Thankss for the reply.

I tried updating to the latest version of libyang 2.1 (2.1.148) to see if that changed any behaviour. The following stack traces are for that version.

I have run the added the command and it seems to happen if we add the WITH_SIBLINGS flag when printing

==29002== at 0x483AD7B: realloc (vg_replace_malloc.c:834)
==29002== by 0x485FA7C: ly_realloc (in /usr/local/lib/libyang.so.2.46.3)
==29002== by 0x4892CC7: ly_write_ (in /usr/local/lib/libyang.so.2.46.3)
==29002== by 0x489520F: json_print_string.part.0 (in /usr/local/lib/libyang.so.2.46.3)
==29002== by 0x489528B: json_print_value (in /usr/local/lib/libyang.so.2.46.3)
==29002== by 0x48964B1: json_print_node.part.0 (in /usr/local/lib/libyang.so.2.46.3)
==29002== by 0x4896F92: json_print_inner (in /usr/local/lib/libyang.so.2.46.3)
==29002== by 0x4896B42: json_print_node.part.0 (in /usr/local/lib/libyang.so.2.46.3)
==29002== by 0x4896F92: json_print_inner (in /usr/local/lib/libyang.so.2.46.3)
==29002== by 0x4896B42: json_print_node.part.0 (in /usr/local/lib/libyang.so.2.46.3)
==29002== by 0x4896F92: json_print_inner (in /usr/local/lib/libyang.so.2.46.3)
==29002== by 0x48964EA: json_print_node.part.0 (in /usr/local/lib/libyang.so.2.46.3)
==29002== by 0x48973EA: json_print_data (in /usr/local/lib/libyang.so.2.46.3)
==29002== by 0x4893772: lyd_print_mem (in /usr/local/lib/libyang.so.2.46.3)
==29002== by 0x119FBC: yang3::data::Data::print_string (data.rs:349)

==32264== LEAK SUMMARY:
==32264== definitely lost: 2,048 bytes in 1 blocks
==32264== indirectly lost: 0 bytes in 0 blocks
==32264== possibly lost: 922,950 bytes in 7,309 blocks
==32264== still reachable: 448 bytes in 2 blocks
==32264== of which reachable via heuristic:
==32264== newarray : 150,032 bytes in 1,044 blocks
==32264== suppressed: 0 bytes in 0 blocks
==32264== Reachable blocks (those to which a pointer was found) are not shown.
==32264== To see them, rerun with: --leak-check=full --show-leak-kinds=all

If I don't use the WITH_SIBLINGS flag the memory leak is not nearly as bad

==30633== LEAK SUMMARY:
==30633== definitely lost: 6 bytes in 1 blocks
==30633== indirectly lost: 0 bytes in 0 blocks
==30633== possibly lost: 922,950 bytes in 7,309 blocks
==30633== still reachable: 448 bytes in 2 blocks
==30633== of which reachable via heuristic:
==30633== newarray : 150,032 bytes in 1,044 blocks
==30633== suppressed: 0 bytes in 0 blocks
==30633== Reachable blocks (those to which a pointer was found) are not shown.
==30633== To see them, rerun with: --leak-check=full --show-leak-kinds=all

[Ebulus7899]

As an added bit of information I have possibly found the leak.

The allocation is happening here

out->method.mem.buf = ly_realloc(*out->method.mem.buf, new_mem_size);
if (!*out->method.mem.buf)
{
 out->method.mem.len = 0;
 out->method.mem.size = 0;
  LOGMEM(NULL);
  return LY_EMEM;
}

so memory is being allocated to out->method.mem.buf

lyd_print_mem(char **strp, const struct lyd_node *root, LYD_FORMAT format, uint32_t options)
{
    LY_ERR ret;
    struct ly_out *out;

    LY_CHECK_ARG_RET(NULL, strp, LY_EINVAL);

    /* init */
    *strp = NULL;

    LY_CHECK_RET(ly_out_new_memory(strp, 0, &out));
    ret = lyd_print_(out, root, format, options);
    ly_out_free(out, NULL, 0);
    return ret;
}

lyd_print_mem calls ly_out_free with destroy of 0 so the buffer will never be cleared

I have created a pull request #2336

[Ebulus7899]

Changing the 0 to a 1 and I get

=85529== LEAK SUMMARY:
==85529== definitely lost: 0 bytes in 0 blocks
==85529== indirectly lost: 0 bytes in 0 blocks
==85529== possibly lost: 931,248 bytes in 7,317 blocks
==85529== still reachable: 448 bytes in 2 blocks
==85529== of which reachable via heuristic:
==85529== newarray : 152,392 bytes in 1,046 blocks
==85529== suppressed: 0 bytes in 0 blocks
==85529== Reachable blocks (those to which a pointer was found) are not shown.
==85529== To see them, rerun with: --leak-check=full --show-leak-kinds=all

Changing it back to a 0 and I get

==88651== LEAK SUMMARY:
==88651== definitely lost: 2,048 bytes in 1 blocks
==88651== indirectly lost: 0 bytes in 0 blocks
==88651== possibly lost: 931,248 bytes in 7,317 blocks
==88651== still reachable: 448 bytes in 2 blocks
==88651== of which reachable via heuristic:
==88651== newarray : 152,392 bytes in 1,046 blocks
==88651== suppressed: 0 bytes in 0 blocks
==88651== Reachable blocks (those to which a pointer was found) are not shown.
==88651== To see them, rerun with: --leak-check=full **--show-leak-kinds=all

[michalvasko]

There is no leak, you must free the output string yourself. It cannot be freed right after printing, you would then get a freed string in your application.

[Ebulus7899]

Sorry my mistake. Thank you for the reply. It seems that the actual culprit would be the yang3 package making use of the libyang library.

    fn print_string(
        &self,
        format: DataFormat,
        options: DataPrinterFlags,
    ) -> Result<String> {
        let mut cstr = std::ptr::null_mut();
        let cstr_ptr = &mut cstr;

        let ret = unsafe {
            ffi::lyd_print_mem(
                cstr_ptr,
                self.raw(),
                format as u32,
                options.bits(),
            )
        };
        if ret != ffi::LY_ERR::LY_SUCCESS {
            return Err(Error::new(self.context()));
        }

        Ok(char_ptr_to_string(cstr))
    }

It does not seem to free the cstr