Crash inside libyang with filtering #1598

vpestret · 2021-05-28T10:03:56Z

Hi, I have Netopeer2 application and when tried filtering I've got crash
get-config --source running --filter-subtree=tests/filters/oven.xml

oven.xml file contents:

<oven/>

stack trace:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007fe294523921 in __GI_abort () at abort.c:79
#2  0x00007fe29456c967 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7fe294699b0d "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#3  0x00007fe2945739da in malloc_printerr (str=str@entry=0x7fe294697d08 "free(): invalid pointer") at malloc.c:5342
#4  0x00007fe29457af0c in _int_free (have_lock=0, p=0x7fe294c4fd50 <__PRETTY_FUNCTION__.6809+16>, av=0x7fe2948cec40 <main_arena>) at malloc.c:4167
#5  __GI___libc_free (mem=0x7fe294c4fd60) at malloc.c:3134
#6  0x00007fe294b5fcd4 in dict_insert (ctx=0x55dbcf4a9dd0, value=0x7fe294c4fd60 "", len=0, zerocopy=1 '\001', str_p=0x7fe274025e08) at /opt/dev/libyang/src/hash_table.c:234
#7  0x00007fe294b5ffb7 in lydict_insert_zc (ctx=0x55dbcf4a9dd0, value=0x7fe294c4fd60 "", str_p=0x7fe274025e08) at /opt/dev/libyang/src/hash_table.c:299
#8  0x00007fe294b75e93 in lyd_create_opaq (ctx=0x55dbcf4a9dd0, name=0x7fe274001f20 "oven", name_len=4, prefix=0x7fe274001550 "", pref_len=0, 
    module_key=0x7fe274017910 "urn:ietf:params:xml:ns:netconf:base:1.0", module_key_len=39, value=0x7fe294c4fd60 "", value_len=0, dynamic=0x7fe292c6c61b "\001\342\177", format=LY_VALUE_XML, 
    val_prefix_data=0x0, hints=0, node=0x7fe292c6c630) at /opt/dev/libyang/src/tree_data.c:844
#9  0x00007fe294b8f2f3 in lyb_parse_subtree_r (lybctx=0x7fe27401fdc0, parent=0x0, first_p=0x7fe292c6c928, parsed=0x7fe292c6c7c0) at /opt/dev/libyang/src/parser_lyb.c:793
#10 0x00007fe294b901db in lyd_parse_lyb (ctx=0x55dbcf4a9dd0, ext=0x0, parent=0x0, first_p=0x7fe292c6c928, in=0x7fe27400ad40, parse_opts=458752, val_opts=0, data_type=LYD_TYPE_DATA_YANG, 
    parsed=0x7fe292c6c7c0, lydctx_p=0x7fe292c6c7b0) at /opt/dev/libyang/src/parser_lyb.c:1100
#11 0x00007fe294b74449 in lyd_parse (ctx=0x55dbcf4a9dd0, ext=0x0, parent=0x0, first_p=0x7fe292c6c928, in=0x7fe27400ad40, format=LYD_LYB, parse_opts=458752, val_opts=0, op=0x0)
    at /opt/dev/libyang/src/tree_data.c:366
#12 0x00007fe294b74966 in lyd_parse_data (ctx=0x55dbcf4a9dd0, parent=0x0, in=0x7fe27400ad40, format=LYD_LYB, parse_options=458752, validate_options=0, tree=0x7fe292c6c928)
    at /opt/dev/libyang/src/tree_data.c:435
#13 0x00007fe294b749e5 in lyd_parse_data_mem (ctx=0x55dbcf4a9dd0, data=0x7fe27401fc10 "lyb\020", format=LYD_LYB, parse_options=458752, validate_options=0, tree=0x7fe292c6c928)
    at /opt/dev/libyang/src/tree_data.c:446
#14 0x00007fe294b8f792 in lyb_parse_subtree_r (lybctx=0x7fe274011f40, parent=0x7fe274025bb0, first_p=0x0, parsed=0x0) at /opt/dev/libyang/src/parser_lyb.c:862
#15 0x00007fe294b8f50a in lyb_parse_subtree_r (lybctx=0x7fe274011f40, parent=0x0, first_p=0x7fe292c6cbd0, parsed=0x7fe292c6cbe0) at /opt/dev/libyang/src/parser_lyb.c:824
#16 0x00007fe294b901db in lyd_parse_lyb (ctx=0x55dbcf4a9dd0, ext=0x0, parent=0x0, first_p=0x7fe292c6cbd0, in=0x7fe2740018f0, parse_opts=196608, val_opts=0, data_type=LYD_TYPE_RPC_YANG, 
    parsed=0x7fe292c6cbe0, lydctx_p=0x7fe292c6cbc8) at /opt/dev/libyang/src/parser_lyb.c:1100
#17 0x00007fe294b74ffe in lyd_parse_op_ (ctx=0x55dbcf4a9dd0, ext=0x0, parent=0x0, in=0x7fe2740018f0, format=LYD_LYB, data_type=LYD_TYPE_RPC_YANG, tree=0x7fe292c6ccb0, op=0x0)
    at /opt/dev/libyang/src/tree_data.c:572
#18 0x00007fe294b752b8 in lyd_parse_op (ctx=0x55dbcf4a9dd0, parent=0x0, in=0x7fe2740018f0, format=LYD_LYB, data_type=LYD_TYPE_RPC_YANG, tree=0x7fe292c6ccb0, op=0x0)
    at /opt/dev/libyang/src/tree_data.c:625
#19 0x00007fe2949232c0 in sr_shmsub_rpc_listen_process_rpc_events (rpc_subs=0x55dbcf4d8100, conn=0x55dbcf4a9bc0) at /opt/dev/sysrepo/src/shm_sub.c:3129
#20 0x00007fe2948e2b2f in sr_process_events (subscription=0x55dbcf6422e0, session=0x0, stop_time_in=0x7fe292c6cde0) at /opt/dev/sysrepo/src/sysrepo.c:3060
#21 0x00007fe29492447f in sr_shmsub_listen_thread (arg=0x55dbcf6422e0) at /opt/dev/sysrepo/src/shm_sub.c:3575
#22 0x00007fe294e906db in start_thread (arg=0x7fe292c6d700) at pthread_create.c:463
#23 0x00007fe29460471f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

my Dockerfile fragment:

# libssh
RUN \
      cd /opt/dev && \
      git clone https://git.libssh.org/projects/libssh.git && cd libssh && \
      git checkout stable-0.9 && \
      mkdir build && cd build && \
      cmake -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE="Release" -DWITH_ZLIB=ON -DWITH_NACL=OFF -DWITH_PCAP=OFF .. && \
      make -j && \
      make install

# libyang
RUN \
      cd /opt/dev && \
      git clone https://github.com/CESNET/libyang.git && cd libyang && \
      git checkout v2.0.7 && \
      mkdir build && cd build && \
      cmake -DCMAKE_INSTALL_PREFIX:PATH=/usr -DCMAKE_BUILD_TYPE:String="Debug" .. && \
      make -j && \
      make install

# sysrepo
RUN \
      cd /opt/dev && \
      git clone https://github.com/sysrepo/sysrepo.git && cd sysrepo && \
      git checkout v2.0.1 && \
      mkdir build && cd build && \
      cmake -DCMAKE_INSTALL_PREFIX:PATH=/usr -DCMAKE_BUILD_TYPE:String="Debug" .. && \
      make -j && \
      make install

# libnetconf2
RUN \
      cd /opt/dev && \
      git clone https://github.com/CESNET/libnetconf2.git && cd libnetconf2 && \
      git checkout v2.0.1 && \
      mkdir build && cd build && \
      cmake  -DCMAKE_BUILD_TYPE:String="Debug" -DCMAKE_INSTALL_PREFIX:PATH=/usr -DENABLE_BUILD_TESTS=OFF .. && \
      make -j && \
      make install

# netopeer 2
RUN \
      cd /opt/dev && \
      git clone https://github.com/CESNET/Netopeer2.git && cd Netopeer2 && \
      git checkout v2.0.0 && \
      mkdir build && cd build && \
      cmake -DCMAKE_INSTALL_PREFIX:PATH=/usr -DCMAKE_BUILD_TYPE:String="Debug" .. && \
      make -j && \
      make install

The text was updated successfully, but these errors were encountered:

vpestret · 2021-05-28T10:45:36Z

Here is full Dockerfile attached

FROM ubuntu:18.04

# defaults
RUN \
	apt-get update && apt-get install -y \
	git \
	curl \
	wget \
	libssl-dev \
	libtool \
	build-essential \
	vim \
	autoconf \
	automake \
	pkg-config \
	libgtk-3-dev \
	make \
	valgrind \
	doxygen \
	libev-dev \
	libpcre2-dev \
	unzip \
	sudo \
	python3 \
	bison \
	flex \
	swig \
	libcmocka0 \
	libcmocka-dev \
	cmake \
	inetutils-syslogd \
	libzmq3-dev \
	supervisor

# Adding netconf user
RUN adduser --system netconf
RUN mkdir -p /home/netconf/.ssh
RUN echo "netconf:1" | chpasswd && adduser netconf sudo


# Clearing and setting authorized ssh keys
RUN \
	echo '' > /home/netconf/.ssh/authorized_keys && \
	ssh-keygen -A && \
	ssh-keygen -t rsa -b 4096 -P '' -f /home/netconf/.ssh/id_rsa && \
	cat /home/netconf/.ssh/id_rsa.pub >> /home/netconf/.ssh/authorized_keys


# Updating shell to bash
RUN sed -i s#/home/netconf:/bin/false#/home/netconf:/bin/bash# /etc/passwd

RUN mkdir /opt/dev && sudo chown -R netconf /opt/dev

# set password for user (same as the username)
RUN echo "root:1" | chpasswd

# libssh
RUN \
      cd /opt/dev && \
      git clone https://git.libssh.org/projects/libssh.git && cd libssh && \
      git checkout stable-0.9 && \
      mkdir build && cd build && \
      cmake -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE="Release" -DWITH_ZLIB=ON -DWITH_NACL=OFF -DWITH_PCAP=OFF .. && \
      make -j && \
      make install

# libyang
RUN \
      cd /opt/dev && \
      git clone https://github.com/CESNET/libyang.git && cd libyang && \
      git checkout v2.0.7 && \
      mkdir build && cd build && \
      cmake -DCMAKE_INSTALL_PREFIX:PATH=/usr -DCMAKE_BUILD_TYPE:String="Debug" .. && \
      make -j && \
      make install

# sysrepo
RUN \
      cd /opt/dev && \
      git clone https://github.com/sysrepo/sysrepo.git && cd sysrepo && \
      git checkout v2.0.1 && \
      mkdir build && cd build && \
      cmake -DCMAKE_INSTALL_PREFIX:PATH=/usr -DCMAKE_BUILD_TYPE:String="Debug" .. && \
      make -j && \
      make install

# libnetconf2
RUN \
      cd /opt/dev && \
      git clone https://github.com/CESNET/libnetconf2.git && cd libnetconf2 && \
      git checkout v2.0.1 && \
      mkdir build && cd build && \
      cmake  -DCMAKE_BUILD_TYPE:String="Debug" -DCMAKE_INSTALL_PREFIX:PATH=/usr -DENABLE_BUILD_TESTS=OFF .. && \
      make -j && \
      make install

# netopeer 2
RUN \
      cd /opt/dev && \
      git clone https://github.com/CESNET/Netopeer2.git && cd Netopeer2 && \
      git checkout v2.0.0 && \
      mkdir build && cd build && \
      cmake -DCMAKE_INSTALL_PREFIX:PATH=/usr -DCMAKE_BUILD_TYPE:String="Debug" .. && \
      make -j && \
      make install

ENV EDITOR vim
EXPOSE 830

COPY tests/oven.yang /home/netconf
RUN sysrepoctl -i /home/netconf/oven.yang

COPY configs/supervisord-m-plane.conf /etc/supervisord.conf

CMD ["/usr/bin/supervisord", "-c", "/etc/supervisord.conf"]

also supervisor file

[supervisord]
nodaemon=true
logfile=/var/log/supervisord.log
loglevel=debug

[program:syslogd]
command=/usr/sbin/syslogd
autorestart=false
startsecs=0
redirect_stderr=true
priority=3

[program:netopeer2-server]
command=/usr/bin/netopeer2-server -d
autorestart=true
redirect_stderr=true
priority=3

enjoy!

by the way is there a documentation about libyang library architecture?

Fixes #1598

michalvasko · 2021-05-31T09:23:03Z

Okay, the crash should be fixed but note that you should not use subtree filters with elements without a namespace. In all likelihood they inherit some namespace from the RPC, which is not what you want.

is there a documentation about libyang library architecture?

I am not sure libyang has some specific architecture. But the documentation is here.

vpestret · 2021-05-31T09:50:52Z

Thank you very much for looking into this. Thank you for the good advice about namespaces. I've found this documentation already and started to study, in fact, there was a reference from the main page. As much as I can I'm glad to help improving such a great project!

vpestret · 2021-05-31T10:18:00Z

The bugfix provided helped to fix the crash asked but there is another crash appear:

#0  0x000055e5ac61df27 in filter_xpath_buf_append_node (node=0x7f3e58025db0, buf=0x7f3e6b4b4a88, size=1) at /opt/dev/Netopeer2/src/common.c:730
#1  0x000055e5ac61e0d8 in filter_xpath_buf_add_r (node=0x7f3e58025db0, buf=0x7f3e6b4b4a88, size=1, filter=0x7f3e6b4b4b30) at /opt/dev/Netopeer2/src/common.c:761
#2  0x000055e5ac61e368 in op_filter_subtree2xpath (node=0x7f3e58025db0, filter=0x7f3e6b4b4b30) at /opt/dev/Netopeer2/src/common.c:837
#3  0x000055e5ac61f388 in np2srv_rpc_get_cb (session=0x7f3e58002370, UNUSED_sub_id=113, op_path=0x55e5acd51670 "/ietf-netconf:get-config", input=0x7f3e58025bb0, event=SR_EV_RPC, UNUSED_request_id=1, 
                                                                     output=0x7f3e5800ad40, UNUSED_private_data=0x0) at /opt/dev/Netopeer2/src/netconf.c:249
#4  0x00007f3e6d16a7d9 in sr_shmsub_rpc_listen_call_callback (rpc_sub=0x55e5acdcb670, ev_sess=0x7f3e58002370, input_op=0x7f3e58025bb0, event=SR_SUB_EV_RPC, request_id=1, output_op=0x7f3e6b4b4cc0, 
                                                                                      err_code=0x7f3e6b4b4c88) at /opt/dev/sysrepo/src/shm_sub.c:2897
#5  0x00007f3e6d16b528 in sr_shmsub_rpc_listen_process_rpc_events (rpc_subs=0x55e5acd4e100, conn=0x55e5acd1fbc0) at /opt/dev/sysrepo/src/shm_sub.c:3182
#6  0x00007f3e6d12ab2f in sr_process_events (subscription=0x55e5aceb82e0, session=0x0, stop_time_in=0x7f3e6b4b4de0) at /opt/dev/sysrepo/src/sysrepo.c:3060
#7  0x00007f3e6d16c47f in sr_shmsub_listen_thread (arg=0x55e5aceb82e0) at /opt/dev/sysrepo/src/shm_sub.c:3575
#8  0x00007f3e6d6d86db in start_thread (arg=0x7f3e6b4b5700) at pthread_create.c:463
#9  0x00007f3e6ce4c71f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Anyway when I've tried to supply correct subtree filter file with namespaces both crashes not happen.

<oven xmlns="urn:sysrepo:oven"/>

michalvasko · 2021-05-31T10:21:03Z

You just need to update netopeer2, I have already fixed that, too, just not mentioned it.

vpestret · 2021-05-31T10:46:07Z

When I've checked out Netopeer2 version 91f862f94f60b9bc112f6810e8fe293b18a8d9a5 (VERSION bump to version 2.0.2) and libyang to version 0b8573e (VERSION bump to version 2.0.8 ) then it worked thanks!!!

vpestret changed the title ~~Crash inside libyang with fitering~~ Crash inside libyang with filtering May 28, 2021

michalvasko added a commit that referenced this issue May 31, 2021

data tree BUGFIX always use dynamic value

4cd0a60

Fixes #1598

vpestret closed this as completed May 31, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Crash inside libyang with filtering #1598

Crash inside libyang with filtering #1598

vpestret commented May 28, 2021 •

edited

Loading

vpestret commented May 28, 2021 •

edited

Loading

michalvasko commented May 31, 2021

vpestret commented May 31, 2021

vpestret commented May 31, 2021

michalvasko commented May 31, 2021

vpestret commented May 31, 2021

Crash inside libyang with filtering #1598

Crash inside libyang with filtering #1598

Comments

vpestret commented May 28, 2021 • edited Loading

vpestret commented May 28, 2021 • edited Loading

michalvasko commented May 31, 2021

vpestret commented May 31, 2021

vpestret commented May 31, 2021

michalvasko commented May 31, 2021

vpestret commented May 31, 2021

vpestret commented May 28, 2021 •

edited

Loading

vpestret commented May 28, 2021 •

edited

Loading