Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make it easier to integrate dfuzzer into CI #20

Open
evverx opened this issue Apr 17, 2022 · 13 comments
Open

Make it easier to integrate dfuzzer into CI #20

evverx opened this issue Apr 17, 2022 · 13 comments

Comments

@evverx
Copy link
Member

evverx commented Apr 17, 2022

It was briefly discussed in https://github.com/matusmarhefka/dfuzzer/pull/13#issuecomment-1092258508

I think in general dfuzzer isn't ready to be integrated into CI due to some limitations like https://github.com/matusmarhefka/dfuzzer/issues/19 but given that it's capable of discovering crashes like

#0  0x00007f5de18fe88c in __pthread_kill_implementation () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install avahi-0.8-14.fc35.x86_64
(gdb) bt
#0  0x00007f5de18fe88c in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x00007f5de18b16a6 in raise () from /lib64/libc.so.6
#2  0x00007f5de189b7d3 in abort () from /lib64/libc.so.6
#3  0x00007f5de1a8bb16 in _dbus_abort.cold () from /lib64/libdbus-1.so.3
#4  0x00007f5de1ab2250 in _dbus_warn_check_failed () from /lib64/libdbus-1.so.3
#5  0x00007f5de1aa14ef in dbus_message_iter_append_basic () from /lib64/libdbus-1.so.3
#6  0x00007f5de1aa29c6 in dbus_message_append_args_valist () from /lib64/libdbus-1.so.3
#7  0x00007f5de1aa2c1d in dbus_message_append_args () from /lib64/libdbus-1.so.3
#8  0x0000561af879063f in dbus_select_common_methods.part ()
#9  0x0000561af8791e53 in msg_server_impl ()
#10 0x00007f5de1a964a9 in dbus_connection_dispatch () from /lib64/libdbus-1.so.3
#11 0x0000561af8796b44 in dispatch_timeout_callback.lto_priv ()
#12 0x00007f5de1b54d28 in avahi_simple_poll_dispatch () from /lib64/libavahi-common.so.3
#13 0x0000561af8788dff in main ()

in less than 5 seconds even without ASan or Valgrind I think it should be possible to utilize it in its current form.

@evverx
Copy link
Member Author

evverx commented Apr 27, 2022

I reported that bug in avahi/avahi#375

@evverx
Copy link
Member Author

evverx commented May 1, 2022

@mrc0mmand FWIW dfuzzer managed to mess up dbus-broker somehow:

dbus-broker-launch[2660]: dbus-broker: ../src/util/user.c:195: user_free: Assertion `c_assert_result && "c_rbtree_is_empty(&user->usage_tree)"' failed
Sat 2022-04-30 19:45:04 UTC 81338   81   81 SIGABRT none     /usr/bin/dbus-broker                   n/a
Sat 2022-04-30 19:49:33 UTC 81669   81   81 SIGABRT none     /usr/bin/dbus-broker                   n/a
Sat 2022-04-30 19:57:28 UTC 81995   81   81 SIGABRT none     /usr/bin/dbus-broker                   n/a

It wasn't triggered by any dbus method in particular. It just crashed when it was stopped/reloaded by systemd. The backtraces aren't available unfortunately.

@evverx
Copy link
Member Author

evverx commented May 5, 2022

@mrc0mmand FYI Looks like dfuzzer was integrated into os-autoinst-distri-opensuse in os-autoinst/os-autoinst-distri-opensuse#14669 a month ago

@evverx
Copy link
Member Author

evverx commented May 11, 2022

FWIW dfuzzer managed to mess up dbus-broker somehow

Looks like it was reported yesterday in bus1/dbus-broker#288 and fixed in bus1/dbus-broker@608b259

evverx referenced this issue in evverx/systemd May 11, 2022
There are memory leaks there bus1/dbus-broker#289
and it crashes from time to time
https://github.com/matusmarhefka/dfuzzer/issues/20#issuecomment-1114097840
so let's just skip it by analogy with dbus-daemon to avoid
reports that have nothing to do with systemd itself.

It's kind of a part of systemd#22547
yuwata referenced this issue in systemd/systemd May 12, 2022
There are memory leaks there bus1/dbus-broker#289
and it crashes from time to time
https://github.com/matusmarhefka/dfuzzer/issues/20#issuecomment-1114097840
so let's just skip it by analogy with dbus-daemon to avoid
reports that have nothing to do with systemd itself.

It's kind of a part of #22547
keszybz referenced this issue in systemd/systemd-stable May 25, 2022
There are memory leaks there bus1/dbus-broker#289
and it crashes from time to time
https://github.com/matusmarhefka/dfuzzer/issues/20#issuecomment-1114097840
so let's just skip it by analogy with dbus-daemon to avoid
reports that have nothing to do with systemd itself.

It's kind of a part of systemd/systemd#22547

(cherry picked from commit d0880fa)
mrc0mmand referenced this issue in mrc0mmand/rhel-9 Jun 13, 2022
There are memory leaks there bus1/dbus-broker#289
and it crashes from time to time
https://github.com/matusmarhefka/dfuzzer/issues/20#issuecomment-1114097840
so let's just skip it by analogy with dbus-daemon to avoid
reports that have nothing to do with systemd itself.

It's kind of a part of systemd/systemd#22547

(cherry picked from commit d0880fa)
Related: #2087652
mrc0mmand referenced this issue in mrc0mmand/rhel-9 Jun 14, 2022
There are memory leaks there bus1/dbus-broker#289
and it crashes from time to time
https://github.com/matusmarhefka/dfuzzer/issues/20#issuecomment-1114097840
so let's just skip it by analogy with dbus-daemon to avoid
reports that have nothing to do with systemd itself.

It's kind of a part of systemd/systemd#22547

(cherry picked from commit d0880fa)
Related: #2087652
systemd-rhel-bot referenced this issue in redhat-plumbers/systemd-rhel9 Jun 14, 2022
There are memory leaks there bus1/dbus-broker#289
and it crashes from time to time
https://github.com/matusmarhefka/dfuzzer/issues/20#issuecomment-1114097840
so let's just skip it by analogy with dbus-daemon to avoid
reports that have nothing to do with systemd itself.

It's kind of a part of systemd/systemd#22547

(cherry picked from commit d0880fa)
Related: #2087652
@evverx
Copy link
Member Author

evverx commented Apr 25, 2023

@mrc0mmand I'm not sure if you're planning to extend dfuzzer but it would probably be interesting to try to figure out how to get it to discover stuff like avahi/avahi#455, avahi/avahi#454, avahi/avahi#453 and avahi/avahi#451.

avahi/avahi#452 is a bit different in the sense that that particular method is somewhat hidden and can't be discovered by dfuzzer. (BTW systemd hides its methods too sometimes).

@mrc0mmand
Copy link
Member

@mrc0mmand I'm not sure if you're planning to extend dfuzzer but it would probably be interesting to try to figure out how to get it to discover stuff like lathiat/avahi#455, lathiat/avahi#454, lathiat/avahi#453 and lathiat/avahi#451.

lathiat/avahi#452 is a bit different in the sense that that particular method is somewhat hidden and can't be discovered by dfuzzer. (BTW systemd hides its methods too sometimes).

Hah, that's indeed interesting, I'll definitely look into that, thanks for the links.

@evverx
Copy link
Member Author

evverx commented Apr 25, 2023

FWIW Sending big-endian messages on little-endian machines (and vice versa) is interesting too: https://gitlab.freedesktop.org/dbus/dbus/-/issues/417 :-) That would be involved though.

@mrc0mmand
Copy link
Member

mrc0mmand commented Apr 25, 2023

Oh well, this reminds me of a pitfall I ran into when playing around with a potential "vfuzzer" (systemd/systemd#23785 (comment)) which is the (sad) reality that the quality of the introspection data depends on the bus' willingness to put the necessary stuff together.

In Avahi's case, however, this is a bit more complicated, since the objects seem to be tied to a specific peer, or at least that's what I got from a dbus capture.

For example, creating a new EntryGroup

# busctl call org.freedesktop.Avahi / org.freedesktop.Avahi.Server EntryGroupNew
o "/Client15/EntryGroup1"

emits

‣ Type=method_call  Endian=l  Flags=4  Version=1 Cookie=2  Timestamp="Tue 2023-04-25 21:15:06.302187 UTC"
  Sender=:1.2197  Destination=org.freedesktop.Avahi  Path=/  Interface=org.freedesktop.Avahi.Server  Member=EntryGroupNew
  UniqueName=:1.2197
  MESSAGE "" {
  };

‣ Type=method_return  Endian=l  Flags=1  Version=1 Cookie=2837  ReplyCookie=2  Timestamp="Tue 2023-04-25 21:15:06.302227 UTC"
  Sender=:1.1067  Destination=:1.2197
  UniqueName=:1.1067
  MESSAGE "o" {
          OBJECT_PATH "/Client15/EntryGroup1";
  };

where :1.2197 is "us". Since calling busctl again creates a new session, trying to call "IsEmpty()" on the newly created group object fails:

# busctl call org.freedesktop.Avahi /Client14/EntryGroup1 org.freedesktop.Avahi.EntryGroup IsEmpty
Call failed: Method "IsEmpty" with signature "" on interface "org.freedesktop.Avahi.EntryGroup" doesn't exist

since we have a different ID:

‣ Type=method_call  Endian=l  Flags=4  Version=1 Cookie=2  Timestamp="Tue 2023-04-25 21:17:04.432179 UTC"
  Sender=:1.2198  Destination=org.freedesktop.Avahi  Path=/Client14/EntryGroup1  Interface=org.freedesktop.Avahi.EntryGroup  Member=GetState
  UniqueName=:1.2198
  MESSAGE "" {
  };

‣ Type=error  Endian=l  Flags=1  Version=1 Cookie=2838  ReplyCookie=2  Timestamp="Tue 2023-04-25 21:17:04.432217 UTC"
  Sender=:1.1067  Destination=:1.2198
  ErrorName=org.freedesktop.DBus.Error.UnknownObject  ErrorMessage="Method "GetState" with signature "" on interface "org.freedesktop.Avahi.EntryGroup" doesn't exist
"
  UniqueName=:1.1067
  MESSAGE "s" {
          STRING "Method "GetState" with signature "" on interface "org.freedesktop.Avahi.EntryGroup" doesn't exist
";
  };

i.e. now "we" are :1.2198.

However, if I create a simple python script, that uses one bus connection for both calls, I get the expected result:

#!/bin/python3

from pydbus import SystemBus

bus = SystemBus()
avahi = bus.get("org.freedesktop.Avahi", "/")
obj_name = avahi.EntryGroupNew()
print(obj_name)
obj = bus.get("org.freedesktop.Avahi", obj_name)
print(obj.IsEmpty())
# ./repro.py 
/Client16/EntryGroup1
True

So, uh, yeah, this makes things slightly more complicated :)

@mrc0mmand
Copy link
Member

mrc0mmand commented Apr 25, 2023

And to expand on the introspection stuff - for example, in systemd, we append a list of child nodes to the introspection data when generating the response [0]. However, Avahi serves a static XML for introspection requests [1][2][...] so even if we somehow managed to deal with the peer-related stuff above, the object still wouldn't be "automagically" discoverable.

[0] https://github.com/systemd/systemd/blob/main/src/libsystemd/sd-bus/bus-introspect.c#L113
[1] https://github.com/lathiat/avahi/blob/d1e71b320d96d0f213ecb0885c8313039a09f693/avahi-daemon/dbus-protocol.c#L1003-L1004
[2] https://github.com/lathiat/avahi/blob/6395c45bcd1210cbbb23fc92294fd61e968daaf2/avahi-daemon/dbus-entry-group.c#L104-L106

@evverx
Copy link
Member Author

evverx commented Apr 25, 2023

systemd, we append a list of child nodes to the introspection data

Unless systemd tries to hide something: systemd/systemd@fb22861 :-)

Seriously though I think the problem is that for those nodes to be generated dfuzzer has to create them first (for example dfuzzer can't easily catch issues like systemd/systemd#24114) so those nodes are effectively hidden. I agree that it's different from avahi though. With avahi it should probably be possible to infer methods by capturing messages its clients normally send but it can't be automated easily.

@mrc0mmand
Copy link
Member

systemd, we append a list of child nodes to the introspection data

Unless systemd tries to hide something: systemd/systemd@fb22861 :-)

Hah, that one slipped past me. And I can't even run dfuzzer on it, since we always do an introspection, which, of course, doesn't work here:

# dfuzzer -n org.freedesktop.systemd1 -o /org/freedesktop/MemoryAllocation1 -i org.freedesktop.MemoryAllocation1 -t GetMallocInfo
[SESSION BUS]
[PROCESS: /usr/lib/systemd/systemd]
[CONNECTED TO PID: 698]
Object: /org/freedesktop/MemoryAllocation1
 Interface: org.freedesktop.MemoryAllocation1
Error while calling method 'org.freedesktop.DBus.Introspectable.Introspect': GDBus.Error:org.freedesktop.DBus.Error.UnknownObject: Unknown object '/org/freedesktop/MemoryAllocation1'.
[SYSTEM BUS]
[PROCESS: /usr/lib/systemd/systemd]
[CONNECTED TO PID: 1]
Object: /org/freedesktop/MemoryAllocation1
 Interface: org.freedesktop.MemoryAllocation1
Error while calling method 'org.freedesktop.DBus.Introspectable.Introspect': GDBus.Error:org.freedesktop.DBus.Error.UnknownObject: Unknown object '/org/freedesktop/MemoryAllocation1'.
Exit status: 1

Maybe we should just run in a "one-shot" mode if we get a full bus quadruplet (i.e. bus, object, interface, method).

Seriously though I think the problem is that for those nodes to be generated dfuzzer has to create them first (for example dfuzzer can't easily catch issues like systemd/systemd#24114) so those nodes are effectively hidden. I agree that it's different from avahi though. With avahi it should probably be possible to infer methods by capturing messages its clients normally send but it can't be automated easily.

Given the systemd case I though about adding something like "user-guided" mode to dfuzzer, where you would use a file containing quadruplets (bus, object, interface, method) to guide dfuzzer which methods to fuzz, to possibly cover these corner cases. But apart from it being way too manual, it still wouldn't work for Avahi, since even though we would use one bus for all calls, the object name is still dynamic, sigh.

@evverx
Copy link
Member Author

evverx commented Apr 26, 2023

Maybe we should just run in a "one-shot" mode if we get a full bus quadruplet (i.e. bus, object, interface, method)

I think it makes sense in the sense that it shouldn't be necessary to call Introspect in this particular case but on the other hand it's probably easier to just use busctl call/gdbus call for that thanks to their bash autocompletion and so on.

Given the systemd case I though about adding something like "user-guided" mode to dfuzzer, where you would use a file containing quadruplets (bus, object, interface, method) to guide dfuzzer which methods to fuzz

I think it should help but it seems that there should be a way to go further than that by making it possible to lead dfuzzer to certain points (by, say, replaying more or less valid messages) and then throwing gibberish. It doesn't work with services like networkd where actual leases should be created outside of its D-Bus API though.

But apart from it being way too manual, it still wouldn't work for Avahi, since even though we would use one bus for all calls, the object name is still dynamic

Agreed.

@evverx
Copy link
Member Author

evverx commented Jun 14, 2023

To judge from https://bugs.launchpad.net/ubuntu/+source/dbus-broker/+bug/2015538/comments/6 the Ubuntu folks took it to the next level. I guess at some point it should be documented that to test stuff on Ubuntu the AppArmor stuff should be relaxed a bit to actually fuzz stuff there (instead of testing whether AppArmor confines things properly).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants