Errors on Linux

[0xdea]

Hi,

A rhabdomancer user has reported the following problem on their Arch Linux x86_64 system:

export IDASDKDIR=/opt/ida-pro-9.0/idasdk90
cargo install rhabdomancer
rhabdomancer --help
rhabdomancer: error while loading shared libraries: [libida.so](http://libida.so/): cannot open shared object file: No such file or directory
...
fd [libida.so](http://libida.so/) /opt/ida-pro-9.0
/opt/ida-pro-9.0/idasdk90/lib/x64_linux_gcc_64/libida.so
/opt/ida-pro-9.0/libida.so

I could quickly reproduce it on a Kali Linux x86_64 VM. This was caused by the fact that the /opt/ida-pro-9.0/idasdk90/lib/x64_linux_gcc_64/ path wasn't present in ld.so.conf. It's easily fixed by adding such entry and running ldconfig -v again. Perhaps we should add this gotcha to the documentation? Or should we force rustc to build a static binary? What do you think?

After I worked around this problem, I unfortunately spotted a more serious issue with the latest idalib:

kali@kali:~/idalib$ export IDASDKDIR=/home/kali/Desktop/idasdk90/ 
kali@kali:~/idalib$ cargo run --example=comments_ls              
   Compiling autocxx-build v0.27.0
   Compiling idalib-build v0.3.0+9.0.240925 (/home/kali/idalib/idalib-build)
   Compiling autocxx v0.27.0
   Compiling idalib-sys v0.3.0+9.0.240925 (/home/kali/idalib/idalib-sys)
   Compiling idalib v0.3.0+9.0.240925 (/home/kali/idalib/idalib)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 48.95s
     Running `target/debug/examples/comments_ls`
Trying to open IDA database...
zsh: segmentation fault (core dumped)  cargo run --example=comments_ls
...
kali@kali:~/idalib$ gdb ./target/debug/examples/comments_ls core.1630393 
GNU gdb (Debian 15.2-1) 15.2
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./target/debug/examples/comments_ls...
[New LWP 1630393]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `target/debug/examples/comments_ls'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  idalib_check_license () at src/kernwin_extras.h:90
90        auto res = manager->_vtbl->check(manager, 0, 0);
warning: Missing auto-load script at offset 0 in section .debug_gdb_scripts
of file /home/kali/idalib/target/debug/examples/comments_ls.
Use `info auto-load python-scripts [REGEXP]' to list them.
(gdb) bt
#0  idalib_check_license () at src/kernwin_extras.h:90
#1  0x0000559f3027eb94 in cxxbridge1$idalib_check_license () at /home/kali/idalib/target/debug/build/idalib-sys-c1c136a50975a2a4/out/autocxx-build-dir/cxx/gen2.cxx:867
#2  0x0000559f3027c6d7 in idalib_sys::ffix::idalib_check_license () at idalib-sys/src/lib.rs:738
#3  0x0000559f3027d6d0 in idalib_sys::ida::is_license_valid () at idalib-sys/src/lib.rs:1192
#4  0x0000559f30277489 in idalib_sys::ida::open_database_quiet<&std::path::Path> (path=..., auto_analysis=true) at idalib-sys/src/lib.rs:1293
#5  0x0000559f30276ca1 in idalib::idb::IDB::open_with<&str> (path=..., auto_analyse=true, save=false) at idalib/src/idb.rs:57
#6  0x0000559f30276842 in idalib::idb::IDB::open<&str> (path=...) at idalib/src/idb.rs:46
#7  0x0000559f3027291e in comments_ls::main () at idalib/examples/comments_ls.rs:7
(gdb) 

Not sure if this is caused by the floating license or if this happens also with other (more common) configurations. Also not sure if this is specific to the latest idalib version or if it was present also in (some) earlier releases. The segmentation fault happens regardless of the presence of a valid license or not.

I don't have a Linux x86_64 VM handy for testing at the moment, but I can provide you with the coredump file for debugging and I'm available in case you need additional details.

[xorpse]

Addressed via e1689b9. Below is an explanation of what's going on in case others run into this issue and/or are interested:

1. We use a build.rs with idalib and rhabdomancer that has a fallback behaviour when it cannot find an IDA installation either at the location specified by IDADIR or at the "default" location (for Linux this is $HOME/ida-pro-9.0):

   idalib/idalib-build/src/lib.rs

   Line 9 in a5a7b9c

   return PathBuf::from(env::var("HOME").unwrap()).join("ida-pro-9.0");

   
   Since the Kali install has IDA installed at /opt/ida-pro-9.0 and no IDADIR set, we will hit the fallback behaviour, this results in not setting the rpath to the .so files in the IDA installation directory as we would if we found the directory successfully:

   idalib/idalib-build/src/lib.rs

   Lines 132 to 144 in a5a7b9c

   	#[cfg(target_os = "linux")]
   	{
   	println!(
   	"cargo::rustc-link-arg=-Wl,-rpath,{},-L{},-l:libida.so",
   	install_path.display(),
   	stub_path.display(),
   	);
   	println!(
   	"cargo::rustc-link-arg=-Wl,-rpath,{},-L{},-l:libidalib.so",
   	install_path.display(),
   	stub_path.display(),
   	);
   	}

   
   Given this situation we will hit the first error (no shared libraries found).

2. The workaround to add the shared libraries bundled with the SDK indeed helps resolve the problem of missing shared libraries, but these libraries are just stub libraries, e.g., for get_license_manager we have:

.text:0000000000016B5A get_license_manager proc near           ; DATA XREF: LOAD:0000000000004C60↑o
.text:0000000000016B5A ; __unwind {
.text:0000000000016B5A                 push    rbp
.text:0000000000016B5B                 mov     rbp, rsp
.text:0000000000016B5E                 nop
.text:0000000000016B5F                 pop     rbp
.text:0000000000016B60                 retn
.text:0000000000016B60 ; } // starts at 16B5A
.text:0000000000016B60 get_license_manager endp

In idalib_check_license, we expect manager to take the return of get_license_manager (the global license manager instance), and with this stub function we have to hope rax is set to 0 otherwise the check below will pass and we will get a segfault on line 90:

idalib/idalib-sys/src/kernwin_extras.h

Lines 86 to 90 in a5a7b9c

	if (!manager) {
	return false;
	}
	auto res = manager->_vtbl->check(manager, 0, 0);

To work around this, we should instead expose /opt/ida-pro-9.0 to the dynamic linker, e.g., via LD_LIBRARY_PATH or /etc/ld.so.conf{,d}, or set IDADIR to /opt/ida-pro-9.0 at build time.

[0xdea]

Awesome, thank you! Just one note about the cargo::warning addition to the build.rs script. As explained in the build scripts reference, warnings printed out in crates.io crates are not emitted:

cargo::warning=MESSAGE
The warning instruction tells Cargo to display a warning after the build script has finished running. Warnings are only shown for path dependencies (that is, those you’re working on locally), so for example warnings printed out in crates.io crates are not emitted by default. The -vv “very verbose” flag may be used to have Cargo display warnings for all crates.

This is of no concern for idalib itself, but might be important for binary crates that include it as a dependency, such as rhabdomancer, when they are installed via cargo install. I was unable to find a workaround other than using cargo -vv, which is annoying.

[xorpse]

Thank you :)--I've updated the README to highlight this caveat--it's indeed a bit annoying to have this behaviour hidden behind "very verbose".