yara-4.3.0-rc1 test-pe and test-dotnet fail for the s390x platform #1855

xambroz · 2023-01-04T09:48:35Z

Describe the bug
Hello,
I just wanted to let you know that the test-pe and test-dotnet fail for the s390x platform (Big-Endian).
https://kojipkgs.fedoraproject.org/…log
There is probably something in the code, which is big-endian/little-endian specific and fails on the other architecture.

Same build for the x64 and others went fine for this 4.3.0 rc1 .
https://koji.fedoraproject.org/…987

Best regards
Michal Ambroz

To Reproduce
Steps to reproduce the behavior:

On the s390x platform run the checks

make check

Expected behavior
All code should deal with binaries in architecture independent way, and all test should pass.

Screenshots

Please complete the following information:

OS: Fedora rawhide (probably affects also RHEL on s390x)
YARA version: 4.3.0 rc1

plusvic · 2023-01-10T08:53:28Z

@xambroz Could you paste the logs produced by the failing tests? The logs should be in the test-suite.log file.

xambroz · 2023-01-18T10:44:59Z

Hello Victor @plusvic
here it is with more details from the mentioned logs
https://kojipkgs.fedoraproject.org//work/tasks/8324/96288324/build.log

+ cat ./test-suite.log
==================================
   yara 4.3.0: ./test-suite.log
==================================
# TOTAL: 20
# PASS:  18
# SKIP:  0
# XFAIL: 0
# FAIL:  2
# XPASS: 0
# ERROR: 0
.. contents:: :depth: 2
FAIL: test-pe
=============
tests/test-pe.c:688: rule does not match contents of'tests/data/079a472d22290a94ebb212aa8015cdc8dd28a968c6b4d3b88acdd58ce2d3b885' (but should)
FAIL test-pe (exit status: 1)
FAIL: test-dotnet
=================
tests/test-dotnet.c:87: rule does not match contents of'tests/data/756684f4017ba7e931a26724ae61606b16b5f8cc84ed38a260a34e50c5016f59' (but should)
FAIL test-dotnet (exit status: 1)
+ echo '===== test-pe.log'
===== test-pe.log
+ '[' -f ./test-pe.log ']'
+ cat ./test-pe.log
tests/test-pe.c:688: rule does not match contents of'tests/data/079a472d22290a94ebb212aa8015cdc8dd28a968c6b4d3b88acdd58ce2d3b885' (but should)
FAIL test-pe (exit status: 1)
+ echo '===== test-dotnet.log'
===== test-dotnet.log
+ '[' -f ./test-dotnet.log ']'
+ cat ./test-dotnet.log
tests/test-dotnet.c:87: rule does not match contents of'tests/data/756684f4017ba7e931a26724ae61606b16b5f8cc84ed38a260a34e50c5016f59' (but should)
FAIL test-dotnet (exit status: 1)

PR #1768 added RVA field to function details in PE module. The new code had the following line: ``` rva_address = yr_le64toh(import_descriptor->FirstThunk + (sizeof(uint64_t) * func_idx)); ``` The `yr_le64toh` should be used for converting the value of `import_descriptor->FirstThunk` from little-endian to the host's endianness *before* performing the add operation. However, the addition was performed before the conversion. This may be the cause of some test cases failing in big endian platforms.

plusvic · 2023-01-20T09:35:44Z

Hi @xambroz, I think I've found the cause of at least one of the issues. Could you try the branch https://github.com/VirusTotal/yara/tree/fix_1855 and let me know if fails in the s390x platform? I think it may fix at lest the test-pe.c test case.

Fixing test-dotnet.c seems a lot harder, quite a few changes have been introduced that don't seem to take endianess into account.

xambroz · 2023-01-20T17:59:01Z

Hi @plusvic ,
unfortunately it did not fix the test-pe on BigEndian platforms.
On the other hand it doesn't seem to break it on Little-Endian platforms.

I have used https://github.com/VirusTotal/yara/commit/90c43e24f0dedd130bea199e6c23094271c3f491.patch as a patch to 4.3.0 rc1.

Full build log:
https://kojipkgs.fedoraproject.org//work/tasks/1695/96431695/build.log

+ cat ./test-suite.log
==================================
   yara 4.3.0: ./test-suite.log
==================================
# TOTAL: 20
# PASS:  18
# SKIP:  0
# XFAIL: 0
# FAIL:  2
# XPASS: 0
# ERROR: 0
.. contents:: :depth: 2
FAIL: test-pe
=============
tests/test-pe.c:688: rule does not match contents of'tests/data/079a472d22290a94ebb212aa8015cdc8dd28a968c6b4d3b88acdd58ce2d3b885' (but should)
FAIL test-pe (exit status: 1)
FAIL: test-dotnet
=================
tests/test-dotnet.c:87: rule does not match contents of'tests/data/756684f4017ba7e931a26724ae61606b16b5f8cc84ed38a260a34e50c5016f59' (but should)
FAIL test-dotnet (exit status: 1)
===== /proc/cpu
+ echo '===== /proc/cpu'
+ head -n 35 /proc/cpuinfo
vendor_id       : IBM/S390
# processors    : 2
bogomips per cpu: 3241.00
max thread id   : 0
features	: esan3 zarch stfle msa ldisp eimm dfp edat etf3eh highgprs te vx vxd vxe gs vxe2 vxp sort dflt sie 
facilities      : 0 1 2 3 4 6 7 8 9 10 12 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 30 31 32 33 34 35 36 37 38 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 57 58 59 60 61 69 70 71 72 73 74 75 76 77 80 81 82 128 129 130 131 133 134 135 146 147 148 150 151 152 155 156 168
cache0          : level=1 type=Data scope=Private size=128K line_size=256 associativity=8
cache1          : level=1 type=Instruction scope=Private size=128K line_size=256 associativity=8
cache2          : level=2 type=Data scope=Private size=4096K line_size=256 associativity=8
cache3          : level=2 type=Instruction scope=Private size=4096K line_size=256 associativity=8
cache4          : level=3 type=Unified scope=Shared size=262144K line_size=256 associativity=32
cache5          : level=4 type=Unified scope=Shared size=983040K line_size=256 associativity=60
processor 0: version = FF,  identification = 354D88,  machine = 8561
processor 1: version = FF,  identification = 354D88,  machine = 8561
cpu number      : 0
physical id     : 0
core id         : 0
book id         : 0
drawer id       : 0
dedicated       : 0
address         : 0
siblings        : 1
cpu cores       : 1
version         : FF
identification  : 354D88
machine         : 8561
cpu MHz dynamic : 5200
cpu MHz static  : 5200
cpu number      : 1
physical id     : 1
core id         : 1
book id         : 1
drawer id       : 1

Seems OK for the other (all little-endian) platforms:
https://koji.fedoraproject.org/koji/taskinfo?taskID=96431531

plusvic · 2023-01-20T18:45:22Z

I've updated https://github.com/VirusTotal/yara/tree/fix_1855 with a new change. It turns out that the issue I fixed before was replicated at some other part of the code. My first attempt fixed it for 64-bits PE files only, but now I've fixed it for 32-bits PE as well. Can you give it a second try?

xambroz · 2023-01-21T01:07:57Z

Cool ... test-pe failed, but there is progress, it is failing in another test.

make  check-TESTS
make[2]: Entering directory '/builddir/build/BUILD/yara-4.3.0-rc1'
make[3]: Entering directory '/builddir/build/BUILD/yara-4.3.0-rc1'
PASS: test-arena
PASS: test-alignment
PASS: test-atoms
PASS: test-api
PASS: test-rules
FAIL: test-pe
PASS: test-elf
PASS: test-version
PASS: test-bitmask
PASS: test-math
PASS: test-stack
PASS: test-re-split
PASS: test-async
PASS: test-string
PASS: test-exception
PASS: test-macho
PASS: test-dex
FAIL: test-dotnet
PASS: test-magic
PASS: test-pb
============================================================================
Testsuite summary for yara 4.3.0
============================================================================
# TOTAL: 20
# PASS:  18
# SKIP:  0
# XFAIL: 0
# FAIL:  2
# XPASS: 0
# ERROR: 0
============================================================================
See ./test-suite.log
Please report to [email protected]
============================================================================
make[3]: *** [Makefile:2507: test-suite.log] Error 1
make[3]: Leaving directory '/builddir/build/BUILD/yara-4.3.0-rc1'
make[2]: Leaving directory '/builddir/build/BUILD/yara-4.3.0-rc1'
make[2]: *** [Makefile:2615: check-TESTS] Error 2
make[1]: *** [Makefile:2960: check-am] Error 2
make[1]: Leaving directory '/builddir/build/BUILD/yara-4.3.0-rc1'
make: *** [Makefile:2962: check] Error 2
+ echo '===== ./test-suite.log'
===== ./test-suite.log
+ '[' -f ./test-suite.log ']'
+ cat ./test-suite.log
==================================
   yara 4.3.0: ./test-suite.log
==================================
# TOTAL: 20
# PASS:  18
# SKIP:  0
# XFAIL: 0
# FAIL:  2
# XPASS: 0
# ERROR: 0
.. contents:: :depth: 2
FAIL: test-pe
=============
tests/test-pe.c:739: rule does not match contents of'tests/data/mtxex_modified_rsrc_rva.dll' (but should)
FAIL test-pe (exit status: 1)
FAIL: test-dotnet
=================
tests/test-dotnet.c:87: rule does not match contents of'tests/data/756684f4017ba7e931a26724ae61606b16b5f8cc84ed38a260a34e50c5016f59' (but should)
FAIL test-dotnet (exit status: 1)

Full log
https://kojipkgs.fedoraproject.org//work/tasks/8751/96458751/build.log

Other (little-endian) platforms unaffected.
https://koji.fedoraproject.org/koji/taskinfo?taskID=96458665

plusvic · 2023-01-27T09:27:59Z

Not all issues has been fixed with 2b631d0. The remaining issues are going to be harder to debug. My guess is that they are related to the difference in endianness.

@xambroz could you disable the test in test-pe.c line 739 and run the test cases again. I'm interested in knowing all the tests in that must be disabled in s390x for the test suite to pass.

See #1855.

plusvic · 2023-02-09T09:12:51Z

After 32ae80d, 64a201f and ff20b39, most of the issues should be already solved. There's a pending is issue (#1874), though.

xambroz · 2023-02-09T12:50:43Z

Sorry for no reply ... I will test

xambroz · 2023-02-10T00:39:15Z

@xambroz could you disable the test in test-pe.c line 739 and run the test cases again. I'm interested in knowing all the tests in that must be disabled in s390x for the test suite to pass.

Yep .. fedora granted me interactive access to the s390x machine I can walk through manually.

xambroz · 2023-02-10T02:33:57Z

So I took last commit 32ae80d as a base
Tests report these errors:
tests/test-elf.c:263: rule does not match contents of'tests/data/elf_with_imports' (but should)

everything else except this test will pass.

plusvic · 2023-02-22T12:42:15Z

With the latest changes in bdd3980 this issue should be completely fixed.

xambroz · 2023-02-24T10:38:36Z

tested, works OK. Thank you.
https://copr.fedorainfracloud.org/coprs/rebus/infosec/build/5562286/

xambroz added the bug label Jan 4, 2023

xambroz mentioned this issue Jan 19, 2023

yara-4.3.0-rc1 test-pe fails on RHEL9 having OpenSSL with disabled SHA1 cert validation #1864

Closed

plusvic mentioned this issue Jan 27, 2023

Fix #1855 #1870

Merged

plusvic closed this as completed in 2b631d0 Jan 27, 2023

plusvic reopened this Jan 27, 2023

plusvic added a commit that referenced this issue Feb 8, 2023

Fix endianness issues in dotnet module.

ff20b39

See #1855.

plusvic added a commit that referenced this issue Feb 8, 2023

Fix endianness issues in PE module.

64a201f

See #1855.

plusvic added a commit that referenced this issue Feb 8, 2023

Fix endianness issues in ELF module.

32ae80d

See #1855.

plusvic closed this as completed Feb 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

yara-4.3.0-rc1 test-pe and test-dotnet fail for the s390x platform #1855

yara-4.3.0-rc1 test-pe and test-dotnet fail for the s390x platform #1855

xambroz commented Jan 4, 2023 •

edited

Loading

plusvic commented Jan 10, 2023

xambroz commented Jan 18, 2023 •

edited

Loading

plusvic commented Jan 20, 2023

xambroz commented Jan 20, 2023

plusvic commented Jan 20, 2023

xambroz commented Jan 21, 2023

plusvic commented Jan 27, 2023

plusvic commented Feb 9, 2023

xambroz commented Feb 9, 2023

xambroz commented Feb 10, 2023

xambroz commented Feb 10, 2023

plusvic commented Feb 22, 2023

xambroz commented Feb 24, 2023

yara-4.3.0-rc1 test-pe and test-dotnet fail for the s390x platform #1855

yara-4.3.0-rc1 test-pe and test-dotnet fail for the s390x platform #1855

Comments

xambroz commented Jan 4, 2023 • edited Loading

On the s390x platform run the checks

plusvic commented Jan 10, 2023

xambroz commented Jan 18, 2023 • edited Loading

plusvic commented Jan 20, 2023

xambroz commented Jan 20, 2023

plusvic commented Jan 20, 2023

xambroz commented Jan 21, 2023

plusvic commented Jan 27, 2023

plusvic commented Feb 9, 2023

xambroz commented Feb 9, 2023

xambroz commented Feb 10, 2023

xambroz commented Feb 10, 2023

plusvic commented Feb 22, 2023

xambroz commented Feb 24, 2023

xambroz commented Jan 4, 2023 •

edited

Loading

xambroz commented Jan 18, 2023 •

edited

Loading