Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory limiting does not work on macOS #379

Open
s3rvac opened this issue Aug 23, 2018 · 1 comment
Open

Memory limiting does not work on macOS #379

s3rvac opened this issue Aug 23, 2018 · 1 comment

Comments

@s3rvac
Copy link
Member

s3rvac commented Aug 23, 2018

The limiting of memory via --max-memory does not work properly on macOS. For example, if you run

$ retdec-fileinfo --max-memory=4096 FILE

the analysis (properly) fails on Linux and Windows, but finishes successfully on macOS. The reason is that on macOS, the following piece of code from src/utils/memory.cpp compiles, runs, returns 0 (success), but does actually not do anything:

struct rlimit rl = {
    .rlim_cur = limit,        // Soft limit.
    .rlim_max = RLIM_INFINITY // Hard limit (ceiling for rlim_cur).
};
auto rc = setrlimit(RLIMIT_AS, &rl);

From what I was able to find, setrlimit() with RLIMIT_AS is broken on macOS and simply does not work.

What is the proper way of programmatically limiting virtual memory on macOS? Feel free to submit a PR that fixes the implementation of limitSystemMemoryOnMacOS() in src/utils/memory.cpp.

s3rvac added a commit to avast/retdec-regression-tests that referenced this issue Aug 23, 2018
Memory limiting does not work correctly on macOS (see
avast/retdec#379).
catap added a commit to catap/retdec that referenced this issue Apr 22, 2022
We can't limit memory on macOS. Before macOS 12
limitSystemMemoryOnPOSIX() does not actually do anything on macOS.
Anyway, it just succeed, since macOS 12 it returns error and retdec
can't start.

To be honest Apple can control memmory limit via so-called the ledger()
system call which is private. An old verison which was opened to
OpenSource (from 10.9-10.10?) using setrlimit() but at some point
setrlimit() was broken and not ledger(). Probably at macOS 12 the
setrlimit() was completley broken.

Because we haven't got any other choose just return true which haven't
change anything.

See: avast#379
Fixes: avast#1045
catap added a commit to catap/retdec that referenced this issue Apr 22, 2022
We can't limit memory on macOS. Before macOS 12
limitSystemMemoryOnPOSIX() does not actually do anything on macOS.
Anyway, it just succeed, since macOS 12 it returns error and retdec
can't start.

To be honest Apple can control memmory limit via so-called the ledger()
system call which is private. An old verison which was opened to
OpenSource (from 10.9-10.10?) using setrlimit() but at some point
setrlimit() was broken and not ledger(). Probably at macOS 12 the
setrlimit() was completley broken.

Because we haven't got any other choose just return true which haven't
change anything.

See: avast#379
Fixes: avast#1045
@s3rvac
Copy link
Member Author

s3rvac commented Apr 28, 2022

In #1074, we have disabled the limiting of the maximum memory on macOS until there is a working way of doing so.

PeterMatula pushed a commit that referenced this issue Dec 5, 2022
We can't limit memory on macOS. Before macOS 12
limitSystemMemoryOnPOSIX() does not actually do anything on macOS.
Anyway, it just succeed, since macOS 12 it returns error and retdec
can't start.

To be honest Apple can control memmory limit via so-called the ledger()
system call which is private. An old verison which was opened to
OpenSource (from 10.9-10.10?) using setrlimit() but at some point
setrlimit() was broken and not ledger(). Probably at macOS 12 the
setrlimit() was completley broken.

Because we haven't got any other choose just return true which haven't
change anything.

See: #379
Fixes: #1045
PeterMatula added a commit that referenced this issue Dec 5, 2022
* Update Capstone to v4.0

* [Capstone-next] Update to capstone-next branch

* [Capstone-next] Update to Capstone-Next Branch
-[ARM]
    -Add ARM_INS_MOVS support
-[ARM64]
    -Remove vess.
        -It overlaps with ARM64_VAS
    -Fix A64SysReg_* into ARM64_SYSREG_*
-[PowerPC]
    -Fix PPC_REG_X2 into PPC_REG_XER
-[X86]
    -Remove X86_INS_FADDP
        -In capstone-next, faddp is actually fadd, both belong to
            "ID 15(fadd)"

* [tests][capstone2llvmir][arm] Fix MOVW Unit Test
- In test, "movw r0, #0xabcd" do not read any register
    and the result is 0xabcd not 0x1234abcd

* [tests][capstone2llvmir][arm] Fix Nop test
- In arm, the NOP instruction is HINT instruction
- Also, in capstone, the cs_insn->id of nop is point to
    HINT(ID: 63)
- So, an error will be occurred when looking for a translate
    instruction method because it is points to nullptr

* [Capstone2llvmir][arm64] Add ADDCS Support

* [capstone2llvmir][arm64] Add ADDS Support

* [capstone2llvmir][arm64] Add ANDS Support

* [capstone2llvmir][arm64] Add SUP Support

* [capstone2llvmir][arm64] Add BICS Support

* [capstonellvmir][PowerPC] Update Register Name

* [capstone2llvmir][PowerPC] Update Register Name

* [capstone2llvmir][PowerPC] Fix CMP Support

* [capstone2llvmir][PowerPC] Add CMPL Support

* [capstone2llvmir][PowerPC] Fix CMPL

* [capstone2llvmir][PowerPC] Add BLT Support

* [capstone2llvmir][PowerPC] Add  Branch mnemonics incorporating
conditions Suppport

* [capstone2llvmir][PowerPC] Fix RLWINM
- RLWINM and clrlwi are same ID

* [tests][capstone2llvmir][PowerPC] Fix Crand Tests

* [capstone2llvmir][PowerPC] Fix bdzla BUG

* [capstone2llvmir][PowerPC] Remove BDZLA TODO

* [capstone2llvmir][x86] Fix ud2b

* [capstone2llvmir][X86] Fix FADD/FADDP

* [capstone2llvmir][x86] Fix FADD/FADDP

* [capstone2llvmir][x86] Fix FXCH
- when transalte "FXCH instruction, in the value of loadOpFloatingBinaryTop Function,
    "top" is equal to idx, which causes the value to be written to top
    twice when exchanging data.

* clean code

* Update Capstone to v5.0

* [capstone2llvmir][x86][PowerPC] Clean code

* [capstone2llvmir][PowerPC] Clean code

* [capstone2llvmir][PowerPC] Remove BUN* and BNU*
-In CapstoneV5, they are both equivalent to BSO* and BNS*

* [capstone2llvmir][PowerPC] Fix rlwinm
- In capstone V5, rlwinm is equivalent to to clrlwi

* [capstone2llvmir][PowerPC] Fix BNL*

* [capstone2llvmir][PowerPC] Add PPC_REG_ZERO

* [capstone2llvmir][PowerPC] Add comment

* Fix merge conflict

* Update YARA to 4.2.X

* Add dll_name from export directory to output

* llvm/CMakeLists: Manually-specified variables were not used by the project.

The following variables were set in CMakeLists, however, they
were not used by the LLVM project build:

LLVM_USE_CRT_DEBUG
LLVM_USE_CRT_RELEASE

* CHANGELOG.md: add entries for #1060 #1061 PRs

* Fixed loading import directory that is modified by relocations

* Fixed comment

* Remove useless trailing whitespace

There is absolutely no reason for it being in the code.

* pelib: Fix a typo in a comment in PeLib::ImageLoader::Load()

* Add a CHANGELOG entry for #1063

* Move signing certificate to separate object

* Updated authenticode parser to the newest version

* Fix uninitialize free, use finer sanity checks in auth. parser

* Add a directory for RetDec-related publications

The list of publications has been originally placed on
https://retdec.com/publications/ (https://retdec.com/ has been redirected
to https://github.com/avast/retdec, and we wanted to keep the list somewhere).

* Fix the wording for an invalid max-memory error in scripts/retdec-unpacker.py

There are the following two reasons for the fix:
- The check only verifies whether the passed value is an integer.
- The parameter can be 0 (i.e. a non-negative integer). It does not have to a
  positive integer.

* Never try to limit memory on macOS

We can't limit memory on macOS. Before macOS 12
limitSystemMemoryOnPOSIX() does not actually do anything on macOS.
Anyway, it just succeed, since macOS 12 it returns error and retdec
can't start.

To be honest Apple can control memmory limit via so-called the ledger()
system call which is private. An old verison which was opened to
OpenSource (from 10.9-10.10?) using setrlimit() but at some point
setrlimit() was broken and not ledger(). Probably at macOS 12 the
setrlimit() was completley broken.

Because we haven't got any other choose just return true which haven't
change anything.

See: #379
Fixes: #1045

* Remove a redundant period from CHANGELOG

* utils: Improve the wording of a comment in getTotalSystemMemoryOnMacOS()

* Add a CHANGELOG entry for #1074 and #1045

* Update authenticode-parser, use-after-free, signedness issues

* Using multistage build for Dockerfile, reduces container size by ~1.5G

* Check for possible overflow when checking for segment overlaps. Fix incorrect range exception message

* Fix parameter and return types for dynamically called functions

Calls to dynamically-linked functions go through the procedure linkage
table (PLT).  RetDec turns a PLT entry into a function, say
malloc@plt, that appears to do nothing but call the external function,
say malloc (though the assembly code will do a jump rather than a
call). User code that logically wants to call malloc instead calls
malloc@plt (and sets up arguments as if calling malloc). The
malloc@plt code first jumps to the dynamic linker which modifies it so
that subsequent calls to malloc@plt will jump directly to malloc. We
say that malloc@plt wraps malloc.  The call to malloc in malloc@plt
will not have any arguments setup, so malloc will appear to have
no parameters or returns (unless that information is provided by
link-time-information, debug information, or name demangling), but it
needs to have the same parameter types and return type as
malloc@plt. The propagateWrapped methods copy the argument information
from the DataFlowEntry of the wrapping function to the wrapped
function. Then, when the calls to the wrapping function are inlined
(in connectWrappers), effectively the call to the wrapping function is
changed into a call to the wrapped function.

The motivation for this change is the programs that analyze the
output of RetDec (either the C code, or the LLVM code) want to
recognize library functions and treat them specially. This
change makes it so that the library function names are used
directly (rather than the plt version) and they are passed
their parameters correctly.

* Upgrade to Capstone release 4.0.2

* Add additional patch on capstone 4.0.2 for PPC Signed 16 bit immediates

Capstone version 4.0.2 has a bug when disassembling a powerpc instruction
with a signed 16-bit immediate.
See capstone-engine/capstone#1746 and
capstone-engine/capstone#1746 (comment).

This change adds to the capstone patch to fix this problem.

* Treat endbr32/endbr64 instructions as NOPs

* capstone2llvmir/powerpc: remove PPC_INS_BDZLA hack fix

As Capstone was updated, the fix in capstone-engine/capstone#968 took effect and the original RetDec fix is not needed - in fact, it caused problems.

* Handle Procedure Linkage calls for 32bit x86 from gcc

This case is for x86 32 bit compiled with GCC. Its PLT entries are in
sections .plt.sec or .plt.got. An entry is of the form:

jmp *offset(%ebx)

When this code is encountered register %ebx has been loaded with the
address of the start of the Global Offset Table (.got) section.
This change handles that case.

* Add ability to process PNG icons for perceptual hash calculation (#1090)

* Add ability to process PNG icons for perceptual hash calculation

* Use SCOPE_EXIT for deallocation

* In generated C, add prototypes for dynamically-linked functions without headers

When the program involves dynamically-linked functions like _Znwj
(operator new) that return a pointer, it is necessary to have
prototypes for them, since otherwise they will be implicitly deduced
to return "int" which cannnot be dereferenced.

Previously RetDec was emitting comments telling which functions were
dynamically linked. This change moves them up before the functions are
emitted and instead emits prototypes for the functions. However,
RetDec also inserts includes of headers for functions for with know
headers. We do not emit prototypes for functions with headers as that
would be redundant.  As a result, some dynamically-linked functions
that used to show in the comments no longer appear as the included
header will declare them.

The section header comment for dynamically-linked functions is only
produced if some prototypes are written for dynamically-linked
functions.

A related PR will have added tests as well as changes needed for
existing tests.

* Add printing of analysis time to retdec-fileinfo output

* Yara: inherits linker flags

* Use provided libtool via `CMAKE_LIBTOOL`

* Added missed `${RETDEC_INSTALL_BIN_DIR}` to `pat2yara`

* Added sanity check for page index when loading pages from broken samples

There are certain samples where page index might go beyond available
pages when trying to load them which will be prevented with this patch.

* Virtual Size overflow is now handler properly

* Fixed error code

* Updated yaramod

* Fix removeZeroSequences

* README.md: add "limited maintenance mode" note

Co-authored-by: Peter Kubov <[email protected]>
Co-authored-by: houndthe <[email protected]>
Co-authored-by: Peter Matula <[email protected]>
Co-authored-by: Ladislav Zezula <[email protected]>
Co-authored-by: Petr Zemek <[email protected]>
Co-authored-by: Marek Milkovič <[email protected]>
Co-authored-by: Kirill A. Korinsky <[email protected]>
Co-authored-by: me <me>
Co-authored-by: Richard L Ford <[email protected]>
Co-authored-by: 未赢 <[email protected]>
PeterMatula added a commit that referenced this issue Dec 5, 2022
* Update Capstone to v4.0

* [Capstone-next] Update to capstone-next branch

* [Capstone-next] Update to Capstone-Next Branch
-[ARM]
    -Add ARM_INS_MOVS support
-[ARM64]
    -Remove vess.
        -It overlaps with ARM64_VAS
    -Fix A64SysReg_* into ARM64_SYSREG_*
-[PowerPC]
    -Fix PPC_REG_X2 into PPC_REG_XER
-[X86]
    -Remove X86_INS_FADDP
        -In capstone-next, faddp is actually fadd, both belong to
            "ID 15(fadd)"

* [tests][capstone2llvmir][arm] Fix MOVW Unit Test
- In test, "movw r0, #0xabcd" do not read any register
    and the result is 0xabcd not 0x1234abcd

* [tests][capstone2llvmir][arm] Fix Nop test
- In arm, the NOP instruction is HINT instruction
- Also, in capstone, the cs_insn->id of nop is point to
    HINT(ID: 63)
- So, an error will be occurred when looking for a translate
    instruction method because it is points to nullptr

* [Capstone2llvmir][arm64] Add ADDCS Support

* [capstone2llvmir][arm64] Add ADDS Support

* [capstone2llvmir][arm64] Add ANDS Support

* [capstone2llvmir][arm64] Add SUP Support

* [capstone2llvmir][arm64] Add BICS Support

* [capstonellvmir][PowerPC] Update Register Name

* [capstone2llvmir][PowerPC] Update Register Name

* [capstone2llvmir][PowerPC] Fix CMP Support

* [capstone2llvmir][PowerPC] Add CMPL Support

* [capstone2llvmir][PowerPC] Fix CMPL

* [capstone2llvmir][PowerPC] Add BLT Support

* [capstone2llvmir][PowerPC] Add  Branch mnemonics incorporating
conditions Suppport

* [capstone2llvmir][PowerPC] Fix RLWINM
- RLWINM and clrlwi are same ID

* [tests][capstone2llvmir][PowerPC] Fix Crand Tests

* [capstone2llvmir][PowerPC] Fix bdzla BUG

* [capstone2llvmir][PowerPC] Remove BDZLA TODO

* [capstone2llvmir][x86] Fix ud2b

* [capstone2llvmir][X86] Fix FADD/FADDP

* [capstone2llvmir][x86] Fix FADD/FADDP

* [capstone2llvmir][x86] Fix FXCH
- when transalte "FXCH instruction, in the value of loadOpFloatingBinaryTop Function,
    "top" is equal to idx, which causes the value to be written to top
    twice when exchanging data.

* clean code

* Update Capstone to v5.0

* [capstone2llvmir][x86][PowerPC] Clean code

* [capstone2llvmir][PowerPC] Clean code

* [capstone2llvmir][PowerPC] Remove BUN* and BNU*
-In CapstoneV5, they are both equivalent to BSO* and BNS*

* [capstone2llvmir][PowerPC] Fix rlwinm
- In capstone V5, rlwinm is equivalent to to clrlwi

* [capstone2llvmir][PowerPC] Fix BNL*

* [capstone2llvmir][PowerPC] Add PPC_REG_ZERO

* [capstone2llvmir][PowerPC] Add comment

* Fix merge conflict

* Update YARA to 4.2.X

* Add dll_name from export directory to output

* llvm/CMakeLists: Manually-specified variables were not used by the project.

The following variables were set in CMakeLists, however, they
were not used by the LLVM project build:

LLVM_USE_CRT_DEBUG
LLVM_USE_CRT_RELEASE

* CHANGELOG.md: add entries for #1060 #1061 PRs

* Fixed loading import directory that is modified by relocations

* Fixed comment

* Remove useless trailing whitespace

There is absolutely no reason for it being in the code.

* pelib: Fix a typo in a comment in PeLib::ImageLoader::Load()

* Add a CHANGELOG entry for #1063

* Move signing certificate to separate object

* Updated authenticode parser to the newest version

* Fix uninitialize free, use finer sanity checks in auth. parser

* Add a directory for RetDec-related publications

The list of publications has been originally placed on
https://retdec.com/publications/ (https://retdec.com/ has been redirected
to https://github.com/avast/retdec, and we wanted to keep the list somewhere).

* Fix the wording for an invalid max-memory error in scripts/retdec-unpacker.py

There are the following two reasons for the fix:
- The check only verifies whether the passed value is an integer.
- The parameter can be 0 (i.e. a non-negative integer). It does not have to a
  positive integer.

* Never try to limit memory on macOS

We can't limit memory on macOS. Before macOS 12
limitSystemMemoryOnPOSIX() does not actually do anything on macOS.
Anyway, it just succeed, since macOS 12 it returns error and retdec
can't start.

To be honest Apple can control memmory limit via so-called the ledger()
system call which is private. An old verison which was opened to
OpenSource (from 10.9-10.10?) using setrlimit() but at some point
setrlimit() was broken and not ledger(). Probably at macOS 12 the
setrlimit() was completley broken.

Because we haven't got any other choose just return true which haven't
change anything.

See: #379
Fixes: #1045

* Remove a redundant period from CHANGELOG

* utils: Improve the wording of a comment in getTotalSystemMemoryOnMacOS()

* Add a CHANGELOG entry for #1074 and #1045

* Update authenticode-parser, use-after-free, signedness issues

* Using multistage build for Dockerfile, reduces container size by ~1.5G

* Check for possible overflow when checking for segment overlaps. Fix incorrect range exception message

* Fix parameter and return types for dynamically called functions

Calls to dynamically-linked functions go through the procedure linkage
table (PLT).  RetDec turns a PLT entry into a function, say
malloc@plt, that appears to do nothing but call the external function,
say malloc (though the assembly code will do a jump rather than a
call). User code that logically wants to call malloc instead calls
malloc@plt (and sets up arguments as if calling malloc). The
malloc@plt code first jumps to the dynamic linker which modifies it so
that subsequent calls to malloc@plt will jump directly to malloc. We
say that malloc@plt wraps malloc.  The call to malloc in malloc@plt
will not have any arguments setup, so malloc will appear to have
no parameters or returns (unless that information is provided by
link-time-information, debug information, or name demangling), but it
needs to have the same parameter types and return type as
malloc@plt. The propagateWrapped methods copy the argument information
from the DataFlowEntry of the wrapping function to the wrapped
function. Then, when the calls to the wrapping function are inlined
(in connectWrappers), effectively the call to the wrapping function is
changed into a call to the wrapped function.

The motivation for this change is the programs that analyze the
output of RetDec (either the C code, or the LLVM code) want to
recognize library functions and treat them specially. This
change makes it so that the library function names are used
directly (rather than the plt version) and they are passed
their parameters correctly.

* Upgrade to Capstone release 4.0.2

* Add additional patch on capstone 4.0.2 for PPC Signed 16 bit immediates

Capstone version 4.0.2 has a bug when disassembling a powerpc instruction
with a signed 16-bit immediate.
See capstone-engine/capstone#1746 and
capstone-engine/capstone#1746 (comment).

This change adds to the capstone patch to fix this problem.

* Treat endbr32/endbr64 instructions as NOPs

* capstone2llvmir/powerpc: remove PPC_INS_BDZLA hack fix

As Capstone was updated, the fix in capstone-engine/capstone#968 took effect and the original RetDec fix is not needed - in fact, it caused problems.

* Handle Procedure Linkage calls for 32bit x86 from gcc

This case is for x86 32 bit compiled with GCC. Its PLT entries are in
sections .plt.sec or .plt.got. An entry is of the form:

jmp *offset(%ebx)

When this code is encountered register %ebx has been loaded with the
address of the start of the Global Offset Table (.got) section.
This change handles that case.

* Add ability to process PNG icons for perceptual hash calculation (#1090)

* Add ability to process PNG icons for perceptual hash calculation

* Use SCOPE_EXIT for deallocation

* In generated C, add prototypes for dynamically-linked functions without headers

When the program involves dynamically-linked functions like _Znwj
(operator new) that return a pointer, it is necessary to have
prototypes for them, since otherwise they will be implicitly deduced
to return "int" which cannnot be dereferenced.

Previously RetDec was emitting comments telling which functions were
dynamically linked. This change moves them up before the functions are
emitted and instead emits prototypes for the functions. However,
RetDec also inserts includes of headers for functions for with know
headers. We do not emit prototypes for functions with headers as that
would be redundant.  As a result, some dynamically-linked functions
that used to show in the comments no longer appear as the included
header will declare them.

The section header comment for dynamically-linked functions is only
produced if some prototypes are written for dynamically-linked
functions.

A related PR will have added tests as well as changes needed for
existing tests.

* Add printing of analysis time to retdec-fileinfo output

* Yara: inherits linker flags

* Use provided libtool via `CMAKE_LIBTOOL`

* Added missed `${RETDEC_INSTALL_BIN_DIR}` to `pat2yara`

* Added sanity check for page index when loading pages from broken samples

There are certain samples where page index might go beyond available
pages when trying to load them which will be prevented with this patch.

* Virtual Size overflow is now handler properly

* Fixed error code

* Updated yaramod

* Fix removeZeroSequences

* README.md: add "limited maintenance mode" note

Co-authored-by: Peter Kubov <[email protected]>
Co-authored-by: houndthe <[email protected]>
Co-authored-by: Peter Matula <[email protected]>
Co-authored-by: Ladislav Zezula <[email protected]>
Co-authored-by: Petr Zemek <[email protected]>
Co-authored-by: Marek Milkovič <[email protected]>
Co-authored-by: Kirill A. Korinsky <[email protected]>
Co-authored-by: me <me>
Co-authored-by: Richard L Ford <[email protected]>
Co-authored-by: 未赢 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant