32bit version of winpmem.exe fails on 64bit Win 10

[johnlp]

Built from source on Master, 11/4/20, VS 2019 16.7.1
64bit exe works as expected
32bit exe on 64bit Win 10, version 2004 build 19041.572 (bare hardware) fails with following output:

**D:\Work\DevProjects\PmemMaster\kernel\executable\Debug>winpmem.exe d:\temp\ram32-1.raw
WinPmem64
Extracting driver to C:\Users\John\AppData\Local\Temp\pmeD628.tmp
Driver Unloaded.
Loaded Driver C:\Users\John\AppData\Local\Temp\pmeD628.tmp.
Deleting C:\Users\John\AppData\Local\Temp\pmeD628.tmp
Failed to get memory geometry,: The program issued a command but the command length is incorrect.

The system time is: 16:46:30
Driver Unloaded.**

Also, when run on a Win 7 32 VM (version 6.1 Build 7601: Service Pack 1), it initially fails by incorrectly selecting the PTE method. Out as follows:

**C:\Users\Adam\Downloads\PMem\win32>winpmem.exe C:\Users\Adam\Downloads\PMem\win3
2\ramcapturepmem20200103.raw
WinPmem32
extract_driver

• service_name: pmem
• filename: C:\Users\Adam\AppData\Local\Temp\pme5D76.tmp
• Extracting driver to C:\Users\Adam\AppData\Local\Temp\pme5D76.tmp
  Driver Unloaded.
  Loaded Driver C:\Users\Adam\AppData\Local\Temp\pme5D76.tmp.
  Deleting C:\Users\Adam\AppData\Local\Temp\pme5D76.tmp
  Failed to set acquisition mode 2 : A device attached to the system is not functi
  oning.

PTE Remapping
Driver Unloaded.**

When run with the -1 option, forcing to PhysicalMemory method, ram capture succeeds.

[vivianezw]

You didn't read #12 did you?

[johnlp]

I did read #12. #12 suggests that PTE is not available 32bit, but thought one of the other modes might work, making some capture available from a 32 bit app. Also, that PTE is selected by default on a 32bit Win7 (causing a failure) seems independent of PTE not working from a 32bit app on a 64bit OS.

[scudette]

We probably should be setting the correct default based on the arch here:

WinPmem/kernel/executable/main.cpp

Line 85 in 134441c

unsigned __int32 mode = PMEM_MODE_PTE;

[vivianezw]

Agreed. Add that to the current todo list? (Needs to be done some time before February, preferably with some time for public testing).

[vivianezw]

Wait, that was not even the problem. (but true nevertheless.) There's more to it. He wrote: 64bit exe works as expected. 32bit exe on 64bit Win 10, version 2004 build 19041.572 (bare hardware) fails with following output: [...]

He executed the 32 bit exe on a 64 bit OS. It already was a 64 bit OS and PTE was the right choice, and the 32 bit usermode program did correctly load the Winpmem64 driver, and the 64 bit driver started...

The error was Failed to get memory geometry,: The program issued a command but the command length is incorrect. That is a STATUS LENGTH MISMATCH error, the input buffer was incorrectly formatted in only the 32 bit usermode part executable.
From the first look, maybe some sizeof() or similar screwed it.

[vivianezw]

I wonder if that also happens with my testsigned version, because I tested it on 32 Bit and it worked.

[johnlp]

2 different issues. One is failure of 32bit app on 64bit OS (bad geometry). 2nd is the wrong default selection of PTE method when running 32bit app on 32bit OS. "Also, when run on a Win 7 32 VM (version 6.1 Build 7601: Service Pack 1), it initially fails by incorrectly selecting the PTE method. Output as follows..." See the initial comment.

[vivianezw]

and you compiled them from this latest source?

The first is an error in the usermode component (the second also, but trivial, it's simply not implemented and that should be the message back to the user). For the first one, the driver rejected the input buffer as badly formatted. Maybe a missing precompiler statement or a bad sizeof(int), or something like that, that would only screw it for 32 bit compilation.

If I ever find some time, the first one would be on my priority list. ;)

[scudette]

This is obvious - running any 32 bit binaries on 64 bit windows is not supported and never will be - the 32 bit binaries have different struct layouts for communicating with the driver and therefore can not parse the ioctl packet.

This is not a supported use case - perhaps we could refuse to run in that case?

[vivianezw]

We could

1. refuse to run and tell user to use the other one.
2. adapt the struct to be formatted correctly in the 32 bit executable. (e.g., by Precompiler statement)
3. Spawn the 64 bit version if the user started the 32 bit version.

[scudette]

1 is the easiest thing to do. 2 wont work because the binary is compiled for 32 bit so any ifdef will select the 32 bit structs - it is just run on 64 bit at runtime. So we could support this scenario but it would mean having 2 separate structs built in and choosing the right one at runtime.

3 might work but seems a bit like a hack - we would need to include the 64 bit binary in the 32 bit binary as a resource.

[johnlp]

If running 32bit exe on 64bit OS is not supported, fair enough. Maybe the How to Use section on the github page should be updated. It states "There are two winpmem executables: winpmem_mini_x86.exe and winpmem_mini_x64.exe. Both versions contain both drivers (32 and 64 bit versions)." Since even the 32bit version contains both drivers, it suggests that the above scenario is supported.

[vletoux]

If that can I help, I think I fixed it.

First, I'm allocating a buffer which is large enough to handle both 32 and 64 bits output.

Then, if running 32bits under 64bits, I'm copying partially the structure, else copy it fully

The trick is that the variable field KPCR which is causing the problem, is not used in the program

[vivianezw]

That could be solved easily, it's only the usermode code. I think the problem is the math in the amd64 switch. Shouldn't that be 64 instead of 32 in the math of RtlCopyMemory? We do not even need to throw them out.

I used to think the KCPRs were of some use in the now antiquated volatility 2, but I never knew exactly why we have these KPCRs in the code. :D We might as well throw them out if no follow-up tool needs them anymore.

[vletoux]

yes, removing KPCR should fix definitively the problem, but if I recompiled it, I needed a kernel mode signature, that why I proposed a fix.
The 32 offset, is computed with 64 - 32 (so from the 64 bits version minus the 32 bits version), not from the 32 version.

[vivianezw]

I also remember I encountered a problem with getNativeSystemInfo (https://docs.microsoft.com/en-us/windows/win32/api/sysinfoapi/nf-sysinfoapi-getnativesysteminfo). I think the problem was about returning an unexpected architecture when under wow64 on a 64-Bit system. (I think the problem was that the querysysteminfo on processor info surprisingly returned 32 bit on a 64 bit architecture when running under wow64 emulation. Expected: 64 bit. )

The IsWow64Process function would also solved this question and seems more safe.

It's a 2x2 question matrix: OS (32, 64) and usermode application (32, 64). Asking for wow64 solves the question more efficiently.

BOOL IsWow64Process(
  HANDLE hProcess,
  PBOOL  Wow64Process
);
https://docs.microsoft.com/en-us/windows/win32/api/wow64apiset/nf-wow64apiset-iswow64process 

Edit: we need both, there could be a 32 bit native application running on a 32 bit OS.

[vivianezw]

I'm not entirely sure about the switch cases in the usermode program code. There are 3 possibilities (wow64+x64, native+64, native+32).

[vletoux]

you missed the ifdef _WIN64 at the begining of the 2nd screenshot.

If compiled under win64, just copy the mem.
Then under win32:
if run under x64, do the hack
else, just copy the mem.

Also I reused this function because this was the one invoked in main.cpp

[vivianezw]

Ughh. I get it. But I don't like it, tbh. ;) Here is another solution: the only case we must avoid is running under wow64 emulation. Wow64 creates an ugly situation where the pairing does not fit because of bitness mismatch. Wow64 is the root cause of our problems. What about this: We check for wow64 emulation, and if yes, we call create process for winpmem64.exe. That way, we get a native pairing with fitting bitness. Problem solved.

It feels like quite some limitation to never be able to touch the driver again. ... I woukd like to throw out the KPCRs now. :D

[vletoux]

Sorry, but I don't get it: "Wow64 creates an ugly situation where the pairing does not fit because of bitness mismatch"
The driver interface is the same, whatever you are running 32 or 64 bits process on x64

[vivianezw]

The driver interface is different depending on the bitness. We are transferring physical memory ranges after all!

It's not only about this:

// xxx: Support up to 32/64  processors for KPCR. Depending on OS bitness
  #if defined(_WIN64)
  LARGE_INTEGER KPCR[64];
  #else
  LARGE_INTEGER KPCR[32];
  #endif
  // xxx: For what exactly do we need all those KPCRs, anyway? Sure, they look nice.

The ioctl code for getting the memory range differs, because we ask for Physical addresses, and they are 64 bit wide on 64 bit OS and 32 bit wide on 32 bit OS, no matter we are running under wow64 emulation ... I don't know if this problem has been dealed with, but I didn't. The only clean way seems to me to simply assert we talk in same bitness.

[vivianezw]

Heck, I'm not sure about what I wrote. Sure, physical addresses on 64 bit OS are 64 bit (..oh really..) and the process under wow64 influence might believe it is running 32 bit and behave like that (concerning sizes), but is this a problem or does the LARGE_INTEGER solve this problem already? A process that believes to run under 32 bit but isn't will still get the LARGE_INTEGER (PHYSICAL_ADDRESS), and the only thing that needs to happen is that he is using the full LARGE INTEGER (the full 64 bit size no matter it thinks it is 32 bit) for creating read requests.

Maybe solved already, thinking about it gives me headaches. ... maybe it's really only the KPCRs that need to get solved.

[vletoux]

Maybe I'm missing something (and I'll learn something new), but the IOCTL code are bitness independant

(you are talking about this ?)

)

If the address would have been addressed as a DWORD, I'll sure agree with you.
But in both case, this is LARGE_INTEGER (x64 addresses) even on 32 bits and the program is handling it perfectly.

I've written a couple of programs, drivers running both x64 & x86 and from the source code, I didn't see such compability problem.
The generated file seems also successfully loaded in memprocfs, but of course, it may exist a hidden error.

[vivianezw]

Hmmm... yes. Another case of thinking to complicated. I thought there was something though in the driver interface at some ioctl a while ago, but then again, everything seems to be fine right now. Remains the KPCR issue.

I still don't like the solution too much, but it's a solution. Still I would rather vote to disallow wow64 emulation (and have 32 KPCRs on native 32 bit, and up to 64 KPCRs on native 64 bit, that was the original idea though we do not use them at all anymore), but Cohen can decide about how to deal with this, even throwing them out and get the driver signed again.
Throwing them out would be the best thing, if we can.

[scudette]

I dont think it is a huge problem - if it works let's rebuild the userspace. I dont see it as a problem to transfer different structs for 32 and 64 bits. Wow64 is a terrible hack and in Velociraptor we straight out refuse to run as 32bit process on 64 bits. It can lead to weird crashes around unaligned access that are hard to always debug. Although this program is probably simple enough to have good coverage so it might be fine here.

@vletoux , can you please send a PR?

[vletoux]

See PR #36 for a fix to enable 32 bits version of winpmem communicate with the 64 bits version of the driver.
The default methods still is PMEM_MODE_PTE for both 32 & 64 bits, I didn't change that.