Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The request could not be performed because of an I/O device error (4.0rc1) #9

Open
igorrogov opened this issue Oct 12, 2020 · 23 comments

Comments

@igorrogov
Copy link

Hello,

I've just tried the new 4.0rc1 release on my laptop, but it produced an error while trying to capture RAM.

00% 0x00100000 ....Failed to Read memory.
: The request could not be performed because of an I/O device error. 

I've tried all three methods, but all of them either produced the above error or sent my computer to BSOD.

This is Windows 10 Pro x64 with 2004 update (exact version is 10.0.19041.508). Full output is attached.

method_2_output.txt

@igorrogov igorrogov changed the title The request could not be performed because of an I/O device error (4.0cr1) The request could not be performed because of an I/O device error (4.0rc1) Oct 12, 2020
@scudette
Copy link
Contributor

Thanks for reporting - can you please capture kernel debug log using https://docs.microsoft.com/en-us/sysinternals/downloads/debugview - select capture kernel messages and try the acquisition again?

@ricoulele
Copy link

Hi
i have the same message than igorrogov (Windows 10 Pro x64 with 2004 update)

@vivianezw
Copy link
Collaborator

vivianezw commented Oct 12, 2020

This is a successfully catched but real error. Though it never occurred on my system and I have 2004.
All three cases? (Impossible, I had said but ...)

I need the precise ntstatus error to know what went wrong.
Since this is a 'real' error, it should have a Dbgprint statement associated. (Error Dbgprints should never be disabled.)

1.) Can you use Dbgview[1], to directly see the message from the driver? You must do it elevated, with all verbose kernel capure settings enabled. (in menu), and put me the debug print output here.

Current diagnosis:
The driver loaded, but it never read anything. Something bad happened and it was handled for the non-BSOD case, that means it was a known error case.

@ edit:

Did you do this on a barebone machine?
Was this on a VM? If yes, were "guest services" drivers of any sorts installed on the VM (Virtualbox, etc)?

In case of a BSOD, the exception codes would also help a lot.

[1] https://docs.microsoft.com/en-us/sysinternals/downloads/debugview

@ricoulele
Copy link

sorry but now i have a blue screen of death....

@ricoulele
Copy link

and it is not on a VM machine

@vivianezw
Copy link
Collaborator

vivianezw commented Oct 12, 2020

There is something really strange going on. My 2004 was a VM, not a barebone.

Did you try with or without memory compression?

Does anyone of you have a 1) Dbgview (does not help against BSOD), or better, 2), WinDbg[2]?

With Windbg, you load the crashdump (I think in c:\windows\memory.dmp per default) that was made when the OS crashed and launch a !analyze -v. That will give generally be enough output to nail down the error.

In case it gets really weird, I have to organize a win10 2004 barebone and must hope to reproduce it.

@ edit: already trying that.

[2] The most easiest way to get WinDbg is via the app store of Microsoft: type "Windbg preview" (Microsoft Corporation) to find it. That should do the job. Does not need a Microsoft online account to download it. Alternatively, it's in any SDK, and it's also in the WDK10 or WDK7600. Windbg can load the post-mortem crashdump that should be "c:\windows\memory.dmp" to tell what went wrong (using !analyze -v).

@vivianezw
Copy link
Collaborator

vivianezw commented Oct 12, 2020

I cannot reproduce it on a Win10 2004 x64. It simply worked. But then again it that was a self-compiled driver.
The iospace method currently fails with systems that have a Hyper-V layer underneath them.
But at least PTE and the physical memory worked always and I just tested it.

@edit: collegue reported to run normally on a Win10 2004 x64, using both PTE and phsyical memory methods. Everything runs completely normal. (He self-compiled & testsigned from this git-cloned source.)

@vivianezw
Copy link
Collaborator

reproduced it. Happens only one real barebone machines, which makes it difficult to study. That needs to wait until tomorrow.

@igorrogov
Copy link
Author

Hi guys,

Yes, this is a real machine, not a VM. But Hyper-V service is enabled. I use VMware and Windows Sandbox sometimes. I'm trying the 2nd method (PTE remapping). Please see the logs from DebugView attached.

This is the command that I'm running:
winpmem_mini_x64 -2 laptop.mem > output_method_2.txt

It works fine on the second machine which I haven't updated to 2004 yet. I'm going to update it soon and check if it works there.

output_method_2.txt
method_2_debug_view.LOG

@scudette
Copy link
Contributor

Ok I think the error is related to VSM mode. When running in this mode, winpmem driver literally can not read the protected memory range and needs to break up the buffer and read around the holes.

We did have code to do this in here
https://github.com/Velocidex/c-aff4/blob/657dc28b805a55e44696125e6b3627e64dfc653a/tools/pmem/win_pmem.cc#L438

but this has not been replicated in the very simple mini imager.

Maybe I will prepare another release of the old aff4 imager to get around this issue. It would also be nice to be able to write an elf image so we can skip all the padding regions - a machine i tested with had a couple gb of padding.

@igorrogov
Copy link
Author

Update: I have upgraded my older desktop PC to Win10 2004 and 4.0rc1 works fine there, no issues. 3.3rc2 doesn't work on this PC because of this issue - Velocidex/c-aff4#144

So, the only issue that remains is being able to capture RAM on a PC that has both Win10 2004 and VSM mode.

Maybe I will prepare another release of the old aff4 imager to get around this issue

Thanks, that would be great!

@vivianezw
Copy link
Collaborator

It was also fine on my gaming PC, which has no virtualization at all. :D

Approach: I think we might be able to catch the error and leave out the VSM-failing page, and return a sufficient amount of zero bytes instead.

I also need a better securing. There is VSM interference to be taken into account.

@vivianezw
Copy link
Collaborator

Here is a physical read one, before I fixed it:

FAULTING_SOURCE_LINE:  c:\src\driver\WinPmem\kernel\read.c

FAULTING_SOURCE_FILE:  c:\src\driver\WinPmem\kernel\read.c

FAULTING_SOURCE_LINE_NUMBER:  74

FAULTING_SOURCE_CODE:  
    72: 
    73:     if (NT_SUCCESS(NtStatus)) {
>   74:       RtlCopyMemory(buf, mapped_buffer + page_offset, to_read);
    75:       ZwUnmapViewOfSection((HANDLE)-1, mapped_buffer);
    76: 
    77:     } else {
    78:       DbgPrint("Failed to Map page at 0x%llX\n", offset.QuadPart);
    79:       RtlZeroMemory(buf, to_read);

I think I fixed all three now. I am testing myself and then will put it on my repo & PR...
You will have to use again my testsigned ones. ;)

@scudette
Copy link
Contributor

I added a slow read path when buffers can not be read in #14 . Can you please test the rc2 binary with VSM enabled? You can find the binary in the releases.

It will now fall back to a slow read where it gets read errors from the driver. This will display an x in the progress instead of . (each character corresponds to 16mb buffers).

@vivianezw
Copy link
Collaborator

vivianezw commented Oct 13, 2020 via email

@scudette
Copy link
Contributor

Which patch?

@vivianezw
Copy link
Collaborator

vivianezw commented Oct 13, 2020 via email

@vivianezw
Copy link
Collaborator

vivianezw commented Oct 13, 2020 via email

@igorrogov
Copy link
Author

Yes, I can confirm that 4.0rc2 works fine on my laptop now (method 2). I can see those "x" symbols in the log. I assume this is where it was failing in rc1.

image

Thanks a lot!

@thomasp1392
Copy link

We tested 4.0rc2 on a physical device with Win10 Build 2004. Unfortunately, we still get an BSOD (Memory management) when we try to access memory. If it is helpful, we have a mini-dump and full-dump of the crash available.

@vivianezw
Copy link
Collaborator

vivianezw commented Oct 22, 2020

Yes, that's exactly why I pushed my update.

Currently please use my testsigned drivers in the pull request!

For convenience, here: https://github.com/Velocidex/WinPmem/files/5386510/winpmem_testsigned_15_okt_2020_2.zip
pw: betatest.

I fixed that already! (But only in the testsigned drivers.)

@scudette
Copy link
Contributor

Does this crash occur with the -2 flag? Using PTE mode?

When you say "trying to access memory" - are you trying to take an image or are you trying to use the driver in your own code?

If so - the only stable access method is the PTE mode which you will need to switch to using the required ioctrl. This is the only mode that captures the hyper V page faults correctly. Vivian added those same checks for the other modes in a later PR but that is not present in the signed driver.

@vivianezw
Copy link
Collaborator

vivianezw commented Oct 22, 2020

I'm pretty sure he used the old signed drivers.

My patched current build is pretty invisible right now. (Hanging in a closed pull request.)

The physmem method ist also safe but only in my testsigned build. The iospace method is safe if no kernel debugger is attached (only in my latest testsigned build).

@scudette: pleeease put my testsigned nice driver to a more visible place. :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants