Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Unknown Issue] DirectX11 Resize Buffer Hook; Crashing on OBS but functioning properly. #4

Open
dxgi opened this issue Jul 13, 2021 · 2 comments

Comments

@dxgi
Copy link

dxgi commented Jul 13, 2021

I've been trying to figure out what's wrong with my resize buffer hook, It's functioning properly, as usual, However, there's an issue with it that I don't know which causes it. If I were to hook the resize buffer and then use obs game capture It would crash after resizing.

HRESULT(__stdcall* original_resize_buffers)(IDXGISwapChain*, UINT, UINT, UINT, DXGI_FORMAT, UINT);

HRESULT new_resize_buffer(IDXGISwapChain* swap_chain, UINT buffer_count, UINT width, UINT heigth, DXGI_FORMAT new_format, UINT flags) {
    if (initialized) {
        SetWindowLongPtr(original_window, GWLP_WNDPROC, (LONG_PTR)original_window_proc);

        if (main_render_taget_view) {
            main_render_taget_view->Release();
            main_render_taget_view = nullptr;
        }

        initialized = false;
    }

    return original_resize_buffers(swap_chain, buffer_count, width, height, new_format, flags);
}
kiero::bind(13, (void**)&original_resize_buffers, new_resize_buffer);

I've also tried referencing to Microsoft's documentation about DXGI Click for documentation!

@Revester
Copy link

Revester commented Oct 14, 2021

Because your function is wrong. Kiero correctly hooks the function. In the documentation link that you provided, it tells you that before actually calling ResizeBuffers, you first need to release all outstanding references to the swap chain's buffers.

And then in the code it shows that you first need to set render targets to null, release all outstanding references to the swap chain's buffers, only then call ResizeBuffers, and then create render target view again and set render targets to your render target view:

case WM_SIZE:
        if (g_pSwapChain)
        {
            g_pd3dDeviceContext->OMSetRenderTargets(0, 0, 0);

            // Release all outstanding references to the swap chain's buffers.
            g_pRenderTargetView->Release();

            HRESULT hr;
            // Preserve the existing buffer count and format.
            // Automatically choose the width and height to match the client rect for HWNDs.
            hr = g_pSwapChain->ResizeBuffers(0, 0, 0, DXGI_FORMAT_UNKNOWN, 0);
                                            
            // Perform error handling here!

            // Get buffer and create a render-target-view.
            ID3D11Texture2D* pBuffer;
            hr = g_pSwapChain->GetBuffer(0, __uuidof( ID3D11Texture2D),
                                         (void**) &pBuffer );
            // Perform error handling here!

            hr = g_pd3dDevice->CreateRenderTargetView(pBuffer, NULL,
                                                     &g_pRenderTargetView);
            // Perform error handling here!
            pBuffer->Release();

            g_pd3dDeviceContext->OMSetRenderTargets(1, &g_pRenderTargetView, NULL );

            // Set up the viewport.
            D3D11_VIEWPORT vp;
            vp.Width = width;
            vp.Height = height;
            vp.MinDepth = 0.0f;
            vp.MaxDepth = 1.0f;
            vp.TopLeftX = 0;
            vp.TopLeftY = 0;
            g_pd3dDeviceContext->RSSetViewports( 1, &vp );
        }
        return 1;

So that's what you gotta do, because during your initialization and rendering hook you get a device, immediate context, create your own render target view and set render targets yourself:

HRESULT __stdcall hkPresent(IDXGISwapChain* pSwapChain, UINT SyncInterval, UINT Flags)
{
	if (!init)
	{
		if (SUCCEEDED(pSwapChain->GetDevice(__uuidof(ID3D11Device), (void**)& pDevice)))
		{
			pDevice->GetImmediateContext(&pContext);
			DXGI_SWAP_CHAIN_DESC sd;
			pSwapChain->GetDesc(&sd);
			window = sd.OutputWindow;
			ID3D11Texture2D* pBackBuffer;
			pSwapChain->GetBuffer(0, __uuidof(ID3D11Texture2D), (LPVOID*)& pBackBuffer);
			pDevice->CreateRenderTargetView(pBackBuffer, NULL, &mainRenderTargetView);
			pBackBuffer->Release();
			oWndProc = (WNDPROC)SetWindowLongPtr(window, GWLP_WNDPROC, (LONG_PTR)WndProc);
			InitImGui();
			init = true;
		}

		else
			return oPresent(pSwapChain, SyncInterval, Flags);
	}

	ImGui_ImplDX11_NewFrame();
	ImGui_ImplWin32_NewFrame();
	ImGui::NewFrame();

	ImGui::Begin("ImGui Window");
	ImGui::End();

	ImGui::Render();

	pContext->OMSetRenderTargets(1, &mainRenderTargetView, NULL);
	ImGui_ImplDX11_RenderDrawData(ImGui::GetDrawData());
	return oPresent(pSwapChain, SyncInterval, Flags);
}

You need to do the same(as shown in Microsoft's documentation) for your objects in your ResizeBuffers hook respectively(or in your WindowProc function responding to the WM_SIZE event, as Microsoft does it in their code). First, you need to set your render targets view to null and release all outstanding references to the swap chain's buffers, then call original resize buffers, then create render target view again, and set the render render targets to your render target view, again:

HRESULT hkResizeBuffers(IDXGISwapChain* pSwapChain, UINT BufferCount, UINT Width, UINT Height, DXGI_FORMAT NewFormat, UINT SwapChainFlags) 
{
	if (mainRenderTargetView) {
		pContext->OMSetRenderTargets(0, 0, 0);
		mainRenderTargetView->Release();
	}

	HRESULT hr = oResizeBuffers(pSwapChain, BufferCount, Width, Height, NewFormat, SwapChainFlags);

	ID3D11Texture2D* pBuffer;
	pSwapChain->GetBuffer(0, __uuidof(ID3D11Texture2D), (void**)&pBuffer);
	// Perform error handling here!

	pDevice->CreateRenderTargetView(pBuffer, NULL, &mainRenderTargetView);
	// Perform error handling here!
	pBuffer->Release();

	pContext->OMSetRenderTargets(1, &mainRenderTargetView, NULL);

	// Set up the viewport.
	D3D11_VIEWPORT vp;
	vp.Width = Width;
	vp.Height = Height;
	vp.MinDepth = 0.0f;
	vp.MaxDepth = 1.0f;
	vp.TopLeftX = 0;
	vp.TopLeftY = 0;
	pContext->RSSetViewports(1, &vp);
	return hr;
}

It works, and works with all the macaronian hooks out there: Steam Overlay hooking game's functions like Present and ResizeBuffers, MSI Afterburner hooking Steam Overlay's functions, your dll hooking MSI Afterburner's hooks, then something like OBS hooking your dll's hooking. Or in any other order, like if you add your DLL to the import table of the exe, which will then hook before all the other dlls.

@yousafkhan143
Copy link

hi everyone i am also facing this resize buffer when i full my game the cheaat menu everything disappear can anyone help me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants