Add Media Foundation encoder by cgutman · Pull Request #681 · LizardByte/Sunshine

cgutman · 2023-01-03T01:02:16Z

Description

This PR introduces support for Media Foundation encoders (h264_mf and hevc_mf). This enables partial hardware acceleration on Windows systems where NVENC, QSV, and AMD AMF are unsupported, like ARM platforms. I also implemented support for GPU accelerated color conversion when encoding from CPU memory and enabled that feature for Media Foundation.

Screenshot

Issues Fixed or Closed

Type of Change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Dependency update (updates to dependencies)
Documentation update (changes to documentation)
Repository update (changes to repository files, e.g. .github/...)

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have added or updated the in code docstring/documentation-blocks for new or existing methods/components

Branch Updates

LizardByte requires that branches be up-to-date before merging. This means that after any PR is merged, this branch
must be updated before it can be merged. You must also
Allow edits from maintainers.

I want maintainers to keep my branch updated

docs/source/about/advanced_usage.rst

psyke83 · 2023-01-03T16:26:08Z

I tested this on my RX 6600. While I understand that MF is inferior to AMF, I noticed an issue with delayed frames that can't be resolved with the existing ffmpeg encoder parameters.

I've opened a PR that should help: LizardByte/build-deps#72

This will require you to add the low_latency param to this PR for changes to take effect.

brad-richardson · 2023-01-03T17:38:16Z

I'm getting a segfault on line 1030 of video.cpp with an Intel dGPU

Here's some of my log: https://gist.github.com/brad-richardson/4ad89fbd492ad211463f367c462a7875

Any ideas for configs to try or that I'm missing?

cgutman · 2023-01-04T02:25:17Z

@brad-richardson Yeah, I saw the same issue on my ARM machine. There is a bug on FFmpeg where it double-releases an internal Media Foundation pointer if initialization fails to set an output format. For my ARM system, it didn't like the rate control mode I had set and setting it to cbr fixed the crash.

FFmpeg's MF support seems pretty poor. It doesn't have a lot of the config parameters plumbed, crashes when init fails, doesn't support GPU surfaces, etc. I think I'm going to put this on hold and work on QSV instead.

src_assets/common/assets/web/config.html

Isalight · 2023-01-20T07:53:25Z

On AMD Vega 6 don't work H264 (black screen when streaming). H265 works fine.

[2023:01:20:02:38:16]: Info: Compiling shaders...
[2023:01:20:02:38:16]: Info: Compiled shaders
[2023:01:20:02:38:16]: Info: Trying encoder [mf]
[2023:01:20:02:38:16]: Info: Set GPU preference: 1
[2023:01:20:02:38:16]: Info: 
Device Description : AMD Radeon(TM) Graphics
Device Vendor ID   : 0x00001002
Device Device ID   : 0x00001636
Device Video Mem   : 495 MiB
Device Sys Mem     : 0 MiB
Share Sys Mem      : 7867 MiB
Feature Level      : 0x0000B100
Capture size       : 1920x1080
Offset             : 0x0
Virtual Desktop    : 1920x1080
[2023:01:20:02:38:16]: Info: Desktop resolution [1920x1080]
[2023:01:20:02:38:16]: Info: Desktop format [DXGI_FORMAT_B8G8R8A8_UNORM]
[2023:01:20:02:38:16]: Info: Color coding [Rec. 601]
[2023:01:20:02:38:16]: Info: Color range: [JPEG]
[2023:01:20:02:38:16]: Error: Couldn't read packet: Invalid data found when processing input
[2023:01:20:02:38:16]: Info: 
Device Description : AMD Radeon(TM) Graphics
Device Vendor ID   : 0x00001002
Device Device ID   : 0x00001636
Device Video Mem   : 495 MiB
Device Sys Mem     : 0 MiB
Share Sys Mem      : 7867 MiB
Feature Level      : 0x0000B100
Capture size       : 1920x1080
Offset             : 0x0
Virtual Desktop    : 1920x1080
[2023:01:20:02:38:16]: Info: Desktop resolution [1920x1080]
[2023:01:20:02:38:16]: Info: Desktop format [DXGI_FORMAT_B8G8R8A8_UNORM]
[2023:01:20:02:38:16]: Info: Color coding [Rec. 601]
[2023:01:20:02:38:16]: Info: Color range: [JPEG]
[2023:01:20:02:38:16]: Error: Couldn't read packet: Invalid data found when processing input
[2023:01:20:02:38:16]: Info: 
Device Description : AMD Radeon(TM) Graphics
Device Vendor ID   : 0x00001002
Device Device ID   : 0x00001636
Device Video Mem   : 495 MiB
Device Sys Mem     : 0 MiB
Share Sys Mem      : 7867 MiB
Feature Level      : 0x0000B100
Capture size       : 1920x1080
Offset             : 0x0
Virtual Desktop    : 1920x1080
[2023:01:20:02:38:16]: Info: Desktop resolution [1920x1080]
[2023:01:20:02:38:16]: Info: Desktop format [DXGI_FORMAT_B8G8R8A8_UNORM]
[2023:01:20:02:38:16]: Info: Color coding [Rec. 601]
[2023:01:20:02:38:16]: Info: Color range: [JPEG]
[2023:01:20:02:38:16]: Error: Couldn't read packet: Invalid data found when processing input
[2023:01:20:02:38:16]: Error: Couldn't read packet: Invalid data found when processing input
[2023:01:20:02:38:17]: Info: 
Device Description : AMD Radeon(TM) Graphics
Device Vendor ID   : 0x00001002
Device Device ID   : 0x00001636
Device Video Mem   : 495 MiB
Device Sys Mem     : 0 MiB
Share Sys Mem      : 7867 MiB
Feature Level      : 0x0000B100
Capture size       : 1920x1080
Offset             : 0x0
Virtual Desktop    : 1920x1080
[2023:01:20:02:38:17]: Info: Desktop resolution [1920x1080]
[2023:01:20:02:38:17]: Info: Desktop format [DXGI_FORMAT_B8G8R8A8_UNORM]
[2023:01:20:02:38:17]: Info: Color coding [Rec. 601]
[2023:01:20:02:38:17]: Info: Color range: [JPEG]
[2023:01:20:02:38:17]: Info: 
Device Description : AMD Radeon(TM) Graphics
Device Vendor ID   : 0x00001002
Device Device ID   : 0x00001636
Device Video Mem   : 495 MiB
Device Sys Mem     : 0 MiB
Share Sys Mem      : 7867 MiB
Feature Level      : 0x0000B100
Capture size       : 1920x1080
Offset             : 0x0
Virtual Desktop    : 1920x1080
[2023:01:20:02:38:17]: Info: Desktop resolution [1920x1080]
[2023:01:20:02:38:17]: Info: Desktop format [DXGI_FORMAT_B8G8R8A8_UNORM]
[2023:01:20:02:38:17]: Info: Color coding [Rec. 601]
[2023:01:20:02:38:17]: Info: Color range: [JPEG]
[2023:01:20:02:38:17]: Info: 
Device Description : AMD Radeon(TM) Graphics
Device Vendor ID   : 0x00001002
Device Device ID   : 0x00001636
Device Video Mem   : 495 MiB
Device Sys Mem     : 0 MiB
Share Sys Mem      : 7867 MiB
Feature Level      : 0x0000B100
Capture size       : 1920x1080
Offset             : 0x0
Virtual Desktop    : 1920x1080
[2023:01:20:02:38:17]: Info: Desktop resolution [1920x1080]
[2023:01:20:02:38:17]: Info: Desktop format [DXGI_FORMAT_B8G8R8A8_UNORM]
[2023:01:20:02:38:17]: Info: Color coding [Rec. 709]
[2023:01:20:02:38:17]: Info: Color range: [JPEG]
[2023:01:20:02:38:17]: Error: Could not open codec [h264_mf]: Invalid argument
[2023:01:20:02:38:17]: Info: 
Device Description : AMD Radeon(TM) Graphics
Device Vendor ID   : 0x00001002
Device Device ID   : 0x00001636
Device Video Mem   : 495 MiB
Device Sys Mem     : 0 MiB
Share Sys Mem      : 7867 MiB
Feature Level      : 0x0000B100
Capture size       : 1920x1080
Offset             : 0x0
Virtual Desktop    : 1920x1080
[2023:01:20:02:38:17]: Info: Desktop resolution [1920x1080]
[2023:01:20:02:38:17]: Info: Desktop format [DXGI_FORMAT_B8G8R8A8_UNORM]
[2023:01:20:02:38:17]: Info: Color coding [Rec. 709]
[2023:01:20:02:38:17]: Info: Color range: [JPEG]
[2023:01:20:02:38:17]: Error: Could not open codec [hevc_mf]: Invalid argument
[2023:01:20:02:38:17]: Info: 
Device Description : AMD Radeon(TM) Graphics
Device Vendor ID   : 0x00001002
Device Device ID   : 0x00001636
Device Video Mem   : 495 MiB
Device Sys Mem     : 0 MiB
Share Sys Mem      : 7867 MiB
Feature Level      : 0x0000B100
Capture size       : 1920x1080
Offset             : 0x0
Virtual Desktop    : 1920x1080
[2023:01:20:02:38:17]: Info: Desktop resolution [1920x1080]
[2023:01:20:02:38:17]: Info: Desktop format [DXGI_FORMAT_B8G8R8A8_UNORM]
[2023:01:20:02:38:17]: Info: Color coding [Rec. 601]
[2023:01:20:02:38:17]: Info: Color range: [JPEG]
[2023:01:20:02:38:17]: Error: Couldn't read packet: Invalid data found when processing input
[2023:01:20:02:38:17]: Error: Couldn't read packet: Invalid data found when processing input
[2023:01:20:02:38:17]: Info: 
Device Description : AMD Radeon(TM) Graphics
Device Vendor ID   : 0x00001002
Device Device ID   : 0x00001636
Device Video Mem   : 495 MiB
Device Sys Mem     : 0 MiB
Share Sys Mem      : 7867 MiB
Feature Level      : 0x0000B100
Capture size       : 1920x1080
Offset             : 0x0
Virtual Desktop    : 1920x1080
[2023:01:20:02:38:17]: Info: Desktop resolution [1920x1080]
[2023:01:20:02:38:17]: Info: Desktop format [DXGI_FORMAT_B8G8R8A8_UNORM]
[2023:01:20:02:38:17]: Info: Color coding [Rec. 601]
[2023:01:20:02:38:17]: Info: Color range: [JPEG]
[2023:01:20:02:38:18]: Warning: mf: h264 missing sps->vui parameters
[2023:01:20:02:38:18]: Info: // Testing for available encoders, this may generate errors. You can safely ignore those errors. //
[2023:01:20:02:38:18]: Info: 
[2023:01:20:02:38:18]: Info: // Ignore any errors mentioned above, they are not relevant. //
[2023:01:20:02:38:18]: Info: 
[2023:01:20:02:38:18]: Info: Found encoder mf: [h264_mf, hevc_mf]
[2023:01:20:02:38:18]: Info: Configuration UI available at [https://localhost:50990]
[2023:01:20:02:38:18]: Info: Registered Sunshine mDNS service
[2023:01:20:02:38:26]: Info: /CN=NVIDIA GameStream Client -- verified
[2023:01:20:02:38:27]: Info: /CN=NVIDIA GameStream Client -- verified
[2023:01:20:02:38:34]: Info: /CN=NVIDIA GameStream Client -- verified
[2023:01:20:02:38:35]: Info: /CN=NVIDIA GameStream Client -- verified
[2023:01:20:02:38:35]: Info: /CN=NVIDIA GameStream Client -- verified
[2023:01:20:02:38:36]: Info: /CN=NVIDIA GameStream Client -- verified
[2023:01:20:02:38:36]: Info: /CN=NVIDIA GameStream Client -- verified
[2023:01:20:02:38:36]: Info: Executing [Desktop]
[2023:01:20:02:38:36]: Info: CLIENT CONNECTED
[2023:01:20:02:38:37]: Info: 
Device Description : AMD Radeon(TM) Graphics
Device Vendor ID   : 0x00001002
Device Device ID   : 0x00001636
Device Video Mem   : 495 MiB
Device Sys Mem     : 0 MiB
Share Sys Mem      : 7867 MiB
Feature Level      : 0x0000B100
Capture size       : 1920x1080
Offset             : 0x0
Virtual Desktop    : 1920x1080
[2023:01:20:02:38:37]: Info: Desktop resolution [1920x1080]
[2023:01:20:02:38:37]: Info: Desktop format [DXGI_FORMAT_B8G8R8A8_UNORM]
[2023:01:20:02:38:37]: Info: Capture format [DXGI_FORMAT_B8G8R8A8_UNORM]
[2023:01:20:02:38:37]: Info: Color coding [Rec. 709]
[2023:01:20:02:38:37]: Info: Color range: [MPEG]
[2023:01:20:02:38:52]: Info: /CN=NVIDIA GameStream Client -- verified
[2023:01:20:02:38:52]: Info: /CN=NVIDIA GameStream Client -- verified
[2023:01:20:02:38:52]: Info: CLIENT DISCONNECTED

psyke83 · 2023-01-21T07:18:02Z

src/platform/common.h

   */
-  virtual int set_frame(AVFrame *frame, AVBufferRef *hw_frames_ctx) {
+  virtual int set_frame(AVFrame *frame, AVBufferRef *hw_frames_ctx, int target_format) {
    BOOST_LOG(error) << "Illegal call to hwdevice_t::set_frame(). Did you forget to override it?";


This error message is triggered when encoder = software is set (and causes the software encoders to become unavailable).

psyke83 · 2023-01-21T07:31:43Z

src/video.cpp

  }

-  int set_frame(AVFrame *frame, AVBufferRef *hw_frames_ctx) {
+  int set_frame(AVFrame *frame, AVBufferRef *hw_frames_ctx, AVPixelFormat target_format) {


changing AVPixelFormat here to int (or perhaps updating in all other cases to AVPixelFormat) should fix the issue with software rendering.

psyke83 · 2023-01-21T07:38:18Z

Apart from the software rendering regression that needs fixing, this looks good to me... with the caveat that my AMD card seems to violate the target bitrate by at least 3x. I'm not really sure why this happens, and manually testing other encoder parameters doesn't help.

Of course, don't take my observation re: bitrate as a reason to block merging, as MediaFoundation is never going to be preferred over AMF for AMD cards.

ReenigneArcher · 2023-03-10T02:40:39Z

I think I'm going to put this on hold and work on QSV instead.

@cgutman Any thoughts on resuming work on this one (if anything else is needed before merging)?

cgutman · 2023-03-10T03:44:33Z

I'm concerned that GPUs might hit this fallback due to race conditions during driver installation or monitor hotplugging. I want to do some work first to minimize the chance of that happening (and fix the bug in software encoding).

ReenigneArcher · 2023-03-28T01:15:38Z

Updating w/ rebase to make this easier to fix conflicts after adjusting clang-format rules

cgutman · 2023-04-12T03:28:17Z

I think there may be a leak in the Media Foundation encoder in FFmpeg or maybe something in the Intel driver that I'm testing with, but memory usage increases each time the encoder loads and seems to never drop back to normal. That will probably need to be investigated before this is merged.

LizardByte-bot · 2023-07-11T10:06:20Z

This PR is stale because it has been open for 90 days with no activity. Comment or remove the stale label, otherwise this will be closed in 10 days.

LizardByte-bot · 2023-07-22T10:05:50Z

This PR was closed because it has been stalled for 10 days with no activity.

cgutman marked this pull request as draft January 3, 2023 01:11

ReenigneArcher force-pushed the mediafoundation branch from 5ae7775 to 910ad82 Compare January 3, 2023 02:02

ReenigneArcher reviewed Jan 3, 2023

View reviewed changes

docs/source/about/advanced_usage.rst Outdated Show resolved Hide resolved

psyke83 mentioned this pull request Jan 3, 2023

ffmpeg: add low_latency encoder param for mediafoundation LizardByte/build-deps#72

Merged

11 tasks

ReenigneArcher reviewed Jan 4, 2023

View reviewed changes

src_assets/common/assets/web/config.html Show resolved Hide resolved

Quackdoc mentioned this pull request Jan 6, 2023

WINDOWS AMF: cursor not shown when streaming with AMF #513

Closed

2 tasks

cgutman force-pushed the mediafoundation branch from 910ad82 to 6120051 Compare January 19, 2023 02:51

cgutman marked this pull request as ready for review January 19, 2023 02:55

psyke83 reviewed Jan 21, 2023

View reviewed changes

cgutman marked this pull request as draft March 10, 2023 03:44

ReenigneArcher force-pushed the mediafoundation branch from 6120051 to 9ef2add Compare March 28, 2023 01:15

LizardByte-bot added the autoupdate label Mar 28, 2023

cgutman added 2 commits April 11, 2023 21:45

Add Media Foundation encoder

dab4114

Use GPU accelerated RGB->YUV conversion for Media Foundation

cf64312

cgutman force-pushed the mediafoundation branch from 9ef2add to cf64312 Compare April 12, 2023 03:05

LizardByte-bot removed the autoupdate label Apr 12, 2023

cgutman marked this pull request as ready for review April 12, 2023 03:14

cgutman marked this pull request as draft April 12, 2023 03:28

LizardByte-bot added the stale label Jul 11, 2023

LizardByte-bot closed this Jul 22, 2023

Uh oh!

Conversation

cgutman commented Jan 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Screenshot

Issues Fixed or Closed

Type of Change

Checklist

Branch Updates

Uh oh!

Uh oh!

psyke83 commented Jan 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

brad-richardson commented Jan 3, 2023

Uh oh!

cgutman commented Jan 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Isalight commented Jan 20, 2023

Uh oh!

psyke83 Jan 21, 2023

Choose a reason for hiding this comment

Uh oh!

psyke83 Jan 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

psyke83 commented Jan 21, 2023

Uh oh!

ReenigneArcher commented Mar 10, 2023

Uh oh!

cgutman commented Mar 10, 2023

Uh oh!

ReenigneArcher commented Mar 28, 2023

Uh oh!

cgutman commented Apr 12, 2023

Uh oh!

LizardByte-bot commented Jul 11, 2023

Uh oh!

LizardByte-bot commented Jul 22, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

cgutman commented Jan 3, 2023 •

edited

Loading

psyke83 commented Jan 3, 2023 •

edited

Loading

cgutman commented Jan 4, 2023 •

edited

Loading

psyke83 Jan 21, 2023 •

edited

Loading