Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix the input pixel format when using GPU video encoder #3426

Closed
wants to merge 1 commit into from

Conversation

mthrok
Copy link
Collaborator

@mthrok mthrok commented Jun 8, 2023

StreamWriter's encoding pipeline looks like the following

  1. convert tensor to AVFrame
  2. pass AVFrame to AVFilter
  3. pass the resulting AVFrame to AVCodecContext (encoder) and AVFormatContext (muxer)

When dealing with CUDA tensor, the AVFilter becomes no-op, as we have not added support for CUDA-compatible filters.

When CUDA frame is passed, the existing solution passes the software pixel format to AVFilter, which issues warning later as what AVFilter sees is AV_PIX_FMT_CUDA.

Since the filter itself is no-op, it functions as expected. But this commit fixes it.

See #3317

@pytorch-bot
Copy link

pytorch-bot bot commented Jun 8, 2023

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/audio/3426

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 12 Pending, 1 Unrelated Failure

As of commit d8cdced:

NEW FAILURE - The following job has failed:

BROKEN TRUNK - The following job failed but were present on the merge base a7fea8a:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

StreamWriter's encoding pipeline looks like the following

1. convert tensor to AVFrame
2. pass AVFrame to AVFilter
3. pass the resulting AVFrame to AVCodecContext (encoder) and AVFormatContext (muxer)

When dealing with CUDA tensor, the AVFilter becomes no-op, as
we have not added support for CUDA-compatible filters.

When CUDA frame is passed, the existing solution passes the software pixel
format to AVFilter, which issues warning later as what AVFilter sees is
AV_PIX_FMT_CUDA.

Since the filter itself is no-op, it functions as expected. But this commit
fixes it.
@mthrok mthrok force-pushed the fix-gpu-encoder branch from 302acb4 to d8cdced Compare June 8, 2023 16:13
@facebook-github-bot
Copy link
Contributor

@mthrok has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

1 similar comment
@facebook-github-bot
Copy link
Contributor

@mthrok has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@mthrok merged this pull request in 30afaa9.

@github-actions
Copy link

github-actions bot commented Jun 9, 2023

Hey @mthrok.
You merged this PR, but labels were not properly added. Please add a primary and secondary label (See https://github.com/pytorch/audio/blob/main/.github/process_commit.py).


Some guidance:

Use 'module: ops' for operations under 'torchaudio/{transforms, functional}', and ML-related components under 'torchaudio/csrc' (e.g. RNN-T loss).

Things in "examples" directory:

  • 'recipe' is applicable to training recipes under the 'examples' folder,
  • 'tutorial' is applicable to tutorials under the “examples/tutorials” folder
  • 'example' is applicable to everything else (e.g. C++ examples)
  • 'module: docs' is applicable to code documentations (not to tutorials).

Regarding examples in code documentations, please also use 'module: docs'.

Please use 'other' tag only when you’re sure the changes are not much relevant to users, or when all other tags are not applicable. Try not to use it often, in order to minimize efforts required when we prepare release notes.


When preparing release notes, please make sure 'documentation' and 'tutorials' occur as the last sub-categories under each primary category like 'new feature', 'improvements' or 'prototype'.

Things related to build are by default excluded from the release note, except when it impacts users. For example:
* Drop support of Python 3.7.
* Add support of Python 3.X.
* Change the way a third party library is bound (so that user needs to install it separately).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants