Create master playlist for cast

[uvjustin]

Proposed change

Google Cast receivers don't seem to play HLS video streams with no accompanying audio. This may be able to be worked around by creating a master playlist and specifying the codecs used.

Type of change

• Dependency upgrade
• Bugfix (non-breaking change which fixes an issue)
• New integration (thank you!)
• New feature (which adds functionality to an existing integration)
• Breaking change (fix/feature causing existing functionality to break)
• Code quality improvements to existing code or addition of tests

Example entry for configuration.yaml:

# Example configuration.yaml

Additional information

• This PR fixes or closes issue: fixes #
• This PR is related to issue: Streaming camera to Google Home is broken in 0.115 #40376
• Link to documentation pull request:

Checklist

• The code change is tested and works locally.
• Local tests pass. Your PR cannot be merged unless tests pass
• There is no commented out code in this PR.
• I have followed the development checklist
• The code has been formatted using Black (black --fast homeassistant tests)
• Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

• Documentation added/updated for www.home-assistant.io

If the code communicates with devices, web services, or third-party tools:

• The manifest file has all fields filled out correctly.
  Updated and included derived files by running: python3 -m script.hassfest.
• New or updated dependencies have been added to requirements_all.txt.
  Updated by running python3 -m script.gen_requirements_all.
• Untested files have been added to .coveragerc.

The integration reached or maintains the following Integration Quality Scale:

• No score or internal
• 🥈 Silver
• 🥇 Gold
• 🏆 Platinum

To help with the load of incoming pull requests:

• I have reviewed two other open pull requests in this repository.

[probot-home-assistant]

Hey there @hunterjm, mind taking a look at this pull request as its been labeled with an integration (stream) you are listed as a codeowner for? Thanks!
_{^{(message by CodeOwnersMention)}}

[uvjustin]

@hunterjm Can you see if it allows casting a video stream with no audio?
Also, see if you can try the commented out alternative codec_string function. Getting the exact codec string is a bit cumbersome, and even in the long version we only map to a few codec profiles/levels. The simpler commented out codec_string function actually omits the profile details and levels. While this isn't standard, if it happens to work I think it is preferable as it is simpler.

[hunterjm]

I'll give it a shot. Also, if the master playlist also works in browser it might just be simpler to always render that, which points to the media playlist, which points to the segments. At least that way the logic will be consistant all around.

[uvjustin]

Sure we can do that.
The only concern there would be if the imperfect codec mapping and bandwidth estimation actually prevent some streams from playing properly. We would probably be able to catch that pretty quickly though.

[hunterjm]

Update here - this seems to work, but you are pulling the codec info from the source so my camera with G.711 audio is saying it has MP3 because it’s caught by the else condition.

The fully qualified codec string is required, so the commented out method did not work.

I’m also actually playing around with another implementation that reads this information from the moov header in the segment as it’s all contained there. I’ve gotten pretty far tonight and hope to keep tinkering tomorrow and this weekend.

[uvjustin]

Ok, I removed the commented out method and added a check that resets the audio_codec when the buffer doesn't support it.
If you can get the information from the segment that might be a cleaner alternative too. The most kludgy part is the profile mapping. Maybe we just do it with dicts?

[uvjustin]

On a tangentially related side note, I was playing around with proxying HLS streams but am leaning against doing it.
There are 2 ways to do it:

1. Don't download segments to HA at all and just copy over the manifests (making sure the urls in the manifest are fully qualified). This would be the simplest/most efficient, but it's not really related to stream at that point and should just be its own component.
2. Download the segments to HA without remuxing them. The filenames would be changed and the manifests would be recreated. The benefit is that since we are not remuxing anything, we avoid losing any of the problems when remuxing (eg the loss of the audio track since we can't currently apply the bitstream filter). The segments would still be in stream, so all the functionality like recorder and lookback would be there.
   I was working on this 2nd approach but realized it doesn't really do much except get around the audio problem we saw and avoid the overhead of remuxing. I had thought it would help with lag, but now I think it won't change the lag too much for two reasons: 1) each segment would still have to be downloaded and subsequently served, same as now and 2) I don't think the remuxing adds significant lag as the buffer in the hls input stream should be available to be read quickly and not limited to real time. Given this, the main benefit of doing this way of proxying would be to get around the audio problem, but that problem may be taken care if we get the ability to apply bitstream filters (I saw your bump on the topic in the PyAV project).

[hunterjm]

Quick update: I've finished implementing the box parsing for H264 and H265 video codecs. Still need to implement mp4a. I think this is the only real way we can go about it and keep compatibility and not blow up a self-maintained list with edge cases:

def get_codecs(segment: io.BytesIO) -> str:
    codecs = []

    # Find moov
    moov_location = next(find_box(segment, b"moov"))

    # Find tracks
    for trak_location in find_box(segment, b"trak", moov_location):
        # Drill down to media info
        mdia_location = next(find_box(segment, b"mdia", trak_location))
        minf_location = next(find_box(segment, b"minf", mdia_location))
        stbl_location = next(find_box(segment, b"stbl", minf_location))
        stsd_location = next(find_box(segment, b"stsd", stbl_location))

        # Get box
        segment.seek(stsd_location)
        stsd_length = int.from_bytes(segment.read(4), byteorder="big")
        segment.seek(stsd_location)
        stsd_box = segment.read(stsd_length)

        # Base Codec
        codec = stsd_box[20:24].decode("utf-8")

        # Handle H264
        if (
            codec in ("avc1", "avc2", "avc3", "avc4")
            and stsd_length > 110
            and stsd_box[106:110] == b"avcC"
        ):
            profile = stsd_box[111:112].hex()
            compatibility = stsd_box[112:113].hex()
            level = stsd_box[113:114].hex()
            codec += "." + profile + compatibility + level

        # Handle H265
        elif (
            codec in ("hev1", "hvc1")
            and stsd_length > 110
            and stsd_box[106:110] == b"hvcC"
        ):
            tmp_byte = int.from_bytes(stsd_box[111:112], byteorder="big")

            # Profile Space
            codec += "."
            profile_space_map = {0: "", 1: "A", 2: "B", 3: "C"}
            profile_space = tmp_byte >> 6
            codec += profile_space_map[profile_space]
            general_profile_idc = tmp_byte & 31
            codec += str(general_profile_idc)

            # Compatibility
            codec += "."
            general_profile_compatibility = int.from_bytes(
                stsd_box[112:116], byteorder="big"
            )
            reverse = 0
            for i in range(0, 32):
                reverse |= general_profile_compatibility & 1
                if i == 31:
                    break
                reverse <<= 1
                general_profile_compatibility >>= 1
            codec += hex(reverse)[2:]

            # Tier Flag
            if (tmp_byte & 32) >> 5 == 0:
                codec += ".L"
            else:
                codec += ".H"
            codec += str(int.from_bytes(stsd_box[122:123], byteorder="big"))

            # Constraint String
            has_byte = False
            constraint_string = ""
            for i in range(121, 115, -1):
                gci = int.from_bytes(stsd_box[i : i + 1], byteorder="big")
                if gci or has_byte:
                    constraint_string = "." + hex(gci)[2:] + constraint_string
                    has_byte = True
            codec += constraint_string

        # Handle Audio
        elif codec == "mp4a":
            oti = stsd_box
            dsi = stsd_box

        codecs.append(codec)

    return ",".join(codecs)

[uvjustin]

Nice, that definitely looks like a more complete solution. I'll let you take it from there.

[hunterjm]

I went ahead and pushed the final commit here for determining the codec string. I've also got small updates to add to media_player and cast, but I need to open an upstream PR in pychromecast before I can push those commits.

We should also probably clean this up and remove the video_codec and audio_codec attributes and modifications to the worker since we are no longer using them.

[uvjustin]

Nice, good work..I had a look at the mp4a part after your commit last night but after opening up a hex editor and trying to scour the web for info on esds boxes I realized how hard it was to find the relevant information. I was able to find the oti but didn't get tot he dsi. Just for curiosity, were you able to find a good source the mp4 format for those boxes or was there a lot of hex/binary scanning involved?

[hunterjm]

I borrowed heavily from https://github.com/gpac/mp4box.js which is the JS port of the command line MP4Box command from gpac. Their solution is a lot more complete than mine, and as you can see I just ended up scanning for the 0x04 and 0x05 sections for the esds box instead of attempting to parse the entire thing.

All the bit shift opeations were stolen from https://github.com/gpac/mp4box.js/blob/master/src/descriptor.js#L118-L131

Before trying to implement my own parser I scoured for a python implementation, but everything I found wasn't fully featured and more importantly lacked parsing for the info we needed :(

[uvjustin]

Should we check if len(codec)>4 before appending?

[hunterjm]

I thought about that, but the MP4box library does fall back to returning the base codec string of the additional features aren’t present. While we know that won’t work for chromecast, neither will leaving the codec out...

[uvjustin]

That makes sense. Since we're generating the segments ourselves hopefully there isn't much variability which would break the parsing.

[hunterjm]

Can you check this against your setup? I made the master playlist the default, so it should work in Lovelace as well. I tested against a generated H265 video, but didn’t change the codec on any of my cameras.

[uvjustin]

Tested against an online H264 stream and a H265 camera stream. Both worked with no problem in Lovelace. Checked the codec profiles/constraints against what an online parser returns and they match.

[hunterjm]

While this is a pre-requisite to get cast working again, I realized that this PR is actually able to be merged as-is (standalone), so I marked it as ready for review. There will probably be more discussion around how I'm implementing the changes in the core media_player in order to enable passing these attributes down, so it should be a separate PR.

[uvjustin]

@hunterjm Tested and works on my end. Great job