HDR videos look pale on mobile

[s0nerik]

The bug

Looks like video_player package doesn’t support HDR colors in videos. The respective issue comments seem to suggest that an alternative package, called media_kit, exists, which doesn’t have this problem.

The OS that Immich Server is running on

Raspbian

Version of Immich Server

v1.86.0

Version of Immich Mobile App

v1.86.0 build.126

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

-

Your .env content

-

Reproduction steps

1. Shoot an HDR video on any iPhone supporting it.
2. Sync it in original quality into Immich.
3. Open synced video in the Immich app on that same iPhone.
4. Colors in the video look pale.

Additional information

No response

[mertalev]

We've been looking into media_kit as a better alternative for HDR and hope to switch to it. However, the impression we got was that while its color reproduction is definitely better, it still isn't HDR. Based on this patch in their repo, it seems there's a hardware decoding limitation involved. HDR seems to be an open issue in the Flutter ecosystem, unfortunately.

[s0nerik]

I think another alternative could be to use the native view for a video player on iOS, without copying frames into a texture. I guess native_video_player does that, but not sure as I haven’t tried it.

[mertalev]

I think platform views with Metal are the only way to get HDR on iOS. media_kit describes its player as native too, so there seems to be more to it than that. I'm not sure what this player uses.

[alexmercerind]

I saw this discussion.

Based on this patch in their repo, it seems there's a hardware decoding limitation involved.

That patch addresses a regression. 4K HDR plays flawless (performance wise) on iOS based on the feedback we have so far.

We have discussed few things in a new ticket:

media-kit/media-kit#615

You can improve overall rendering result on normal displays with tone mapping.

I believe default one isn't best on iOS/macOS right now.

[mertalev]

Thank you for clarifying that! I think the --tone-mapping flag can help as well here, but we already transcode and tone-map videos, so the problem only occurs when users want to view the original. Does this flag have any color or performance impact on SDR videos, or is it safe to enable universally?

I also wonder if there's any hope for hardware-decoded HDR using Flutter. From the discussion you linked, it seems the limitations are on Flutter textures along with libmpv. Am I right in thinking that platform views with Metal would allow for HDR? If so, are you aware of any player that takes this approach?

[yotis1982]

Sorry if my knowledge is not at best to fully understand this issue, but is this topic being addressed on the roadmap or at least a fix expected in the future releases? Is a bit weird to look at my videos on iPhone and look like they are washed out.

Thank you!

[mertalev]

Do you have transcoding disabled? HDR videos have to be transcoded for the colors to look normal on mobile.

We do have a PoC with media_kit that can render these videos better (albeit not HDR), but it has some other issues we haven't fixed yet.

[yotis1982]

@mertalev thanks for fast reply, transcoding settings are pretty much the default ones; for the Trandcode Policy I have "Only videos not in desired format". Should I have it on "All videos" to transcode everything?

Thanks!

[mertalev]

Hmm, it should be transcoding on default settings. Could you try running a "missing" job for transcoding? If that doesn't transcode anything then it might just be a bug.

[yotis1982]

I tried your suggestion, the task went fast, I don't think anything was transcoded. I'll check more on. Thanks!

[yotis1982]

I did a test today and it seems to work fine (aka colors looks fine in the HDR), but is interesting how it works at least for me. I'll do my best to explain and maybe somebody with more technical knowledge in this area will find it useful.

First the Immich server Settings for Transcode Policy is set to "Only videos not in desired format"; everything else is default as per installation.

I took a video on my iPhone and played the video through the Immich app.

1. Video is only local on iPhone (backup icon is cloud cut) - colors are faded (see below for details):

The Info shows "is remote: false" and "storage : AssetState.local"

2. Video is local on iPhone and backup to Immich (backup icon is cloud and checkmark) - colors are faded (see below for details)

The Info shows "is remote: true" and "storage : AssetState.merged".

3. Video is only backup on Immich (backup icon is just cloud) and deleted from Iphone - colors are normal (see below for details).

The info shows "is remote: false" and "storage : AssetState.remote"

I think everything is fine, because my only concern was when I was about to check on backup videos that the colors will not look ok. If the video is still on my iPhone, I can play it via the native iOS app.

Thanks and sorry for the large images.

[benbeton]

thank you for the testing an Details. Nevertheless in my opinion the colors should never be shown faded, other users may ask themselves the same questions and maybe worried about. But I think It's really helpful to have your analysis

[aureateflux]

Is this only an IOS issue? I'm seeing this on my android app as well. I have a video that is marked "10-bit HDR" in Google Photos that looks washed out in the Immich app on Android, but colors are normal on the Immich web view (the web view doesn't appear to be HDR, but the colors look similar to the native video).

[baldemar-wuda]

Same here (iOS) - HDR video colors are "off" in the app, but fine when played via web browser.

[s0nerik]

Is this only an IOS issue?

I'm seeing the same issue on Android as well with HDR videos recorded by iPhone 13 mini.

[mertalev]

I believe the issue affects both iOS and Android. The underlying native players for both platforms support HDR, but the way Flutter interacts with them doesn't allow for it.

[blue-kaleidoscope]

Using Immich on iPadOS HDR looks good and not pale. 🤔

[s0nerik]

Using Immich on iPadOS HDR looks good and not pale. 🤔

You're probably viewing the processed videos (where HDR was converted to SDR). The problem is with viewing the original HDR video.

[blue-kaleidoscope]

That is possible, as there was some time difference between watching an HDR video on my iPhone, and then on my iPad.

[sannidhyaroy]

Yes, I'm facing this as well. HDR 10 videos (shot on Galaxy S23) looks washed out in the Immich mobile app that looks perfectly fine in the web app. Will this issue not be resolved unless the core Flutter team works on flutter/flutter#91241?

[mertalev]

Will this issue not be resolved unless the core Flutter team works on flutter/flutter#91241

We will probably switch to the media_kit player, which doesn't have the issue of washed out videos since it tone-maps them.

Of course it'd be better if we could get proper HDR, but that might require writing native iOS / Android implementations and nobody in the team has a background in that.

[serfriz]

I just want to confirm that I'm having the same issue with my Pixel 8 Pro (Android 14). HDR videos look washed out.

[televisedBow]

Will this issue not be resolved unless the core Flutter team works on flutter/flutter#91241

We will probably switch to the media_kit player, which doesn't have the issue of washed out videos since it tone-maps them.

Of course it'd be better if we could get proper HDR, but that might require writing native iOS / Android implementations and nobody in the team has a background in that.

Tone mapping would be a good stop gap solution until the proper solution can be achieved. It's a shame that the flutter problem is literally years old at this point. I would hope that the prevalence of HDR on almost all new phones would cause some activity.

[Austinzveare]

I just started using Immich and ran into this problem immediately. I have somewhat of a niche, but high quality workflow where I shoot ProRes Log on my iPhone 15 Pro Max, export from Davinci Resolve in HDR10 with metadata, then view in Plex, but ultimately would prefer Immich. The video plays back with correct color in Safari Immich on my iPhone and MacBook, and is just the app that doesn't display HDR properly, I have video transcoding effectively disabled. I really hope this can be worked on because I'd say for most, if not all iPhone users it's kind of a deal breaker in terms of video playback through Immich.

Quick side note. I know true HDR photography is still barely supported in any OS/software, but ever since Adobe Lightroom had editing support I've been waiting for the day for ANY viewer to actually display HDR JXL/AVIF properly. One can only hope that the next MacOS & iOS improve this.

[rinlv]

same with me, it only happens on the mobile app

[hkrob]

+1 here, Samsung S24 - non-transcoded video looks washed out on mobile, fine in desktop (Windows) browser.
As a stop-gap, would it be possible to add an option to open the file in a local player on mobile?

[d1scolor]

Not sure about technical implementations but -
I have a Jellyfin server which has some HDR videos taken by iPhones.
In the Jellyfin iOS app there is a "Use Native Video Player" beta option. When enabled, the iOS app plays the untranscoded iPhone HDR videos flawlessly.

[d1scolor]

In addition to the pale look, the framerate of video playback appear to be lower than the source.

[Titaniumtown]

I've experienced something like this with photos as well. But I'm also setting up a new camera so it could be a configuration issue :p

Edit: Camera is a Canon R50 and I'm shooting HEIF. Seems to be an actual display issue with the HDR PQ images.

[mertalev]

Just to give a little update on this: we're experimenting with native_video_player, which from a PoC actually renders HDR and at a higher frame rate than the current player. But there are a few issues with it that we still need to figure out: it frequently crashes in debug mode (but not in the release version), and it doesn't let you add HTTP headers for network requests (for e.g. authorization).

[joaopedrocg27]

Hey @mertalev

Did you guys have any progress on this?

Thanks

[mertalev]

As a matter of fact, yes! I'm currently doing testing and polishing on iOS. There are no major issues, just smaller things left - wakelock doesn't get disabled when you leave the video, the video position widget gets choppy when you scrub, etc. I'm hoping to get it merged very soon.

[joaopedrocg27]

That is awesome! Thank you very much!

[Austinzveare]

This is great, thanks for the update! Slightly related, is there progress on HDR photo support? Ideally JXL, or AVIF, JPG gain map etc.

[mertalev]

This is great, thanks for the update! Slightly related, is there progress on HDR photo support? Ideally JXL, or AVIF, JPG gain map etc.

No, we still need to look into alternative photo viewers that can display HDR. This is on my mind, but I personally won't be able to work on it in the short-term.

[RdWing]

Discovered the other day that viewing Photos via icloud.com renders photos in HDR, at least while using Edge on Windows. The gain maps are being properly displayed and interpreted. Glaringly, attempting the same task via Immich leads to disappointment as HDR photos are nowhere to be found. Now that ISO 21496-1 is a standard, that can be brought into Immich?

[mertalev]

I'm excited to say that the next release will have HDR video playback on both Android and iOS!

[alxjsn]

I'm excited to say that the next release will have HDR video playback on both Android and iOS!

This is exciting! Do we have to force a re-encode of all videos to reap the benefits?

[mertalev]

I'm excited to say that the next release will have HDR video playback on both Android and iOS!

This is exciting! Do we have to force a re-encode of all videos to reap the benefits?

No, but whether the video is rendered in HDR depends on where it's sourced. If the video exists on the phone locally, it will always play the (HDR) original. However, if the app needs to stream the video from the server and the server transcoded this video, it will serve this SDR transcode instead.

This will change in the next release - you'll be able to prefer streaming the (HDR) original even if there's a transcode available. Later releases will likely adapt transcoding to be more HDR-friendly now that the client support is better.

Edit: Oh, the video preference is already available in the latest release.