Add support for ZSTD compression for all LUTs files #1062
Conversation
|
Hi @satgit62 Let me know if there are any memory issues when using compressed LUTs (problems may only manifest themselves when decompressing them during LUT change). I expect decompression to take about 1 sec but you will find it in your logs as it depends on your hardware. |
awawa-dev
force-pushed
the
compressed_lut
branch
from
4e43acb to
a47d436
Compare
Hello, |
|
Hi @awawa-dev However, once decompressed, and you only watch channels that require decompressed LUT, such as SDR, HDR or DV, there is no delay at all. Only when switching between the LUTs. I think this is okay, and I believe that the newer devices with more powerful CPUs/SoC need less time for this. |
|
Hmm..on x64 it's a split of second :-( Maybe I can add support for zstd too https://community.centminmod.com/threads/compression-comparison-benchmarks-zstd-vs-brotli-vs-pigz-vs-bzip2-vs-xz-etc.12764/ |
|
Or I can use multithreaded version lzma_stream_decoder_mt for xz let me check it |
awawa-dev
force-pushed
the
compressed_lut
branch
3 times, most recently
from
c3b3023 to
ea30d4c
Compare
awawa-dev
changed the title
awawa-dev
force-pushed
the
compressed_lut
branch
from
ea30d4c to
9abb717
Compare
awawa-dev
force-pushed
the
compressed_lut
branch
from
9abb717 to
d68503f
Compare
|
@satgit62 I finally migrated XZ to ZSTD. The compression is slightly worse but it's much faster. I've also updated the manual. 7-zip does not have full ZSTD support so under Windows you must use its fork 7-Zip-zstd. |
|
Hello @awawa-dev |
|
Thank you for the feedback @satgit62 You can experiment with the compression level as it does not have negative impact on the decompression speed (diffent story than XZ). 20 shoul be safe because archlinux use it on default, 22 also should be fine but it will take very long time to compresss each LUT. To be safe verify decompression speed on your device for LUT with higher compression level. |
I will experiment with level 20 or 22 and report back. |
awawa-dev
changed the title
|
@awawa-dev Experimenting with levels 20 and 22 creates an additional 40MB of free storage space. However, the times remain almost unchanged, which is a good thing. See log with compression level 22. UHD-HDR |
|
Yes switching to NV12 Seems everything is fine so far so I merge the changes into the master branch. |
|
If you were to prepare a ready distribution package for the described conditions, you could even zero out (e.g. with Linux dd command) the unused first and last table, which would significantly reduce the zstd archive size. And it would probably also speed up the decompression of table no. 1 by about a second, since table no. 0 would be a trivial zero. But I don't know if it's worth messing around with. |
|
Hi @satgit62 The effect is spectacular when you compress it later 😃 |
|
Hello @awawa-dev, you are right, the effect is spectacular if you compress the LUT files later. The files are then under 10 MB. I then deleted the lut_lin_tables.3d.zst, which I had previously prepared with “fsutil file setzerodata”, and created a new file without first compressing it again with “fsutil file setzerodata” (the resulting file size is approx. 50 MB) and lo and behold, it works. I don't know what's wrong here. |
That means that there is still something wrong with the backend.Either it turns off tone mapping and loads the table with index 2 (with NV12 enabled) or tone mapping is active but turns off NV12 and loads the table with index 0. You will have the answer in the logs. Both tables with indexes 0 and 2 are zeroed by these commands, so the effect will be a black screen. |
|
Hi @awawa-dev, I have attached one for you . Yes, it is index 2 for YUV, and HDR Global is deactivated logs |
|
Yes, it's backend. Maybe there is other place that enables/disables tone mapping or changing lut filename in the backend that needs to be fixed. Here "SDR" lut_lin_tables.3d was passed as filename with tone mapping disabled (via JSON command): HDR has tone mapping enabled correctly (via JSON command): |
|
Or the default tone mapping is disabled in HyperHDR (in webos build configuration) and JSON sets the filename but doesnt set tone mapping state. |
|
No, it must be a backend. Filename and HDR state is set in the same method: HyperHDR/sources/api/HyperAPI.cpp Line 977 in 1ce4c3a Are you sure you have installed fixed backend? |
Yes, the hyperon-webos backend has been fixed by sundermann and NV12 has been implemented. See commits: webosbrew/hyperion-webos@289ca60 |
|
Or should I enter in line 202: |
|
No, it's fine. Could you add these 3 lines before HyperHDR/sources/api/HyperAPI.cpp Line 979 in 1ce4c3a |
|
Yes, I will do that. |
|
Hi @awawa-dev, logs |
|
As you can see something set tone mapping to disabled via JSON API: Nothing else comes to my mind but the old not-fixed backend. |
|
For testing you can rename lut_lin_tables.3d in the backend to: lut_lin_tables_sdr.3d, upload changed backend, reset the device and see if the changes has been applied: HyperHDR should receive lut_lin_tables_sdr.3d for SDR. Maybe the old backend is installed somewhere and is still active? |
|
Hi @awawa-dev, I think the log is OK now. 2025-01-24T19:31:01.501Z [CORE] TTY is not attached to the log output |
|
I think it's not related, that value returns status: 0 = OK |
Yes, this works perfectly and the compressed LUTs are loaded in real time. No idea why it works with this value. |
|
To be sure, I suggest checking the backend logs now to see if they are littered with false errors (sending the command was successful but now the method returns status 1). And additionally changing this value ("ret") back to 0 and compiling and installing this version to be sure that this value was the problem and not the old backend. I simply have no idea where this effect is coming from 😃 |
Hi, it's not the backend, it works even if the value stays at 0. “int ret = 0;” 2025-01-25T12:33:09.958Z [FLATBUFSERVER] Setting user LUT filename to: 'lut_lin_tables_hdr.3d' |
|
Yes, webos could be tricky especially with that cold restarts (Quick start I think). Hopefully this isn't due to an older backend version in the repository, but newer compressed LUTs should clear this up for users quickly. Decompression time is very good on your device. |
|
Hi, @awawa-dev, I barely slept to find out before recommending it to someone with webOS. I have no idea why it didn't work out in the end. But everything is fine now. I also completely reset both my LG devices and lo and behold, they are working fine again. So the HyperHDR master branch is running and the new PicCap v0.5.0 with the NV12 backend I built is also running. https://github.com/satgit62/piccap/tree/NV12 |
|
Yes, support ZigBee is probably last change before beta2. A matter of days or one week maybe. I will restore dynamic brightness there, but I have to significantly modify my fork zigbee2mqtt and test it. In the meantime zigbee2mqtt released version 2.0.0 which changes a lot and additionally I was discouraged by this non-working touchlink in Silicone Labs. Unfortunately, Internet opinions that Zigbee-E is not such a good replacement for Zigbee-P have been confirmed in practice. Hardware alone is not everything. |
I can then do testing of all your changes in the webOS branch. |
|
OK, @satgit62 HyperHDR 21.0.0.0beta2 is ready for testing (master branch) |
|
@satgit62 while testing I found a bug in HA wizard :-( Please rebuild it, the master branch is fixed now. |
* Add support for XZ compression for all LUTs file * Replace XZ with ZSTD * Update 3RD_PARTY_LICENSES
Recommended usage:
Fork of 7-zip with a support of ZSTD compression https://github.com/mcmilk/7-Zip-zstd
Use 7-zip ZS. Right click on the uncompressed LUT file:
7-zip ZS⇒Add to archive...You should see the 7-zip compression settings.
HyperHDR should automatically recognize that the uncompressed file does not exist and will try to find a compressed file with the additional
.zstextension.