Imagecodecs

[hmaarrfk]

Improvements to conda-forge:

• Current package imagecodecs-lite is defunct and on pypi, is a pointer to imagecodecs built with no options.
• This package works with python to open compressed tiff images. Something currently impossible with imageio and scikit-image being installed with conda-forge.
• This package includes as many options in imagecodecs as conda-forge currently has.
• This package uses jpeg9 instead of jpeg-turbo to be consistent with the rest of the conda-forge ecosystem.
• Worked with upstream to include options to overwrite their previously hardcoded options to options that would be amenable to conda forge.
  • (WIP) Fix conda build path cgohlke/imagecodecs#2

Previous checklist for work that

TODO:

• Try without custom build script for windows.
• Try withoout JPEG turbo

Needs:

• Add jxrlib from #6842 #10327
• Add Windows build support with Cmake lcms2-feedstock#3 or Try autotools conda win lcms2-feedstock#4 lcms2 for windows
• Windows build libaec-feedstock#5 libaec for windows
• needs jpeg-turbo, will not find enum constants like JCS_EXT_RGBA in libjpeg
  • Basically i just disabled any jpeg. Seems to be OK. Will have to see what the user experience is like
• libzopfli Add libzopfli #10513

May benefit from:

• charls #10330
• Add libjpeg turbo12 from #6842 #10328 -- its apparently not a good idea to have libjpeg-turbo12 and libjpeg-turbo on the same program.

Checklist

• License file is packaged (see here for an example)
• Source is from official source
• Package does not vendor other packages. (If a package uses the source of another package, they should be separate packages or the licenses of all packages need to be packaged)
• If static libraries are linked in, the license of the static library is packaged.
• Build number is 0
• A tarball (url) rather than a repo (e.g. git_url) is used in your recipe (see here for more details)
• GitHub users listed in the maintainer section have posted a comment confirming they are willing to be listed there

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/imagecodecs) and found it was in an excellent condition.

[Czaki]

I would like to share my experience with building imagecodecs wheels. I got the point with support of @cgohlke so setup.py is modified to simplify maintaining.

Current setup.py detects that there is non windows build with all dependencies if LD_LIBRARY_PATH or LIBRARY_PATH environment variable is defined

I decide to not install libraries globally, this allow to try build on own machine without need of cleaning.

In my repossitotory there are few scripts:

1. To build all depends libraires I need to install few things.
   1. On ubuntu: nasm autopoint libhdf5-dev
   2. On macos (from brew): pcre openssl nasm automake libtool. Also default compiler does not support openmp. I found that it is enough to install gcc@9 and use it as compiler.
2. build_utils/download_libraries.sh - download libraries in proper version. Downloaded libraries land in build_utils/libs_src directory.
3. build_utils/build_libraries.sh - it builds all needed libraries. Builded libraries land in build_utils/libs_build directory. During build there are few patches apply (to allow build on macos). patches are in build_utils/*.patch)
   Macos needs to set path to openssl with

export LDFLAGS=-L/usr/local/opt/openssl/lib
export CPPFLAGS=-I/usr/local/opt/openssl/include

4. Because macos introduce SIP (System Integrity Protection) and then using of LD_LIBRAY_PATH is not possible then I upgrade loading path with build_utils/fix_macos_lib.sh

I also create docker images (base on manylinux2010). Their build recipes are in build_utils. They are also available on https://hub.docker.com/search?q=bokota&type=image (for speedup). Maybe they will be useful for theses builds?

the upgrade to 2019.11.28 can be seen here: Czaki/imagecodecs/pull/2 (i'm waiting on tests result).

[Czaki]

State is accepted by @cgohlke and merged to master. Wheels created in this pipeline are uploaded to pypi https://pypi.org/project/imagecodecs/#files

[hmaarrfk]

nice, good update. Excited to try building again for a new release!

[Czaki]

Next release is ready.

[hmaarrfk]

don't we need jrxlib?
nobody merged that in yet. so i'm kinda stuck.

[Czaki]

I think that best start case is to get version without additional extension (lines from 149 in setup.py).
Then maybe add optional extensions.

[hmaarrfk]

hmm ok, i've been able to use imagecodecs-lite for now. Not sure I have time to get a working recepie for this immediately. Wanna give it a stab?

[Czaki]

I do not use anaconda currently. I work with classical CPython and install everything from pypi.

So I create pipeline to build wheel. Then I can share knowledge what I need to build wheel.

I do not know how such package should be introduced to anaconda. how many recipes it should be created and otthers.

Christopher really fast upgrade dependencies. There is no option for create recipe based on wheels?

[hmaarrfk]

Christopher really fast upgrade dependencies

Do you mean that Christopher changes the dependencies often?

I'll see what I can do over the weekend.

[Czaki]

I see that he uses newest version of libraries which are available. But also it is possible that code may work with older versions of libraries but it is not checked.

[hmaarrfk]

Conda forge has REALLY updated versions of libraries.

Typically, library versions are not a problem.

[Czaki]

I think that I misunderstand your question.
_imagecodecs module depends on jxrlib
In my configuration this lib is represent by 'jxrglue', 'jpegxr' in libraries variable.

[hmaarrfk]

hmm, cool, the package just got merged, so we can try again later.

[hmaarrfk]

how do I disable jxrlib?

[hmaarrfk]

It seems the problem is that the recipe has been regenerated after merged in staged recipe.

https://github.com/conda-forge/jxrlib-feedstock

it builds locally with jxrlib

[npavlovikj]

Hmm, the jxrlib-feedstock is empty?

[npavlovikj]

Looks like there is an issue with staged-recipes, and all packages merged today are not uploaded nor have feedstocks created, including jxrlib. We need to wait until the core team fixes this issue before restarting this build.

[hmaarrfk]

xref: #10438

[hmaarrfk]

Needs: #10513

[hmaarrfk]

@Czaki @cgohlke is there any way to disable certain libraries from being used? It seems that the _imagecodecs.pyx file tries to include all the plugins irrespectively.

[cgohlke]

is there any way to disable certain libraries from being used? It seems that the _imagecodecs.pyx file tries to include all the plugins irrespectively.

Only codecs that cannot be built using current Debian packages are optional and disabled by default: jpeg12, jpegls, jpegxl, and zfp.

[hmaarrfk]

Thanks for the reply and clarification.

Do take this as a kind request to attempt to make new libraries optional if the packagers decide to explicitly disable them. I know it can be more maintenance work, but keeping up with the wide array of formats you support is challenging!

[Czaki]

From my side, on MacOS I meet conflict between zopfli library and zopflipy package (when testing) when both are compiled with gcc/g++ from brew.
I use g++ to have OpenMP support for build but i need to compile zopflipy with clang.

[sdvillal]

Actually I very much like the practical approach you have taken to just have a custom setup.py instead of the patch, I do not want to deal with these pesky line endings either. We can just remove the workaround when imagecodecs sees a new release with the patch.

[hmaarrfk]

It seems that you have a slightly different organization.

I would prefer that we tested your patch since you already submitted it.

Lets wait 10 mins until this build passes, then test your take.

[hmaarrfk]

I finally got my version working. testing now with yours. I think it is the same, but you know, there are always bugs and errors.

[cgohlke]

Not sure why you are still patching/overriding imagecodecs's setup.py instead of providing your own customize_build function in an importable imagecodecs_distributor_setup module?

[hmaarrfk]

That would hvae been my preferred solution, but you decided to graciously include our options upstream.

We wish to work with you to get our changes included in your future distributions so that we stay in synch.

Thats all.

Your choice of line endings is totally fine, I was just venting my frustration at tools, not at your choice of them!

[hmaarrfk]

@jakirkham outstanding issues with jpeg have been resolved!

A merge would be greatly appreciated!

[cgohlke]

Does the testsuite pass? E.g. install imagecodecs and it's test requirements (most are optional) and run test_imagecodecs.py from within the tests directory included in the sdist.

[hmaarrfk]

I'm trying locally now before pushing.

@cgohlke do you have an interest in being a maintainer. I think it would simply make it easier to ping you if we continue to have issues with future updates.

[hmaarrfk]

Seems like many tests are running, but some seem to be getting skipped

test_imagecodecs.py::test_tifffile[deflate-u1] SKIPPED                   [ 99%]
test_imagecodecs.py::test_tifffile[deflate-u2] SKIPPED                   [ 99%]
test_imagecodecs.py::test_tifffile[deflate-f4] SKIPPED                   [ 99%]
test_imagecodecs.py::test_tifffile[lzma-u1] SKIPPED                      [ 99%]
test_imagecodecs.py::test_tifffile[lzma-u2] SKIPPED                      [ 99%]
test_imagecodecs.py::test_tifffile[lzma-f4] SKIPPED                      [ 99%]
test_imagecodecs.py::test_tifffile[zstd-u1] SKIPPED                      [ 99%]
test_imagecodecs.py::test_tifffile[zstd-u2] SKIPPED                      [ 99%]
test_imagecodecs.py::test_tifffile[zstd-f4] SKIPPED                      [ 99%]
test_imagecodecs.py::test_tifffile[packbits-u1] SKIPPED                  [ 99%]

[cgohlke]

Seems like many tests are running, but some seem to be getting skipped

The more test requirements are installed, the less tests are skipped.

If imagecodecs is compiled with all features and all test requirements are installed, there should be about 4563 passed, 1631 xfailed, and no skipped test cases.

[hmaarrfk]

got it.

I guess for now these are the results:
Linux

=============== 2088 passed, 3281 skipped, 823 xfailed in 58.70s ===============

Mac

========== 2087 passed, 3282 skipped, 823 xfailed in 63.62s (0:01:03) ==========

Windows

========= 2088 passed, 3281 skipped, 823 xfailed in 77.13s (0:01:17) ==========

Seems like many are getting skipped for only forgetting a few dependencies. But I guess that is enough for now.

[cgohlke]

Yes, looks good to me.

[jakirkham]

Do we still need this workaround now that setup.py has been corrected?

Suggested change

	LIBRARY_INC="bogus" $PYTHON -m pip install . -vv
	$PYTHON -m pip install . -vv

[hmaarrfk]

thanks

[jakirkham]

Should this be in host as well?

[hmaarrfk]

wow thanks for the catch!

[jakirkham]

Is this file in the recipe somewhere?

[hmaarrfk]

just added it back/

[jakirkham]

Thanks for the work all! Had a couple follow-up questions based on the recent changes.

[jakirkham]

Thanks @hmaarrfk for all of the work here and everyone for the reviews! Let's pick up anything else in the feedstock. Thanks again all 😄

[csachs]

Thank you all for pushing this forward 😄