BMFF/JPEG-XL patent discussion

[ghost]

Although Exiv2 supports ISOBMFF, it is currently disabled by default, and is guarded by the CMake option and enableBMFF().
The README file also has a warning text:

Attention is drawn to the possibility that bmff support may be the subject of patent rights. Exiv2 shall not be held responsible for identifying any or all such patent rights. Exiv2 shall not be held responsible for the legal consequences of the use of this code.

I asked JPEG XL developers about the patent issues (since BMFF container is also used in JXL), and they believe that it is fine to use and support BMFF.

According to @jonsneyers (JPEG XL core developer):

ISOBMFF was never patented and it is in any case old enough to have all unknown patents expired (JPEG 2000 uses it, so it is at least 21 years old, probably older)

[clanmills]

Thank You for this information. We made -DEXIV2_ENABLE_BMFF=0 the default for 0.27-maintenance and =1 for 'main' (which will mains to be Exiv2 v1.00 eventually).

The reason to restrict this in 0.27-maintenance is simple caution. Although nobody has provided a solid case that reading BMFF is illegal, the default ensures that existing applications using exiv2 v0.27.4 (and any later 0.27 releases) are not exposed to a legal disk without their knowledge. To use Exiv2 with bmff support requires an application to either:

• Build the application with Exiv2 v0.27.4
  and call Exiv2::enableBMFF()
  and build the library with -DEXIV2_ENABLE_BMFF=0
  or
• Build the application with Exiv2 v1.00

In the unlikely event that exiv2/bmff support is illegal, the application has to actively request this support. It has not been pushed into the application by a platform library update.

I've changed the name of this issue from "Regarding ISOBMFF support in Exiv2 and concerns over patent issues" to "BMFF/JPEG-XL patent discussion"

I'm going to rename and close this issue. It has been logged in the release notes for Exiv2 v0.27.4 RC3 which will ship on 2021-05-22. #1018 (comment)

[ghost]

I see. Thank you very much!

[jonsneyers]

I think this caution is erring on the side of paranoia, and possibly based on a misunderstanding.

ISOBMFF is pretty old, old enough for it to be impossible to still have applicable patents (patents expire after 20 years). It is a simple box-based container format that is used by many formats, including MP4, JPEG 2000, and JPEG XL.

One particular use of ISOBMFF is HEIF, which is more recent, and for which there actually are known patent claims by Nokia. HEIF does use the generic ISOBMFF box structure and extends it by defining mechanisms to do cropping, layering, grids, rotation etc, which are described in the HEIF spec. HEIF can be used with various payloads: when used with HEVC it is called HEIC, when used with AV1 it is called AVIF.

While most people would consider patents on a container format ridiculous, it is a fact that, at least in theory, if you use HEIF, you might risk patent litigation.
Note that it would not be the application implementor who could be sued, but the end-user who uses the implementation and thereby possibly needs a patent license from Nokia.

This is only true specifically for HEIF though, not for ISOBMFF in general. Parsing the MP4 or JPEG 2000 container (which do not use HEIF) carries absolutely no risk in terms of patents, since they only use ISOBMFF, not HEIF. The same is true for JPEG XL, which has explicitly avoided using the HEIF container exactly for this reason.

TL;DR: ISOBMFF is OK, HEIF might be risky.

[clanmills]

@jonsneyers You know how it is. I'm damned if I do, and damned if I don't. bmff support was to be added in Exiv2 v0.27.3 (which shipped on 2020-06-30). When the subject of patent infringement was raised, I cancelled the bmff feature. The consequences were dire. In June 2020, I got about 100 emails in a week asking "What's the problem? We must have bmff support for CR3 and HEIF.".

The implementation in bmffimage.cpp is based on my reading of the ISO spec. Exiv2 is concerned with 4 types of metadata Exif, ICC, XMP and IPTC. The storage is clearly explained in the standard. Exiv2 has no code to understand boxes involved with cropping, rotation and so on.

The original patent objection was raised by a member of Team Exiv2 who works for a major linux distribution. His concern was that we could expose his employer to litigation. I have tried to please everbody by providing the capability, yet restricting the distribution and use of the code. The objection was sustained by a couple of users who stroked their gray beards and effectively said: "Trust me, I know enough about MPEG to be very cautious.". The reality is only a court case can prove the case.

Your statement about Nokia and HEIF is the only message I have received which discusses possible enfringement. All the other emails discussed reason why reading bmff is legal. For example:

1 The age of the bmff container design
2 The wide-spread use of bmff containers
3 Many open source projects read bmff with impunity
4 Exiv2 reads unencrypted metadata written to public standards

I believe that the start of legal proceedings are for the patent holder to issue a "cease and desist letter". If that were to happen, we could remove bmffimage.cpp and related code and ship a new dot release.

Thank you for pointing out that it is the end user who needs a license from the patent holder. There are lots of players in the game. Exiv2, the distro, the end-user, the application developer, the ISO standard, the patent holder. Which patent or patents are involved? Only a court could resolve this complexity.

So, I've tried hard (and believe me, this has been a lot of work) to provide the community with the code they have requested while adopting caution to respect the opinion of those who raised the subject of patents.

[jonsneyers]

So just to be clear: reading BMFF is not an issue, it is over 20 years old so even if it was an issue in the past (it wasn't) it now certainly isn't anymore.

Reading HEIF-specific boxes is something else, but as long as Exiv2 is not doing that, there obviously is no issue either. Note that other FOSS tools like libheif and libavif actually do read those boxes, and they do seem to get away with it (but I can understand why you wouldn't want to do that; those boxes are from 2015 and Nokia does claim patents on them so it will only really be 'safe' to use that functionality in 2035).

So while I appreciate the caution, I think it's OK to just enable the BMFF code by default (perhaps have an option to disable it, if someone is still for some reason worried, but imo that would be an unfounded worry). Otherwise most of the modern image codecs (jp2, jxl, avif and heic) end up being unsupported by default, for no good reason, which seems a suboptimal situation.

[clanmills]

Your thinking is parallel to mine. Exiv2 v1.00 uses the default -DEXIV2_ENABLE_BMFF=1. So in future, bmff support is enabled by default and can be optionally disabled.

The first version of ISOBMFF was 2004 and the container format was first used by MPEG in the 1990s. So, even if a patent has been infringed, the holder's "ownership" has almost certainly expired.

I have retired. I was 70 in January and have been trying for years to stop working on Exiv2. The bmff code is my final C++ contribution. Because of the visibility of the bmff issue, several new folks have joined Team Exiv2 and will take the project into the future. It's curious that while working on bmff, the puzzle of Exiv2 maintenance seems to have been solved. To enable me to retire, I have written a book: Image Metadata and Exiv2 Architecture. Part of the book deals with bmff files: http://clanmills.com/exiv2/book/#BMFF