KHR_draco_mesh_compression

[fanzhanggoogle]

Hi,

We made an extension for using Google Draco geometry compression library with glTF. It could also be easily extended to more compression libraries if it suits well to glTF 2.0.

Let us know if there is any flaws in the design. Thank you very much.

We also provide an alternative approach in the appendix in case anyone is interested in discussing it.

@ondys
@FrankGalligan

[lexaknyazev]

This extension is proposed by Google, so should it use GOOGLE (or EXT) prefix instead of KHR?
Also, since it extends mesh object, maybe it should be called _mesh_draco_compression to be aligned with other extension names?

[fanzhanggoogle]

We talked to Patrick and Neil and the plan was to preferably propose it as KHR extension and integrate it to glTF 2.0 if we can work something out before the 2.0 release soon. But we can start with GOOGLE and see how it goes it that's better?

The draco library could also compress point cloud, in that case the primitive will just not have indice accessor. Calling it geometry will be better since it includes mesh and point cloud.

[lexaknyazev]

In glTF, point clouds are represented via mesh object by using different mesh.primitive.mode value. So they're meshes in glTF terms. I think that extension's name should reflect its formal relations with core spec.

CC @pjcozzi

[fanzhanggoogle]

I changed the name back to mesh. And removed geometryType in extension. Added description for restriction in README.

[lexaknyazev]

Why not use bufferView object to address Draco stream instead of "re-inventing" the same concept (buffer, byteOffset, and byteLength) here? Such bufferView would have byteStride and target properties undefined.

[fanzhanggoogle]

We thought about that. The reason for not using bufferView is that we want to reduced the number of newly introduced layers. But I think it makes sense to use bufferView here. Changed here and the json.

[lexaknyazev]

attributesComponentTypes and attributesType must be switched to match core spec definitions.

attributesOrder, attributesComponentTypes, and attributesType arrays must be of the same size and must list elements in the same order, right? So why not combine these properties into one structure:

{
    "attributes" : [
        {
            "semantic": "POSITION",
            "type": "VEC3",
            "componentType": 5126
        },
        {
            "semantic": "NORMAL",
            "type": "VEC3",
            "componentType": 5126
        },
        {
            "semantic": "TEXCOORD_0",
            "type": "VEC2",
            "componentType": 5126
        },
    ]
}

type: "VEC3" and componentType: 5126 could be made default, since we could expect them to be the most frequent values.

[fanzhanggoogle]

Good idea! Changed.

[lexaknyazev]

String-based enums should use uppercase.

[fanzhanggoogle]

Done

[lexaknyazev]

Also, geometryType property duplicates mesh.primitive.mode, so could we remove it and add needed restrictions on valid core enum values (i.e. it's either TRIANGLE_LIST, or POINTS)?

[lexaknyazev]

Probably should add that all restrictions on semantics names from the core spec still apply (i.e., TEXCOORD_%d instead of just TEXCOORD).

[fanzhanggoogle]

Done

[lexaknyazev]

UNSIGNED_INT value isn't allowed for attributes, so it shouldn't be here.

[fanzhanggoogle]

Done. Also removed unrelated description.

[lexaknyazev]

Looks like buffer, byteOffset, and byteLength should be removed from example.

[fanzhanggoogle]

Done

[pjcozzi]

For the extension name prefix, let's just start with KHR_. This will avoid churn in the extensions namespace and this will have multiple implementations quickly.

Beyond that the complete extension name is up to the Draco team.

[pjcozzi]

@sbtron @bghgary perhaps you also want to review this?

[pjcozzi]

Suggested tweak: "accessor in an extended primitive no longer points to a bufferView"

[fanzhanggoogle]

Done

[pjcozzi]

I think this was discussed offline, but to confirm - does this create a scenario for bottom-up loaders where they might fill an accessor with zeros thinking it is for a sparse accessor, but then later it is used for compressed data? Probably not, right? Because a loader will probably only access buffer and bufferView bottom, then access primitive before touching an accessor?

[fanzhanggoogle]

Yes. For bottom-up loading, even if the accessor is filled with zeros, it will be accessed through primitive --> attributes where the primitive is extended, so the decompressed data will be used instead.

[FrankGalligan]

Draco bitestream version 2.2

[fanzhanggoogle]

Done

[FrankGalligan]

If a primitive contains an extension property and the extension property defines it's KHR_draco_mesh_compression property, then the Draco geometry compression must be used.

[fanzhanggoogle]

Unless the loader doesn't support Draco and use the fallback uncompressed data?

[FrankGalligan]

The Draco extension points to the buffer that contains the compressed data.

If the uncompressed version of the asset is not provided, then KHR_draco_mesh_compression must be added to extensionsRequired.

 "extensionsRequired" : [
     "KHR_draco_mesh_compression"
 ]
 

If KHR_draco_mesh_compression is set in the extensionsRequired property then the primitive should only contain the Draco compressed data.

If a Draco compressed version of the asset is provided then KHR_draco_mesh_compression must be added to extensionsUsed.

 "extensionsUsed" : [
     "KHR_draco_mesh_compression"
 ]
 

[fanzhanggoogle]

Done.

[FrankGalligan]

Below is an example of what a part of a glTF file may look like if the Draco extension is set. Note that all other nodes stay the same except primitives:

[fanzhanggoogle]

Done.

[FrankGalligan]

properties

Should we just remove this line?

[fanzhanggoogle]

Removed.

[FrankGalligan]

* If the loader does support the Draco extension, but will not process `KHR_draco_mesh_compression`, then the loader must load the glTF asset ignoring `KHR_draco_mesh_compression` in `primitive`.
* If the loader does support the Draco extension, and will process `KHR_draco_mesh_compression`, then:
  * The loader must process `KHR_draco_mesh_compression` first. The loader must get the data from `KHR_draco_mesh_compression`'s `bufferView` property and decompress the data using a Draco decoder to a Draco geometry.
  * Then the loader must process `attributes` and `indices` properties of the `primitive`. When loading each `accessor`, go to the previously Draco decoded geometry in the `primitive` to get indices and attributes data. A loader must use the decompressed data to overwrite `accessors` or render the decompressed geometry directly (e.g. ThreeJS).

[fanzhanggoogle]

Done

[FrankGalligan]

Remove this section

[fanzhanggoogle]

Done

[FrankGalligan]

I think we can remove this section as well and replace it with this (which will be part of the list above.

  * If  additional attributes are defined in `primitive`'s `attributes`, but not defined in `KHR_draco_mesh_compression `'s `attributes`, then the loader must process the additional attributes as usual.

[fanzhanggoogle]

Done

[FrankGalligan]

I think we can remove this line with the re-worked paragraph above.

[fanzhanggoogle]

Done

[FrankGalligan]

All Khronos specifications should contain the standard Copyright Header:
https://www.khronos.org/members/login/groups/Agreements%20and%20Licenses/Open%20Source%20Repository%20Resources/Khronos%20Specification%20Copyright%20License%20Header%20V3%20May17.txt

[FrankGalligan]

As the Draco extension is referring to significant external documents – namely the Bitstream specification we should also include, and be consistent with the additional header wording for normative clarity:
https://www.khronos.org/members/login/groups/Agreements%20and%20Licenses/Open%20Source%20Repository%20Resources/Khronos%20Copyright%20License%20Header%20Addition%20for%20NORMATIVE%20CLARITY%20Oct17.txt Note - this has some boiler plate that needs to be tweaked to fit the spec.

The key text in the header is:

Some parts of this Specification are purely informative and do not define requirements
necessary for compliance and so are outside the Scope of this Specification. These
parts of the Specification are marked by the "Note" icon or designated "Informative"
<replace/insert specific conventions for the specification here>.

And

Where this Specification includes normative references to external documents, only the
specifically identified sections and functionality of those external documents are in
Scope. Requirements defined by external documents not created by Khronos may contain
contributions from non-members of Khronos not covered by the Khronos Intellectual
Property Rights Policy.

[fanzhanggoogle]

Done

[FrankGalligan]

"indices propertiesof theprimitive" -> "indicesproperties of theprimitive`"

[fanzhanggoogle]

Done

[FrankGalligan]

Remove this "Draco geometry compression library could be used for primitive by adding an extension property to a primitive, and defining its KHR_draco_mesh_compression property."

[fanzhanggoogle]

Done

[zellski]

Having just recently added morph targets to my glTF generation, I'm suddenly noticing that they're absent from Draco considerations. It's likely out of scope and in any case too late to add this to the initial release of the extension, but may I join @donmccurdy in advocating for its usefulness?

Sparse accessors salvage morph targets from extreme bloat, but they're still likely to take up a disproportionate amount of space when all the other static geometry has been compressed.

As a corollary, I know the Draco team is at least thinking about the temporaral dimensions viz. animations, and so it's worth noticing here that the 'weights' target for animations is likely very compressible.

[donmccurdy]

This is probably too close to final, as you said, but even if it weren't — if we added support for morph targets with the Draco format as it is now, I assume each target would need to be treated as a separate mesh by the decoder. This would limit the performance benefit, compared to using a temporal dimension per the long-term roadmap.

So let's give feedback on the use cases in the Draco repository (say, google/draco#126) and consider bringing Draco-compressed morph targets to glTF as an official KHR_ extension when those dependencies are in place.

In the meantime it would certainly be possible to use vendor extensions for short-term needs.

[JeremieA]

Hello,
Great work on this very useful and well-designed extension !
I have two suggestions below, but as it is indeed very late in the process it would be fine to only consider them for the next version. However I think it would be great to include them if possible, as they are simple but quite useful.

For our use-case (physics-based simulations) we are very interested in animations, hence I created a proof-of-concept implementation of adding morph targets to this extension. It is actually pretty simple, the attributes within targets can be compressed in the same way as the regular attributes, and the extension format for them can follow exactly the same principle. Because in glTF the morph targets are stored as displacements from the initial mesh, they are actually very efficient to compress, without changing anything in the Draco encoder itself :)

Please see the simple additions I propose to add support for morph targets in the extension text and json schema in 13c1d79.

To demonstrate the usefulness of this feature, you can have a look at :

• Implementing morph targets decompression in three.js GLTFLoader can be done in 19 lines (after the refactoring discussed below)
• Implementing morph targets compression in a simple experimental online tool (credits to @donmccurdy for the great viewer!) can be added in 43 lines
• A water animated mesh (credits to 3dartel for the model) can be compressed from 32.8MB to 5.6MB (total size as GLB). For reference, compressing only the attributes leaves a 31MB file (which is not even valid as there is no easy way at least in the javascript version to apply the indices permutation from the Draco encoder)
• Combining morph targets and skinning works great (credits to Rob Allen for the model). The overall savings (2.48MB -> 1.83MB) are not as good because of the included texture, but still better than not compressing the morph targets (2.34MB)

(you can use Pack ZIP then Save to have a look at the glTF json and data, or inspect gltfContent.gltf from the console)

Note that one of the reason why supporting this feature can be simple is by refactoring DRACOLoader as used within three.js GLTFLoader to be a transparent preprocess step when loading the data of the glTF accessors, rather than as a replacement for all the mesh primitives parsing code. The corresponding commit is 41b75f0 (with a later update in 14e91e8 after merging with the dependencies refactoring that was recently pushed to three.js). This design requires that each different compressed mesh uses unique glTF accessor ids, which is currently not an explicit requirement in the extension, but does make sense (given that information such as min/max must still be given for each attribute), and it is the case for all draco-compressed glTF created by current encoders as far as I could test. The added benefit is that this adds support for mixing compressed and uncompressed attributes without any code required. It does also make implementing a decompress tool quite simple (as the list of accessors would be unchanged, only bufferViews would be added back)
To make this explicit in the extension text, I propose to add a paragraph such as 2ff0ffa

[donmccurdy]

Great comments, thanks @JeremieA!

I'll leave aside the question of whether it is too late in the process — I suspect that is so, but would defer to others. Noticing that you've used a DracoEncoderModule.GENERIC attribute to encode the morph targets, there may be better savings possible with changes to the Draco encoder to factor in morph target and animation data as recognized types. That's probably on the longer-term roadmap for Draco, as mentioned in google/draco#126. If so, and given that it's easier to iterate on the encoding/decoder/loader code than on a published specification, it might be best to do this in a follow-up version.

These changes for implementation within THREE.GLTFLoader and THREE.DRACOLoader look promising in their own rights, as well. By the way, how are you doing .ZIP export? THREE.GLTFExporter hasn't added that option yet although we're interested. 🙂

The restriction on accessor IDs looks reasonable to me... is it considered OK for an extension to narrow the core spec when in use? I'm inclined to say this should be an implementation note, which we could still add after the extension is finished... This also buys me some time to work on implementation: I've been meaning to try to get DRACOLoader running in a Web Worker, with multiple workers in parallel for multiple meshes, which might get around the question of preprocessing.

[donmccurdy]

/cc @lexaknyazev for the question of accessor ID requirements.

[pjcozzi]

I think this is ready to merge. Any other feedback?

[donmccurdy]

Looks good to me. 👍

[donmccurdy]

About unique IDs on accessors — the pending three.js implementation does not require unique IDs, but I am fine with including the restriction in some form. Morph targets will be left for a future version of the extension (or new, separate extension, TBD) covering animation, and should be omitted from the proposed text.

A question about implications — suppose my model has three (3) compressed meshes, and each mesh is identical except for an application-specific attribute (say, _ROUGHNESS) that will differ between instances of the mesh. To use Draco compression, what can I do?

a. Duplicate all compressed data for each mesh, in a new Draco-compressed bufferView.
b. Encode multiple roughness attributes (_ROUGHNESS_0, _ROUGHNESS_1, ...) within a single Draco-compressed bufferView. All three meshes will reference the same Draco data, but will use different IDs into the compressed data for their _ROUGHNESS attributes.
c. Leave the _ROUGHNESS attribute uncompressed, pointing to a different accessor for each mesh.

If I understand correctly, (a) and (b) are fine, but (c) would be disallowed with this requirement.

[FrankGalligan]

@donmccurdy Yes a. and b. are fine, c. is disallowed. You can file a feature request for a future version if you think this use case is useful.

[donmccurdy]

I don't have any need for (c), just wanted to verify that restricting accessor IDs to be unique-per-mesh would have the implications expected. 👍

[pjcozzi]

Is this ready to merge?

[JeremieA]

Congrats on the official launch of this extension !

I will update my tools to reflect the final standard (disabling the morph targets support by default).

Regarding the attributes from multiple mesh instances, option (b) is what I implemented in my compressor (see here). All primitives sharing the same indices and positions data are compressed as a single Draco-compressed mesh, as I assumed that it would be most efficient. While testing this compression on various glTF assets, I did not notice this case of two primitives using different attributes while referring to the same positions and indices. I suspect most exporters would duplicate all attributes in this case. But I know of at least GearboxAssy from glTF-Samples-Model with two primitives referring to the same attributes (but different materials).

Sorry @donmccurdy for not answering your earlier questions. The ZIP export that I implemented was done as a completely separate code from THREE.GLTFExporter, but it shouldn't be too difficult to use something similar. The relevant code is here.