Implement the Vulkan Kompute framework for GPU-accelerated machine learning and advanced data processing

[axsaucedo]

Describe the project you are working on:

Vulkan Kompute is a GPU processing framework built to augment Vulkan's computing capabilities, standardising the boilerplate code identified across a broad range of projects that aim to bring Vulkan to enable mobile and cross-vendor GPU compute capabilities.

I have added an integration with the Godot Engine, both through the GdNative Library and the Godot Custom Module approaches. Both of these approaches are outlined in this blog post I released recently.

I would like to propose how GPU accelerated processing could be exposed through the Godot game engine whether as tools or best practices. This would be an an opportunity to further the discussion around leveraging the underlying Vulkan capabilities specifically for the purpose of GPU compute. The current implementation I shared above is built on Godot 3.2.x, so it doesn't leverage the Godot Vulkan instance - it would be interesting to explore what are the ways it could be integrated, and more generally how some of these workflows for GPU accelerated processing to be exposed for developers.

Here is also another article I wrote that covers this from a higher level, talking about the design principles of the framework, as well as the underlying implementation intuition on the ML algorithm used.

Describe the problem or limitation you are having in your project:

Currently it's not clear what's the best way to integrate with Godot, hence why it would be interesting to explore further.

Describe the feature / enhancement and how it helps to overcome the problem or limitation:

This would involve providing an option for Godot developers to leverage not only the underlying Vulkan Instance, but more specifically leverage the compute features of Vulkan through interfaces that encourage best practices, and reduce repeated/boilerplate code.

Describe how your proposal will work, with code, pseudocode, mockups, and/or diagrams:

The architecture of kompute augments the Vulkan architecture, and introduces terminology that provides a GPU compute focused approach. Currently the way that it is integrated with Godot is quite superficial, but it would be interesting to explore what would be the best practices when developers want to leverage the GPU compute requirements using the Vulkan API. This would be quite powerful espcially with the cross-platform / cross-vendor GPU support that vulkan provides.

If this enhancement will not be used often, can it be worked around with a few lines of script?:

It would be possible to extend the current implementation to enable for CPU specific implementation of the compute backend, however it would depend on what level of abstraction this is to be introduced.

Is there a reason why this should be core and not an add-on in the asset library?:

I think there may be an opportunity to explore this across both areas. It would be interesting to explore if the Vulkan instance can also be exposed through GdNative Libraries, such that other compute applications can also leverage the underlying vulkan instance/appliction. Similarly, it would be useful for developers that want to use GPU acceleration for compute specific tasks, to have some best practices or guidelines when performing these integrations. The initial results integrating Vulkan Kompute has provided some workflows that could be abstracted for individuals that are interested to integrate it into their game development workflows

[fire]

How does one use Kompute when there's an existing system in tensorflow or pytorch?

https://github.com/fire/deep-motion-editing This is my fork of the library made compatible with windows pytorch cpu.

My approach was to use onnx-runtime, but it was too difficult to do in one session.

[axsaucedo]

@fire very good question (and interesting project, thanks for sharing). I would say there are two parts to your question that will be important to address, namely:

1. Support for advanced data processing & ML in the GPU with Vulkan
2. Support & integration with existing ML libraries

1. Support for advanced data processing & ML in the GPU with Vulkan

This is partially answered by the "Motivations" Section. Basically it states that Kompute "started after seeing that a lot of new and renowned ML & DL projects like Pytorch, Tensorflow, Alibaba DNN, between others, have either integrated or are looking to integrate the Vulkan GPU SDK to add mobile GPU (and cross-vendor GPU) support".

This should provide an insight of where Kompute sits - it's currently designed to be an underlying backend for higher level interfaces that would expose GPU compute capabilities. This is why it focuses on a level that could be quite interesting in context of providing Godot developers with access to the compute capabilities of Vulkan. Of course, there are a broad range of things to consider when it comes to an underlying compute layer for Godot, which is why at this point this discussion would most certainly go beyond Kompute itself, and initially be more of a design and conceptual discussion on what level could make most sense to expose this functionality to developers.

2. Support & integration with existing ML libraries

This point may or may not be directly relevant, but it would pertain to the higher level interoperability and bindings with existing libraries. In theory it would be less about the underlying GPU compute capabilities that should be exposed to the developers, and more around the lines of what is the best / most efficient way to create bindings and integrations with existing libraries. I have given it some thought, and there is also a broad range of options for this - from building C++ bindings to the relevant frameworks, to using the Godot Python capabilities - either way, as suggested in the referenced issue #30613 it seems to fall more under the plugin/module territory.

[GeorgeS2019]

@axsaucedo

to using the Godot Python capabilities

Microsoft is now seriously look into and has started to integrate Tensorflow.NET as a c# binding option to the tensorflow 2.x

I agree, this is under the plugin/module territory. This will provide the C# godot community to have access to an increasing more complete coverage of the tensorlfow 2.x with GPU support in Windows platform and eventually to mobile device through TensorflowLite binding.

=> A recent comparison studies (watch the youtube link above) have shown that the C# Tensorflow.NET out perform python tensorflow in Window platform.
=> This is to support there is merit to use the C# Tensorflow.NET (implemented as Godot Plugin) as an alternative to the Godot Python capabilities.

[WilliamTambellini]

Hello @axsaucedo
Although I would also encourage godot to move toward the "machine learning pattern" whatever the task (physics, animation, rendering, ...), I dislike the fact that such a module/plugin rely entirely on 1 technology (here Vulkan) targeting mainly only 1 type of hardware (GPU) considering:

• the godot current architecture is doing its best to be backend independent: that s what Godot call a 'server'. Some rendering 'servers' (GLES, vulkan, ..), physics does the same (godot physics engine, Bullet, and possibly other engines), same for audio, ... :
  https://github.com/godotengine/godot/tree/master/servers
• the godot src code base already embed a DNN library :
  https://github.com/godotengine/godot/tree/master/thirdparty/oidn/mkl-dnn
  now called
  https://github.com/oneapi-src/oneDNN
• Vulkan is as today 1 standard/technology, similar to GL/GLES.

Consequently, I would advice for such project to opt for a 'server' like API to be more generic meaning backend neutral : multiple implementations being possible based on different third parties techs (VulkanKomp, oneDNN, ONNX, ...).
Consequently, should nt the first step be to design a generic Godot "MLServer" or "DNNServer" API ?
Kind
W.

Refs:
https://github.com/Unity-Technologies/ml-agents/tree/latest_release

Some additional refs showing the arrival of ML/DNN in game engines:
Ubisoft replacing newton based physics engine with some DNN:
https://www.youtube.com/watch?v=atcKO15YVD8
Character animation:
https://www.youtube.com/watch?v=7c6oQP1u2eQ
Models generation:
https://www.youtube.com/watch?v=0JEb7knenps
Clouds rendering:
https://www.youtube.com/watch?v=Hk--xpI9b5U
...