Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add a Bevy-native tool to generate (and bake) global illumination for light maps #12233

Open
alice-i-cecile opened this issue Mar 1, 2024 · 4 comments
Open

Add a Bevy-native tool to generate (and bake) global illumination for light maps #12233

alice-i-cecile opened this issue Mar 1, 2024 · 4 comments
Labels
A-Assets Load files from disk to use for things like images, models, and sounds A-Editor Graphical tools to make Bevy games A-Rendering Drawing game state to the screen C-Feature A new feature, making something new possible D-Complex Quite challenging from either a design or technical perspective. Ask for help!

Comments

@alice-i-cecile
Copy link
Member

What problem does this solve or what need does it fill?

Light maps (cube maps, irradiance volumes, reflection probes) are a class of assets used to "fake" pretty but expensive global illumination. Bevy has support for these now, but as #10057 says, we have no way to generate them ourselves:

An easy way to generate reflection probe cubemaps is to bake them in Blender and use the export-blender-gi tool that's part of the bevy-baked-gi project. This tool takes a .blend file containing baked cubemaps as input and exports cubemap images, pre-filtered with an embedded fork of the glTF IBL Sampler, alongside a corresponding .scn.ron file that the scene spawner can use to recreate the reflection probes.

What solution would you like?

  1. Add features for global illumination to Bevy, even if they're slow.
  2. Add a tool to generate the lightmaps and export them as assets.
  3. Teach users how to use this workflow.

This is tagged with A-Editor, as while it is possible to do this using a non-GUI solution, it makes more sense as a component of a larger scene editor.

What alternative(s) have you considered?

Existing third-party light baking tools exist, and can be used with Bevy.

These are generally harder to customize / integrate, and cannot be used to dynamically generate lightmaps as part of procedurally generated environments.

Additional context

Pre-existing work:
@alice-i-cecile alice-i-cecile added C-Feature A new feature, making something new possible A-Rendering Drawing game state to the screen A-Assets Load files from disk to use for things like images, models, and sounds A-Editor Graphical tools to make Bevy games D-Complex Quite challenging from either a design or technical perspective. Ask for help! labels Mar 1, 2024
@pcwalton
Copy link
Contributor

pcwalton commented Mar 1, 2024

In order of difficulty, what would need to be done is:

  1. Write a reflection probe baker. This is pretty simple: just load a scene, render all 6 sides of a cubemap, run the results through the glTF IBL Sampler, and save the results to .ktx2.

  2. Write an irradiance volume baker. This is a bit more involved than (1), but still not too hard. The easiest approach would be to, for every voxel location, render a cubemap in which each side is, say, 32x32, and then average all the colors of every side together to reduce the cubemap to 1x1. This would work, but would be slow (however, it wouldn't be that slow: it would still probably only take a few seconds).

  3. Write a lightmap baker. This is a lot more involved. You'll want to start with some sort of GI solution, for example bevy_solari or strolle. For each lightmapped object, initialize a blank light map texture. Then path trace every pixel on that texture, and save the results as an image. As a followup, you'll probably want to integrate xatlas to generate the UVs and a denoiser like OpenImageDenoise.

@JMS55
Copy link
Contributor

JMS55 commented Mar 1, 2024

I made a PR to replace gltf-ibl-sampler, but it didn't work well on all environment maps and I've shelved it for now due to lack of motivation #9414.

@viridia
Copy link
Contributor

viridia commented Mar 4, 2024

I've contemplated posting my environment maps to Discord and asking if anyone would be willing to convert them :)

@viridia
Copy link
Contributor

viridia commented Mar 4, 2024

BTW, three.js has a PMREMGenerator written in JavaScript which does something similar to what we want:

https://github.com/mrdoob/three.js/blob/master/src/extras/PMREMGenerator.js

pcwalton added a commit to pcwalton/bevy that referenced this issue Jun 13, 2024
@pcwalton
Implement omnidirectional cameras for real-time reflection probes.
fc9e41c 
This commit introduces a new type of camera, the *omnidirectional*
camera. These cameras render to a cubemap texture, and as such extract
into six different cameras in the render world, one for each side. The
cubemap texture can be attached to a reflection probe as usual, which
allows reflections to contain moving objects. To use an omnidirectional
camera, create an [`OmnidirectionalCameraBundle`].

Because omnidirectional cameras extract to six different sub-cameras in
the render world, render world extraction code that targets components
present on cameras now needs to be aware of this fact and extract
components to the individual sub-cameras, not the root camera component.
They also need to run after omnidirectional camera extraction, as only
then will the sub-cameras be present in the render world. New plugins,
`ExtractCameraComponentPlugin` and `ExtractCameraInstancesPlugin`, are
available to assist with this.

Each side of an omnidirectional camera can be individually marked as
active via the `ActiveCubemapSides` bitfield. This allows for the common
technique of rendering only one (or two, or three) sides of the cubemap
per frame, to reduce rendering overhead. It also allows for on-demand
rendering, so that an application that wishes to optimize further can
choose sides to refresh. For example, an application might wish to only
rerender sides whose frusta contain moving entities.

In addition to real-time reflection probes, this patch introduces much
of the infrastructure necessary to support baking reflection probes from
within Bevy as opposed to in an external program such as Blender, which
has been the status quo up to this point. Even with this patch, there
are still missing pieces needed to make this truly convenient, however:

1. Baking a reflection probe requires more than just saving a cubemap:
   it requires pre-filtering the cubemap into diffuse and specular parts
   in the same way that the [glTF IBL Sampler] does. This is not yet
   implemented in Bevy; see bevyengine#9414 for a previous attempt.

2. The cubemap needs to be saved in `.ktx2` format, as that's the only
   format that Bevy presently knows how to load. There's no
   comprehensive Rust crate for this, though note that my [glTF IBL
   Sampler UI] has code to do it for the specific case of cubemaps.

3. An editor UI is necessary for convenience, as otherwise every
   application will have to create some sort of bespoke tool that
   arranges scenes and saves the reflection cubemaps.

The `reflection_probes` example has been updated in order to add an
option to enable dynamic reflection probes, as well as an option to spin
the cubes so that the impact of the dynamic reflection probes is
visible. Additionally, the static reflection probe, which was previously
rendered in Blender, has been changed to one rendered in Bevy. This
results in a change in appearance, as Blender and Bevy render somewhat
differently.

Partially addresses bevyengine#12233.

[glTF IBL Sampler]: https://github.com/KhronosGroup/glTF-IBL-Sampler

[glTF IBL Sampler UI]: https://github.com/pcwalton/gltf-ibl-sampler-egui
pcwalton added a commit to pcwalton/bevy that referenced this issue Jun 14, 2024
@pcwalton
Implement omnidirectional cameras for real-time reflection probes.
f0d12c5 
This commit introduces a new type of camera, the *omnidirectional*
camera. These cameras render to a cubemap texture, and as such extract
into six different cameras in the render world, one for each side. The
cubemap texture can be attached to a reflection probe as usual, which
allows reflections to contain moving objects. To use an omnidirectional
camera, create an [`OmnidirectionalCameraBundle`].

Because omnidirectional cameras extract to six different sub-cameras in
the render world, render world extraction code that targets components
present on cameras now needs to be aware of this fact and extract
components to the individual sub-cameras, not the root camera component.
They also need to run after omnidirectional camera extraction, as only
then will the sub-cameras be present in the render world. New plugins,
`ExtractCameraComponentPlugin` and `ExtractCameraInstancesPlugin`, are
available to assist with this.

Each side of an omnidirectional camera can be individually marked as
active via the `ActiveCubemapSides` bitfield. This allows for the common
technique of rendering only one (or two, or three) sides of the cubemap
per frame, to reduce rendering overhead. It also allows for on-demand
rendering, so that an application that wishes to optimize further can
choose sides to refresh. For example, an application might wish to only
rerender sides whose frusta contain moving entities.

In addition to real-time reflection probes, this patch introduces much
of the infrastructure necessary to support baking reflection probes from
within Bevy as opposed to in an external program such as Blender, which
has been the status quo up to this point. Even with this patch, there
are still missing pieces needed to make this truly convenient, however:

1. Baking a reflection probe requires more than just saving a cubemap:
   it requires pre-filtering the cubemap into diffuse and specular parts
   in the same way that the [glTF IBL Sampler] does. This is not yet
   implemented in Bevy; see bevyengine#9414 for a previous attempt.

2. The cubemap needs to be saved in `.ktx2` format, as that's the only
   format that Bevy presently knows how to load. There's no
   comprehensive Rust crate for this, though note that my [glTF IBL
   Sampler UI] has code to do it for the specific case of cubemaps.

3. An editor UI is necessary for convenience, as otherwise every
   application will have to create some sort of bespoke tool that
   arranges scenes and saves the reflection cubemaps.

The `reflection_probes` example has been updated in order to add an
option to enable dynamic reflection probes, as well as an option to spin
the cubes so that the impact of the dynamic reflection probes is
visible. Additionally, the static reflection probe, which was previously
rendered in Blender, has been changed to one rendered in Bevy. This
results in a change in appearance, as Blender and Bevy render somewhat
differently.

Partially addresses bevyengine#12233.

[glTF IBL Sampler]: https://github.com/KhronosGroup/glTF-IBL-Sampler

[glTF IBL Sampler UI]: https://github.com/pcwalton/gltf-ibl-sampler-egui
@pcwalton pcwalton mentioned this issue Jun 14, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-Assets Load files from disk to use for things like images, models, and sounds A-Editor Graphical tools to make Bevy games A-Rendering Drawing game state to the screen C-Feature A new feature, making something new possible D-Complex Quite challenging from either a design or technical perspective. Ask for help!
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@pcwalton @viridia @alice-i-cecile @JMS55