Add Custom Projection to Camera3D

[dxur]

Add custom_projection property to Camera3D

• Bugsquad edit, closes: Add custom projection mode to Camera3D, make Camera3D store its CameraMatrix and expose CameraMatrix to user code godot-proposals#2713*

[AThousandShips]

Can you test what happens if you create a projection that produces invalid output values, like NaN and what happens if they are injected into the renderer, it could cause serious issues (like what happens when you do the same for audio, where it cuts all audio)

[dxur]

Can you test what happens if you create a projection that produces invalid output values, like NaN and what happens if they are injected into the renderer, it could cause serious issues (like what happens when you do the same for audio, where it cuts all audio)

Actually this implementation affect just the projection matrix it works very well with oblique projection... , then it's for people that want to do some tweaks like 3d looks parallel in one dimensions, then it doesn't cause issues with viewports ..., I think it's so useful.
I started working on global shader overriding which will allow doing stuffs like curved world..., my problem now with the camera frustum.
Edit: you can't past Null value to set_custom_projection;

[AThousandShips]

What do you mean? I'm not asking for that, I'm asking what happens if you provide a broken Projection, that transforms values to NaN, the problem isn't "what limitations does it have?", it is "will adding an invalid projection mess up the rendering or crash?"

[dxur]

What do you mean? I'm not asking for that, I'm asking what happens if you provide a broken Projection, that transforms values to NaN, the problem isn't "what limitations does it have?", it is "will adding an invalid projection mess up the rendering or crash?"

I tested playing with the projection as i can it never crash the rendering.

[AThousandShips]

Try with something like: Projection.create_orthogonal(-1, 1, -1, 1, 0, INF)

If it causes problems then that would be helped by having the methods suggested in:

• Add helper methods to projection #77481

[dxur]

Projection.create_orthogonal(-1, 1, -1, 1, 0, INF)

I tested it, then it does not cause any problems, even with the frustum.

[PitouGames]

I also need this for my project, and I also have my own implementation (roughly the same): https://github.com/PitouGames/godot/tree/custom_projection

I don't understand why the projection matrix is accessible in the editor GUI in this PR after your comment: godotengine/godot-proposals#2713 (comment). User will still be able to show it with scripts like that:

extends Camera3D


@export var projection_matrix: Projection

func _process(delta):
	custom_projection = projection_matrix

There are the camera's functions set_perspective, set_orthogonal, set_frustum but the set_custom to switch the camera to the custom mode from code is missing (or I didn't found it).

Can we also take advantage of this PR to move the field "Current" up? It's in the middle of projection settings.

[Calinou]

Can we also take advantage of this PR to move the field "Current" up? It's in the middle of projection settings.

While we're at it, I'd move Current at the very top and move Keep Aspect just above FOV. The way FOV is interpreted (vertical or horizontal) depends on the value of Keep Aspect.

[dxur]

I don't understand why the projection matrix is accessible in the editor GUI in this PR after your comment

Actually I just make it accessible because it makes it a little bit harder to test some values theoretically then making it accessible from the editor is pointless as the projection must be recalculated if the viewport had resized but still annoying to export another projection as a script variable then make a setter..., I think it's OK now.

[AThousandShips]

See also:

• Permit overriding the projection matrix passed to shaders without affecting culling logic. #85529

[BrunkMortel]

Hello,
I stumbled upon a problem playing with this PR. It appear that shadows doesn't render correctly when the camera projection is set to custom and mimic orthogonal view (even if we just copy the projection matrix from another camera) :

It seems to work just fine when mimic a perspective view :

I'm sure that the two camera have the same projection matrix since I basically did :
$SubViewport2/Camera3D.custom_projection = $SubViewport/Camera3D.get_camera_projection()

I know that the problem as something to do with the shadow rendering but can't figure out what exactly ...

[dxur]

@BrunkMortel I'll check it because I don't have this issue in my projects, can provide which renderer are you using?

[BrunkMortel]

@dxur I'm using the Forward+ renderer and everything else is by default.
Here is a simple scene setup that show the probleme : simple_scene.zip

[Calinou]

Tested locally (rebased against master fa48a51) on NVIDIA. I get a Vulkan device lost error if I adjust the Camera3D's projection matrix in the editor while the project is running:

ERROR: Vulkan: Did not create swapchain successfully. Error code: VK_NOT_READY
   at: prepare_buffers (drivers/vulkan/vulkan_context.cpp:2491)
ERROR: Vulkan: Cannot submit graphics queue. Error code: VK_ERROR_DEVICE_LOST
   at: swap_buffers (drivers/vulkan/vulkan_context.cpp:2571)

This only occurs if SDFGI is enabled. VoxelGI doesn't have these errors but renders incorrectly if any values in the custom projection are modified.

Also, running a project with a Camera3D that has its projection mode to Custom prints the following error (even if run outside the editor):

ERROR: Camera is not inside the scene tree.
   at: get_camera_projection (./scene/3d/camera_3d.cpp:212)

Directional shadow rendering also breaks if you change any values in the custom camera projection, with the following errors printed:

ERROR: Condition "!res" is true. Returning: false
   at: get_endpoints (./core/math/projection.cpp:502)
ERROR: Condition "!res" is true. Continuing.
   at: _light_instance_setup_directional_shadow (./servers/rendering/renderer_scene_cull.cpp:2153)

Point light shadow rendering breaks with certain values too (even though the rest of the scene still renders), but not as often as directional shadows. There are also no errors printed in this case.

Testing project: test_pr_84454.zip

[BryceBarbara]

@dxur what's the status on this? Is there work to be done or are you waiting on more feedback?

[otonashixav]

@dxur I would like to continue working on this PR, since it's nearly complete and I would benefit from it.

[dxur]

I will fix all the issues that people face with this PR as soon as possible so I would like to be merged before 4.3

[Flarkk]

For completeness, exposing the camera matrix would not only potentially break depth-based rendering effects, but also physics picking and viewport stretching.

This short term approach I pulled together recently works around that : godotengine/godot-proposals#11436.

It may help going beyond the current blocking points.

[JNSStudios]

Just coming in here to bump this PR, as the game I want to make would require access to this to allow me to create a specific appearance akin to Enter the Gungeon. The solution in Unity (which Gungeon was made in) is to scale the vertical axis of the camera's projection matrix, which cannot be done in standard Godot.

[theraot]

I've tested this PR rebased against 1753893 (the last pulled from master at the time, post 4.4). I have not reviewed the code.

I'd wish switching from other camera modes preserved the projection better. Frustum seems to be only transferring correctly. Transferrin from Perspective or Orthogonal seems to change the aspect ratio. Also when switching from Perspective, the gizmo goes to the opposite side of the camera.

As expected - setting the projection in the inspector is not practical or intuitive. However, in my limited tests, it seems to work.

[AdAstraSaturn]

Is there a guide on how to use this pr?

[akien-mga]

I rebased this PR on latest master and squashed the commits, so it's easier to cherry-pick for users interested in using this feature.

The rendering maintainers still have concerns about the mergeability of this PR as is as there's a number of engine features which break when using custom projections, in ways which users would rightfully consider to be bugs.

Still, this is a highly requested feature and we'd like to find a way to eventually merge a version that's "good enough" for common use cases, with documented caveats.

I also included a commit by @reduz which fixes picking/raycasting with custom projection matrices.

[Glorax]

@akien-mga What does this recent activity mean? Is this nearing readiness for merging to the main branch?

[OeilDeLance]

@Glorax I don't think this is gonna be merged anytime soon. As far as I know it is because it breaks some advanced sub systems for 3d rendering and perhaps more.

We are using it in my company in a 3d game that does not make use of advanced rendering and it does a good job.
Perhaps someone more involved with dev on this PR could list us what subsystems don't play well with it ? Perhaps @reduz ?

That could be useful.

[akien-mga]

@akien-mga What does this recent activity mean? Is this nearing readiness for merging to the main branch?

The status for this PR is still what's documented above in #84454 (comment).

I simply rebase the PR when it has conflicts (I just did again) as I know a number of users rely on this implementation, and I want to make it easier for them to cherry-pick it in their fork on top of Godot 4.5-stable or 4.6-dev.

See #110850 for a newer attempt at addressing the use cases of custom camera projection matrices which has recently been discussed by Rendering team contributors.

[huwpascoe]

We are using it in my company in a 3d game that does not make use of advanced rendering and it does a good job. Perhaps someone more involved with dev on this PR could list us what subsystems don't play well with it ?

On testing #110850

• sky
• fog
• scene shader
• LOD
• SSR
• SSAO
• SSIL
• SSS
• TAA
• FSR

At this point, there doesn’t seem to be anything blocking merge aside from a decision to move forward.

[Glorax]

@AThousandShips @akien-mga @Calinou The whole of October and November have passed without anyone reviewing this. Is there any reason this should remain blocked?

[clayjohn]

@AThousandShips @akien-mga @Calinou The whole of October and November have passed without anyone reviewing this. Is there any reason this should remain blocked?

Its not the direction we want to go with this feature and its not finished even if this was the way we wanted to expose a custom projection matrix.

Unfortunately huwpascoe is not being fully honest above and has falsely created expectations about this PR being mergable. If custom projection matrices are used in this way the following effects won't consistently work:

• XR
• Volumetric Fog
• SSR
• SSS
• SSAO
• SSIL
• LOD
• HLOD
• TAA
• FSR2
• MetalFX temporal upscaler
• Custom sky shaders

Additionally, the Forward+ renderer will have bugs with:

• spotlights
• omnilights
• decals
• reflection probes

What makes this approach problematic is that it will appear to work correctly when using a custom projection matrix that is very similar to a regular perspective matrix, but for many different custom matrices, the above effects will just completely break.

To be clear, this PR could be improved to support all the things listed above. But that would be a huge amount of work and clearly the OP isn't willing to do that work since they don't need all those features for their own project. Ultimately though, even if all those cases worked, this approach would result in a huge amount of future maintenance burden for us and would create a lot of tricky bugs for us to solve.

Nobody wants to spend the next 6 months fixing weird edge cases when features break due to users using non-standard projection matrices so we have agreed to go in a different direction that requires more work up front, but will save us (and users) a lot of pain in the long run. That approach is described here: godotengine/godot-proposals#11436

[Glorax]

@clayjohn It seems @huwpascoe incorrectly commented in the wrong thread. It seems they were talking about their own PR. How does that one fare?

I don't know why @huwpascoe commented on @dxur's implementation instead of their own, but even if this one isn't in a good enough place, is the other one?

[Glorax]

@clayjohn Another two things I don’t understand. The commits on @dxur's implementation show that @reduz made contributions to it as recent as this October. If this PR is as hopeless as you say, and will never be merged, then why is Juan spending any time on it at all? That fact that Juan contributed to it suggests he still thinks this PR is worth developing.

And @huwpascoe's PR identifies itself from the start as a "starting point" for the different direction you have agreed to go in. It's the closest thing existing to an attempted implementation of the "practical approach". They seem to be the only person who has attempted to do so, and I'm not sure why you didn't mention that they are ostensibly following your proposal.

[clayjohn]

@clayjohn It seems @huwpascoe incorrectly commented in the wrong thread. It seems they were talking about their own PR. How does that one fare?

I don't know why @huwpascoe commented on @dxur's implementation instead of their own, but even if this one isn't in a good enough place, is the other one?

Its further away from being ready than this one is.

@clayjohn Another two things I don’t understand. The commits on @dxur's implementation show that @reduz made contributions to it as recent as this October. If this PR is as hopeless as you say, and will never be merged, then why is Juan spending any time on it at all? That fact that Juan contributed to it suggests he still thinks this PR is worth developing.

We had a client that wanted to use it but needed support for ray casting. So Juan added support for ray casting to this PR so they could use it for their game. They didn't need support for any other features, so Juan didn't add them. However, in order for us to merge something and provide it out of the box for all Godot users it needs to support all features.

And @huwpascoe's PR identifies itself from the start as a "starting point" for the different direction you have agreed to go in. It's the closest thing existing to an attempted implementation of the "practical approach". They seem to be the only person who has attempted to do so, and I'm not sure why you didn't mention that they are ostensibly following your proposal.

Unfortunately that PR doesn't follow the proposal any more than this one does. It just implements one way of setting a custom projection matrix that we discussed in a rendering meeting one time. I'm not sure why you think I should have mentioned it earlier.

[Glorax]

@clayjohn

Unfortunately that PR doesn't follow the proposal any more than this one does. It just implements one way of setting a custom projection matrix that we discussed in a rendering meeting one time. I'm not sure why you think I should have mentioned it earlier.

I wrote that reply in haste. I regret not having phrased that better. I just am confused why your initial reply didn't explain the following discrepancy:

Unfortunately huwpascoe is not being fully honest above and has falsely created expectations about this PR being mergable.

Nobody wants to spend the next 6 months fixing weird edge cases when features break due to users using non-standard projection matrices so we have agreed to go in a different direction that requires more work up front, but will save us (and users) a lot of pain in the long run. That approach is described here: godotengine/godot-proposals#11436

And now we go to @huwpascoe's proposal:

Starting point for godotengine/godot-proposals#11436

This is why I am confused. Because to me it seems that huwpascoe is the only person currently attempting to implement godotengine/godot-proposals#11436, and I'm not sure why that wasn't mentioned in your initial reply. You say that it doesn't follow the proposal any more than this one does, but how can that be the case when huwpascoe claims their work is a starting point for said proposal? That seems like a very important thing to mention, even if huwpascoe isn't being fully honest about it, and has failed to properly implement the proposal.

You also implied that neither of these PRs is salvageable, but why? If the idea is to limit the alterations that can be applied to the camera matrix, wouldn't said limits simply be added on top of the existing work? Why not communicate this to huwpascoe directly and tell them that the entire proposal must be followed before anything can be merged?

Your proposal has existed for just under a year, and as far as I'm aware, nobody attempted to implement it until huwpascoe created their PR a few months ago. And already it's been judged not just merely incomplete, but fundamentally flawed. I guess I am just confused, but it seems like nobody has directly told huwpascoe there was a problem with the PR. Is it just supposed to sit there gathering dust until people forget about it? Is there nobody ensuring this proposal gets implemented according to the guidelines? I don't understand why this PR has sat in a flawed state for several months while people eagerly wait for it to merge, which they think it will soon because of huwpascoe's prior comments. Yet nobody has come in there to disabuse them of this notion.

The most recent comment on huwpascoe's PR is from yesterday, and it reads:

Just tried this PR alongside godot-rust and it's working perfectly for my purposes! Thank you so much, can't wait to see this merged.

But nobody has seen fit to break it to these people that the PR will never be merged? And it seems nobody has told huwpascoe what they must to do to get the PR to a mergable state either.

[Flarkk]

I am the author of godotengine/godot-proposals#11436.

Chiming in to react to

And now we go to @huwpascoe's proposal:

Starting point for godotengine/godot-proposals#11436

huwpascoe isn't being fully honest about it, and has failed to properly implement the proposal.

Please let's assume positive intentions to every contributor. Each attempt - even incomplete - has value and can serve as foundations for the next ones.

If the idea is to limit the alterations that can be applied to the camera matrix, wouldn't said limits simply be added on top of the existing work?

This is the exact idea of the proposal. It suggests that the limits are implemented as an interface between the user and the matrix, at both UI and API levels, in order to restrict degrees of freedom.

I see no reason this PR cannot evolve to something closer to the proposal, by adding this interface. I'm able to help anyone willing to take a stab at it, as I used to do with @huwpascoe.

[huwpascoe]

Yeah, I've just checked in here from the other thread. SIGH I'm not being dishonest, you haven't actually tested any of those features have you? I have, and they work. You only think they don't for whatever reason. Try it and see.

Edit: Just realized this isn't my PR, lol. But still, check my PR. Actually test it. You'll see everything I've checked off works.

[Glorax]

@Flarkk

Please let's assume positive intentions to every contributor. Each attempt - even incomplete - has value and can serve as foundations for the next ones.

I was only echoing what ClayJohn said earlier, and I intentionally phrased it as a hypothetical. I didn't want to think the worst of huwpascoe, but I also didn't want to outright contradict the head of rendering on a matter of which I knew nothing. And he was fairly unambiguous in his assessment:

Unfortunately huwpascoe is not being fully honest above and has falsely created expectations about this PR being mergable.

And he later went even farther than that in his extensive comment to huwpascoe, which I assume you're aware of by now: #110850 (comment)

We don't merge unfinished work and call it experimental. Full stop. Laziness is not an excuse to release unfinished code. As a rule, we only merge features once they are working and everyone is happy with the implementation. This PR is nowhere near being finished, you are refusing to make the changes suggested by Bastiaan and it cannot be merged until those changes are done.

For the record, I do not endorse the tone or content of this message.