Add stencil support to spatial materials

[apples]

Adds stencil buffer support to spatial materials.

Linked Issues

• Supersedes [WIP] Added stencil function and depth function #78542
• Resolves Expose an intuitive subset of stencil operations godot-proposals#7174
• Bugsquad edit: Resolves Add support for depth function in spatial materials and/or shaders (GL_GREATER, …) godot-proposals#1298
• Currently integrates Add support for depth function in spatial materials #73527 The depth function design included here has deviated from its original PR.

Testing with the sample project https://github.com/apples/godot-stencil-demo

Screenshots

Standard outline, standard x-ray, and a custom outline effect.
Made entirely with the Material settings.

A more interesting x-ray example with a custom material.

ww_light.webm

A cheap light effect that mimics the lighting effects in Wind Waker.

campfire_480.webm

A simple stylized fire effect implemented entirely with StandardMaterial3D.

Example shader

shader_type spatial;
render_mode depth_draw_never, cull_back, unshaded;
stencil_mode write_depth_fail, 2;

void fragment() {
	ALPHA = 0.0; // Currently needed because we can't control color mask.
}

Differences from the design

• Added compare operator ALWAYS. The ALWAYS operator is needed for some effects.
• Added outline and xray presets ahead of schedule because it was easy and made testing easier.
  • The way that next_pass materials are handled is a bit strange right now, but I tried to make it as non-destructive as possible.

Currently implemented

• Added stencil_mode to shaders, which works very similarly to render_mode.
  • read - enables stencil comparisons.
  • write - enables stencil writes on depth pass.
  • write_depth_fail - enables stencil writes on depth fail.
  • compare_(never|less|equal|less_or_equal|greater|not_equal|greater_or_equal|always) - sets comparison operator.
  • (integer) - sets the reference value.
• Modified the depth_test_disabled render mode to be split into depth_test_{default,disabled,inverted} modes.
  • depth_test_default - Depth test enabled, standard sorting.
  • depth_test_disabled - Depth test disabled, same behavior as currently implemented.
  • depth_test_inverted - Depth test enabled, inverted sorting.
  • VisualShader now has special handling for depth_test_ modes: The disabled mode is kept as-is and presented as a bool flag, while the other two modes are presented as a enum mode dropdown which excludes the disabled mode.
• BaseMaterial3D stencil properties.
  • depth_test - Determines whether the depth test is inverted or not. Hidden when no_depth_test is true.
  • stencil_mode - choose between disabled, custom, or presets.
  • stencil_flags - set read/write/write_depth_fail flags.
  • stencil_compare - set stencil comparison operator.
  • stencil_reference - set stencil reference value.
  • stencil_effect_color - used by outline and xray presets.
  • stencil_outline_thickness - used by outline preset.
• BaseMaterial3D stencil presets.
  • STENCIL_MODE_OUTLINE - adds a next pass which uses the grow property to create an outline.
  • STENCIL_MODE_XRAY - adds a next pass which uses depth_test_disabled to draw a silhouette of the object behind other geometry.

Supported Renderers

• Forward+
• Mobile
• Compatibility

Depth Prepass

Stencil effects work well when rendered with a second opaque pass immediately following the current opaque pass. However, with depth prepass enabled, the depth information from any stencil materials is not available for screen-space effects. For instance, with SSAO enabled, opaque stencil materials have strong visual artifacts. Due to this, there might not be a good way to support opaque stencil effects when depth prepass is enabled.

This has been resolved by simply not supporting opaque-pass stencil-read materials.

GLES3

More work is needed for GLES3 support. The depth buffer attachment for the scene FBO must be changed to a depth-stencil attachment. Making this change without breaking existing projects will be difficult.

• See f2ec38d for an experimental implementation, and find concerns in this comment: Add stencil support to spatial materials #80710 (comment).

Update: GLES3 and therefore the Compatibility renderer are now included in this PR. It may potentially cause problems for XR, but requires further testing.

Sorting Problems

Currently the biggest annoyance with sorting, is that next_pass materials aren't necessarily rendered immediately after their parent. Users have to rely entirely on render_priority to get correct sorting.

Individual stencil effects which have material passes in both the opaque and the alpha passes will have more difficulty being correctly sorted. This might make effects such as portals and impossible geometry more challenging to implement.

[QbieShay]

Woahh awesome @apples ! Thank you so much! I am busy busy until next week but I'll review it then!

[Calinou]

Tested locally (rebased on top of master ff5c884), it mostly works as expected. MSAA, TAA and FXAA correctly smooth out stencil rendering as expected too.

In the demo project, the Wind Waker Light appears much darker when using the Mobile rendering method though. I assume this is due to the use of a RGB10A2 color buffer instead of RGBA16, but maybe this is due to the 2× color multiplier that has to be applied? It doesn't affect the x-ray outline on the second screenshot further below.

Forward+	Mobile

Forward+	Mobile

I also noticed the stencil modes don't feature syntax highlighting and autocompletion in the shader editor, unlike the render modes:

Shader syntax validation also doesn't work if the stencil type has anything written after write, even if it makes no sense:

Here, I don't get a shader compilation error but I logically should.

---

I also suggest changing the default outline color in the material preset to be opaque black (or opaque white), so that you can see it immediately after setting it up. Right now, the default color is fully transparent.

PS: Using a Material Overlay property, it's possible to have an object that has both an outline and x-ray at the same time 🙂

Also, if you're curious about how a stencil outline compares with an inverted hull outline. Inverted hull on the left, stencil outline on the right:

[Calinou]

Is it possible to create stencil shadows that don't appear through walls with this PR? Logically, it seems feasible, but I've tried to modify the Wind Waker light shader to no avail.

[apples]

Is it possible to create stencil shadows that don't appear through walls with this PR?

@Calinou My intuition says that stencil shadows would require the write operations INCREMENT/DECREMENT to be exposed, but there might be some clever way to implement them without those operators.

Right now, since the write operator isn't exposed, it's just set to REPLACE internally.

the Wind Waker Light appears much darker when using the Mobile rendering method

I noticed this too and haven't had a chance to debug it. It's something to do with how the raw color values are being mixed (multiply), it seems to be clamping the values to 1.0 at some point. The xray/outline effects in my demo just use standard alpha blending with regular colors, so the problem doesn't affect them.

EDIT: If we want to expose those write operations for stencil shadows, we'll likely need to expose the compare/write masks as well. This can't really be done in the material properties for bloat reasons, but we could expose them in the shader stencil_mode without any trouble. (edit again: this is assuming it's necessary at all, which it might not be)

[QbieShay]

Tested with my project, I didn't attempt windwaker lights because I have completely forgot how the setup for that is supposed to work.

I tested my "particles passing through hole in the ground" scene and also the uotline and xray mode.

My own particles worked great! I am glad that this syntax seems indeed to be an update from the full stencil one:

Outlines&Xray: works great! The default configuration however seems to use stencil reference 0 and not work, it started working as soon as i changed it to something that isn't 0 (maybe that has to do with the mask like you were saying @apples ?)

[QbieShay]

RE: stencil shadows

I have not seen a huge demand amongst users for this. Since it seems to add some complexity to the API, i'd leave it for a future PR in case there's need for it. I'd specifically wait to see if anyone needs those features also for other reasons.

The internet seems to think stencil shadows are a bit outdated ad not "worth it" with current programmable pipelines?
(based on this, https://www.reddit.com/r/opengl/comments/4mjkv1/comment/d3w8m28/?utm_source=share&utm_medium=web2x&context=3 )

[QbieShay]

Tested on my test project for stencil, looks good!

I've noticed this PR also adds depth test operations, which I'm assuming they're needed for the stencil write if depth fail.

While exposing the whole thing just for this may be overkill, I see no harm in adding it for the users, I think they'll find ways to use it (plus i think the requested windwaker thing isn't possible without it).

Thank you Apples! My comments on default configurations can be tweaked in a later PR (and doesn't need to be you since changing defaults is something good for a first contributor as well ^^ )

[Calinou]

The internet seems to think stencil shadows are a bit outdated ad not "worth it" with current programmable pipelines?

This is true, but sometimes art direction prevails above all else 🙂

[QbieShay]

Oh. Didn't know they looked different! I really know nothing of them

[Lielay9]

RE: stencil shadows

I have not seen a huge demand amongst users for this. Since it seems to add some complexity to the API, i'd leave it for a future PR in case there's need for it. I'd specifically wait to see if anyone needs those features also for other reasons.

Wasn't the original proposal #3373 made explicitly because of stencil shadows? I can attest to that at the very least one of the 40 thumbs ups was for those. If they won't be supported the proposal should probably be reopened. 👀

The internet seems to think stencil shadows are a bit outdated ad not "worth it" with current programmable pipelines? (based on this, https://www.reddit.com/r/opengl/comments/4mjkv1/comment/d3w8m28/?utm_source=share&utm_medium=web2x&context=3 )

I wouldn't put much weight to a 7 year old comment with 5 likes. As Calinou alluded to, stencil shadows have a distinct look and properties that are better fit for stylized games. I've commonly encountered them in platforming games, where the environment uses shadow maps and characters have down pointing stencil shadows. I could've swore some of the mario games worked like this but I can't recall which.

[QbieShay]

@Lielay9 could you test this PR, and see if it supports them? I don't understand how they work, so I can't quite test them myself.

You can go on the "checks" tab and at the right there is an "artifacts" menu drop down - you'll find a build for this PR there.

Wasn't the original proposal godotengine/godot-proposals#3373 made explicitly because of stencil shadows? I can attest to that at the very least one of the 40 thumbs ups was for those. If they won't be supported the proposal should probably be reopened. 👀

The proposal was indeed for stencil shadows specifically, but they cannot be done without stencil at all. It should be reassessed with this API.

[apples]

@QbieShay

The default configuration however seems to use stencil reference 0 and not work, it started working as soon as i changed it to something that isn't 0 (maybe that has to do with the mask like you were saying @apples ?)

Nothing to do with masks, it's just that the buffer is initialized to all zeros every frame, so choosing 0 as a default reference value is kind of useless. It could easily be changed to 1 as the default.

I've noticed this PR also adds depth test operations, which I'm assuming they're needed for the stencil write if depth fail.

Some stencil effects (Wind Waker lights in particular) will need the depth test operator exposed. Since #73527 seems like it's on track to be merged, I included it in this PR for testing purposes. I'm hoping that PR gets merged first so I can just rebase this one on top of it.

[apples]

Regarding stencil shadows:

The internet seems to think stencil shadows are a bit outdated ad not "worth it" with current programmable pipelines?

Modern shadows are typically based on shadow maps, which can be a bit fuzzy/pixelated since they are stored as textures. Stencil shadows, in comparison, use geometry-based stencil volumes instead of textures, so they have infinitely crisp edges. Some users will desire this for artistic reasons, typically because they're trying to follow the style of retro games.

If they [stencil shadows] won't be supported the proposal should probably be reopened.

I don't think that'll be necessary. This PR is a good foundation, and will be very easy to expand upon in future PRs.

Additionally, stencil shadows might require some other changes not directly related to stencils, such as exposing depth clamping, which is needed for infinite shadow volumes. Also, finding ways to generate the infinite shadow volumes in the first place and avoid culling them is outside the scope of this PR.

Rest assured: I am one of those people who wants stencil shadows. It is not something forgotten with this PR. It just might take multiple PRs to fully support them.

[QbieShay]

Some stencil effects (Wind Waker lights in particular) will need the depth test operator exposed. Since #73527 seems like it's on track to be merged, I included it in this PR for testing purposes. I'm hoping that PR gets merged first so I can just rebase this one on top of it.

@clayjohn ?

[Lielay9]

2023-08-24.01-26-39.mp4

Well, I guess this is best you can do now. Works if volumes don't overlap; a simple torus, for example, will break it.

Calinou My intuition says that stencil shadows would require the write operations INCREMENT/DECREMENT to be exposed, but there might be some clever way to implement them without those operators.

These are indeed necessary for shadows. Besides that, the only thing missing would be culling the shadows correctly which is outside the scope of this PR. Thereafter, stencil shadows are viable, if not exactly painless to use.

Extra note: Some primitive shapes seem to have small holes/gaps and therefore didn't play too nicely with the code generating the shadow meshes.

[Calinou]

Extra note: Some primitive shapes seem to have small holes/gaps and therefore didn't play too nicely with the code generating the shadow meshes.

Could you post your code somewhere? I think the current primitive meshes are fine as they are, although some of them have hard seams as you'd typically expect (e.g. boxes aren't smooth-shaded).

[Delsin-Yu]

Any solutions to the face separation for sharp corners? I havn't found any non-shader appraoch to this yet.

Because these faces are being extruded away from their vertex normals, and since these vertices have sharp edges, their normals are not rounded. There are several existing approaches to address this problem.

See Pixel-Perfect Outline Shaders for Unity/Handling sharp edges for more info.

Sharp edges in 3D meshes are achieved by duplicating vertices along the edge. Each side of this cube has four unique vertices with normals oriented perpendicular to those of adjacent sides. When we translate these vertices’ positions along their normals, it’s equivalent to exploding the sides of the cube.

1. One way to address this problem is to make the edges of the cube smooth, and bevel them. This will impact the look of the cube itself.
2. Another option is to use a separate mesh that has smooth normals exclusively for drawing the outline. This requires us to author or generate in a script a separate mesh for each mesh that has sharp edges that we’d like to outline.
3. Finally, we can store smooth normal data in another channel of the mesh we aren’t using, e.g. in the vertex colors.

[SomeRanDev]

Does this PR also expose the stencils to compute shaders in any way? Is there some buffer texture somewhere that can be sampled similar to how Depth is sampled using RenderSceneBuffersRd.GetDepthLayer()

At least for compatibility, it appears the stencil values are contained on the depth buffer?? But I'm far from an expert on this, so I'm not sure how it works. If there is a way to sample from the stencil buffer through a CompositorEffect (or even better, on gdshader), could someone document it?

[yoyotam3]

It was mentioned earlier in the PR that increment/decrement operations were not exposed, meaning it is not possible to implement stencil shadows. Is this still the case?

[QbieShay]

@shafnaz that's an issue with any mesh with sharp edges unfortunately.

@yoyotam3 increment and decrement are indeed not exposed. please open a proposal to detail your usecase, in particular what are the benefits of stencil shadows (linking other articles it's fine too). I suspect we'll run into the current limitation of reading being possible only in the alpha pass and where stencil happens in the pipeline, but I may be wrong.

We need to understand if exposing those feature works with the current limitations, otherwise we will expose them later down the line.

[QbieShay]

@JeeeJeee there's been no explicit work to expose to compositor, not sure if it works with compute.

[tmilker]

@QbieShay

in particular what are the benefits of stencil shadows (linking other articles it's fine too)

The reason increment is needed is almost as old as stencil shadows themselves: Without it, you get double(or more) darkening from overlapping parts of the model. I guess you could maybe get around it with an if in your shader but those aren't good for performance in shaders, especially fragment ones.

This is an image from a slide show by Mark Kilgard in 1998. Here's the full talk, still archived 27 years later.

This is just the simple version of stencil shadows too. For rendering shadow volumes properly, you increment and decrement the stencil buffer. See https://ogldev.org/www/tutorial40/tutorial40.html

[QbieShay]

Please put that in a proposal so it doesn't get lost

[yoyotam3]

@QbieShay, the benefits of stencil shadows themselves, as mentioned by others, are mostly stylistic. They do also allow for volumetric shadowing though which I don't think is as easily achievable with shadowmaps. I don't think only being available in the alpha pass should be an issue, given the current system can be used to implement wind waker-style lighting which uses the stencil buffer in a similar manner. But I'm not entirely sure, I haven't implemented stencil shadows before.

[yoyotam3]

I have created a proposal for the required features here, please tell me if there are any issues with it.

[QbieShay]

Thank you! No worries, style benefits are in fact very very important - we make games and games are art :)

[eimfach]

Will this be in 4.5 still ? Or is it there already ? Can't see to read smth about it..

[AThousandShips]

It is already in 4.5, it's mentioned in the beta1 release

[GustJc]

Is there any way to change the stencil ref with a shader uniform?
Or change it with code somehow?

I can do it using the StandardMaterial3D with material_override.stencil_reference for example.
But I can't seem to find any way to do it with a gdshader on a ShaderMaterial.
stencil_mode seems to be a static thing.

Is there any way around this? Or is there any open issues about this?
The same is true for the render_mode flag now that I think about it, so maybe this can't be solved.

Edit:
Also, is there any way to use stencils with CompositorEffects?
I'd like to apply post-effects selectively using CompositorEffect's glsl shaders.
But I'm not sure how that would work as of now.
Is this feature/issue tracked anywhere?

Edit2:
Also, is there any plans to let us sample the stencil ref of a given pixel of the screen in the shader?
It would be helpful for making shaders that does something different depending on the stencil ref on the screen.

[apples]

@GustJc

Is there any way to change the stencil ref with a shader uniform? Or change it with code somehow?

No, the ref value can only be set on the material. It cannot be set within a shader.

Also, is there any way to use stencils with CompositorEffects?
Is this feature/issue tracked anywhere?
Also, is there any plans to let us sample the stencil ref of a given pixel of the screen in the shader?

Not currently.
These are possible to implement, but just need a bit of careful work, since any implementation would also impact how the depth buffer is accessed.
I'm not aware of any trackers for these.

[zomby138]

I would really love it if there was some way to use the stencil buffer in a post process effect.

Right now I have a post process in a ColorRect with a canvas_item material applied to it. If I could just add:
stencil_mode read, compare_equal, 1;
To the top of a canvas_item shader then I would be happy.

[BurningFluffer]

Can we have write_depth_success as well? I'd like to modify the buffer after meshes that do get drawn. For example, for an outline that is then passed as a stencil value to a screen shader, which uses it as a blur mask. I reeeealy wanna make fluffy edges like that.
It would also be great if we could conditionally modify DEPTH of the fragment and then have stencil read/write after that. This would enable outlines that show over the mesh as well, kinda like in here. I would like to combine the best of both worlds (outlines on top and intersection outlines of stencils) but I can't seem to figure out a way with the current stencil implementation.
Also, is it possible to check stencil for one value, but then write another? As these are separate operations, I feel like they should not be tied to limit each other.

[zomby138]

Can we have write_depth_success as well?

I'm pretty sure that they are calling write is what you are calling write_depth_success I prefer your name, or perhaps write_depth_pass, which is what the description for write says.

The real problem is the lack of ability to then use the stencil in a full screen shader.

[GreenCrowDev]

Sorry if this is not the best place to ask, but is the stencil buffer not supported when using Alpha Scissor or Alpha Hash in a spatial shader?
Is this something that will be implemented in the future or it's impossible because of the pipeline?

The problem I'm encountering is that I'm not able to apply a stencil mask to a tree with alpha scissored leaves while preserving the correct ordering of the leaves, and also preserve shadows.

This is what I'd like to achieve (the circle that masks the leaves), but the ordering of the transparent leaves is incorrect, and there's no shadows.

[GustJc]

This is what I'd like to achieve (the circle that masks the leaves), but the ordering of the transparent leaves is incorrect, and there's no shadows.

The workaround I've found for this is to use Depth Pre-Pass with depth-write always (to preserve the mesh order).
And then set the transparency to 0.01.

That way, you'll have a working stencil read with shadows.

The only problem is that the shadows are not occluded and are always visible no matter if the stencil is showing the object or not.
I think we could fix this using the new IN_SHADOW_PASS in shader, but we'd need to have access to the stencil ref (stencil texture) inside the shader, which I we don't have yet.

I don't think it is possible to mask shadows with the stencil. (to have shadows only inside impossible geometry for example)
I don't know how the passivestar doors in the 4.5 post did this, I tried a lot of things. I think they faked it somehow, maybe they are baked into the texture idk.

[w2zyx]

@Randomised-gamedev don't worry about it too much, we just try to keep issues "tidy" so that all discussion that's visible is technical and about the feature itself. Nevertheless I am happy to hear you're enjoying stencil!

yeah but i like it will cool to make this with a game maybe i can get a idea But i need to brain storm but if i make using this now ill make it free and open soure

[murrayvrichard-ux]

Does anyone know of a way in the current system to deal with this overlapping issue?
Basically this "portal" is using the stencil effect to hide and show rooms through doorways. But the issue is that objects that are hidden get drawn over the doors, instead of only being seen "through" them. I have tried messing with the depth rules and render priority but they all seem to be ignored once it comes to the "read" and "write" steps of this system. Video included...
https://github.com/user-attachments/assets/c41e592b-2932-497f-9642-0bafd83f6233