All right.
Good afternoon, everybody.
I hope you're having a good Unreal Fest so far.
My name is Kevin, and together with Thais and Vignesh
from CD Projekt, we will take you behind the scenes
on the topic of large-scale animated foliage,
specifically in the context of the Witcher 4 Unreal
Gen 5 tech demo.
This is the demo we showcased at Unreal Fest Orlando
just a few months ago.
So the story arc of today's session
is us telling you about the difficulties of rendering
foliage to early experiments, leading into solutions,
and finally talk about how this collaboration and demo
culminates in engine features for you guys to try out in 5.7.
So before we kick things off, here's
a brief snippet from the demo just to set the tone.
It is running on a base PS5 at 60 FPS
with all the bells and whistles that UE5 has to offer.
So a bit of a spoiler for those who haven't seen it.
This is kind of the middle section of the demo
where we go on rails and we just kind of fly through,
look at all the awesome foliage and vegetation tech
that we built.
And in case you haven't seen it, it's online.
You can see it in its full glory.
And as we move up here, we're going
to catch up with our protagonist and continue
the demo, but we're going to skip that part for now.
So I'm going to hand it over to Thijs to start us off
with the challenges of rendering large-scale animated forests.
And I'll be back at the end to tell you more about 5.7 stuff.
All right, take it away, Thijs.
THIJ SINGHALAVYSKI- All right.
So from the very start, we set ourselves two goals.
First of all, we wanted the player to feel
like they're in a huge forest, but we also
wanted the forest to feel alive.
So basically, it responds naturally to forces
from the environment, especially wind.
And we wanted this to work not just on a small scale,
but really push it as far as we can.
But first, we had to start with basically from nothing,
because there was no nanite foliage at that time.
So why is foliage hard?
Well, foliage has always been a challenge in video games.
The state-of-the-art solution hasn't changed much
in a very long time.
There were some improvements made, for sure.
But fundamentally, we were making trees and foliage
the same way.
We were using cards.
We were using alpha testing.
And many of the advantages that Nanite gave,
they didn't apply to foliage.
So first, we tried cards.
We started with the known solution.
On the screen, you can see the same tree made with cards
and made with geometry.
It's the reference mesh.
So the cards approach, I would not recommend it in any case.
It's still around three to four times slower
than trying to mesh it out and let Nanite handle the geometry.
But if you do use cards, we found that it's, well,
Try to make the geometry match the cards as closely as possible.
So don't have big planes.
Try to cut it out if you can.
But generally, we recommend staying away from alpha testing.
So why is alpha testing slow?
Well, especially in the software resorizer,
the main reason is the barycentrics.
So Nanite, it handles all the geometry
in one fixed function pipeline.
But with alpha testing, this is no longer possible.
The reason it's slow is that in the inner loop
of the software rasterizer, we need
to calculate the barycentrics to get the UVs
and then sample the alpha mask.
You can see this Nanite rasterization common
file for the logic.
But because it's in the inner loop of the software rasterizer,
there's a heavy performance penalty to pay.
That is not easy to get around.
So we tried the other approach, we tried geometry.
This had some advantages.
We didn't use alpha, it was faster to render
because it didn't have this problem with the barycentrics.
It also looked better because we didn't have to fake the normals,
which vegetation artists have gotten very good at over the years But this kind of gave us good server scattering and all that out of the box But there was another problem mainly the disk size and the RAM So Nanite geometry it rendered quite fast
but we were using up to 63 megabytes of VRAM for a single tree
versus 6 megabytes of cards.
So a 10x increase in memory usage and even more on disk, a 30x increase.
so we would be shipping a huge game if we made all our foliage this way.
Next, there is also just the inherent problem of foliage,
which is that it's hard to build good LODs for it.
Nanite has the preserve area flag, which is an absolute must for foliage,
otherwise it will just disappear, but even with the flag,
it doesn't do a great job of keeping the shape at the distance.
As you can see here on the left you see a rock and on the right you see a tree.
So clearly the quality of the rock LOD is much higher where the tree basically becomes a triangle soup.
And that's just an inherently hard problem to try to model a tree with a very low triangle count.
Plus all the holes that you can see in the geometry, they also cause excessive overdraw.
So it's not just worse lots, they're also more expensive.
And then we have animation.
So we started out, again, with kind of industry best practice,
which was vertex shader-based wind.
There was, you know, friendly out-of-the-box solutions available.
We started with Pivot Painter 2, which was a vertex shader-based wind,
where the artist would store basically in the textures,
the pivots, and the hierarchy of the tree
that we then evaluate at runtime in the vertex shader.
Actually, it reminded us a lot of skeletal animation
how this approach was doing it,
but it was all happening in the vertex shader.
As you may remember from earlier slides,
if we go with geometry, we have a lot of vertices.
Even with cards, we're around 100k,
but with geometry, over a million vertices
that need to be transformed at runtime.
So it was quite clear that the work that we need to do
for each vertex needs to be absolutely minimal.
And there's another problem, which
is that using vertex shader logic to move things around,
or world position offset, the way to handle the cluster
bounds in the night is that you need to inflate them.
So in this case, we can imagine the tree on the right
being our kind of reference tree,
and the rotated tree is the one that's animated.
The little green box that you see,
those are the cluster bounds of that particular cluster up top,
which is used for culling within nanite,
and it is very, very important for performance.
Now, if we want to move those vertices to that point,
we have to inflate those cluster bounds
so that they fit the original position.
So that means they need to be absolutely huge.
And while you may say, well, foliage is not good
at occlusion culling anyway, also remember
that these cluster bounds are used for invalidating VSM
pages.
So clearly, vertex shader animation
was not the way to go.
So we needed some new solutions.
We saw potential in Nanite, especially
of geometry, but it was also clear that we had to develop
on it to make it work.
So one of the first things we did was skin foliage.
This was quite quickly proving to be a success.
We switched from doing everything in the vertex shader
to a two-pass system, where we first
calculated all the bones, their transformations,
did animations on a compute shader,
and then in the vertex shader just move the vertices around.
This was quite cheaply done.
We also went with rigid skinning.
This was a good enough trade-off for us
where we only had one bone influence one vertex.
This could also result in potential benefits later,
such as with ray tracing.
And we ended up using skin foliage in the demo We also tried to make alpha testing work We didn go full in on geometry right away
The first thing that we tried was Nanite, but with custom LODs.
So basically, we were using Nanite as it works out of the box,
but we had manual LODs at certain distances.
This did bring good perf improvement, around 20% to 30%,
But it also came with downsides like brought back popping
and all the other issues associated with cards.
So this was not enough for us to achieve that vision
that we had.
Then with the help of Epic, we also
tried to optimize the software restorizer
as much as we could to make the alpha testing as fast as we can.
And significant improvements were made,
but we started hitting diminishing returns.
So at that point, it was kind of clear
that alpha testing was just a no-go.
One other thing that we considered but didn't pursue
was inspired by opacity micro maps.
If you're familiar with it, it's a technique
used in ray tracing to speed up the ray tracing of cards
and alpha testing.
This idea could be transferred to the Nanite software
rasterizer to basically get rid of that expensive
barycentric calculation that I was talking about earlier and to use a more
efficient way of well basically checking if this pixel is opaque or not. But in
the end we didn't pursue this direction because the geometry was showing more
potential. So there's no alpha-tested foliage in a demo anywhere, not even the
small foliage. Everything is using the same pipeline and nowhere did we have to
use cards.
So back to geometry.
So fully modeling a tree was not really
feasible due to memory costs.
But actually, the artists were already
constructing trees in a very modular way,
where they were using node systems
to combine different pieces and scatter them
around to create a tree.
So this was something we could make use of.
So the first thing we tried was, well, why not just use
instancing, right?
Just instance a bunch of tree parts all over the place
and try to construct trees with that.
It worked quite well memory-wise,
but we quickly hit some limits.
For example, just hitting the instance limit in the GPU scene
due to the very large number of parts that we had to instance.
And we also had to pay for the cost
that comes with an instance.
And Rails stores instances in the GPU scene,
which actually stores quite a lot of data.
And really, we didn't need this data.
What we needed was very lightweight instances.
So now I will hand it over to Viknesh,
who will explain how we overcame these limitations.
So Viknesh.
All right.
So we could not use the regular instancing,
like I mentioned, because of the instance limit.
So we came up with virtual instancing.
So it involves composing a tree out of lightweight instances
out of a set of archetype meshes.
These lightweight copies, they do not
count towards the instance limit, so are very free.
What we see here is the set of archetype meshes
that I mentioned before.
We compose this tree entirely out of this set of meshes.
These do not count towards the instance limit,
and they're quite lightweight.
And this is the structure we use to store
those lightweight instances.
We just have a opaque pointer towards the archetype mesh,
a pointer towards a transform, and the bone it belongs to
in the skeleton.
So I think right now we fixed it, right?
Because right now the memory is pretty good.
We don't affect the instance limit.
But we ran into another issue.
This is the Nanite clusters visualization that you can see.
So what you see right now is the clusters stopping to simplify.
They have reached the closest map right now in the MIP type.
So those clusters which are gray, they cannot simplify anymore.
And they do it pretty quickly.
This is because each of this virtual instance it a separate entity these virtual instances do not have any kind of geometry connections with each other So they do not combine together to form a simplified continuous geometry basically
And one of the side effects of this was the explosion in the number of virtual instances.
That's the first thing.
The second thing was the explosion in the triangle count.
In especially larger scenes, we have very small, like 128 triangle clusters all over the place.
And this was causing performance issues.
So this is the Nanette instances visualization.
What we tried next was basically creating levels of detail for these trees and swapping
them in based on the distance from the camera to avoid this explosion of triangles.
So these are what we call the LODs for the virtual instances basically.
I'll go back and front to see the not just the amount of virtual instances, but these
levels of details are much less in number of triangles as well.
Let's look at some numbers.
So what you see here is the amount of time taken by the GPU to fill this buffer.
And on the y axis, we see the distance from the camera from the near to the far.
And as you can see here, near, there is not much difference between these three cases.
The first case is a tree without any kind of LODs.
The second is the tree with the LODs.
And the third is the reference mesh, which is the fully modeled tree.
In the nearby case, there is not much differences between these because things are still simplifying.
as the distance increases the tree without simplification has this huge explosion in
the number of triangles in the scene and this contributes to the very big numbers in the
WIS buffer. So we can see the clear winner here is both the reference tree and the tree
with simplification. They are pretty close in all those cases. But now let's take a look
memory. So it's a similar case but this is the 9.8 streamed in memory and as you
can see the reference tree mesh up close it's streaming in so many high quality
clusters from the MIP tile that it takes almost 165 megs while the tree with
simplification just has to pay for the architect meshes as well as the LODs.
But we did not end up using virtual instances in the demo as is.
It's not because it is the failure of the tech,
but we wanted to do it properly with
Epic with better integration into the engine
without any kind of manual load generation that we saw.
So the second thing,
Thais previously mentioned about distance representation,
but we have poor shape preservation.
with triangles, even with preserved area turned on, as well as inefficient
inefficiency due to holes in the aggregate geometry that is poor occlusion
curling and generally trying to make a model out LODs for aggregate geometry is
quite difficult. So we wanted a different representation and point clouds is
something we tried out. So what we did was during the build time when the import
when the foliage is imported. We created point out clusters of the virtual
instances. At runtime, we just replaced the virtual instances with the point
close clusters at a certain distance from the camera. We also added a pass
that these point load clusters are just ladded onto the Nanite Vis buffer.
Visually, it looks really good.
It preserves the area much better.
The one on the right is the point clouds, while on the left is the triangles.
As you can see, the area preservation is much better.
But we did not end up using this in the demo due to performance reasons.
This is because of the nature of how we write, how we do atomic writes to the WIS buffer.
We are writing to the whisper for all over the place with this new point clouds.
It was causing too much traffic in the memory controller,
and that was causing lots of performance issues.
So we decided not to go this route.
So, what we ended up doing with the tech demo.
We did three things.
The first is voxels, that is for the better distance LOD in terms of efficiency as well
as shape preservation.
The second is assemblies for the reduced disk size, as well as at the same time having the
ability to have high fidelity assets.
And the third is fast and efficient skinning.
So let's have a quick overview on how voxels are used.
So all these subpixel triangle clusters we have during the 9.8 build process are now voxel clusters.
That is a simplification error for each cluster.
And if it falls below a certain threshold, then instead of having them as triangles, we make them into voxels.
This happens during the 9.8 builder process.
These voxels are stored in 4x4x4 bricks and it's represented by a 64 bit field basically.
And these bricks are traced at runtime during the 9x8 trasterization pass depending on if
a cluster is a triangle or a voxel.
And these bricks are actually bent in depth buckets from front to back for early depth
listing.
So this helps a bunch with old raw as well.
And the video you see right here, it's been exaggerated to show the voxels.
So in practice you won't see the voxels because they're usually subpixel.
Next is assemblies.
So what you see here, each of the separate colored entity is assembly part.
So the image at the bottom would seem familiar because the input we give to assemblies is
the same as the input we gave to virtual instances.
We didn't have to change anything.
We just had to re-import with the new assembly builder.
So each tree is composed of this instance assembly paths.
These are what we call assembly paths.
And you can see on the right the number of instances of each assembly path that's been
used in that tree.
And this shows how the assemblies are simplified.
Each of the colored part eventually combined together to form a simple continuous geometry.
This is the advantage of assemblies
or what we had with virtual instances
where we had to generate our own LODs.
Here it's already taken care by the Nanite Builder
in the offline build process.
And now the primitive scene data
has a range of assembly transforms.
And during runtime,
the cluster has assembly transform index,
and it just picks this assembly transform,
this assembly transform and flattens it so that we know the final world space
transform of the assembly.
And now it's a quick comparison between the alpha cards, the fully modeled
geometry and the assemblies and voxels. The assemblies have the performance
characteristic of the fully modeled tree while maintaining a fraction of the
memory footprint. And voxels makes it much faster than the fully fully
model geometry by helping with the lot inefficiency
solar distance as well as
having a better
shape preservation.
Keep in mind that
assemblies and voxels, they don't need
to be used together. There could be a non-assembly
tree that can be voxelized.
It just doesn't need to be a tree.
It can be any kind of
mesh.
Especially like aggregate geometry,
it's really helpful. At the same time,
the other ways around is true where
a tree can be made of assemblies but it doesn't necessarily have voxels enabled.
Next is skinning. How did we skin these assemblies?
So all the assembly parts, they are transformed rigidly. All the clusters within the assembly
parts are transformed rigidly. So they are pretty watertight. There is no cracks between them.
Each of the assembly part can be influenced by up to eight bones But in the case of foliage our assembly parts are usually associated with one branch So in our case we just use the single bone influence basically
We did see in one of the previous slides that assembly parts simplify to form a continuous
geometry. And this is during that process. This simplified geometry that's represented
in blue that undergoes per-vertex skinning instead of rigid transformation.
The bone influences of these vertices, they just pick it up from the
assembly part they originally belong to basically.
So in code it's pretty straightforward. At each primitive on being added to the world,
it receives a skinning header and we just get the transform buffer offset and load the bone
transforms and move the clusters basically.
And what you're seeing right here is one of the optimizations we did
to help with skinning lots of foliage in the world.
That is this default min animation screen size
which is used to stop animating skin meshes in the world which fall below the
certain threshold of screen size and this helped a lot in our case.
And this is the same C-Var in motion.
pretty pretty difficult to see the trees that have stopped moving because of most of the foliage is
just noise if you look at it so yeah the next is animation our animation is fully compute based
it's quite fast this is one of our bigger trees that we used in the demo it has more than a
a thousand bones. There is no pre-break animation involved. The wind is evaluated for each bone
every frame. And this is our debug view to see the bones. So how it works is we dispatch a thread
in shader for each bone in the skeleton and we recursively accumulate the position and rotation
all the way from the bone to the root and this happens for all the bones so for example the bone
b just needs to accumulate just its portion accumulation sorry orientation while bones are
the leaf nodes they have to go all the way and we do this every frame basically and in code it
looks something something like this that the skeleton data has the current bone index and
the parent bone index and we do the accumulation in a recursive manner where we start from
the current bone and go all the way up to the root bone.
And this is pretty fast.
It just takes around 300 microseconds on the critical path, but with the async computer
enabled we basically hit it.
It's less than 20 microseconds.
The next is animation instancing.
So you'd have noticed that with around 500,000 plus foliage animated, 300 microseconds is
quite small.
That's because we don't run the wind evaluation for all the 500,000 foliage.
So this is one of the shots from the demo, and I would like to show the debug view of
the bones.
So there is no way to realistically animate this many bones.
And we did try to do it with a naive approach, the render threat suffers the most.
The skinning scope takes almost 20 milliseconds on PS5, more than 20 milliseconds on PS5.
So we introduced animation instancing.
So animation instancing is needed because we wanted to have a sense of directionality for our trees We didn want trees to be moving in random directions We did want to have the trees follow a certain direction along the bend So we evaluate eight variants of the
animation for each unique skeleton, not each unique primitive, each unit skeleton in the world
and select an appropriate template based on the direction of the skeleton. So let's say
Say I have a primitive in the world that's facing 160 degrees in the z-axis, then we'll
probably select the skeleton that's been simulated for the octane 4.
It was a good approximation, it didn't look bad, and we were happy with the results.
And this shows a high level overview of what's happening.
We have the unique foliage skeletons and the wind shader is run per unique skeleton and
it generates eight templates.
And these templates are used on the instances in the world based on their orientation.
When a primitive is added to the world, an animation template is assigned to it based
on the direction it's facing.
The template just corresponds to an offset into the skinning buffer.
So it's a matter of just like sampling the proper bones.
So using this, the number of unique items
that we processed in the skinning scene
went down from almost 500,000 to just 150,
which is the number of unique skeletons in the world.
This is around 200% improvement in the render thread.
And that is how we also got the very low amount of time
on the GPU because we just simulate
115 to like
8 variants
and that amounts to almost
like 100,000 bones per frame.
So, and we heat it with
async compute so it's almost free.
So next up,
Kevin,
we'll be coming up to show some numbers
because we love numbers.
Thank you.
Alright, so with all these
juicy details and beautiful content,
What did we actually end up paying for all of this?
So in this vista shot here, we have about 20,000 trees post-cull.
I mean, the source triangle count coming into nanite here is in the billions.
That doesn't really matter with nanites.
Across the whole valley, we have scattered over 500,000 trees
and 1.7 million shrubs and 7 million grasses,
and that's across these 26 square kilometers.
We have about 28 species of vegetation in a couple of different biomes.
So the tree assets, they kind of range from below a million to about 10 million triangles
and up to several hundred bones per tree.
That kind of varies with the state of the tree.
If it's a dead tree, obviously you don't need that many leaves on it,
or maybe it doesn't even move at that point.
We also had a very big tree, which had 40 million triangles,
which was kind of the hero tree where we showed off some other debug view modes.
on.
And here I put some of the numbers
from the actual foliage section that we saw earlier.
So in this particular shot, we have about 100,000 bones.
I think it was 135,000 that updates in 0.1 milliseconds
on the GPU.
So that's very fast and scalable, what Vignesh outlined.
I put here the nanite timings roughly on average for this,
what is this, like 10,000 frames?
This is kind of the whole forest section.
We had a budget of around four milliseconds
for the nanite passes.
That excludes the shadow passes that had its own kind of budget.
So what's interesting here is that the big arrow I put here
points to the actual frame of this shot.
And I put the numbers for that particular frame
in the image there.
And it's actually a little cheaper
than the average of the whole thing.
You can also see the kind of the dip in the graph.
And it's because you'd think that because you're
seeing so many trees and everything is in frame at once,
it would be more expensive.
But Nanite and VSM actually scales fairly well
with the distance.
And that's also thanks to the voxel tech.
All right.
So that's what we did for the demo.
And let's go through what's coming in the engine for you
guys to play with So starting in 5 we got some experimental things for you And that mainly to give us some more time to polish things and take some feedback for future releases as well
So there's kind of a long list here with 5.7.
We've got a brand new Win plugin.
We've got support for an edit assemblies through our USD importer paths
and some in-editor tools if you want to stay in-editor.
We've got some example tree assets from Quixel on Fab
that are using best practices, easy for you to use.
And finally, we have a brand new tree authoring tool
directly inside of Unreal Engine.
So for the Dynamic Wind plugin,
you can easily enable it and start playing with it right away.
And the idea is that you should be able to get your Nanite foliage
up and running with wind easily.
So it supports procedural wind just like what we showed
in the actual demo here.
And you can configure your wind easily,
and it will affect instant skin mesh assemblies and support the whole wind direction batching
and physical parameters like Vignesh mentioned.
So in our USD importer in the engine, we have now support for nanodassemblies.
So through exposing some USD schemas to use in DCC tools to mark up your scenes,
you can actually build your own nanodassembly structures and import them straight into the engine.
So with this, you can actually create your own pipelines in any tool that supports USD.
We also have some in-editor functionality to create nanite assemblies with.
So in this new plugin called nanite assembly editor utils, you'll find things such as right-click
actions to create nanite assemblies from a selection in your viewport, and a blueprint
API for manually building your own nanite assemblies and saving them out to disk as a
mesh.
That's kind of the bottom one here.
We also added support for Nanet assemblies
right in the procedural content generation tools.
You might be familiar with those.
We've got an example here in the middle.
So that can also kind of spit out your Nanet assemblies
as a mesh, which is actually quite powerful.
Because I mean, the Nanet assemblies
goes beyond foliage.
You can actually kind of instance.
It's a micro-instancing solution, right?
So you can use that to kind of build up any kind of mesh
if you want to.
And for content examples, to get you started out of the box,
Quixel is providing a pack of preset trees,
which are optimized for nano assemblies
and the foliage pipeline as a whole.
And these are procedurally generated.
They're gamer-ready assets, which
are extensively customizable through the new tooling
coming in 5.7.
And there will be more additions and foliage packs over time
to Fab, so keep an eye out on that.
All right.
So note that it will come on FAB in 5.7 when it goes live
for a preview one.
If you're downloading that, these things
will be in the plugging itself, in case you're
looking for that.
And the tool I'm mentioning is the new procedural vegetation
editor.
It's kind of built on Quicksilce extensive experience
with procedural foliage growing algorithms.
And using this tool, you can edit and create
new variations of existing trees with graph-based rules
and logic, so right in the editor,
similar to how you do things with geometry scripting
or PCG, tools you might already be familiar with.
And this tool will produce Nana assembly assets
ready for you to use right in your projects.
And that will also support the Dynamic Win plugin
right out of the box.
And over time, the idea is for this
to become your kind of one-stop shop for foliage generation,
even stuff from scratch.
To learn more about how to set up your content using these new
Engine features.
You can go see Simon and Hassan from Quixel
in session room, this room, I think, tomorrow at 2.30,
so the same time as this talk.
They will walk you through all the new content examples
and tools, as well as content pipelines and best practices
with Nanite Foliage.
Highly recommend that talk as a follow up to this one.
And that's it.
I want to thank everybody for listening here.
And we also want to extend a special thanks
to everybody who worked on this ad epic and CD project.
We wanted to put an acknowledgment slide up,
but there was just going to be too many names on it.
So you'll have to do with this.
I think we have time for some Q&A.
Yeah, let's do that.
Thank you.