PBR-for-Photogrammetry--Marso-with-Reali

Hey there, I'm Wallace Wainhouse from Epic Games, and I am a product specialist for RealityScan Desktop. And I'm Elliot Round, the co-founder and CTO of MXR, and we build measured PBR technology.

These are some of the results

that Wallace has been capturing more recently

using Marceau and RealityCapture together. Rendered in Unreal.

What is this talk about? It is basically about making PBR textures and how to make scans look real. That involves making materials that behave correctly according to light.

If you have any experience of making PBRs, you might know that that can involve using

extremely expensive rigs with special hardware, special expertise, or just a lot of artistic guesswork.

Before we go into the workflow, probably best to start off with what are materials.

In 3D, you have your mesh, but the Engine needs to understand how best to render that mesh.

This is where shaders and materials come into play. To render a pixel, the Engine has to know, does a way of pass through?

Does a way of bounce off the surface? Does a way of reflect? Can it have a mission? We use materials to create a whole wealth of different styles.

Some of these could be fictitious, They could be fantasy, cartoons. You can do all kinds of different things. But today, we are going to be talking about PBR,

which is a sort of umbrella of Material types.

This is a scene that we put together. Lots of 3D scans. They are all real objects. At the moment, none of these have PBR properties. We have the shape. We have got lighting. We have got some simple shading.

But they all look very flat. They all look like they are made out of clay. you do not really get a sense of what each object is made from. As soon as we add the PBR properties into the mix, they start interacting with light. They start to look a lot more realistic, and things come to life.

Comparing the two, we can see what things are made out of metal,

what things are leather.

We can almost get a sense of a degree of tactility.

They give things a grounding within the scene.

Something that I find really cool about PBR is just how well you can almost feel the objects.

On one of these clips, we have this metal tin.

It is black.

But in the PBR version, we can see this as a painted enamel.

It is quite old. It is quite flaky.

You can almost imagine what it would feel like

to run your hand across the surface.

The way that PBR does this, PBR stands for Physically Based Rendering, and it is simulating the rules of reality. It is simulating how real rays of light

interact with the surface of a 3D object and ensure a degree of energy conservation. So when you have a 3D object, and you place it into your scene,

and you have accurate lighting, the object will reflect that lighting correctly.

It won bounce more light off than the light that hits the object itself And if you mix this in with things like HDRIs which are lighting environments that have been captured from the real world then these objects look even more grounded

in a virtual environment,

and we can almost tell what they are made from and feel what they are made from as well.

What makes up a PBR object? Well, there are two key shaders under the umbrella of PBR. You have a specular workflow, and you have a metallic workflow. You could argue that they are both doing the same thing, but optimized in different ways. Today, we are going to go through the Specular Workflow Shader.

The first thing that we have here is our base mesh. It is quite low poly, it is rather smooth, and it does not have any complex shading.

When we add normals into the mix, normals describe what is happening between the vertices. It adds a little bit of micro detail that we do not have to describe as a high poly mesh.

When we turn that on, we can start to see all the scuffs and scratches and the little flicks of paint coming off of this old petrol can.

Our first true PBR property that we have is roughness. Roughness denotes when a ray of light hits the surface of an object,

does that ray of light scatter, or does that ray of light reflect in a smooth line? When it scatters, it is very rough.

We denote this with a white value, When it is smooth, we denote this with a black value. With this old, rusty petrol cam, we can see that the rusty areas are white,

and the painted areas that are quite smooth are more of a black.

We can also see these thin white strips, which are all the scratches. Then when we apply that to the 3D object, we can see which areas are reflective and which areas are not. The other chop property we have is specular. Specular relates to the light that hits the object,

how much of that light is reflected, how much of that energy is actually absorbed. A lot of the specular reflection does tend to have a slight color to it.

You see this a lot with paints, metals like gold and brass, where the specular reflection is white, it might be more of an orange or a yellow.

When you turn that on, you start to see the real difference between the highly reflective areas like the paint and the not-so-reflective areas like the Rust. Then finally, we have the Diffuse value. The Diffuse is our base color,

and this gives it the main appearance. You mix that in, and you have got your final 3D asset, and it looks pretty realistic.

Here is another video rendered in Unreal. In a Quixel scene, by the way.

That object was done in Marceau with RealityCapture. You can see with the normal map,

you get really fine details on the surface.

And that's a different petrol can to the one we're actually going to show you today, because me and Elliot have both got petrol cans that we scan in our garages.

So you might be wondering, why aren't PBR scans everywhere?

And the answer is that they're actually not particularly easy to make.

With normal photogrammetry, the way you get a texture is you just get a diffuse. It's a sort of attempt at an albedo texture. And essentially it made by just projecting and mixing every single camera onto the surface and blending them all together So you don get any of that material information about the reflections and roughness

So traditionally, here's two of the main ways

people might try and make a PBR material. There's a third one I'll go into later. By hand, which we'll go into in a sec, and also using very, very complicated hardware

with cross-polarization and photometric stereo. The manual way is kind of like an educated guess.

What people will do is they'll take the diffuse map. A lot of you might have had a go at this. Then when you have an understanding of what the roughness is going to be, so basically the darker parts are shinier

and the lighter parts are rougher. In this case, I made the decision

that I wanted the grooves in those tiles to be rougher. so I inverted it and then messed around with it a bit and got a sort of vague roughness map there. It's kind of arbitrary. The bump map, I think I've made a little bit dark there, but essentially, you know, the darker parts sink further away, so I've got the sort of grooves in the tiles there. Then you can combine it together, and it definitely looks a lot more realistic than not doing anything at all.

Now, here we get into the far more complicated ways. and one way you can have a go at making a roughness is using cross-polarization, where you use two polarized filters,

one in one orientation on the flash and one at 90 degrees to it on the lens, and what that does is it blocks out all the glare.

Now, imagine if you do that from every single camera position and you have a sort of glare map from every picture, then you can blend them all together

and you get a glossiness map, and then if you invert it, it's going to be akin to a roughness map.

But you can see that it doesn't work perfectly every time. You can see I've got that white enamel badge on there,

and you can see in the glossiness map, that kind of should be white because it should be glossy, but it's actually turned out black, so it's got that completely wrong. So yeah, it doesn't work all the time,

but you can get amazing results like this. And then if you want to get a normal map, there's an old technique called photometric stereo,

but it's an absolute pain in the ass to do. Oh, I've lost my mic. Because you need to take an image from the same camera position with several different lights, and then the math will figure out a normal map from that.

So you can imagine that when you're trying to make a PBR material like this, you end up with terabytes of data, and it's not actually that easy to process either.

This is a third one I was going to talk about. And basically, this is easy because you just take your diffuse and you just click give me my materials.

But it is kind of flawed and you are going to get really, really mixed results. And also it does not work on 3D.

These maps look nice, but actually when you put them together it did not actually look as good

as the one that I just messed around with the diffuse for.

So we are going to show you how you can capture 3D PBR using Marso Measure and Reality Scan all by yourself

using available hardware. Before we do so, I am quite interested to get a show of hands

who has a camera and who also has a flash,

so like a point light or a ring flash. Okay, quite a few of you. So, using Marso, you will be able to do just this.

You will be able to capture full PBR objects.

We built Marso really as a way to try and bring PBR to as many craters as possible Previously PBR capture has been incredibly tricky or using very expensive hardware has been limited

to large game and film studios.

We have come up with a method where you do not need any complex hardware, so no cross-polarization, no fancy rigs,

no filters and triggers. All you need is a camera and a point light.

This could come in many different variants.

It could be a DSLR with a little point light on top.

It could be a ring flash. It could even be your smartphone with the light within it.

This means that it could be a AAA game studio, an indie studio, hobbyist, student, or even just someone that likes doing 3D scans, and you can all capture PBR properties to the same level.

By simplifying this to just a camera and a flash, it also acts as a standard for capture.

If you are capturing assets from different locations,

three different vendors, you can bring all of these assets

into the same scene, and they will look identical.

We do this with flash photography,

and this is done by measuring how a 3D object interacts with light. We measure the PBR properties based off of this reflection.

By measuring how it bounces off, we can look at the specularity, we can look at the roughness,

and we can also understand the base color. We designed Masso for capturing 3D objects that you want to put into your larger 3D scenes.

It is not really designed for capturing rooms, but anything that you can get around with a camera,

you can do a full PBR scan.

We have tried this with fashion apparel, so shoes and handbags, small to medium-sized objects for film and games,

so lots of props, even the odd cultural heritage piece, where you do not have to take the artifact out of a museum,

you simply take the glass cabinet off and take a few pictures. and we have even captured a few small to medium-sized pieces of furniture.

What can you expect for a mass-side measure? The software itself delivers the full array of PBR properties for both the metallic and the specular workflow. We give you diffuse, specular, albedo, metallic, roughness,

gloss, index refraction, and normals. These can be delivered all the way up from 512 pixels up to 16K, and these can be delivered as JPEGs,

PNGs, and even 32-bit EXR files. It works best probably with freehand.

A lot of the scans that we have done tend to be with the ring flash, but you can also use a rig if you put a small LED on the top and do your scan that way.

But to get the best results, you need to get good coverage, and you are going to hear me say this a lot. the better you can get around your 3D objects, the more likely you are to capture all of those reflections.

If any of you are photogrammetry veterans, this is something that you are probably not used to doing or doing it in a slightly different way.

But if you want to get a Marceau measured scan, just please follow along,

and you will get a good measured scan at the end.

Wallace and I, over the next few slides, are going to show you actually how to do one of these.

We are going to cover setting up your scene, how to actually capture. The better quality your 3D scan source photography is, the better the result will be out the other end. We are then going to show you how to process this through Reality Scan, how to process the PBR through Marceau, and then finally bringing it all together within Unreal Engine.

Those of you out there who are used to doing photogrammetry, you might find that this capture method is a bit different to normal. Now, for one, you're probably going to end up having to underexpose things. Now, the reason for that is Marceau needs

to be able to get values through the whole range of the brightness of the object. So those specular highlights, they can't blow out. They need to be below 1 in value, which

can obviously make the rest of the object pretty dark. But Marceau uses raws, and they've got big color depth,

so you can pull back a lot of that information. Now, the other thing we often tell people is don't bother using shiny objects.

But that's kind of the point here. Often the most exciting PBR materials are going to be things with specular highlights and some reflections. Now, lots of images.

That's not breaking a rule because obviously we do encourage people to take as many photos as they possibly can because you can always not use them. But I think you're going to be surprised how many you need here.

It's a lot more than you would need to get a really nice photogrammetry scan.

There are four things that you need to think about when you are doing a MASO measures scan. The first thing is the camera. When you are using the camera, it has to be a camera that can shoot RAW files. The reason for this is so that we can keep as much of that dynamic range as possible, so all of the highlights, all of the shadows, which will often get clipped if you are using a JPEG.

The other thing is the exposure. We want to slightly underexpose the image. The reason for this is to, again, maintain that highlight. If we expose normally, we are going to clip the highlights and we are going to expose for the majority of the 3D scene.

Also, locking off your exposure. A camera that can shoot in manual is really important. If you have the exposure changing throughout the scan, it is going to absolutely mess things up.

Find your exposure, lock it off, and then move forward with your scan. The other thing is lighting. Because we are doing a measured scan and we are measuring the response to light, the lighting is key.

What we want to do is get an image where the ambient lighting has no effect on the image. It is entirely coming from the point of view of the flash. If you are using a point light source, like a continuous light,

it is best to capture this in a really dark room. If you are using a flash, like what Wallace has here, you can usually pump it up enough to overpower the ambient lighting in the room.

Coverage, keep going on about this, but you want to get enough views of the object.

Usually, we will do a scan where you capture it from a few rings around the object, and then punch in and get a few of the details, and then finally scale.

This is something that usually people do not have to worry about too much when you are doing a 3D scan, but for a MarsO scan, scale is key. The reason for this is when you are doing a picture

with the flash, the closer you are to the object, the quicker that object is going to burn out and it is going to look really bright.

As you move further away, the lighting is going to fall off more quickly. There is going to be a difference in how bright some of the images are. If we can measure the scale of the scene,

Marceau can actually account for that. A few ways that we can do this is either using some markers,

which I think we have here.

Markers that can be put into Reality Scan, where you know the distance between them.

If you have got markers, you can put a ruler in the scene. Alternatively, you can even just measure a part of the object and do it that way.

When you are getting your exposure, the easiest way to get this right is to actually take two pictures.

The first image you will take is with the flash turned off. Ideally this will be completely black If it is not you can tweak your exposure setting accordingly Then the second image take with the flash turned on If that second image looks nice and slightly underexposed

as you can see here, then you are good to go. What you can see from this point of view

is that reflected region. By capturing this reflection here, we can understand what the

PBR values are. We want to get enough images where we almost cover the whole object in that specular reflection.

We pick this object because it has quite a few interesting material properties on it. Mindflip is quite hard for me to

get through airport security on my way here. When we are doing a scan, we will often do a few loops just

to make sure you have got the general shape.

but when you are capturing something quite planar like this,

it is usually good to scrub left and right, up and down,

and that will just ensure that you have got

all of the reflective regions in your data set.

This is just an example of what that looks like. Scrubbing left and right, you can see if you were to almost add all those images together, It is like you are painting the object with light, so the whole thing becomes covered. If you do not take enough pictures, so this is a sparse scan, what you will end up with is regions

where you have got the PBR property and regions where you have not. Your Roughness Pass or your Specular will have peaks and troughs within it. The general rule of thumb is the shinier the object is,

the more images that you will want to take, and the more matte the object is, you can probably get away with a few less.

Just for example, all the scans that we have done of this

It is around, I think, about 600 to 700

for things like shoes that are quite matte, like leather. You can get away with about 300 images.

We have done the capture. Now we can go through the rest of the process. Just top level of the workflow, do your scan. You process the data initially through Marceau. You then hand that over to RealityCapture, RealityScan for the 3D.

You then go back to Marceau, where you do the PBR predictions, and then you can take both of those elements and bring them into Unreal Engine.

The first stage within Marceau is creating a job. A dialog will come up where you can input the name of your scan, and this allows you to come back to it many times. From there, we can also say where our raw photography is. When we first load in this raw photography, we can actually change the brightness of the image. This has absolutely no effect on the PBR predictions

that you get out the other end. It just allows you to have a proxy file that you can run through Reality Scan. If it is too bright or if it is too dark, you can make a quick adjustment here

and then export them to either a JPEG or a EXR file.

When you send your JPEGs through or EXRs through to Reality Scan, you have an option to change the exposure.

As I showed earlier, you can just let those highlights blow out at this stage so we can actually get a better solve from all of the cameras.

I am going to go through processing in RealityScan and a slightly more advanced method that you won't actually need all the time. It is going to be very similar to normal processing

in RealityScan, aligning, meshing, scaling, simplifying, smoothing, unwrap is important, and then obviously exporting the files that Marceau is going to need to make those PBR materials You going to probably be using shiny objects

As I was saying earlier, they're not great for scanning.

Specular highlights cause issues in alignment and meshing. The reason for that is in alignment, the way alignment works is it finds common features on the surface of the object. Then using those, it can triangulate where the cameras are.

So in alignment, we figure out where all of the camera positions are. Now in meshing, we assume that we know where those cameras are,

and then we go in on a pixel-by-pixel basis, look for similar pixels in other images,

and then figure out a depth map. So we have a depth value for every single pixel on the object. And then when we have a depth for every camera,

we can make a nice mesh with millions and millions of triangles. But obviously, because we're looking for similar pixels,

if it goes to look for a similar pixel, and there's a big reflection there, that is going to create a mess in the meshing as well.

We have problems in both alignment and meshing. How do we help the alignment?

Markers are really helpful. I am not sure why I have used the circular ones here, because we actually have the option to generate April tags as well, which would be a better option. Imagine those are April tags and not the circular ones.

Circular ones are fine, but April tags tend to be a little bit better. The other thing we can do is you can put stuff in the background. You can see on the stand here,

I have put this highly textured material. Reality Scan is going to find loads of tie points in that,

which will help the alignment. It does not matter if the object itself is helping Reality Scan figure out where the cameras are. It can be anything in the scene. The markers are going to help,

and any highly textured surface you can put in and around your object will help with the alignment as well.

Now, helping in meshing. Now, actually, if we look, oh, no. Yes, I have not got the mesh there, but actually this mesh turned out fine If you'd simplified it and smoothed it, it would have been okay. But sometimes you'll get massive holes in your geometry, and obviously that's no good at all. So what you can do with meshing is you can actually get some scanning spray,

spray your object, and we'll align those images together with the ones from Marceau, like the shiny ones, and we can just use the sprayed images for our mesh.

So we get super nice geometry. Then obviously, we only send the images for Marceau back to Marceau.

The markers are going to help in this situation if you want to use this sprayed workflow, because they are going to help. It is not going to see the same features on the surface, because one set is sprayed and one set is not. The markers are going to have to be the same

in both sets of images. Every time I have tried it, everything is all aligned together in one go. You can see my mesh there. That is meshed using only the sprayed images.

Okay, so scaling. As we said earlier, you can set a scale between the markers.

But yeah, Elliot also pointed out, if you know the distance between two points on the object, Reality Scan also has features to do that. That is really important for Marceau

to actually figure out how much light is involved, because it needs to know how far the light traveled from the flash to the object, bounce back, hit the sensor,

and it can't make any calculations without knowing that. Scaling is super important. You can't just have an arbitrary scale,

which is what you'll get in Reality Scan unless you actually scale it.

It's totally arbitrary. Simplifying and smoothing.

You can see my original mesh here with 69 million triangles. I smoothed to 2 million I think it probably overkill as well But Marceau makes a really nice normal map very similar to what you get with photometric stereo So your actual final mesh

as long as it doesn't have massive holes in it and it's smooth and nice and not too many triangles, Marceau is going to do a nice job with it.

It has to be unwrapped.

Yeah, because Marceau also needs to have an unwrap to work with. So unwrapped to a single texture, The default in Reality Scanner is a single 8K texture, but I don't think it actually matters what size it is.

It could be 2K, 4K, 8K, or 16K.

Then finally, we're going to export our files to Masso. The main thing, obviously, is a mesh, and it needs to obviously be unwrapped as well, like I said earlier. Masso also has an option for a mask mesh. In this situation, you definitely want one,

because the images underneath are going to be occluded. So when Masso tries to take measurements for anything near the bottom of that, if the table is in the way, it's going to get a completely wrong measurement because it's looking at a table and not the actual object.

So if you send the mask in, Masso knows what is the object and what isn't the object. We need ST maps as well, so it knows what to do with the distortions.

It has a perfect understanding of the distortion of the object. Something we can also do at this stage is if for whatever reason you want to UV unwrap your mesh slightly differently,

retapologize it, clean it up a little bit, you can do that to the Alambic file and send that also into Marceau, provided you maintain all of the cameras in the scale.

We have got our 3D data from Reality Scan. The first thing we are going to do before setting that up is actually putting in our exposure settings. One of the first things you do during a scan is actually take a picture of a gray card. You would pop a gray card up, take a picture, measure the distance, and then Marceau will understand what that light falloff is.

You only have to do this once. If you are using the same camera and light setup, do that once and then you can reuse it many times. You can have lots of these just in a list, so different pairings of exposure and light brightness, and you can pick those. Once you have picked that, you then state which version of photogrammetry application you are using. Here we are using Reality Scan. You then state where your mesh file is. You go to your Alambic. You can then optionally state where your mask is, if you want to use a mask, and then finally where your ST Map folder is. Once you have done that, you can then go over to your settings. This is deciding what resolution you would like. We go all the way up to 16K. You can also decide what version of master you want to run. We have got different versions of our models. that give slightly different results. Then you add this to a queue or process it immediately.

Out of that workflow, we have now got a lovely 3D mesh from Reality Scan, and we have got all of the PBR textures from Marceau.

Finally, we got some help from Sebastian from Quixel with this substrate material. I am actually no expert here, but I threw this together, and it has just got a few little tweaks, so you can make some adjustments there, but it is very similar to just making any material in Unreal.

You just load your maps in there.

Finally, we just have a few Unreal renders there,

and that is it. Thank you.

Thank you for coming.