Occlusion Exercise
One of the problems with the basic Maya Software Renderer is that it sometimes ignores the way that light behaves in the real world. As such, objects like the one below can tend to look flat and artificial. One way to increase the realism of a shot is to add an occlusion pass. What occlusion does is calculate the areas where it is unlikely that light would reach and makes them darker. However, the reason that Maya doesn't give you occlusion by default is that it is extremely expensive to calculate. For a complex scene, rendering can take up to 30 minutes per frame.
Setting up an occulsion pass is fairly simple. Go to the Render Layer Editor. Create a new render layer and name it something like "occlusion_pass", then right click the layer and select Attributes. Left click on the Presets button near the layer name and select Occlusion. This should turn all geometry in this layer black. Rendering on this layer will produce an occlusion pass using Mental Ray instead of a normal render.
By default, the quality of the occlusion isn't great, as seen by the image below on the left. It looks grainy and spreads farther than it should.
You can edit the occlusion by accessing the shader that was created for the preset. It can be found in the Hypershade or by selecting one of the objects in the occlusion layer and clicking the appropriate tab in the Attribute Editor. The shader should called something like "surfaceShader1". Follow the Out Color mapping of the Surface Shader. This will bring you to the occlusion node. The three attributes of interest are Samples, Spread, and Max Distance. These values will determine the quality and speed of the occlusion. Set Samples to 250, Spread to 0.5, and Max Distance to 3. If you re-render, you should see that the occlusion is much less grainy and is focused around the nooks and cranies of the geometry.
An occlusion pass isn't much good by itself. Once you have rendered it out, it must be composited with your regular render. Typically this is done in After Effects with an entire series of frames, but for brevity, we will composite a single frame in Photoshop. Place your color on one layer and the occlusion on a layer above it. Set mode of the occlusion layer to Multiply. Simple as that. The difference is subtle, but occlusion adds a lot of richness to a render. The renders below show the difference between a normal render and a render with composited occlusion.
It would be nice to only incur the cost of occlusion once, not for every frame. Lucky for us, Maya provides a process called baking that will allow us to generate an occlusion texture. The first step is to lay out the UVs of your object. We will use an Automatic Mapping for the sake of speed. Go to Windows → Settings/Preferences → Plug-in Manager. Make sure that the RenderMan_for_Maya.mll plug-in is Loaded.
In the Modeling menu set, go to Mesh Display → Batch Bake (mental ray) (Options). Set Objects to bake to Selected, Bake to Texture, and check the Use bake set override box. Change the Color mode to Occlusion and Occlusion rays 256. Verify that the File format is TIFF and the X and Y resolution match the dimensions of your color map. Click Convert and Close to begin the process. This will lock up Maya for a while, but eventually it should finish.
A new occlusion texture should be applied to the object you selected. The new texture should reside somewhere in your default project. If you need help finding it, follow the Incandescence mapping to the File node. If you click View, it should open fcheck, from which you can save the texture somewhere reasonable. All you need to do now is to multipy the occlusion texture on top of your color map. One benefit of doing occlusion per frame is that it can account for the occlusion that appears when objects get near each other, something static occlusion doesn't reflect.
