Baking ambient occlusion: Exporting from Maya into Unity

This is a post I’ve been meaning to write for a long while regarding my experiences baking an ambient occlusion texture onto a model for import into Unity.

First off, baking ambient occlusion is absolutely worth it to producing an appealing result. It makes the details in a scene really pop. (Ambient occlusion captures the self shadowing of close surfaces due to ambient light.)

With ambient occlusion
MallAmbTex

Without ambient occlusion
MallNoTex

Additionally, by baking the effects of ambient occlusion onto the model, there is no additional overhead to the effect beyond displaying textures. Although I suspect that if you are reading this, I’m already preaching to the converted.

Happily, there are a lot of resources for doing this sort of thing, but the potential pitfalls can vary a lot depending on one’s workflow and approach. The approach I took was to

  • Use Mental Ray to compute ambient occlusion for static objects in the scene.
  • Bake the result to texture.
  • Export the model as an FBX with textures embedded.

So let’s give the details and then I enumerate the pitfalls, eh challenges, I experienced while working this out.

Baking ambient occlusion

Step #1 is to compute ambient occlusion for your scene and then bake the result to texture in Maya. There are a lot of resources for this online, but the video below by Josh Robinson is my favorite.

Baking Ambient Occlusion in Maya from Josh Robinson on Vimeo.

If you are unfamiliar or rusty with Maya, it’s really worth watching the video to see how things are done with Maya’s UI. Below is a summary of the steps in the video:

    • Add a surface shader to your objects. Right click on the object, assign new material, choose a surface shader. The object will now look black.
    • Make sure Mental Ray is enabled. If it is not enabled, objects will be rendered black. This can be set in Window -> Settings/Preferences -> Plug-in Manager. Make sure mayatomr.mll is checked. By the way, if you don’t have the FBX plugin enabled, you should make sure it is also checked.
      Plug-inMenu
  • Setup ambient occlusion shading using Mental Ray
    • Add a mib_amb_occlusion shader. Use the Window -> Rendering Editors -> Hypershade dialog.
    • Connect mib_amb_occlusion to the surface shader (middle click and drag from mib_amb_occlusion to the surface shader and select ‘Default’)
    • In the surface shader properties, set the number of occlusion rays to 128
    • Do a test render. You should see ambient occlusion shading.
  • Bake the ambient occlusion effect to texture
    • Under the menu Lighting/Shading -> Batch Bake (mental ray), open the option dialog. Set the file resolution and file type.
    • Under the menu, Window -> Rendering Editors -> Render Settings, make sure the renderer is set to Mental Ray. Use high quality settings for the best looking results.

When things don’t work

This technique relies on your models having good texture coordinates.

If your model remains completely black after baking, your model may not have UV (e.g. texture) coordinates, or they may be setup incorrectly. I ran into this problem with a few old models stored in OBJ format. Setting up good coordinates is beyond the scope of this tutorial (and I am not an expert texturer myself), but this is something to be aware of.

Other problems can occur if texture coordinates are duplicated or misaligned. This writeup gives a few examples with solutions using 3DS Max.

The scene which I was shading had a lot of objects. By default, each object had its own texture, which for me turned out to be problematic since using the same texture resolution for every object was inappropriate. For large scene objects, I needed to increase the resolution to get a better looking result.

Effect of a small resolution texture
AmbSmallTex

Effect of a high resolution texture
AmbBigTex

A final problem — which I have not entirely fixed in my own environment — is occasional seams in the baked textures. As there are other resources more fit to help troubleshoot such problems in Maya, I am going to move on to the potential gotchas importing the model into Unity.

Export the result to FBX

Step #2 is to export the result to FBX. So far, I have never run into a problem with this process and using FBX in Unity has a number of advantages over using Maya’s .MB format

  • Fewer path problems. Embedded textures are unpacked automatically with good relative paths.
  • The FBX file does not require someone to have Maya installed to load it
  • The FBX file is often smaller

If you can’t find the FBX import/export options, you may need to enable the plug-in manually. Under the menu Windows -> Settings/Preferences -> Plug-in Menu, open the Plug-in manager and make sure fbxmaya.mll is checked.

It’s very important to embed the texture assets when you export.

EmbedMedia

Import the FBX into Unity

Step #3 is to import the FBX file into Unity. In general, this is as simple as copying the FBX into your Unity project’s Asset directory.

However, if your textures do not load correctly, you might be running into one of the following

  • The filenames generated by Maya during the baking process may be too long for Unity to load them. To shorten them, you can specify a shorter prefix in the Mental Ray Baking options (under Texture bake settings), or try shortening the project directory from the menu File -> Project -> Edit Current.  Note that if you repeatedly re-bake the textures, Maya will repeatedly prepend the prefix, leading to names that look like “baked_baked_baked_baked_object1Lighting.tiff”.
  • Multiple objects in the scene may be using the same texture name (this leads to one of the objects using the wrong texture. The result is usually big black splotches in the wrong places). To fix, change the model’s import settings in Unity to use prefixes before material names.
    NameMaterials
  • Maya may be connecting the result of ambient occlusion to “incandescence” instead of color, which Unity ignores. We need it to connect to color instead.ShaderNetworkThis can be fixed manually, but if you have a lot of objects, a script is easier. The following script connects the outColor property of each texture (a ‘file’ object) to the color property of the corresponding lambert material.
    import maya.cmds as cmds
    allObjects = cmds.ls(l=True)
    for obj in allObjects:
       if cmds.nodeType(obj) == 'file':
          connections = cmds.listConnections(obj+".outColor", d=True, s=True )
          print obj, "connects", connections
          for connection in connections:
              if cmds.nodeType(connection) == "lambert":
                 print connection
                 try:
                    cmds.connectAttr( obj+'.outColor', connection+'.color' )
                 except:
                    pass
    

Here’s a demo of the environment shown at the start, running in a browser with Unity’s web plugin (warning: it’s a big environment, the plugin is ~7 MB). The camera can be zoomed in and out with the middle mouse button and rotated with Alt-LeftDrag.


MallEnvThumbnail

And here are some screenshots. Up close, some of the objects still have artifacts (seams) which need fixing, but I’m pleased with how well the scene looks at a distance.

MallEnv1

MallEnv2

If you find this writeup helpful, let me know. Also let me know if you have additional tips, gotchas, and/or alternative methods approaches. I will add them.