Baking (« convert to file texture »)
Workarounds




This tutorial is aimed at providing solutions to go around several limitations inherent to Maya's « convert to file texture » command, rayDiffuse's rayBake or Mental Ray's own « convert to file texture ». Not being the author of either of these utilities (obviously) I've no way to « fix » them, but I managed to find several workarounds.

All the exemples here are geared towards using these commands to output data provided by my plugin rayDisplace as reusable maps (image files). The idea behind rayDisplace is to recover information from a high resolution surface back on a low resolution one as displacement maps and normal maps. However rayDisplace only does half the work, it collects the information and outputs so that you can use it in a shading network and see it at render time. To be fully useful you need to be able to write this information to a reusable map (image file) so that you can get rid of the high detail object and work only with the low detail one afterwards. Best way to get an idea of the applications of rayDisplace would be to take a look at the user gallery.

You can check for the latest version of rayDisplace, both for Maya and Mental Ray, on Highend3d, or here : rayDisplace and rayDisplaceMental

Be sure to check you have the last version if you experience problems loading the scenes in this tutorial.

The explainations below suppose you are familiar with the use of rayDisplace, if not you can get a start by looking at the tutorials : rayDisplace tutorial and rayDisplace for Mental Ray tutorial

However it is only one specific use of the « convert to file texture », and the limitations described below are not inherent to rayDisplace, thus you might be able to apply the workflow described below to converting your own nodes/networks.

Also I need to do conversions beetween different coordinate spaces in the exemples below, for that I use the utility node : spaceChange

There are three different tools that I know allowing you to convert the result provided by a shading network to a file image: Maya provided « convert to file texture » command, rayDiffuse « rayBake » utility and Mental Ray for Maya own « convert to file texture ». The process of converting to a file texture is often called « baking » texture and that's how I'll refer to it later on.



Converting raytraced networks on subdivision surfaces :

Only rayBake and Mental Ray's bake can convert/bake raytraced networks. However none of them works on subdivision surfaces. It's not such a big issue, as in Maya anyway subdivs would be tessellated at render time, according the the Tessellation attributes of the shape. So if you convert them to polygons before baking using the same parameters as the ones set for render you won't be causing any difference at all in the generated maps.



To make things easier, I modified rayBake script rayBake.mel so that it will do it in a transparent way. It will generate a tessellated version of the subdivision surface you are baking on, using the Tessellation attributes, then will delete it when bake is over so you will only have to tune the tessellation parameters beforehand like you would for a regular render. There is no need to convert the other subdivision surfaces in the scene, only the one being baked as Maya can render them well and rayDisplace have no issue with them either.


For Mental Ray you need to tessellate every subdivision surface before rendering as it can't handle them yet. For the too I prepared the script orTessellateAllSubdiv that will convert all subdivs in the scene according to their Tessellation attributes, while keeping the original subdiv as an intermediate object. It will preserve the history so that deformed, skinned, and animated subdiv will re-generate correct polygon surfaces at each frame (can be quite slow so it should be a « last before render » action). It will also preserve instances in the scene. The conversion node will be linked to the subdivision surface Tessellation parameters so you can still tune them like you would for a Maya render.

Note : both scripts will also copy Render Stats attributes from the subdivison surface to the polygon surface, it's very important for using rayDisplace in particular that the low level object isn't visible in reflections or you can run into an endless render loop and crash from memory shortage fast. For Mental Ray you need to set the « trace » parameter to « on » or « off », the « derive from Maya » option doesn't seem to work.

Sample scene for use of rayDisplace with rayBake : bakeOnSubdiv_rayBake.zip
Sample scene for use of rayDisplace with Mental Ray : bakeOnSubdiv_MentalRay.zip
Baked maps obtained with these scenes : maps.zip



Low Subdiv


High Subdiv


Displaced Low Subdiv

Note : it is very important for displacement values to be accurate to « bake » with the same tessellation settings that the final render will use. Displacement will be calculated by measuring the distance between both surfaces, and tessellation can change it. However, for normal maps, it's not so critical as normals are « copied » from the high resolution object to the low resolution one.



Expressing normal maps in object space :

Displacement information is a scalar number, which represents the distance the surface will be displaced along its normal. Thus there is no issue of coordinate space with dispalcement, it will always be usable even for animated objects. However, often displacement is not enough. For render engines that do not provide a displacement as good and fast as, for instance, Renderman, the use of a normal map « on top » of displacement greatly increases visual quality at close to no render cost. For video games, typically only the normal map will be used. Unlike bump or displacement maps, normals maps represent vectors, so unlike displacement they need to be expressed in the correct coordinate space to get the expected results.

Check the scene and displacement map image file for the « arm » exemple and see how it looks without normal maps: rotatingArmDisplaceOnly.zip .
Not so nice actually, you can see artefacts created by the baking process on the nurbs cylinder tessellation lines. Either the displacement map needs a bit of manual smoothing in an external image editing program (you can check how Dave Cardwell uses Shake for the job in the user gallery) either the use of a normal map on top can smoothen things out nicely.

In all exemples and tutorials for rayDisplace until now I baked and stored normal in world space. They're usually provided by Maya in camera space but were they stored that way they could only be used for one view. As you can see in the « arm » exemple, we can rotate the camera freely and the normals « stick » to the object.

(Click images on the right for DIVX animations)


But what happens if the object itself is rotating? Then the normals will be off and you'll notice it reacts to light in a very strange way. Check the animation example on the right were I lighted it with only two lights and see how wrong it goes during the rotation. You can retrieve the scene file with the associated image file for the normals map expressed in world space here :
rotatingArmWorldNormals.zip


Then the solution would be to store normals in object space instead. In the file you'll notice we're not using spaceChange to convert back from world space to camera space but from object space to camera space. The image map we're using looks like the colors have changed too, the XYZ vector coordinates represented by the RGB colors in the file have changed, because we're not expressing these coordinates in the same space. Notice how the arm part correctly receives lighting now during its rotation.
Scene file and associated image file normals map in object space : rotatingArmObjectNormals.zip

Note : the normals are still off in the « cutout » area but they will always be unless edited manually as no info has been recovered for these parts, I'm talking about the arm surface area here.


How to obtain that map of the normals in object space? Since we saw in the displacement and normals tutorials how to recover a world space normal map, it should be as easy as converting from word to object space and baking again. Check how spaceChange is used to bake a new map in this scene :
rotatingArmWorldToObjectNormals.zip

Here I used 2 spaceChange nodes in the shading network for clarity, open the scene and check by yourself, the first spaceChange converts from world to object space, then the second from object to camera space. If you do a render all seem well, also the surfaceShaders showWorldnormals and getObjectNormals will show the color differences we expect in the two maps when rendered.

So we should only have to bake the getObjectNormals shader using Maya own bake and we'll get the needed object space normals. Surprise, when baking we get the exact same map... Even Maya's own « convert to file texture » got limitations it seems.

The culprit is the matrixObjectToWorld / matrixWorldToObject that the spaceChange node expects to receive during the rendering process. Correct values are provided when rendering, just strangely an identity matrix seem to be provided when doing a bake / « convert to file texture ».


The solution is quite simple here, we can just connect manually the worldMatrix of the shape we are baking on to the matrixObjectToWorld parameters of every spaceChange we'll use during baking, and the inverseWorldMatrix to the matrixWorldToObject so we'll « force » the information in. Of course if you're baking several shapes in one pass, then you got some shader splitting work ahead... You can download the « fixed » scene here :
rotatingArmWorldToObjectNormalsFixed.zip


Note : Why not directly bake object space normals? Actually rayBake fails to provide the render attributes we'd need (it seems to only provide pointCamera, normalCamera, matrixEyeToWorld and UV coordinates so far, which is luckily just enough to successfully bake world space normals). Of course we could apply the same workaround described here successfully, however as I already described how to get world normals I figured it wasn't necessary to go through the whole process again. Mental Ray bake has other issues though I didn't check this specific point, so it might work and you're welcome to have a try at it! As we don't need raytracing anymore to convert from one coordinate system to another, Maya own « convert to file texture » is nearly ideal...



Expressing normal maps in point or tangent space :

Now what will happen if the same arm surface is animated at the vertex/control point level, like if it's bound to a skeleton? Of course normal maps expressed in object space won't help, we need another coordinate space that will « follow » the surface deformations.

Point space or tangent space is exactly what we need, it is a coordinate space using the point being shaded as origin, and tangentU, tangentV and Normal at this point as its 3 axis. The spaceChange node will allow you to convert to and from point space too (you can check the basic spaceChange documentation too)

I'll divide this section in 2, first obtaining point space normal maps and related issues, then using point normal maps together with displacement maps. That way those of you looking to obtain normal maps for use in real time environment (without the added displacement) only need to read to first part, and check the hack for polygon surfaces.

Obtaining point space normal maps :

For nurbs and subdivision surfaces it's pretty straightforward. All the attributes we need to do the conversion are : matrixWorldToEye, tangentUCamera, tangentVCamera and normalCamera. They're provided by the rendering process and also correctly provided in maya's bake. You can also get them from the samplerInfo node.

The method will be the same as above, we don't need the « matrixWorldToObject » fix as long as we're not converting from or to object space, so it's even simplier, and we don't need raytracing either so all conversion is done from the world normals map we already have using Maya native « convert to file texture ».

Just note that the rebuild UVN attribute in both spaceChange nodes is set to « keepU ». The reason is the point space coordinate system is expected to be orthogonal, depending on how UVs are laid out on the surface, there is no garanty TangentUCamera, TangentVCamera and NormalCamera will be orthogonal to each other. With that setting, the spaceChange node will rebuild TangentVCamera from the provided TangentUCamera and NormalCamera (hence « keepU ») to ensure orthogonality. You can choose to « keepV » instead (thus TangentUCamera will be rebuilt instead), just make sure the spaceChange nodes you will use to convert to and from point space all use the same settings for a given map.

As before I used two spaceChange nodes so you can more easily see what is going on, and check the result on the final shader.

Just baking getPointNormalsShader using Maya's bake will give you the correct point space normals map.

You can download the example scene here :
subdivTorso_WorldToPointNormals.zip


Now you can check on the animation on the right that normals expressed in point space will follow the surface deformations (here a fast smooth skin was done on the torso).

Check also the scene file : subdivTorso_PointNormalsNoDisplace.zip

And the normal map used : torsoSubdPointNormals.zip


Now for polygons we could expect the same simplicity. Alas NOT! There is a very silly issue both in Maya render and Mental Ray render. Maya will provide correct tangentUCamera and tangentVCamera on polygon surfaces during render (as long as shader is dependant on another attribute like normalCamera, even if it has not visible impact, it must just be connected to force the network to update properly), but not during a bake / convert to file texture. Mental Ray for maya won't let you bake surface derivatives either.

This should be all we need to convert, alas though it works for a regular render, it won't during a bake.


This will force us to use a very silly and dirty hack, but I see no other solutions until this issue is fixed, if it ever is. But well, there is a working solution, though it adds unasked complexity.

The « mirror » trick for use with rayBake :

First I use a scene where I have duplicated the low detail object. So we got 2 low resolution torsos, identical and sitting in the exact same position. Note that I also triangulated it as it will be animated later and if Maya (or the render engine used) switches edges order as object is deformed it can mess the normals.


One of the 2 torsos (that I called torsoSourceTangentU) will be assigned a simple shader that will output its tangentUCamera to the color (recovering it from a samplerInfo node) while the other (called torsoLowDest) will use rayDisplace to recover the color of the first.
The first object should have its « Primary Visibility » set to off, while it has « Visible in Reflections » and « Double Sided » set to on. The second object got its « primary Visibilty » to on and (important) « Visible in Reflections » to off (to avoid self reflection). Of course raytracing needs to be enabled (1 level of reflection is enough) and rayBake will need to be used as we'll bake a raytraced shader.

Now first bake that tangentUCamera and save it. Note how I set a key of the camera at frame zero, because we write and use the map representing dP/dU in camera space, it's very important that the camera doesn't move between this step and the last one (you could easily bake a tangentUWorld instead in case you plan to reuse it by adding the necessary spaceChange node though).


Now we'll finally be able to convert these normals, using the same process as explained for nurbs or subdivs, except we use the previously baked tangentUCamera map and connect it to the spaceChange attribute of same name to ensure correct values are available during the bake process. Note that no raytracing is used here so you can bake with Maya's native convert to file texture.

Scene file : cheatPolygonTangentsRaybake.zip
TangentU map : rb_polyTorsoTgU.zip

Note: We only need a map of tangentU, as tangentV is rebuilt by the spaceChange node anyway. To use spaceChange in « keepV » mode, then you must bake tangentV instead.


And again a small animation (smooth skinning the torso to check how the point space normals behave)

scene file : polygonPointNormals.zip


Bit convoluted huh, don't tell me...

For using the same trick with Mental Ray, the only difference is Mental Ray renderer doesn't seem to care at all for object render attributes like « Visible in Reflections », « Double Sided » etc... However it takes « Primary Visibility » allright. I also have an issue in getting the bias paramater to work in the same way in the Mental Ray shader that it does in Maya... So to avoid self reflection I had to offset a bit one of the 2 surfaces (select all faces, use « polygons, move component », and move every facet by a small amount in its z local axis. You can get the scene for Mental Ray (and tangentU map) here : cheatPolyTangentUMental.zip

Known issues : view dependant alteration of the tangent is visible, on the border facets (close to facing othogonally to the view direction). It happens in the same amount whatever of the 2 methods you use.



Update for the 4.51 version of spaceChange : the use of « vector » mode to transform normals in version 4.0 and 4.5 of spaceChange would induce inaccuracy in some cases. However these would mostly go unnoticed when using a spaceChange node to transform both to and from a specific space. With version 4.51 and after the « normal » mode will be more adapted to transforming normals.

Note : converting back from point space without spaceChange.

As spaceChange is a plugin node it won't be converting by many export utilities that handle translation of Maya's nodes for a third party renderer. If you plan to use another renderer than Maya's and you have a mean to convert Maya native shading nodes (like using Maya to Mental Ray conversion or Mayaman, etc...), then it might be useful for you to use the shading network provided here to do the « point space to camera space » conversion instead :

pointToCameraNetwork.zip

Now we'll see how to use these hard earned maps! Note that if you plan to do your final render with a Renderman compatible renderer, best place is to check here :
Shaders for Renderman (the space coordinates change is handled by the shader I wrote for it so you don't need to worry about it there).

Using point space normal maps conjointly with displacement maps :

Now if the destination is not real-time but rather high quality rendered images, normal maps are more likely to be used « on top » of displacement maps to ensure the visual aspect is virtually identical to the one the high resolution surface would have got, while still allowing some reduction of the displacement parameters for weight/speed issues.

Let's go back to the nurbs arm we used before. If we convert its world normals to point normals (no particular trouble with nurbs) and try to use them together with the previously used displacement... Yuck! Something is wrong obviously as you can see in the animation. Normals quite follow the movement allright, but they seem to be somewhat off by a constant amount.


This small quick Maya fix confirmed to me what the problem is. To convert the world normals to point normals we used the tangentU, tangentV and Normal of the surface prior to displacement. When using the point space normal map on the displaced surface, what Maya provides us with are the tangentU, tangentV and Normal post-displacement. That is why the exemple above looks wrong while this one (using pre-displaced surface information) behaves right.

Note : this exemple uses a ugly hack that I would not advise to do, it gets the information it needs through a « pointOnSurfaceInfo » node. Such nodes do not always cope well with shading networks, they're slow, and I seen many cases where they would crash a multiprocessor render (nodes that are not shading nodes and are not marked MP_safe or MP_unsafe, but some turned out to be quite unsafe actually).




------ check back later for more headaches, limitations, roadblocks and hopefully workarounds ;) ----------------

Olivier Renouard

olivier AT drone DOT org