Recovering normal and displacement maps
from existing geometry in Maya

(plugin, sample shaders and scenes)

If you need an explanation about what normal maps are you can check here : normal_maps, however you will have much faster render times using the plugin and methods described here, wether you're after displacement or normal maps (or both!).

January 10, 2005 : recompiled rayDisplace for Maya 6.0, and source code is available

First as a warning I'm doing this for fun and experimentation. I you have a need (and a budget) for a professionnal application of the techniques described here you might check cyslice software by headus.

Some time ago I presented a shading network to recover and use the normals of a high detail mesh on a low detail one. I reused the principles to write a plugin that can get displacement info as well as normal / bump info, and that manages to be about 100 times (really!) faster thanks to using Maya's raytracing API calls.

As this node uses raytracing, you can't bake it using Maya's « convert solid texture ». However Mark Davies great plugin includes rayBake using Tristan Salome's scanCookNode, and is a great tool to bake raytraced shading nodes to image maps : lightEngine3d

As an alternative not using rayBake, you can check the Mental Ray version of that shader I wrote, and see how the process described in detail here can be ported to Mental Ray for Maya in that tutorial : displacement_maps_mental

Also it's a good idea to check in case I did a newer version of rayDisplace.mll. For major releases I'll try to update this main page, but for minor modifications and bug fixes I probably won't have enough time so I'll just mention them here : rayDisplace

(beside posting them on Highend3d)

And finally there are some issues when « baking » (using « convert to file texture » or « rayBake » to output data provided by rayDisplace as an image file) that I can't fix directly as they're caused by limitations of either Maya's, Mental Ray or rayBake « convert to file texture » method. Until these issues are fixed, it might be useful for you to check the workarounds page : rayDisplace_workaround

General idea : given a high detail and low detail version of a polygonal mesh, I'm using this plugin to generate a displacement map and a normal map that once applied to the low detail mesh will give it a rendered appearance as close as possible to the high detail render.

Applications : from a heavy, detailed object (like the one you would get from 3D scanning or the use of Zbrush great detailing tools) you can derive a custom built, animation friendly low resolution object and the displacement and normal maps that will make it quite identical to the high detail version at render times.

Low Def Torso	High Def Torso	Low Def Torso with maps applied (spot the difference if you can!)

The plugin with the AETemplate can be downloaded for Maya 4.0 : rayDisplace.zip
And here for Maya 4.5 : rayDisplace45.zip

Exemple result scene with low detail mesh, shader and maps (for Maya 4.0) : final.zip
And for Maya 4.5 : final45.zip

For both scenes you'll need the baked maps here : final_images.zip

Note that the version 4.0 scenes won't load correctly into Maya 4.5 due the the use of the plusMinusAverage node and the 4.5 changes, so load the correct scene for your version.

How to use rayDisplace : copy the file rayDisplace.mll in your maya plugins directory (or any directory that is in your plugins path), then load it using the plugin manager. As rayDisplace is a shading node so you can create it from the hypershade menu under the section « general utilities ».

A scene with both models set up for calculation and all shaders detailed later included can be downloaded here : rayDisplaceShaders
And for 4.5 : rayDisplaceShaders45

Render settings : raytracing needs to be enabled for the rayDisplace to work, you only need one reflection level though, no refractions or shadows and bias should be left to 0.

Camera settings : some shaders use the camera worldMatrix, so all renders should be done from the « persp » camera or shaders should be modified to use the new camera instead.

Object settings : the primary object is the lowest detail object. It is the one that will be visible in renders. You need « primary visibility », « smooth shading » and possibly « double sided » to be on for it. The highest detail object is the secondary one and need not to be visible in renders, but need to be « seen » by the plugin.You need to have « primary visibility » turned off, but « double sided » and « visible in reflections » turned on. It also needs to have a shader applied to be visible, as we'll see later we use it's color as a way to pass back information, so you should leave the shader that is currently assigned to it called normalMapshader. You could have several high detail objects as anything that is visible in reflections will be accounted for. Thus objects that should not interfere should be hidden before you calculate maps. No light is needed for calculations though.

Displacement settings : the primary object will be displaced, so to allow enough detail you need to set up its Displacement Map parameters. Here the highest object being smoothed twice a value of 16 gives good result for initial sample rate. I had also best result basing the extra tesselation on texture rather than normals.

rayDisplace outputs : there are 3 groups of output that can be used to retrieve information out of a rayDisplace node.

First outDistance will give the distance between the primary object and the secondary one, at the point being rendered. It is the output that will be used to get displacement maps. Under it you'll see 3 attributes : outDistancePositive (distance when the secondary object is « outside » the primary object , this based on primary object normal direction on the primary, and will be null else), outDistanceNegative (when secondary is « inside » primary) and outDistanceUnified (distance is either positive or negative according to secondary being « outside » or « inside » at this point).
outDisplaceVector represents the vector from the rendered point on the primary object to the corresponding point on the secondary object. You'll see I'm not using it in the generation of the displacement maps but ti can be useful for a render than allow to specify displacement as a vector (not necessarily along the normal).
Then outColor returns the color of the secondary object where the intersection was found. I use it as a trick to pass back information from the secondary object to the primary object's shading network. Here by using a surfaceShader like the normalMapshader on the secondary object, it allows me to know the value of the normal on the secondary object intersection point and to generate a corresponding normal map for the primary. You could pass back different information by using a different type of shader on the secondary though.

Note : different shaders demonstrate what we get from these outputs in rayDisplaceShaders. Assign them to the primary object (low detail) and do a render to see how different outputs show in a render.

Using the outDistance attributes :
(shader found in rayDisplaceShader)

Here an expression is used to propagate the positiveMin and positiveMax values (see below in parameters) to the multiplyRange that will thus restore the normalised distances (green connection) to their original values. It could be skipped by directly reading raw distance values too, but it mimics the way baked maps will be used later. After that, the values for the negative distance is substracted from the value for the positive distance to get final displacement value.

rayDisplace parameters : there are several parameters to tweak to get the best results and generate best maps from the rayDisplace node.

returnColor : if not checked the node will only provide outDistance and outDisplaceVector and calculation will be faster. You need it checked if you use the outColor plug.
Bias : when the primary and secondary surfaces are very close to each other, the intersection point is often « missed » by the raytrace method. A small bias value gets rid of these misses. There can be an adverse effect of a too high bias value in places where the primary and secondary object fold very closely on itself, but most of the time you'd need a real high bias to encounter them. I used a value of 0.01 in this exemple, you can scale it accordingly to the size of your object compared to the example here.
positiveMax and positiveMax : the highest values for the positive and negative distance. Distance will still be returned if above these values, but they're used when range or clamp are on to output distance as a normalised (0-1) value, as anything outside that range will be lost when baking the distance as a map. As for now Maya can only generate / use maps in formats using 8bit per color channel, so the color resolution is an issue. To have best result with displacement maps especially, and avoid « stairsteps » effects you need to use the full available greyscale range and setting Max values help you to do so. If these values are too high you'll have poor color resolution, if they're too low you'll loose some of the most displaced zones. See below for tips in settings these values automatically and refining them.
positiveCut and negativeCut : if some intersections are missed, it can happen the ray hits the other side of a close secondary object, giving an artifically high value as a result. This would show up as a long spike in the displacement. By using the cut parameter you can decide to set distances above these values to zero and get rid of the spikes. I didn't have to use these parameters since I introduced bias, but I left them in case you encounter localised errors of this type.
range : if on, then the outputDistance values will be scaled to the 0-1 range using positiveMax and negativeMax.
clamp : if on, distance values above positiveMax and negativeMax are clamped (set to max).
cut : if on, distance values above positiveCut and negativeCut are cut (set to zero).

The rayDisplace attributes :

Tips for setting ranges : to ease up the determination of the positiveMax and negativeMax values I use a locator to store values, a small script to recover them and I do a first low quality / low resolution render to initialise ranges.

Get the scene for range initialisation here : getRanges.zip or here : getRanges45.zip

The locator displacementRanges holds attributes storing highest encountered values. It will update them whenever the shading network is evaluated, unless the attribute « initialised » is on, locking them. Opening the hypershade or the shading nodes attribute editor will generate irrelevent values, so best way is to set them to zero, set initialised to off, and start a test render. Then after this render lock them by setting initialised to on. The expression that will cause the ranges attributes to update is part of a shading network, so you can assign the shading group to the facets for which you want the ranges evaluated and them only. Setting some transparence for the shader allows to evaluate both front and back facets. You'll notice I left some facets out, in the armpit area and the mesh borders area. Borders of non closed meshes like this one can generate high values and I preferred to get erroneous information for these zones than to reduce overall color resolution. Same for the armpit area, we'll see later that the use of normal maps on top of displacement maps can very easily hide small glitches in displacement so small erroneous areas are sometimes a good tradeoff for an increased overall quality.

Once you have baked your maps as we see later, a good way ot evaluate color resolution quality is to display « levels » in Photoshop. You should have levels spanning the whole 0-255 range as much as possible or you're « wasting » some bandwidth. That's why you'll notice I set lower Max values than those you can read on the displacementRanges attributes in my exemple.

You could also sometimes gain quality by using a gamma node to adjust the output, you'll just have to remember to use an inverse gamma (one divided by the value you used) in the shading network when using the baked map later.

You can get about the same result by using both OutDistancePositive and OutDistanceNegative to generate twin maps or use only OutDistanceUnified. However the first method will give you double the resolution you'd get with the second. You can use the getDisplacementPositiveNegativeShader and assign it to the primary object ot preview results and fine tune settings before you bake distance.

Here you see glitches caused by setting a Max value too low : patches are « clamped out ». Don't worry about the faceted look. The intersection doesn't take « smooth shading » into account as it is only a smoothing of the normals. Of course you could crank up the secondary object resolution as much as possible with a polySmooth (or even make a subdivision surface out of it) as you are not going to keep it anyway. But these « facets » are more caused by discontinuities in the displaced object normals than real space position issues and the normal mapping will take care of them nicely.

I also didn't care too much about the glitches at the mesh borders, and preferred to keep them out the Max range. A typical polygonal mesh for an organic character should be closed anyway.

Baking the displacement maps : you have all the shaders you need in the rayDisplaceShaders (rayDisplaceShaders45) scene, but you can also get a stripped « ready to bake scene » here bakeDistance (bakeDistance45).

The shader « getDistanceShader » should be assigned to the primary object, then call the rayBake utility coming with rayDiffuse.

Choose « rayDisplace » as node type. Then you can bake outDistancePositive and outDistance Negative in two passes or save one pass and bake outDistance. In the resulting image, the red channel will be the positive distance, the green for negative and blue for unified.

In the « Output Setup » tab you can choose the texture size, UV range and edge padding. You should bake a reasonnable texture for first tests, then as big a texture as you can (use a size that is a power of two like 512, 1024, 2048, 4096) as displacement is quite sensitive on map quality. It seems that you'd need twice as big a map for a good bump displacement as you'd need for color, depending on how much you plan to allow primary object to be tessellated by displacement. Also as baking textures don't allow you to set sampling rates like a regular render, it might be good to bake a map twice as big as the one you'd want to use, then resize it down afterwards to get extra smoothing. Luckily it's quite fast (I could bake a 4096 on this model in about one hour). UV range depends on how the primary object mapping is done. You'll need proper UVs to generate a map and it will only be as good as your UV layout (UV stretching will show on the resulting map). Also as with Maya's native « convert to solid textures » you might need to correct UV seams manually to make the map « bleed » a bit over them (though edge padding can help you with just that, but it also seems to cause some edge artefacts).

Finally, the primary object must be selected when you start the baking process.

You can check the scene in final.zip for baked distance maps and the shader that will use them. It's quite simple, first multiply them back by their respective Max value (positive or negative) as distances were stored as (0-1) values (as colors), then substract the negative distance from the positive one and you have your displacement value.

There are several factors that will affect the quality of your maps. Color resolution (if you get « banding » or « stairsteps » you need to adjust your Max values). Then size of the map, and quality of the UVs, seams and non UV mapped areas will cause glitches that you might have to correct on the map (you'd need UV mapped caps on the mesh holes to get rid of the small spikes you see in the example). Finally resolution of the secondary object will show to some extent.

One thing you can't totally avoid is that the baking process seems to create faint lines following the low resolution mesh edges on the map, especially in high curvature areas. The problem gets more visible when you use the « edge padding » option too. Baking two maps, one without « egde pad » and one with, then combining them in a paint can help there.

However don't worry too much about these kinds of defects. Normal maps will hide about anything, the only criteria should be, can this defect be seen on the « silhouette » of the model? If yes it involves inacurate space positions and it's worth the time correcting the displacement map, else it just show inaccurate normals and the normal map will sort it.

Baking the normal map : same here the getWorldNormalsShader used for baking is in the rayDisplaceShaders but you can get a stripped « ready to use » scene here : bakeNormals.zip or here bakeNormals45.zip

Here the normalMapShader is assigned to the secondary object and will set its color according the the normal values (you can set and pass negative color values in a render until the final stage where an image file is generated or « clip final color » is used). The shader also checks if we are on front or back of a face to set normals accordingly. The getWorldNormalsShader recovers the information as the outColor from the rayDisplace node, then converts the normals to world space and normalises their value to color range (0-1). You can bake them the same way you baked distances, using rayBake. You won't need such a big map for them (about the same size as you would set a color map for your model and planned shots). The baked map is using the 3 color channels to represent X, Y, and Z of the normal vector. As in the map will be expressed in world coordinates, you might want to convert it to object space for an object whose tansform will be animated, or to point space for animated deformed meshes. These conversions can be done using a spaceChange utility node, and baking the converted map you'll have to take into account some limitations of Maya's bake but you can find workarounds here. You'll see the low detail object with the normal map only looks quite close to the high detail one, apart form the « silhouette » that breaks the illusion.

Now in final_images.zip you'll see the 3 baked maps and in final.zip or final45.zip the shader using them. You have no more need of the secondary / high detail object now and can discard it. As you see the rendered image is quite close to the original. Note that you can still edit the displacement map to correct the small glitches or add detail. Also the normal map don't prevent you from using a regular bump either, just insert the bump the same way that is used to « chain bumps », between the last node of the normals network (the vectorProductWorldToEye) and the shader node. The normal map values will be connected to the « normalCamera » input of the bump node instead of the shader node in that case.

The final shader using the 3 maps on the low detail object (only 281 face!)

Here you can see the little defects caused by mesh border edges in the form of spikes on the borders. In the image on the top of the page, I just used a 2D image editing program to manually iron them out of the displacement map.

Left you can see how to chain normal camera connection to use one (or several) bump « on top »...

Note that you can use any kind of surfaces for the primary and secondary object as long as you can render them. They don't have to be of the same type either, you can recover displacement range from a nurbs onto a polygon or subdivision etc...

Here, a polygon arm model, a simple nurbs cylinder, and check the animation ;)

Click the image for the turnaround	Generated from this simple cylinder...	And this displacement data

As a final comment I'd advise you to use tessellate high level objects as high as your machine can take it before « baking » distances. You'll get best displacement map smoothness due to the better definition of the high resolution object, and this object will be discarded once you have the map anyway so it won't be a cause of added complexity/weight when doing your final render.

You might also have use for that small helper plugin I did while testing these methods : spaceChange

Finally you will likely encounter difficulties working with subdivision surfaces and rayBake, or trying to express normals in point/tangent space, it might be useful for you to check the workarounds page : rayDisplace_workarounds

You are most welcome to write and give me some feedback, and to send your finest samples to the user gallery!

Olivier Renouard

olivier AT drone DOT org