Recovering normal and displacement maps
from existing geometry with Mental Ray
and Maya

(Mental Ray shader, sample shading networks and scenes)

If you need an explanation about what normal maps are you can check here : normal_maps, however you will have much faster render times using the plugin and methods described here.

The Mental Ray shader described here is a port of a custom shading node plugin I wrote for Maya's API described here : displacement_maps. It uses the same parameters and works roughly the same way the original node did.

Also it's a good idea to check in case I did a newer version of the rayDisplace.dll. For manjor releases I'll try to update this main page, but for minor modifications and bug fixes I probably won't have enough time so I'll just mention them here (beside posting them on Highend3d): rayDisplaceMental

And finally there are some issues when « baking » (using « convert to file texture » or « rayBake » to output data provided by rayDisplace as an image file) that I can't fix directly as they're caused by limitations of either Maya's, Mental Ray or rayBake « convert to file texture » method. Until these issues are fixed, it might be useful for you to check the workarounds page.

General idea : given a high detail and low detail version of a polygonal mesh, I'm using this plugin to generate a displacement map and a normal map that once applied to the low detail mesh will give it a rendered appearance as close as possible to the high detail render.

Applications : from a heavy, detailed object (like the one you would get from 3D scanning or the use of Zbrush great detailing tools) you can derive a custom built, animation friendly, low resolution object and the displacement and normal maps that will make it quite identical to the high detail version at render times.

The advantage with using Mental Ray is that you can directly bake maps from shading networks using ray tracing (with « convert to file texture (Mental Ray) » in the « Edit » menu of the hypershade. Also you can bake 16 or even 32 bit per channel images from Mental Ray convert utility, to small a color resolution being an issue for displacement maps. I'd recomend using at least 16bit per channel images (you can create 16bit TIFs that offer a good compromise, being very good color resolution and editable in programs like Photoshop, and that Mental Ray, Maya or Renderman all can use as a displacement map.

Note : Mental Ray can have problems opening and using images saved by other applications like Photoshop saved TIFs, several ways around that include saving .tga or .iff (.iff plugin for Photoshop available on Highend3d), using Maya's image convert or Mental Ray imf_copy utility.

The bad news is you can't render subdivision surfaces in Mental Ray yet, you'd need their Mental Matter plugin. Even with it I'm not sure (actually I'd quite suspect the opposite) you'd be able to render maya's subdivision surfaces straight away as there is no indication the export for subd has been implemented yet in Mental Ray for Maya. I hope the Maya version 5.0 will bring that soon :). However Maya will tessellate all surfaces before render actually starts, so you would not have any differences using the method I described in the workarounds.

Low Def Torso	High Def Torso	Low Def Torso with maps applied

The plugin .dll and the associated .mi file can be downloaded here : rayDisplaceMental.zip

Exemple scene with both mesh, custom shader and shading networks : rayDisplaceMentalShaders.zip

And the baked maps here : maps.zip

Requirements : you'll need Maya 4.5 and Mental Ray for Maya 1.5 (the later can be found on Alias|Wavefront web site and is free for registered users of Maya 4.5 :

How to use rayDisplace : first you need to set an environment variable in order to be able any custom Mental Ray shader (even those already included). You can set it in your maya.env file with other maya specific variables :

MAYA_MRFM_SHOW_CUSTOM_SHADERS = 1

You need also to put the maya_rayDisplace.mi file either in Mental Ray for maya /include subdirectory or set up your custom directory and add it to the path using the environment variable MI_CUSTOM_SHADER_PATH.

Now maya_rayDisplace.dll goes in the /lib subdirectory or use (I think) MI_LIBRARY_PATH.

Now you need to get Mental Ray for Maya to load the custom shader at startup. I think can also do that by entering the path or names in the Custom Globals of the Mental Ray render globals but I'm not positive about it (any information welcome). The way I did it is by editing maya.rayrc file in Mental Ray /plug-ins subdirectory, you'll need to add the 2 lines that I put in red there :

The Maya.rayrc file :

#*****************************************************************************
# Copyright 1986-2002 by mental images GmbH & Co.KG, Fasanenstr. 81, D-10623
# Berlin, Germany. All rights reserved.
#*****************************************************************************
# Evaluated at startup time of the plug-in to fill the mental ray registry.
#*****************************************************************************/

registry "{MAYABASE}" value "C:/Program Files/AliasWavefront/mental ray for Maya 1.5" end registry

registry "{SYSTEM}" value "windows" end registry
registry "{DSO}" value "dll" end registry

$lookup "{MAYABASE}"
$lookup "{SYSTEM}"
$lookup "{DSO}"

registry "{MRMAYA_START}"
link "{MAYABASE}/lib/base.{DSO}"
link "{MAYABASE}/lib/physics.{DSO}"
link "{MAYABASE}/lib/mayabase.{DSO}"
link "{MAYABASE}/lib/contour.{DSO}"
link "{MAYABASE}/lib/maya_rayDisplace.{DSO}"
mi "{MAYABASE}/include/mayabase.mi"
mi "{MAYABASE}/include/base.mi"
mi "{MAYABASE}/include/physics.mi"
mi "{MAYABASE}/include/contour.mi"
mi "{MAYABASE}/include/maya_rayDisplace.mi"
echo "mental ray for Maya - startup done"
end registry

$lookup "{MRMAYA_START}"

I'd love to be able to specify to Mental Ray for maya that this maya_rayDisplace custom shader should replace my custom rayDisplace maya shading node when doing a Mental Ray render (the way Mental Ray for maya have defined equivalents for all maya basic shading nodes) but I've got no clue on how to do it yet, any information welcome!

Now you're ready to start! You can load the scene I provided and start baking the maps with the new « convert to file texture» in the Hypershade « Edit » menu provided with Mental Ray for Maya.

Here are the settings you'll need first (all the procedure is very close to the one described here : displacement_maps ).

Render settings : raytracing needs to be enabled for the rayDisplace to work, reflection level is not important here as long as raytracing is activated. You can set it in Maya render globals or Mental Ray render globals either way. Camera settings : some shaders use the camera worldMatrix, so all renders should be done from the « persp » camera or shaders should be modified to use the new camera instead.

Object settings : the primary object is the lowest detail object. It is the one that will be visible in renders. You need « primary visibility », « smooth shading » and possibly « double sided » to be on for it. The highest detail object is the secondary one and need not to be visible in renders, but need to be « seen » by the plugin.You need to have « primary visibility » turned off, but « double sided » and « visible in reflections » turned on. It also needs to have a shader (any shader, the basic lambert will do fine) applied to be visible. No more need for the special normalMap shader as with Maya method.

Displacement settings : the primary object will be displaced, so to allow enough detail you need to set up its Displacement Map parameters. Here the highest object being smoothed twice a value of 16 gives good result for initial sample rate. I had also best result basing the extra tesselation on texture rather than normals.

If the final render is to be done in Mental Ray as well, you can set the displacement tesselation specifically in Mental Ray instead of letting it derive Maya's parameters.

Note that you need Mental Ray to obtain the displacement and normal maps, but the final render using these maps can be done in Maya, Mental Ray or Renderman indiferently, check here for a Renderman shader using the maps : rendermanDisplace.zip

rayDisplace outputs : there are 3 groups of output that can be used to retrieve information out of a rayDisplace node.

First outDistance will give the distance between the primary object and the secondary one, at the point being rendered. It is the output that will be used to get displacement maps. Under it you'll see 3 attributes : outDistancePositive (distance when the secondary object is « outside » the primary object , this based on primary object normal direction on the primary, and will be null else), outDistanceNegative (when secondary is « inside » primary) and outDistanceUnified (distance is either positive or negative according to secondary being « outside » or « inside » at this point).
outDisplaceVector represents the vector from the rendered point on the primary object to the corresponding point on the secondary object. You'll see I'm not using it in the generation of the displacement maps but it can be useful for a render than allow to specify displacement as a vector (not necessarily along the normal). As for now the displace distance is already computed along the normal so there is no point in using it in this version.
Then outNormal (was outColor in the Maya rayDisplace node) returns the normal of the secondary object where the intersection was found.

Warning : so far outDisplaceVector and outNormal are returned in world space coordinates, whereas Maya node would return them in camera space.

Limitation : with Maya rayDisplace node you could directly feed the outDistance or another parameter to a displacement shading node. Don't try that in Mental Ray as it would just crash Maya. I think it's related to the fact displacement shaders are geometry shaders and geometry shaders can't make calls to raytracing functions... You'll have to bake a map first, then use the map in a displacement.

rayDisplace parameters : there are several parameters to tweak to get the best results and generate best maps from the rayDisplace node.

Bias : this is a leftover from the maya node. It was used as a workaround to avoid missed intersection at places where both object surfaces were very close to each other. The problem don't seem to be present in Mental Ray so I'll either leave it for compatibility if I manage to get the Mental Ray shader to automatically replace the maya shader, or remove it in later versions if it evolves to a purely MR shader.
positiveMax and positiveMax : the highest values for the positive and negative distance. Distance will still be returned if above these values, but they're used when range or clamp are on to output distance as a normalised (0-1) value, as anything outside that range will be lost when baking the distance as a map. With the possibility to bake 16bit maps the color resolution is much less of an issue, but it's still useful to set these quite close to the actual maxima so you can visually see the color zones in the baked displacement map. I didn't notice any adverse quality effect on the final displacement when I used a max value 10 times higher than the actual max (thus meaning I was only using 10% of the available color resolution), but of course the baked map appeared very dark to the eye!
positiveCut and negativeCut : if some intersections are missed, it can happen the ray hits the other side of a close secondary object, giving an artifically high value as a result. This would show up as a long spike in the displacement. By using the cut parameter you can decide to set distances above these values to zero and get rid of the spikes. Mental Ray's baking function seem to sometimes « miss » some texels and this parameter is useful to filter them out. Best is to set it a small amount above the corresponding positive or negative Max parameter.
range : if on, then the outputDistance values will be scaled to the 0-1 range using positiveMax and negativeMax.
clamp : if on, distance values above positiveMax and negativeMax are clamped (set to max).
cut : if on, distance values above positiveCut and negativeCut are cut (set to zero).
useSamplerInfo : Mental Ray got its own way of finding the position of the currenlty shaded point and its associated normal. However you can decide to feed them to the node using the normalCamera and pointCamera input attributes, and check this option to have the node use them instead. Note that you have to manually connect these inputs, to a samplerInfo equivalent outputs for exemple, as Maya won't update them during render the way it would do with a native maya shading node (another thing I have to sort out).

The rayDisplace attributes :

Tips for setting ranges : it 's much easier to set ranges as you don't have to maximize the use of the color resolution the same way you had to using the Maya shading node (provided you bake 16bit maps). Just look out for zones of pixels at full brightness (in one channel, thus full red (positive values) or full green (negative values) in the shading node exemple I provided which would indicate too low max values and unwanted clamping of the highest displacements.

Max is too high, you can barely make out a few non black pixels here!	Would be about ok and and a safe setting (little chance you're clamping anything out there)	Max is obviously too low ain't it?

Note : as for evaluating render, the good old trick of playing with your monitor/video board gamma to check images and spot clamped values values work well here too.

The use of Cut values : here you see typical artefacts created by occasionnaly missed intersections, by setting the Cut values just a little bit above the Max values these dots will be filtered out.

Spot the spots!

Baking the displacement maps : you'll bake the shader named getDistanceShader to recover displacement information and getNormalsShader to recover the normal map (normal values are offset to be expressed in the RGB (0 0 0 to 1 1 1) color range by the shading network, but otherwise unmodified).

Note : maps will be created in the lightMap project directory.

You can check the resulting render with the two shaders provided in rayDisplaceMentalShaders.zip (assign them to the low resolution object of course), lambertDisplaceOnly and lambertDisplaceAndNormals. They use the maps that you can get here : maps.zip

Here I forgot to set the « Cut » values	Still needs some « sandpaper »...	Looks better with the normals map applied

You have no more need of the secondary / high detail object now and can discard it. As you see the rendered image is quite close to the original. Also the normal map won't prevent you from using a regular bump either, just insert the bump the same way that is used to « chain bumps », between the last node of the normals network (the vectorProductWorldToEye) and the shader node. The normal map values will be connected to the « normalCamera » input of the bump node instead of the shader node in that case. The final render can be done in Maya, Mental Ray or any other rendering tool with good displacement, as it's just a matter of using the generated maps (you don't need the custom shader any more either).

Note that you can use any kind of surfaces for the primary and secondary object as long as you can render them. They don't have to be of the same type either, you can recover displacement range from a nurbs onto a polygon or subdivision etc...

Just to show off, see what can be done with a mere nurbs cylinder :

Click the image for the turnaround (divX codec)	Generated from this simple cylinder...	And this displacement data

As a final comment I'd advise you to use tessellate high level objects as high as your machine can take it before « baking » distances. You'll get best displacement map smoothness due to the better definition of the high resolution object, and this object will be discarded once you have the map anyway so it won't be a cause of added complexity/weight when doing your final render.

As said above Mental Ray for Maya current release doesn't yet render subdivision surfaces, also you might likely encounter difficulties trying to express normals in point/tangent space, it might be useful for you to check the workarounds page : rayDisplace_workaround

You are most welcome to write and give me some feedback, and to send your finest samples to the user gallery!

Olivier Renouard

olivier AT drone DOT org