UV smoothing in Maya

(why is it ugly and some workaround)

There has been discussion again on Maya highend3d mailing list about the UV distortion caused by polygon smoothing in Maya. I have been working on this problem and devising solutions for a recent production, and as it's hard to explain verbally in a reasonably short email I thought it would be best to sum it up in some tutorial.

Why does Maya create ugly UV distortions when smoothing polygons?

When you apply a polySmooth in Maya, new vertices will be created, as well as new UV coordinates for these vertices. Both the space coordinates and the UV coordinates of these new vertices will be derived from the old vertices (the original or « low res » mesh). The problem is whereas new vertices position are obtained through smooth interpolation, the new UV coordinates are obtained through simple linear interpolation. This is the cause of disrependancies between these new vertices space coordinates and UV coordinates and thus, map distortion. If I'm telling you something that you already know, then you can jump directly to the « How to make it work » part and skip some of the babbling!

To give a visual example of the difference between smooth and linear interpolation, you can see the effect on a smoothed mesh when setting the polySmoothFace « continuity » parameter to 0. You can see below the difference between « linear » and « smooth » interpolation on the same original mesh :

Original shape	Linearly interpolated shape	« Smoothed » shape

Usually you'll model a low resolution version of it and smooth it using Maya's polySmooth with maximum continuity (that is 1). The new vertices positions will be nicely interpolated and you'll get a smooth model, but the UV coordinates will not, they'll just be linearly interpolated you will get texture « stretching ».

Let's take the example of a polygon head I modeled some time ago, it's a rather light model, nothing fancy in the way it's built : polys are all 4 sided, some vertices are part of 5 edges (which is unavoidable for organic models made of 4 sided polys) and in some occurrence 6 (I'd avoid this actually). You can see the mesh below, and it behaves allright when smoothed as far as volume is concerned:

And that is the UV layout as it has been done (credits to mapper Alban Orlhiac here) on the low resolution mesh :

If we use a grid texture to check UV distortions, we can check how texture will behave on the smoothed object, yuck!

Original shape	What Maya does	What we would like to see

If we take a close up look at the cheek area you'll notice immediatly the texture stretching artefacts caused by the use of polySmooth :

« bad » smoothing	And « good » one

To understand what is happening at UV level, let's check how the smoothed object UV layout compares to the original object UVs :

The polySmooth generated UVs	The UV that would fit the smoothed surface

The UVs generated by polySmooth do not fit the smoothed surface because they're just linearly interpolated (see the « straight lines » in the left image), and they'd need to be smoothly interpolated the way the mesh is, if we superpose the UV layouts we can see where distortions will happen :

Original UV are in white, bad ones in red, and good ones in green

How to make it work?

So the problem is that Maya is not smoothing vertices UVs the same way it smoothes vertices space positions (actually it doesn't smooth UV at all). The idea is then to create a new mesh that expresses UV as space coordinates, smooth that mesh, then use the resulting vertices space coordinates to specify the UVs of our final model.

To create a mesh object that represents the original mesh UV coordinates in 3D space, luckily Imre Tuske created a great plugin, called xyPolyTransferUV that you can find on Highend3d here :

http://www.highend3d.com/maya/plugins/?section=polygon#1842

There is some documentation in the header of the associated MEL script xyPolyTransferUV.mel, by selecting your original (unsmoothed) mesh and calling his script with this command : xyPolyTransferUV(0) you'll create a flat object representing the UV space of the orginal mesh :

After that, just duplicate the original mesh and the « flattened » object, and smooth both of them using the same « continuity » and « divisions » parameters. Set « keep tesselation » to on, you might want to set « keep border » to on also or you will have to rework map edges.

Now we have this smooth « UV object » that represents the UV coordinates we ideally would like see on our smoothed model, because Maya interpolated the position of the new points using it's space smoothing method. Notice how the smoothed flattened UV object differs from the UV layout on the smoothed head :

UVs on the smoothed head	The smoothed flattened UV object

What we have to do is to transfer the UV information back from 3D space to the smoothed head model. Though it is theorically possible to do it with xyPolyTransferUV, there are several issues that keep us from using this plugin to do so. Either it will fail when discontinuous UV hulls are present like in this example, or it will recreate a new high resolution geometry with the correct UV, but then will make it hard to integrate in a full skinned, blended object history like we use for animated characters.

First we'll have to map the flattened UV object so that it's UV coordinates match its space coordinates. It's quite straightforward as the X and Z coordinates of the flattened object correspond to the UV coordinates they represent, thus for a normalised UV layout (set in the 0-1 UV range), a simple planar projection along the Y axis will produce the desired result :

The projection parameters :

The projection

And the resulting UVs

Then using Maya's menu command « Polygons / Transfer » and checking only « UV sets » in the option box, we can transfer back the UVs from the flattened object to the smoothed head :

The Polygons/Transfer options

And the smoothed object with « nice » UVs

Conclusion :

Here we have a way to obtain correctly smoothed UVs, and to apply them back to the smoothed object even when it has a complex history as for animated characters (in that case, the polyTransfer node should come last in the history). Any « divisions » or « continuity » parameter can be used for the smoothing as long as both the base mesh and the flattened UV object are smoothed with the same parameters (it can be a good thing to link the parameters of both polySmooth nodes so you can modify one and have the other update automatically). The polyTransfer node can generate messed UVs as smooth resolution is adjusted up or down, so it's a good idea to delete/re-create the polyTransfer node before the final render.

To make this workaround cleaner and easier to use, you could:

hire developers and « package » the whole procedure in a history friendly updatable Maya node (that's what the company where I first devised this trick did)
petition Maya so they fix their polySmooth node :)
or mail Imre Tuske to tell how much you appreciate the great tool he or she released and if a history friendly way of getting back multi-shell UVs on a given mesh (without creating a new mesh due to animation issues) could be added? (I 'm going to do that actually).

Olivier Renouard

olivier AT drone DOT org