Problem Rendering Mattes with MR

[Lord Futzi Voldemort]

I don´t know, if this is more a comp problem, but since the problem seems to be the same in afx, I´m posting this here.

Ok, simple thing:
I have an rendered image and want to mask out an area for whatever purpose. So i render a mattepass accordingly, using the very same geometry an put black or white constant shaders on, nothing special.



But then I decide to mask out the other part of the image by inverting the mask and -oops- I discover the white area is significantly smaller than the orignial colour area.



No matter what I do mental ray seems to render white areas smaller than the black ones. If I don´t invert the mask, but use it, for example, for masking a colour correction, it looks like this:



Or if I just want to comp my cut out image over the exact same are area in another pass, it looks like this:



I´ve been rendering and using mattepasses all the time, and never had this problem. It looks like the fringes are way more than one pixel, so it can´t be antialiasing only. It´s almost the same in maya, only the fringes seem an bit narrower rendered in maya. The Mask port uses red, because I need to use luma instead of alpha for masking.

All images are rendered with standard sampling, 1, 3, 0,036, triangle filter, 3px as 8 bit tiff. Switching colorprofiles doesn´t do anything either.
Using matte shrink or expand filters is not a solution, since they tend to soften the matte and make it even more inaccurate.

I have no idea at the moment why this happens and why it happens now. Has anyone else this problem? Mattes are quite useless at the moment for me, and that´s a huge problem.

Achim

Edit: It´s SI 2012 SP1 and Maya 2012 Hotfix2 (64 bitch both) by the way.

[Lord Futzi Voldemort]

Noone else has this problem?

Am I missing something?

Achim

[Maximus]

I've did a quick test and i have no problem, but i dont understand whats happening here.
What did you do to render that image? It is only one geometry with 2 shaders with cluster? or are 2 separate geometries?
I used 2 geometries and rendered fine, didnt try with clusters, that might be the problem.

In the framebuffer options in render manager do you have "Premultiply with Alpha" on or off?

It is a bit weird to be honest what happening.

[Lord Futzi Voldemort]

Hi maximus,
it´s separate geometry, the same geometry I use to render the other passes. I´m not using any alpha values here, only color channels. There shouldn´t be any change in the alpha channel anyway between the adjacent polygons of the two objects.
I have this problem in multiple scenes now and no idea what to do with it.
Can you post your test scene? I will post mine, when I´m back in the office.


Achim

[Maximus]

Well i didnt save the scene because it was just basic
Anyway i personally always use MM_Matte_Pass javascript from Felix Geremus script to create mattes, it is amazing, you can find it here:

http://rray.de/xsi/

or on his website in the download section

http://felixgeremus.com/

Could it be some Nuke option? i dont know nuke at all, i've just tried with fxtree.
What i did is create a grid, assigned a lambert to it, extracted the hole and gave another material, then i've created the matte pass with that script and rendered both beauty and matte pass as tiff 8bit rgbA.

thats it

[Zafar Iqbal]

Question for the masses:

Render a sphere. Use the alpha to remove the sphere 100% absolutely completely. Possible? If so then my workflow is wrong because I run into this issue all the time.

[Maximus]

Question for the masses:

Render a sphere. Use the alpha to remove the sphere 100% absolutely completely. Possible? If so then my workflow is wrong because I run into this issue all the time.

What issue? This is pretty straightforward to do.
I guess it depends more on what compositing software you use, but for example in photoshop you can easily do this, i dont see the problem, can you be more specific?

[Maximus]

I've only used the Composite->Composite With Matte Node after i rendered a tiff rgbA picture.
Using the alpha channel from the render to drive the composite node.

Is this what you need?

http://img854.imageshack.us/img854/8869/111dp.jpg

http://img39.imageshack.us/img39/574/000thr.jpg

[luceric]

it's premultiplcation


I don't use Nuke, but perhaps there is a switch in the file input node where you can specify that you image is premultipled. If so, perhaps Nuke will handle everything automatically.

When compositing you needs to un-premultiply the image before doing any Color Correction, and re-premultiply after. the problems would not normally be obvious and you could do a lot of work without bothering with this, but they will be obvious in this case. what's happening is that the mask for the color correction is being applied twice at the edge of the object

[Zafar Iqbal]

What issue? This is pretty straightforward to do.
I guess it depends more on what compositing software you use, but for example in photoshop you can easily do this, i dont see the problem, can you be more specific?

I always have edge related issues but they may not always be apparent. In fact lots of times it appears like my mattes are working just perfect. In the example mentioned, I can't render a sphere, save the file with its corresponding alpha and then in comp, use the alpha to remove the sphere. There is still going to be an edge/outline. Same issue in Photoshop.

I checked old renders last night - some rendered with a different 3d app and the issue is there too. And as said, Photoshop does the same thing but also when paint on a transparent layer and then using the transparency of the layer (as a matte/alpha) to get rid of what was drawn - result is very similar to comp: edges/soft borders from brushes are still there.

[Lord Futzi Voldemort]

it's premultiplcation


I don't use Nuke, but perhaps there is a switch in the file input node where you can specify that you image is premultipled. If so, perhaps Nuke will handle everything automatically.

When compositing you needs to un-premultiply the image before doing any Color Correction, and re-premultiply after. the problems would not normally be obvious and you could do a lot of work without bothering with this, but they will be obvious in this case. what's happening is that the mask for the color correction is being applied twice at the edge of the object

Again, I´m not using any alpha channel at all, and between the small square and the outer square there wouldn´t be any change in the alpha value at all, because it´s just opaque black and white geometry. Also switching from premultiplied to unpremultiplied in nuke doesn´t change anything either, and I get the same effect if I do the test in afx with alpha ignored.

I´ve attached the testscene. It´s relly nothing I haven´t done a hundred times before.


Achim

[luceric]

how can you say you're not using alpha channels?
The merge node uses the alpha, especially in 'over' mode.
It's going to multiply the RGB with the Alpha, and since your image is rendered premultipled in XSI, then the antialiased edge of the objects have already been multipled with the alpha during the mental ray render, and therefore it will be multipled a second time during Merge, resulting in a dark edge.

[Lord Futzi Voldemort]

I´m talking about the matte pass. The compositing over the background works just fine.
If you look at the masking for the color correction, I´m masking the the area of the smaller square in the middle out, using a simple black and white image. There are no edges which are composed over the background . You can see, that the colorcorrected area is smaller than the original one. The CC node doesn´t use any alpha at all. Also when I merge another color pass, using the matte for masking, not the alpha, the area won´t match, check the 4th example: the alpha of both passes is one (1) for the area of the BIG square. The small square, for which I rendered the matte, is merged only by using the mask input, using the red channel of the matte image, so no alpha involved here. If you want I can make a different example for you, that uses only fully opaque images. Just ignore, that in the example the first pass is composited over a solid. That is not the problem.


Achim

Edit: here´s another example making it even more clear:
A sphere over a grid. NO transparencies anywhere. I desaturated the red sphere in nuke using a matte, drived from the very same geometry, only with constant black and white shaders. I´m not rendering a white sphere over the backgroung, but I´m rendering the floor as black geometry. So NO transparencies there either, nothing gets premultiplied with anything here. Alpha is 1 for every pixel. As anyone can see, the matte doesn´t match the area of the sphere, thus leaving a red fringe.

[Maximus]

Oh well, i've just got the same problem, i didnt understand that you werent using the alpha channel.
Doing the same procedure with straight matte black and white mask produces this fringe problem, i've tried in photoshop and in FXTree, happens the same thing.

I'm also interested in solving this now

[Maximus]

so as far as i can understand, this is not a problem of the matte, but is the beauty pass itself.
if you just use the beauty + color correction and move the hue you'll see that fringe edge.

I think its kinda normal because that edge is the part where the 3d program blend the red color with the blue color, it is kinda impossible to remove (i guess). I've tried 32bit float exr and there is a little improvement but still is present.

I think its more an antialias pixel color blend problem, infact i've tried to enable the "Pixel coverage" framebuffer in softimage and you'll see that it render that thin edge line on a complete white background.

_Need to dig it further, any help is appreciated

here is the pixel coverage render

http://img600.imageshack.us/img600/9544/15846698.jpg

[Lord Futzi Voldemort]

so as far as i can understand, this is not a problem of the matte, but is the beauty pass itself.
if you just use the beauty + color correction and move the hue you'll see that fringe edge.

I think its kinda normal because that edge is the part where the 3d program blend the red color with the blue color, it is kinda impossible to remove (i guess). I've tried 32bit float exr and there is a little improvement but still is present.

I think its more an antialias pixel color blend problem, infact i've tried to enable the "Pixel coverage" framebuffer in softimage and you'll see that it render that thin edge line on a complete white background.

_Need to dig it further, any help is appreciated

here is the pixel coverage render

http://img600.imageshack.us/img600/9544/15846698.jpg

In fact the same problem occurs, if you use that matte to composite something over the same area. It occurs in almost any operation you use a matte to mask. Shifting hue always produces weird effects, because you get a false color image, thats why I tested with saturation only.
It seems to me as if there was some antialiasing bias towards brighter or darker pixels.
I would in fact expect a tiny area, where the matte won´t match, but not as big as it is.

Achim

PS: by the way, in any of my posts in this thread there´s a hint, that I don´t use alpha.... ;)