Often one wants to include synthetic1 objects into an existing
photographic background plate filled with real world objects,
for example, adding a yet-to-be-constructed building into an
empty lot, adding a virtual car onto a road, or having a virtual
character walk through a scene and realistically interact with
objects in the real world scene.
Two main shaders exists to facilitate this, mip_cameramap,
which "projects" an image from the camera onto geometry, and
mip_matteshadow which takes care of generating hold-out
mattes, as well as allowing the real world objects in the
photographic plate both cast and receive shadows, as well as
receive reflections and indirect light.
This shader is used to look up a color texture based on the surface
points on-screen pixel coordinate. The shader is similar in function to
mib_lookup_background from the base library,
but with the following important differences:
The map parameter is a color texture to be looked up,
and multiplier a multiplier for that map.
If necessary, the shader can apply an inverse gamma correction to
the texture by the degamma parameter. If gamma is handled
by other shaders in the chain, or if the global mental ray gamma
is used, use a value of 1.0, which means "unchanged".
When per_pixel_match is off, the texture is simply stretched
to fill the scene exactly. When per_pixel_match is on, the
lower left pixel of the map is exactly matched to the lower left rendered
pixel. If the pixel size of the map and the pixel size of the rendering
is different a warning is printed, but the image is still rendered
although the image will be cropped/padded as needed.
Sometimes one wants the shader applied as a background, but still want
to extract information from the alpha channel.
If transparent_alpha is on, the alpha values is always zero.
If it is off, the alpha value from the texture is used.
The shader performs a true "reverse transform" of the shading
point into raster space. While this by necessity results in an on-screen
location for eye rays, this is not so for reflections, refractions etc.
These rays may hit a point on the object which is off screen. If
the offscreen_is_environment is on, these rays will return
the environment. If it is off, the offscreen_color is
returned. In either case, this color is not affected by neither
the multiplier nor gamma.
This shader is used to create "matte objects", i.e. objects who
is used to "represent" existing real world objects in an existing
photographic plate, for the purposes of...
In all above cases the mip_matteshadow is applied to an object
representing the real world object, and the synthetic object is
using a traditional material.
The shader can also function as a "shadows only" shader, i.e. a shader
which only shows how much in shadow a point is compared to the incoming
light, but ignoring the actual amount of incoming light itself (only the
occluded percentage of it).
NOTE: This section will only briefly lists the parameters
and the section "usage tips" on page matteuseagetips gives an in
depth explanation of different use cases.
The shader has multiple outputs (it returns a struct). However, by default
(for compatibility reasons) it only fills in the first item of the struct,
the compound "result". Only if the parameter multiple_outputs is
enabled are the separate results written to the other outputs.
The background parameter is the background color. If neither of the
catch_... options are on, this result is simply returned, including
its alpha, and the shader does nothing. Otherwise it is the base color upon
which all the other operations occur. When using external compositing this is
generally transparent black (0 0 0 0), otherwise one would use the real
world background plate mapped with the mip_cameramap shader.
The catch_shadows option enables other objects to cast shadows
on this object.
The shadows parameter is the color of the shadows. When shadows
are detected, a blend between background and shadows
depending on how much "in shadow" the point is.
The ambient parameter sets a "base light level". It raises the
lowest "in shadow" level. For example, if this is 0.2 0.2 0.2 the darkest
shadow produced will be an 20 percent blend of background to a 80
percent blend of shadow (unless ambient occlusion is enabled).
Turning on the no_self_shadow option (and also using an
instance of the shader as shadow shader) causes any object using
mip_matteshadow not to receive shadows on any
other such object.
The use_dot_nl option defines if the angle to the light is
considered when calculating the incoming amount or not.
If color_shadows is 0.0, all shadows are cast in grayscale.
If it is 1.0, the shadows have full color. For example, if the surface
is lit by one red and one green light, the red light will have a green
shadow, and the green light will have a red shadow.
If ao_on is enabled, a built in Ambient Occlusion
(henceforth simply called "AO") is applied based on the color in
the ambient parameter. The AO respects the
no_self_shadow switch and will not cause AO from objects
with the same material instance as itself.
The ao_dark parameter decides how dark shadows the AO
will cause. The default black is generally adequate, but a lighter
color will cause a less pronounced shading effect.
ao_samples number of AO rays are shot. One can limit the reach
of the AO rays with the ao_distance parameter. If it is zero,
the rays reach infinitely far. Short rays increase performance dramatically
but localizes the AO effect.
The catch_reflections turns on reflections.
The refl_color is the multiplier for reflections. Note that
the alpha value of this color is important in that the reflections will
influence the alpha by this amount. In some cases the reflections may
need to be subtractive, in which case refl_subtractive is
used - see page matteuseagetips for details.
The refl_samples parameters sets the number of glossy
reflection samples. If it is zero, mirror reflections are used.
Otherwise the refl_glossiness sets the ward glossiness
for reflections.
Reflections are seen at most refl_max_dist and the shape
of the falloff is refl_falloff, very similar like the
base library's shader mib_glossy_reflection.
If the option catch_indirect is enabled, indirect light is
gathered and scaled by the indirect color (which one generally
sets to the same as the background color, thereby treating the
background color as a reflectance value for the indirect light).
The multiple_outputs causes the shader to output more than
a single value in the returned struct.
If the catch_illuminators switch is on, the lights listed
as illuminators are tested and allowed to actually light
the scene. This is a simple Lambertian illumination treating the
background parameter as the diffuse color2.
The additional_color is a color input simply added to the
result, for easy shader graph construction. It can be used for anything3.
Finally, mode sets the light inclusion/exclusion mode and
lights is the light list used for casting shadows, just like in
many other shaders.
The shader also has multiple outputs. For compatibility, the shader
does not actually write any values to anything but the main output if the
multiple_outputs parameter is not on. But if this parameter is
enabled, the shader outputs the following values:
All outputs are as "raw" as possible to be maximally useful as layers in post production,
e.g. the reflections have not been multiplied with the reflection color, etc.
This is a material phenomena that embeds mip_matteshadow, and
applies it as surface, shadow and photon shader for a material. It has the same
parameters as mip_matteshadow itself, plus a scalar opacity.
The opacity does not apply to the shadows or photons, but only to the visible
surface shading itself. By using the opacity, one can use painted masks
for irregular edges. For example, use a simple tapered cylinder to roughly
match a forearm, and use a camera-projected mask (applied with the help of
mip_cameramap) for the actual contours of the arm. This also
allows matching motion blur that may be present in the background plate
with the help of masks.
The shaders in this section are concerned with combining a real
world photographic plate with synthetic objects. For the sake of
our examples, we are basing them on the following photograph
of a kitchen counter, with an old beat up plastic salt shaker,
a spice jar, a piece of paper, and some other things on it:
This is our background photograph. This is not a rendering... yet.Introduction
mip_cameramap
declare shader "mip_cameramap" (
color texture "map",
scalar "multiplier" default 1.0,
scalar "degamma" default 1.0,
boolean "per_pixel_match" default off,
boolean "transparent_alpha" default off,
boolean "offscreen_is_environment" default on,
color "offscreen_color"
)
version 4
apply material, texture, environment
end declare
mip_matteshadow
declare shader
struct {
color "result",
color "shadows_raw",
color "ao_raw",
color "refl_raw",
color "indirect_raw",
color "illumination_raw"
} "mip_matteshadow" (
color "background" default 0 0 0 0,
# Shadows
boolean "catch_shadows" default on,
color "shadows" default 0 0 0 1,
color "ambient" default 0.2 0.2 0.2,
boolean "no_self_shadow" default on,
boolean "use_dot_nl" default on,
scalar "colored_shadows" default 1.0,
# AO
boolean "ao_on" default on,
color "ao_dark" default 0.0 0.0 0.0,
integer "ao_samples" default 16,
scalar "ao_distance" default 0.0,
# Reflections
boolean "catch_reflections" default off,
color "refl_color" default 0.2 0.2 0.2 0.2,
color "refl_subtractive" default 0.2 0.2 0.2 0.2,
integer "refl_samples" default 0,
scalar "refl_glossiness" default 10.0,
scalar "refl_max_dist" default 0.0,
scalar "refl_falloff" default 2.0,
# Indirect
boolean "catch_indirect" default off,
color "indirect",
# System
boolean "multiple_outputs" default off,
# Additional illumination
boolean "catch_illuminators" default off,
array light "illuminators",
# Extra input
color "additional_color" default 0 0 0,
# Light linking
integer "mode",
array light "lights"
)
version 6
apply material, texture
end declare
mip_matteshadow_mtl
Usage Tips

![]() mib_lookup_background | ![]() mip_cameramap |
Notice how the left image, which uses mib_lookup_background looks incorrect. It looks transparent rather than reflective. What is the cause of this?
Imagine we are rendering pixel coordinate 200,200 which lies on the sphere. An eye ray is sent that hits the sphere. This hit point is indeed on the 200,200 pixel coordinate. But the ray continues as a reflection ray to hit the ground plane.
This new point (on the ground plane) is not the same as the pixel being rendered, it is somewhere else in the model. Yet mib_lookup_background only concerns itself with the current pixel (200,200) and will return the color at that point from the map.
Conversely, mip_cameramap actually converts this new point to a new set of raster coordinates (e.g. 129,145) and will look up the map at that new position, creating the correct appearance in the reflection.
However, there is a snag; what happens if the reflected point lies outside the screen? What if the calculated raster coordinate is (-45, 39)? The answer is that mip_cameramap either returns the specified color, or, if offscreen_is_environment is on, returns the environment color for that ray direction.
![]() The scene fully lit | ![]() The scene in shadow |
This technique is especially useful since it solves any self-shadowing issues perfectly, it is even advised to have no_self_shadow off in this mode. Here is the result:

Shadows come from unlit version of photo
This gives amazing detail and automatically gives the "correct" color and strength of shadows without any tuning being necessary.
A final way is to handle shadows completely in external compositing. This is accomplished by setting background to 0 0 0 0 (i.e. transparent black, alpha is zero) and shadow to 0 0 0 1 (i.e. opaque black, alpha is one).
This will create a rendering as follows:
![]() The color channels | ![]() The alpha channel |
This image can be composited on top of the background externally since the shadow information is present in the alpha channel.
Since we are using mip_cameramap reflections and refractions
of the background objects work correctly:
Reflection and refraction
Note how even the reflection of the salt shaker is correct.
Note that this also means final gathering will pick up the background
as indirect illumination correctly:
Final gathering lights up our objects
The shader can also show (glossy) reflections of other objects:
Glossy paper reflects our objects
Please note how the reflection only involves the actual synthetic
objects added, the shader already assumes that the real world
photo contains reflections for the other real world objects, i.e.
a mip_matteshadow object never reflects another, nor
does it reflect the environment.
One subtle detail is still missing, though. Notice how the
cylinder is reflecting the paper, but nothing else. This is
because the mip_matteshadow does not know what to
do about rays that hit the ground plane somewhere "off
screen".
No reflections
This is solved by using the offscreen_is_environment
parameter of mip_cameramap, and then utilize
an environment map (for example mip_mirrorball) to
supply an environment map:
Much better
The shader can also receive indirect light such as final
gathering and (if also applied as photon shader) photons.
To demonstrate this we enable catch_indirect and turn
our white objects luminous red and made sure our glass sphere
generates caustics:
Receiving indirect light
As above, the shader makes sure no indirect light is bounced
from one matte object to another, and no indirect light is received
from the environment (since both these effects are assumed to be
present in the original real-world photo).
A final special case for reflection is when the background photo
already contains massive amounts of reflections. Here is
a new background photo on top of a highly reflective surface:
Photograph of our salt shaker on a very reflective surface
If we add our synthetic objects to this scene and use the same settings
we used for the glossy paper, we get the following rather odd looking
results:
Additive reflections - doesn't look right
But this looks very incorrect! One can see the reflection of the
background through the added reflections, and the added reflections
appear much too bright.
To fix this one uses the refl_subtract to set the amount of
attenuation of the background caused by the reflection. Naturally this
can be a painted map to match reflective areas in the image, i.e. a
water puddle in front of a house etc.
Subtractive reflections - that's much better!
The correct result. The synthetic reflections override the
existing reflections and replace them! Also note that even
reflections of objects
behind a real world object inside a real world reflection
works correctly (the white ball
behind the salt shaker is behind it even in the reflection).
Finally, all objects pick up indirect illumination based on the
background plate itself. Only one rectangular area light source exists
in the rendering.
When preparing for external compositing and using transparent black
for background and opaque black for shadows it is
important that reflections also get an alpha channel.
The alpha component of the refl_color parameter defines
how much the reflection is present in the alpha channel.
Likewise the alpha component of the indirect parameter
defines how much indirect light is seen in the alpha channel.
Neither of those have any function if the alpha of the background
is 1.0, i.e. when using a background photo.
If one need the synthetic objects to reflect the real objects but
still want the flexibility of external compositing for shadows (i.e.
one does not want to add in the background already when rendering)
one can create a hybrid approach.
This is accomplished with the help of the mip_rayswitch
shader. Use the shader to switch between using a transparent
black (0 0 0 0) background for eye rays and using the camera mapped
photo for other rays (e.g. reflection, refraction and final gathering).
Here is a simple step-by-step workflow to render synthetic objects into a
background plate. It assumes one has
Prerequisite stuff
Then perform as follows:
The shade tree
Add any CG objects with physically plausible shaders (such as the
mia_material from the Architectural library). Render. Smile.
Render. Smile.Reflections









Reflections and Alpha
Best-of-Both-Worlds Mode
Conclusion and Workflow Tips



Footnotes