When matchmoving stereo images, both the left and right eyes must be tracked and they must be placed in the same stereo space. Typically, the analog encoder stereo meta data from the camera rig is not accurate enough for demanding shots such as set extensions. This is not a criticism of the Pace Camera system. After spending many years operating motion control systems, it became obvious that mechanically measuring the exact sub-pixel position of cameras and optics (outside of laboratory conditions) is almost impossible on a film set. The stereo meta data from the Pace camera will get you close to an accurate stereo reading, but only stereoscopic matchmove will provide an exact result.

If you attend a stereoscopic film (such as Beowulf), you can do an experiment that will illustrate some 3D principles.

First,  remove the 3D glasses from your face. Observe that the stereo effect is caused by the fact that closer objects will display more left right separation on the screen and distant objects will be more "converged". In Imax films, objects at infinity will typically have no divergence and closer objects will diverge. But in other systems, the stereographer will often pan the left and right cameras towards one another minutely, changing the convergence point by using camera toe-in (ry). In this case, both near and far stereo objects will diverge and only a mid point will converge. A small adjustment of the pan angle between the two cameras has a large visual effect on the audience. On a 2k image, the near objects can have no more than 80 pixels of stereo shift and the distant objects can have no more than about 30 pixels of negative shift (wall-eye). The human eye will tolerate more stereo separation for near objects (80 pixels) than for distant objects (30 pixels). This is because the human eye muscles are built to pan towards one another, but not to pan away from one another (wall eye). These pixel separation limits (80 hither, 30 yon) are subjective, approximate and depend on projection techniques, choreography and editorial style.

A surprisingly large amount of vibration between the left and right cameras can exist and the stereoscopic images can still easily be fused by the human eye. For example, on T2-3D, our twin 65mm stereo cameras were placed on a camera car driving on rough terrain. Because of the cantilever design, the two cameras would shake against one another. If you removed your stereo polarizing glasses in the theater, the vibration between the two images was disturbing. But when you put your stereo glasses back on, your eye fuses the images perfectly and the unsteadiness between the stereo images disappears. But if we are talking about a match move vfx shot, then the vibration between the two eyes will not be accurately recorded in the meta data and Unified Solve may be necessary to converge the left and right match moves properly. Even though the Pace stereo camera is a low vibration design, mechanical and optical inconsistencies between the left/right optics can show up as the lens is rack focused, etc.

The job of the stereo match mover is to figure out what the convergence and interocular of the original camera was set at. As mentioned, for critical shots like set extensions, the encoder data will not be good enough and unified solve (to be defined shortly) is needed.

FYI the human eye can easily see a half pixel shift in stereo placement. I will give an example from Ant Bully - 3D. A typical shot would be of an ant walking on the ground, ant and ground elements rendered in different passes and then combined in Nuke. But the uneven terrain was rendered with displacement mapping, which meant that the original smooth, flat ground plane geometry that the character animator originally walked her ant over is now bumpy. The rendered ground then had variegated height that the character animator could not anticipate. In monoscopic, this is not a problem, but in stereoscopic, the CGI ant would often appear to either float above the dirt or to have her feet buried in the dirt. It is generally only practical to fine tune this fix in the final 2D comp, not in the eariler Houdini or Renderman stage. And we found that the eye could sense a stereo mismatch of as little as half a pixel (at 2K).

The important point is that for critical shots, the left and right eyes must be match moved in depth precisely to one another. Many of the JCE shots involved actors floating in air and so their feet would not actually touch the CG set. In this less demanding situation, the stereo meta data from the Pace camera was good enough, after a simple trim in the Maya camera. But when the live action feet are touching the CGI or there is a set extension, then the more accurate Unified Solve is used.

When using meta data, only the right eye is matchmoved and the left eye transform is sent to Maya from the camera encoders. But in Unified Solve, both eyes are matchmoved together and the left and right camera solves "talk" to one another inside of 3DE.

Unified Solve is basically bringing both the left and right plates into 3D Equalizer and tracking a percentage of identical features for both eyes. Unified Solve is especially easy with blue screen markers, since 3DE Marker mode finds the exact center of the dot pretty accurately for both eyes. Finding the same center of the marker for both eyes is important, so that in stereo the resulting matchmove of the blue screen doesn't float in front or behind the correct stereo depth.

On JCE, wind machines often blew the blue screen markers around, so the 3DE translation smoothing value was increased to avoid a noisy motion solve.

In other JCE shots, the actors would walk on rocks. These features require Pattern Tracking (not Marker Tracking) and it requires more human intervention to insure that the feature is tracked for exactly the same spot in the left and right eyes.

Because earlier versions of 3DE already supported using multiple plates and cameras, Unified Stereo Solve has always been a standard feature in 3DE. Unified Stereo Solve is just a nickname for an already existing capability. Nevertheless, special 3DE stereo constraint features were created by Rolf and Uwe to enhance Unified Solve.

When using Unified Stereo Solve, you can combine the Autotracker with manually created tracks. A certain number of common left/right points will need to be tracked with user intervention. This will ensure that the feature is the exact same spot on the set for the left and right eyes. The remainder of the tracks do not need to be common between the left/right eyes and you could optionally use the autotracker.

As mentioned, Unified Stereo Solve is almost always necessary on stereo set extensions since meta data will rarely be perfect enough. You can trim the encoder meta data in Maya, but  you can almost never get all of the features in the eyes lined up without Unified Solve. You could use a "least square fit" (LSF) surveyed solver like rasTrack to improve the meta data for the secondary eye (left, usually), but the 3DE Unified Stereo Solve is ultimately the easy and elegant method.

It is well known that surveyless matchmove does not always create plausible solves. Sometimes you end up with a calculation that looks like an M.C. Escher painting, lovely in 2D, but ludicrously impossible in 3D space. Typically a 3DE user will use Reference Frames to add parallax and solve this problem. Similarly, Unified Stereo Solve may not always provide plausible stereo solves. For this reason, 3DE added stereo constraints. Theoretically, the left and right cameras should be exactly left/right of one another and not at different heights (local camera space) or weird skew angles.

The 3DE stereo constraints ensure that only the i/o (tx) and convergence (ry) between the cameras can be different between the two eyes. All other values (ty, tz, rx, rz) will be constrained to zero.

On JCE, only the convergence was animated on set. but other shows (like T2-3D and Avatar) will have also have animated i/o. 3DE will support this technique in later releases. Animated i/o will be read into 3DE, from an ascii file.

 

 

As you probably know, 3DE has a terrific proxy system for image sequences. Typically, the F5 button is user assigned to bring up full res, the F6 button half res and the F7 button is quarter res. Great feature.

 

 

Often a Unified Stereo Solve will work well without 3DE stereo constraints, or sometimes will be superior. Please experiment, matchmove is not an exact science. Warning: Object tracks in 3DE may not work properly without the 3DE stereo constraints enabled and the object may locate in different space left/right. This is not a problem for camera solves and you may even wish to solve a stereo object as a camera track and then convert to object motion in Maya.

Surveyless Object tracking is always ambiguous when it comes to Scale. In 3DE, you can track multiple moving Objects with cameras. On JCE, we object tracked mine cars. But were they miniature mine cars, close to the camera, or giant mine cars, far from the camera? The monoscopic layout artist can make any subjective decision about scale that she likes, but not so in stereoscopic. Since there are stereo eyes triangulating on the depth of the Object track, the scale of the Object track becomes more "objective". Once the i/o and the convergence of a stereo camera are calculated, the scale of an object must be at a certain value. The vanishing points and epipolars  of stereo cameras demand that the object have a certain scale.

By using the stereo constraints in 3DE (instead of regular Unified Solve), the  Object track will appear correct in both left and right Maya eyes.

 

Multiple stereo cameras and projectors were used, with suitable naming conventions so that the stereo tools would act on all of the cameras and projectors in the Maya scene.

A great feature in Shake and other comping software is a hot key to toggle between two images that need to be compared. In Shake, the hot key is "1".

So in Maya, we reassigned the "one" key so that the view port would also toggle between the left and right stereo eyes. Very useful for rapid A/B stereo comparisons.

(Download Python script  for Maya  "1" button hotkey)
http://michaelkarp.net/lr.zip
 
In 3DE and Maya Live, there is an Autocenter tool that will keep the 2D or 3D feature centered in the viewport. With a MEL script, It is also possible to autocenter anything in Maya. So we set it up so that when we zoomed in the viewport, using the over scan, that the left/right autocenter and over scan were always synchronized.

Maya supports rgbA premult plates in the image planes used in Playblasts. For stereo blue screen work, I would suggest the use of an image plane with the alpha channel enabled. It can be very distracting to judge the stereo effect on a move test with the blue screen not extracted. The visual depth cues are...weird.

So on JCE, we had three pairs of image planes prewired, the rgb (with the plate unaltered), rgbA premult (blue screen extracted) and MayaLive rotoPlane for subpixel accuracy and image caching. We could easily toggle between the three image planes. Typically, matchmove/layout artists would submit two playblasts for dailies, one optimized for tracking and the other for layout. The tracking test would show the original blue screen and markers and the layout test would have the blue screen removed by multiplication/extraction and would be more "artistic".

Surprisingly, almost all matchmove tests for Hollywood 2K feature films are rendered at half res 1K, which is quite adequate for most shots and allows major efficiency increases over full res 2k tests.

It is often useful to adjust the Maya Image plane depth, but mandatory for 2.5 D "projector/card" shots. A premult Image plane depth will generally be near the camera and regular blue screen Image planes will be set for a distant depth. For projector/card shots where the camera translates away from the projector, the image plane depth should be animated so that the image is near the position of the critical of the action. The image plane depth for projector shots will always be a subjective compromise. For mine car shots, the image plane depth of the actors was placed at the front of the mine car, but in other shots, the depth was placed on an important actor.

 

 

There are many shots in JCE of characters carrying flash lights. Often times the beam was not bright enough in the plate or there wasn't enough smoke on the set for the beam to illuminate, so a volumetric light pass would need to be rendered. Beams require smoke or particulates in the air to be visible, but that smoke can interfere with the photography and extraction of a blue screen. So many JCE shots required beams to be created in post and of course to be match moved in stereoscopic.

We created special rigs in Maya for the headlights. Using the plate for the right eye (master), the user animates a locator in Maya that matches up with the position of the light. It is very useful if the headlight locator is rigged as follows:

• A null called Scalar is created and made a child of the right (master) camera. 

  • Scalar trans/rot should be locked at zero. 

  • Scalar uniform scale (XYZ with all the same value) will be animated later. Start with a value such as one (do not use zero for scale).

• A headlight locator is a child of Scalar

  • 
    

  • Aim constrain headlight to the right camera

• Looking through the right camera, animate just the (local) tx and ty of the headlight locator, you can lock the tz at zero

• After the right eye headlight animation looks good, look through the left Maya camera. The depth of the locator will be wrong in the left eye, so animate the uniform scale (XYZ) of Scalar so that the depth from the left eye is correct.

Notice that any adjustments to the left eye will not change the right eye at all. This is a huge time saving technique. You can also 2D track in 3DE and export the track to Maya with Export_Single_MM_PointsV1.3detcl. This script creates a camera in Maya with a locator that follows the 2D track from 3DE. You can then constrain the 2D point Maya camera Scene node to the right Maya camera. Then, reparent (bake) the 2D locator so that it is a child of Scalar. The 3DE 2D tracker is not perfect, but overall is the best 2D tracker GUI and engine available in any 2D or 3D package. So it is very convenient to 2D track in 3DE and then export to Shake, Maya, etc.

By Aim constraining the headlight locator to the right camera, you keep the locator "square" to the right eye.

After you animate the headlight translations for the right eye and set the depth for the left eye, then you can animate the rotations. Create another locator HeadlightRotate that is Point constrained to the headlight locator and then animate only the rotations of the HeadlightRotate node. You can also constrain a cylinder to the HeadlightRotate node, for visual reference of the quality of the animation. Obviously we created a script to automate the creation of each headlight rig.

On one occasion, we needed a different stereo camera for the headlight track than we did for the set extension for the scene. Because the moving headlight was on the actors head and was distant from the stationary set, the matchmove for the set was only suitable for the set extension. In this situation, we duplicated the main matchmove camera and gave special trims so that the headlights would track properly. Since the headlights are much closer to the camera than the set, any minute translation errors on the main matchmove were magnified when tracking the headlights.

 

 

It is common in matchmove to take a Maya node and give it a new parent. For example, an Object track could be converted to a Camera Track, or vice versa. A Maya node could be baked to a new parent, so that the scene is cleaned up for publishing to other departments. Maya constraints and baking are used for this, but we used an automated Reparent script to greatly simplify this process, available here. Be sure to hide the viewports when running the script. It will speed up the bake, since the image plane is not needlessly read in:

http://michaelkarp.net/reparent.zip

 

 

Several shots needed on JCE needed speed changes and time warps. There are two basic approaches.

• The time warp can be rendered to the plate in Shake, Combustion, etc. using Twixtor, etc. and the rerendered plate can be matchmoved

• The time warp can be calculated in Shake, Combustion, but the actual plate that is matchmoved is not altered. Rather, the speed change is exported to Maya as an ASCII file and Maya simulates the speed change.

Tracking a time warped plate is problematic. Time warp creates motion artifacts that may be acceptable to the audience, but confusing to matchmove software. The same goes for 2D repos of plates for matchmove. So we tracked clean plates and then applied the time warp in Maya, using a custom script. This way, the time warp could be modified at any time and we weren't locked into the original time warp decisions. The time warp in Maya was baked back into all of the Maya animation and the final time warp (if different) could be exported back to Shake in a simple ascii file.

 

 

We had a large number of stereo "projector shots". These were two and a half D "card" shots.

• The 3DE matchmove was imported into Maya

• The camera was duplicated and called projector

• Many elegant schemes were used to offset the camera from the projector, both for nodal pans on the projector and other shots where the camera translates away from the projector.

The farther the camera translates from the projector, the more obvious the "cheat" becomes. You may need to aim the projector image plane back at the camera, so the image doesn't get too squished or keystoned. The convergence may need to be trimmed on these fancy projector shots, so that the stereo depth stays correct with the cheated perspective.

When doing a projector shot repo, the plate needs to be rerendered. Either you can rerender the plate in Renderman or you can export a corner pin to Shake. Adapting a corner pin script from HighEnd3D, we exported the four corners of the image planes into a Shake tracker script. Automatic compensation was made for lens distortion, time warps, image padding, etc.

 

A cool stabilization trick is this:

Let's say a crane or hand held shot has too much vibration. If you matchmove the shot, you can take that solve in Maya and make that camera into a projector. Make another stationary Maya camera that looks at the projector camera and the stationary camera will see a completely smooth, vibration free image. Since the 3D solved locators from 3DE are stationary in world space, then a stationary camera would see a perfectly stable image. You can also duplicate the projector camera to a render camera, smooth the render cameras animation and then you will retain the original motion of the plate, with the objectionable bumps removed.

A deeper discussion of matchmove stabilization is here:
http://carbonmedianltech.vox.com/library/post/6-dof-stabilization-using-camera-tracking-and-camera-mapping.html

Many JCE shots had actors walking on treadmills.  We would typically matchmove the camera in 3DE and hand track the treadmill Object motion in Maya. Next we would reparent the camera to be a child of the treadmill. Reparent is completely different than parenting, because the child node animation is baked, so that it's position in world space does not change. The final step is to Mute the animation on the treadmill Object tracking. By muting instead of deleting the treadmill object tracking, the artist can easily undo changes if necessary.

By this simple procedure, an object track is easily converted to a camera track.

 

 

There are two powerful new triangulation tcl scripts from 3DE. These allow 3DE points that won't solve Passive, to still be passively calc'd in 3D space. One script is intended for monoscopic and the other is for stereoscopic.

 

 

There are many shots in JCE where ik human characters were matchmoved over plates of live actors, in stereo. It is typical in character matchmove to carefully pose the ik model for the first frame lineup. This is especially important in stereo, because the character must be correctly posed for both left and right eyes. The most important point is that the i/o and convergence for the camera should be set so that it works well with the posed ik character. The scale and posing of an ik character in stereo must be precise and all other layout and world space decisions must follow downstream from this first and demanding step.

 

 

We had many scenes with actors in a mine car chase. Typically, the three actors (and personal mine car) would be shot in their own blue screen plates and then the three plates would be choreographed and combined in Maya. The actors would be standing on a mine car on a motion base, but the bottom chassis of the mine car would be missing and had to be added as a set extension match move. Set extension in stereo can be complicated, since as soon as you translate the projector image, the nature of the 2.5 D cheat becomes apparent. So one of the three mine cars (the most difficult to set extend) becomes the master and the other mine cars set extensions need be less exact.

The stereo trim of the camera is optimized so that the most demanding mine car projector looks correct in stereo.

The stereo mine car sequence was the most complicated that we worked on and there are many vital subtleties that I am leaving out.

Many object tracks were done in 3DE of a rectangular mine car "chariot". These 3DE solves were very good, but the point cloud wasn't always perfectly perpendicular. Normally we would have put lattices on the mine car in Maya, but this would have broken the kinematics of the pump and wheel linkages of the mine car in Maya. So our brilliant rigger Marc-Andre published standard blend shapes, so that we could deform the model without breaking the fk of the chariot wheels.

Also, the animated motion of the mine car was used as a path to procedurally extrude the mine car rails, bridges and trestle.

 

 

One problem in stereoscopic vfx is creating roto mattes that have the proper matching stereo depth between the eyes. One possible solution to this problem is to place a texture card in the Maya scene and 3D paint the rotoscope split line on the texture card. Instead of rotoscoping the bezier splines twice for the two eyes, you only rotoscope the split once and use the left/right Maya cameras to view the single texture card at different perspectives. If the texture card is placed at a suitable depth in the Maya scene, then the stereo rotoscoping would naturally blend smoothly between left/right images.

 

 

There are several methods of viewing in stereo.

In the vfx facility theater, dual projectors with polarizers will be used or a RealD system can be installed.

At the workstation, three general methods are used:

• Anaglyph
• Shuttered glasses
• Mirror

 

Michael Archambault
Pierre Bonnette
Francis Camacho
Eric Desaulniers
Christian Emond
John Higbie
Daniel Lowenberg
Michael Karp
Muzaffer Korkut
Candida Nunez
Hernan Vietri