QTVR ShockWave3D

©2002 Joachim Baur, Medienwerkstatt Schorndorf

A short table of contents:

• 1) Introduction to cylindrical and cubic VRs
• 2) Initialization of the sprite's behavior and the 3D-world

      Creatung cylindrical QTVRs using Shockwave3D-#cylinder-primitives

• 3) Creating model & resource
• 4) Creating texture & shader
• 5) Navigating the QTVR (by manipulating the camera)
• 5a) Panning
• 5b) Tilting
• 5c) Zooming
• 6) Hotspot-detection and -interaction
• 7) Additional options for animating the VR

      Creatung cubic QTVRs using Shockwave3D-#box-primitives:

• 8) Creating model & resource
• 9) Creating texture & shader
• 10) Navigation (pan, tilt, zoom)
• 11) Hotspot-detection and -interaction

1) Introduction to cylindrical and cubic VRs

The starting point for any VR is a bitmap that shows an image of a complete 3-dimensional room/scene, a so-called "sourcepict", either photographed and "stitched" together using particular software tools or rendered directly by a 3D modeling application. This image is created using either a cylindrical or a cubic projection method. The VR-player software (traditionally Apple's QuickTime-player, but others are available too, even JavaPlayers) displays this projection in reverse, navigable in all directions.

Cylindrical QTVR	Cubic QTVR
• Any QuickTime-version sufficient for playback	• QuickTime-version >= 5 needed for playback
• horizontal (left/right): field-of-view up to 360° • vertical (up/down): field-of-view is limited	• horizontal (left/right): field-of-view up to 360° • vertical (up/down): field-of-view up to 360°
• 1 SourcePict is used for the whole panorama:	• 6 SourcePicts for all sides of the cube are necessary:
above: SourcePict, below: Projection onto inside of a cylinder	above: SourcePicts, below: Projection onto the insides of a cube

As long as the observer is placed exactly at the center of the QTVR-model (cylinder/cube), and only turns/tilts his "head", the bitmaps that are mapped onto the insides of the model create a perspectively correct reproduction of the originally captured (rendered/photographed) room.

The aim of this project is to use the standard souce material which is used for QTVRs, import that into director and display/navigate it using shockwave3D-primitives, also integrating hotspot-functionality. A Hotspot is a arbitrarily shaped area within the QTVR-room by which the user can interact with the room or its "contents" with the mouse (rollover/click).

This is not a dramatically innovative concept - such "simulated" QTVRs have been done since the introduction of quads in Director 7, for cylindrical QTVRs take a look at the QTVRLingoEngine, for cubic QTVRs there's Nonoche's groundbreaking ShockTimeVR.

The really new aspects of such a simulated QTVR are e.g. the integration of any other shockwave3D-models within the shockwave3D-QTVRs. Personally speaking, this project was my first attempt at coping with all the new 3D-Lingo and concepts suddenly available with Dir 8.5. For this purpose this presentation was assembled and held at the German Director User's meeting in Frankfurt at the beginning of March, 2002.

2) Initialization of the sprite's behavior and the 3D-world

There are 2 huge differences when comparing sprites of "traditional" media types (text, bitmap, video, flash, etc) to sprites created from Shockwave3D-members:

• The shockwave3D-sprite (the "3D-world") is always empty (black) in the beginning.
  All the 3D-data (models, textures, light sources, etc) have to be created one by one within the member's 3D-world, either using lingo or importing external 3D-model-files.
  
  
• When the movie is stopped, the 3D-world keeps existing with all its contents, as the world is stored within the 3D-castmember. A regular sprite, on the other hand, is reset by the process of stopping/rewinding the director movie.

Therefore it is an absolut MUST when authoring to manually reset the 3D-world when the 3D-world's sprite is created ("on beginSprite"). Otherwise the lingocode would throw up errors and exceptions all the time (two objects in a 3D-world mustn't have the same name, for example - which would be the case each time a movie is re-started without deleting the previous instance of a model from the 3D-world first). Only when the director application is quit or the director-movie is closed and reopened, the 3D-world is as empty and black as our lingocode expects it to be.

So the "beginSprite"-handler initially looks like this:

ResetWorld() and revertToWorldDefaults() reset the 3D-member referenced by gMy3Dworld,
the global variable gBehavRef stores the reference to the behavior's instance that is created at runtime. This makes debugging a lot easier - to get/set any property of the behavior, just type e.g. "put gBehavRef.pMySprite" into the message window, or alternatively use the watcher window to monitor "gBehavRef.pMySprite"

3) Creating model & resource

The next step is to create a cylinder within the 3D-world.

Generally speaking, each 3D-body is defined by four aspects:

• The model itself
  This property is just a container that holds the remaining three aspects
  
  
• The model-resource
  The resource defines the shape/gestalt of the 3d-body
  
  
• One shader for each surface/plane of the model that
  The shader defines the surface-material and its parameters (shiny, dull,...)
  
  
• One textuer for each shader
  Fnally the texture applies a color, pattern or image to each shader

So for a cylindrical QTVR we deploy a model with a "cylinder"-resource that uses the panorama-sourcePict as a texture attached to an even (unobtrusive) shader.

The absolut measurements of the cylinder (its size in 3D-world-units) are not at all important, as soon as the ratio cylinder-radius to cylinder-height is right, the panorama is rendered perspectively correct. This is because the camera/eye is located in the center of the cylinder, so a 10-unit-radius cylinder fills just the same "camera-view-space" as a 10,000-unit-radius cylinder (the cylinder's radius increases by a factor of 1,000 - but so does it's height).

The radius-to-height-ratio is determined by the dimensions of the sourcePict. To apply this ratio to a cylinder, the cylinder's height is simply set equal to the height of the sourcePict-Bitmap (measured in pixels). The corresponding radius is then calculated by dividing the sourcePict's width by 2*PI (radius = circumference/2*PI).

This yields the following lingo for the model- & resource-creation:

The lingo-command newModelResource creates a new, empty resource of the type #cylinder - that is a built in, predefined body within shockwave3D (called a "primitive"). To this raw cylinder the correct height and -radius has to be applied.
The additional parameter #back means, that this cylinder only consists of an inside - as the camera is placed inside the cylinder, an outside view is not possible, and therefore needn't be definded.

The resource-properties topcap = false and bottomcap = false additionally "remove" the top and bottom cap of the cylinder, so all that's left is an open "pipe" painted on the inside. Finally, resolution = 72.0 (default is 20.0) increases the resolution of the cylinder's circumference – it is now rendered using 72 subdivisions (close enough to an ideal circle).

As long as only the "pure" model is defined (pMyQTVRmodel = gMy3Dworld.newModel("QTVRmodel")), it is not visible in the 3D-world. After applying the necessary resource-information (pMyQTVRmodel.resource = pMyQTVRresource), the cylinder is rendered using a default-texture (the checkered red and white pattern) tied to a default-shader and illuminated by a default-lightsource:

The next step is to replace this default values with our own, visually more appealing texture and shader.

4) Creating texture & shader

The dimensions (in pixels) of an ideal texture (in terms of rendering performance) should be equal powers of 2, e.g. 64x64, 128x128, etc. Basically, any other texture size will do, but has to be scaled first at runtime by the 3D-renderengine. The sourcePict-texture I'm using in this example movie has a size of 1440x440 pixels.
This texture-bitmap is "painted" seamlessly around the inside of the cylinder. The texture-origin (the left of the sourcePict-bitmap) is located on the positive Z-axis at X=0, the projection is done clockwise (seen from above) along the cylinder's circumference:

Caveat: The 3D-world-coordinates (shown gray on the right) are laid out in such a way, that the Z-axis points towards the front. This coordinate-system is used as a world-coordinate-system (the reference system for all objects within the world) that does not move (translate) or rotate. Additionally, every 3D-object in the 3D-world also carries its own, local coordinate-system (shown red), which rotates/translates in accordance with the model itself. When performing rotations and translations, it is therefore essential to reference the coordinate system relative to which the movement should be performed (#self vs, #world).

A texture cannot be applied directly to a model, only by the way of a shader. Each surface of a model needs a shader in order to be drawn/rendered at all. The properties of this shader determine the appearance of the surface, as in reflective, glossy, transparent, etc.
For the QTVR, the #standard-shader is just right, as it renders the texture-bitmap photo-realistically (there are some other shaders which I have never seen in use by anyone - #engraver etc).

Texture and shader are applied in the behavior like this:

The shader-property emissive is set to make the inside-surface "glow" with a white color (rgb(255,255,255)) comparable to a slide that is illuminated from behind (but it's not the same as a lightsource, as it doesn't illuminate other objects or cast shadows on them!). The emissive-property is just a way of achieving a constant brightness of the texture all the way round, so other, real light-sources have no effect on the texture's appearance.

The ambient-lightcolor is accordingly set to black (rgb(0,0,0)) to avoid any bright spots within the cylinder caused by ambient lighting.

Setting the property flat=false is not absolutely necessary, but increases rendering performance as is bypasses the more processor-heavy default-method for light/shadow-calculations of the texture.

After applying texture and shader, the QTVR itself is rendered correctly - only the point of view (camera-position) has to be adjusted, as the default-value is off-center:

5) Navigating the QTVR (by manipulating the camera)

The easiest way to take a look around the newly created cylinder-room is to manipulate (turn/tilt) the camera while the cylinder itself remains stationary.
Prior to that, however, the camera has to be positioned at the right spot, that is the center of the cylinder and also the center of this particular 3D-world: The location where X=0, Y=0 and Z=0:

So the initial camera-view of the QTVR from the center of the "room" looks like this:

Camera-movement is composed of three different components:

• Panning: Turning of the camera to the left/right
  
  
• Tilting: Pointing the camera upwards/downwards
  
  
• Zooming: Increasing/decreasing the visble area of the QTVR-room

All three components can be applied to the camera simultaneously or separately.

They can be triggered using different approaches, too:

• Keypresses
  according to Apple's original QTVR-navigation, the cursor keys are used in the example-movies for panning/tilting and the shift-/Cctrl-keys for zooming)
  
  
• Mouse interaction
  "click&drag" also accordingly to a QuickTimeVR (when dragging the mouse with the mousebutton down, the pano "follows" the mouse via pan/tilt)
  
  
• Other Lingo-Behaviors (e.g. buttonscripts that react to mouseOver or mouseDown)

To be able to combine all triggers with all possible camera-actions independently of each other, the lingo in my example-movie uses flags. These flags are stored in global variables and can be set/cleared by any triggering event (script). The "exitFrame"-handler within the QTVR-behavior then sequentially checks all flags and pans/tilts/zooms the camera accordingly. This is just one way to avoid performance-killing repeat-loops, as for example the "dragActive"-flag is set by a mouseDown-action and cleared again when the mousebutton is released - no "repeat while the mouseDown"-construct is necessary.

The main flags in the example-movie are these:

• gPanFlag
  =  1: the camera is rotated to the left (the room seems to pan to the right)
  = -1: the camera is rotated to the right (the room seems to pan to the left)
  =  0: panning stops
  
  
• gTiltFlag
  =  1: the camera is tilted upwards
  = -1: the camera is tilted downwards
  =  0: tilting stops
  
  
• gZoomFlag
  =  1: the camera zooms in
  = -1: the camera zooms out
  =  0: zooming stops
  

5a) Panning

As soon as the cursor-left/cursor-right-key is pressed, gPanFlag is set accordingly
Further down the script gPanFlag is checked and the camera is rotated by 5 degrees around the WORLD-Y-Axis.

5b) Tilting

For tilting, the camera is again rotated (by 4 degrees), but this time around the MODEL-Y-Axis.
Rotating around the WORLD-Y-Axis would not produce the correct motion, as this drawing illustrates:

The camera in center of the the drawing has already been panned by 90 degrees when the tilt-motion is invoked. So the MODEL-X-Axis (the green coordinates) is now aligned to the WORLD-Z-axis (the gray coordinates), and vice versa the MODEL-Z-Axis is to the WORLD-X-Axis.
Rotating the camera around the WORLD-X-Axis for tilting would result in a rotation around the MODEL-Z-Axis:

camera panned 90 degrees...	rotate(4,0,0,#world) > UPS!	rotate(4,0,0,#self) > ALRIGHT!

Choosing the correct reference-system (#self <> #world) for the rotation is elemntary - but not always apparent.

Another problem is caused by the nature of the cylinder-resource: There's no bottom and no top cap. So after some tilting the upper/lower rim of the cylinder shows up, and the black outer space beyond the cylinder:

To avoid this, a limit for the tilt-angle has to be determined in such a way that the visible field of view of the camera always stays within the cylinder's boundaries. This maximal field of view (maxFoV) is dependent of the sourcePict's height and can be calculated using a little trigonometry:

Parsed into lingo:

Within the pMaxFov-range the camera can be freely tilted up and down - but only after the current field of view (camera.fieldOfView) has been subtracted from pMaxFov, yielding the "remainingTilt"-angle:

The tilt-code is therefore extended as follows:

The current tilt-angle of the camera is stored in the property pTotalTilt. Before actually tilting the camera the newTilt is calculated and compared to pRemainingTilt. Only if the newTilt is less than pRemainingTilt, the camera is rotated and pTotalTilt is updated to the new angle.

5c) Zooming

For zooming in and out, the position of the camera could be moved towards/away from the inside of the cylinder - but the camera has to remain in the center of the cylinder to keep up a perspectively correct impression. So instead the field of view of the camera is changed from a wide-angle-lens (zoomed out, with camera.fieldOfView <= pMaxFov) to a close-up view (zoomed in, with camera.fieldOfView >= pMinFov, pMinFov is an arbitrary number, but must be >0.0)

When zooming out, the same problem is created by the rim of the cylinder as when tilting too much:
If the camera has a narrow field-of-view (is zoomed in) and is tilted downwards to the limit set by pTotalTilt, the black space beyond the cylinder starts to show up as soon as the field-of-view is increased (the camera is zoomed out). Therefore, the tilt-angle has to be checked and adjusted if neccessary when zooming out:

	camera.fieldofview increased, with unadjusted tilt angle	camera.fieldofview increased, with adjusted tilt angle

The zoom-code is therefore extended as follows:

Before a change in the camera's field-of-view is applied, the script checks if the pRemainingTilt caused by this change is still within the range of pTotalTilt . If not, the camera is rotated (pMyCamera.rotate(newTilt,0,0)) before the camera.fieldofview is applied.

At this point, the cylindrical QTVR is fully navigable, the example director movie for download contains additionally to the key-triggered navigation in the script-snippets of this section the mouse-based ("click&drag") functionality which is not further explained here.

6) Hotspot-detection and -interaction

In order to make the "contents" of a QTVR-panorama interactive, hotspots have to be defined, comparable to the hotspot-areas of an image map in HTML.

To define these hotspot-areas, Apple's QTVR uses a hotspot-bitmap (lower image in the illustration above) which is exactly the same size as the panorama-sourcePict (upper image in the illustration above). For performance reasons, this hotspot-bitmap is created using only an 8-bit color palette - traditionally the "System Mac"-palette, but any other will also do fine.
On the white background of the hotspot-bitmap each hotspot is painted using a different palette-color, so the hotspot-number is determined by the palette index of its color.

Important:

• The hotspot-areas must not have feathered/anti-aliased edges!
  

  anti-aliased edge = does not work!	hard edge = exactly right

  
  
• When importing hotspot-bitmaps into director, the option "Trim white space" must be turned off!

Detecting the hotspots in a panorama requires three steps:

1. What 3-dimensional point (dx, dy, dz) of the cylinder is the mouse currently over?
   
   
2. To which pixel (x,y) of the hotspot-bitmap "belongs" this point (dx, dy, dz)?
   
   
3. What palette-index has this pixel (x,y)?

Step 1 is best solved using the 3D-Lingo function modelsUnderLoc(). When this function is called and a point (x,y) within the 3D-sprite's rect is passed along as a parameter and returns a long list of partly useful, partly mysterious properties:

The most interesting property for our purpose is the #isectPosition: This vector contains the 3D-coordinates dx, dy, dz of the point within the cylinder's surface that is eqivalent to the testLoc-point in the script-snippet above. Because the exact position and size of the cylinder is also known to us, the corresponding pixel (x,y) of the cylinder's texture and therefore also of the cylinder's hotspot-bitmap can be calculated.

The iSectPosition is composed of the three components dx, dy und dz.

The easiest value to calculate is the y-coordinate of the bitmap-pixel - because the height of the cylinder is equal to the height of the sourcePict measured in pixels, the y-coordinate can be obtained directly from the dy-value:

Obtaining the x-coordinate of the bitmap-pixel is slightly more complicated. Because the texture's width is mapped along the cylinder's circumference, the #isectPositon's angle (determined again by dx/dy) has to be calculated relative to the texture's origin (where the texture's x-coordinate equals 0):

The texture is wrapped around the cylinder clockwise (seen from above) starting at 0°	Texture's origin: vector(0, 0, 1) isectPositions-vector: vector(dx, 0, dz)	angleBetween() returns values from 0...180 degrees for both halves of the circle.

Thanks go to Ullala for the angleBetween()-hint – I calculated this first with a more complicated and slower atan()-approach!

The resulting values for the angle ranging from 0...180 in each direction have to be mapped to values from 0...360 in order to determine the correct hotspot-bitmap-pixel's x-coordinate. All the parts thrown together look like this:

As long as the mouse is over the 3D-model, the current mouseLoc minus the upper left sprite-corner (which is stored in the property pMyOrigin) is used to calculate x and y (the x- and y-coordinates of the corresponding hotspot-bitmap-pixel) and determine the hotspot-bitmpa-pixel's color value via getPixel.

Using this method as a starting point, all kinds of hotspot-interaction can be scripted – changing the cursor when the mouse is over a hotspot, animations and actions when entering, clicking or leaving a hotspot, etc.

7) Additional options for animating the VR

With the shockwaveVR being just one model within a whole shockwave3D-world, special effects and animations can be added that can't be achieved when using the traditional method of importing a QuickTimeVR-movie into director.

Three kinds of animation can be applied to the shockwaveVR:

1. Modifications of the 3D-model's properties
   
   
2. Modifiyng the texture's bitmap with Imaging Lingo
   
   
3. Adding other 3D-models to the shockwave3D-primitive

One example for a modification of the 3D-model's properties is shown below with simply switching the "emissive"-property from white to blue:

shader.emissive = rgb(255,255,255) -- white	shader.emissive = rgb(102,102,204) -- blue

As a result, the room appears to be lit by a weak blue lightsource – in movies and on TV this "look" is often used for night-time shots.

Modifying the texture's bitmap with Imaging Lingo is a way to "re-decorate" literally everything in the room.
To make the bitmap accessible for Imaging Lingo manipulations, the shader's texture has to be declared using the #fromImageObject-property instead of #fromCastMember:

The image object is stored in the global variable gQTVRimage, making it possible for code other than the behaviour itself to access the image also. To retain the original (unmodified) sourcePict-image, not the castmember's image is referenced, but instead a copy is created in memory at runtime with image.duplicate() - otherwise changes made with Imaging Lingo would be irreversible, at least when authoring...

As an example of a practical use of this method, the lamp on the desk is to be switched on and off by clicking on it. For this purpose, an additional bitmap is rendered with only the size of the switched-off desklamp:

This bitmap (imported into castmember "LampeAus") is copied into gQTVRimage the when the desklamp-hotspot is clicked once, a second click on the hotspot restores the bitmap-area from the original sourcePict-bitmap (so the lamp is switched on again):

After modifying the image object with copyPixels, the image object has to be explicitly re-assigned to the texture (pMyQTVRtexture.image = gQTVRimage), otherwise the changes won't show up in the 3D-world.

As an alternative to this 2-dimensional animation, any number of additional 3D-models can be integrated into the scenery. As the camera remains stationary in the center of the room, the 3D-model automatically is rendered perspectively correct in relation to the surrounding panorma. It only has to be scaled/rotated/illuminated in such a way that it is also naturally integrated within the contents of the rendered scenery (else it appears to be floating in the air...)
I've added a simple #box to the scene and moved it to a position (paketModel.translate(68,-88,-168)) which makes it appear to be placed correctly on the table:

The resulting composition is missing the shadow that the parcel would normally cast onto the table. The parcel also has to be illuminated with a separate light-source that adjusts the object's appearance to the overall lighting of the pre-rendered room.

Another example for an animation created with an adiitional 3D-model is the cloudy sky that moves outside the room. This is done by making the windows in the sourcepict's texture transparent using an alpha-mask and placing a second, lager cylinder with the sky/cloud-texture around the QTVR-room. When rotating the outer cylinder, the clouds appear to move - but always in the same direction though (from right to left), regardless of the direction the window actually points to.

8) Creating model & resource

The same techniques and structures of a cylindrical VR also apply to a cubic VR, so I will only shwo the main differences in the code of the cubic VR.

First of all, a model and its resource is created, this time of the type #box:

Instead of a cylinder's radius and height, the dimensions of the cube are defined by the three parameters height/length/width
The cube's size is again determined by the dimensions of the sourcepict's bitmaps so as to make the hotspot-calculations simpler. I also "invented" the property pCubeRadius that is not known in regular geometry but is quite handy as shorthand for pTextureSize/2

9) Creating texture & shader

For displaying 6 sides of the cube 6 shaders with 6 different textures are necessary. Therefore the shaders are attached to the model with
pMyQTVRmodel.shaderlist = pMyQTVRshaderList instead of
pMyQTVRmodel.shader = pMyQTVRshader

The shaders (and corresponding textures) are ordered like this:

The 4 "walls" of the cube are followed by the top-cap and finally the bottom-cap. The orientation of top and bottom-cap is indicated by the numbers above to demonstrate how the textures of the walls 1-4 adjoin to the top and bottom textures.

The two lines in the script that are disabled by comments
-- pMyQTVRshaderList[img].textureTransform.scale(1.001,1.001,1.001)
-- pMyQTVRshaderList[img].textureRepeat = False
are a way of avoiding visible hairlines along the cube edges. Un-comment those lines hairlines appear in your panorama, as shown here

These hairlines are (often) caused by the textures being scaled and by the "nearfiltering"-calcuations that are used to apply the texures to a model. This nearfiltering smooths the display of the texture-bitmaps but also can cause dark edges at the texture borders (if you tunr nearfiltering off, the hairlines disappear while the texture looks blockier).

Setting the textureTransform.scale(1.001,1.001,1.001 makes the texture just a little bit larger than the plane (the shader), so the texture's borders are not drawn as dark hairlines any more.

10) Navigation (pan, tilt, zoom)

With the cube having no upper or lower limit when tilting the camera, all the calculations that are concerned with pRemainingTilt are obsolete. Instead, a pMaxTilt-value is introduced to constrain the upwards/downwards tilting at 90 degrees, so the room doesn't flip over when looking up or down. pMinFov and pMaxFov are both still used, but can be set to arbitrary numbers (if pMinFov is chosen too small, the image becomes very pixelated, if pMaxFov is chosen too large, the appearance of the room becomes rather surreal ;-)

Here's the simplified navigation-script for the cube:

11) Hotspot-detection and -interaction

A cubic VR utilizes a hotspot-bitmap for each side of the cube (with the same dimensions in pixels as the sourcepict):

Detecting hotspots via the isectPosition is even easier when working with a cube than with a cylinder - as each side of the cube is perpendicular to one of the three axis, one of the three isectPosition-coordinates dx, dy and dz is alway equal to pCubeRadius. The other two coordinates just have to be mapped to texturepixel-coordinates, shown here for side #2 of the cube:

In this case, the dx-coordinate is equal to the pCubeRadius, and the dz-coordinate corresponds to the texture's x-value, while the dy-coordinate corresponds to the texture's y-value.
The hotspot-script for all 6 sides of the cube goes like this:

12) Animations (cubic VR)

Same techniques as mentioned in animations for cylindrical VR ;-)

Comments, suggestions and critique regarding this article are welcome, please send them to:

Joachim Baur | medienwerkstatt | grafik-design