// Introduction

GEM, Graphics Environment for Multimedia is a Pd Library and comes ready to use as a part of Pd-extended. GEM makes it is possible to generate and manipulate 2D and 3D graphics and animations, process and modify media like images and videos and generate particles. GEM-patches are Pd patches. They use a special set of objects that are not native to Pd, but are provided by the GEM library.GEM objects are not standalone objects and you will have to use severl GEM-objects together to get your diesired result. You can use GEM to create and play back videos and still images, mix videos, draw shapes in 2D and 3D, move objects and shapes around. Because it is part of Pd, you can make your visuals react to sounds, generate them from sounds themselves.

You can look at GEM examples via the Pd Help Browser (in the Help menu, under Browser...), under examples/Gem or have a look at the GEM manual in manuals/GEM.

gemwin

The gemwin object is the one object that must be included in your patch to make GEM work. The gemwin object provides the window on which the computer draws your graphics. This object controls the timing of your graphics, by scheduling wheb frames should be drawn based on framerate. By default, it also clears the window every frame and ssets it to a background color.

Messages to gemwin change the size of the window, start and stop the rendering process, alter the position from which you look at your 3D , and control various other aspects of the window, such as antialising.


To create the GEM-window, you have to send the create message to gemwin. To destroy the window you have to send the destroy message.To display something on the window you have to turn on rendering. This can be done by sending a 1 to gemwin.You can only turn rendering on when there is already a GEM window.

gemhead

While gemwin creates a window for rendering, gemhead connects a set of GEM objects to this rendering context. gemhead is the start of a chain of graphics operations connected by patch cords that should be executed every frame. Drawing operations, including video effects, cascade from the top down, adding to each other flow downwards across objects. This chain of operations is triggered invisibly by gemwin according to the framerate you have set. gemhead triggers an output, the Gem-list at your framerate. You can turn this automatic rendering off by sending gemhead the message 0. Additionally, gemhead can be triggered manually by a bang, which is useful when you want to control the order in which your graphics chains are drawn. With several gemheads, you can force this execution order by either giving them an argument or set their order number. Lower numbers are triggered first. The default ordering number is 50.

Drawing something

The most basic Gem-objects are called Geos, which are primitive shapes. They are drawables that tell the renderer to draw a certain shape. One example of a drawable is the square object:


The sqare has two inlets. the first is for the Gem-list. The second can be used to set parameters. Keep in mind though that the only parameter that can be set for the square is the length of its edge.

Exercise

1. Open Pd
   
   

   ---

2. Create a new patch and save it to a folder and square_with_options
   
   

   ---

3. Create a create message with a second message of 1.This will create the window and start rendering:
   
   
   

   ---

4. Next create a gemwin object and connect the inlet to the outlet of the create message
   
   In order to open up a window for drawing, you have to create an object called gemwin which is the render context and you send it the messages create to create the window and 1 to start the rendering.
   
   

   ---

5. Add a destroy object and connect it to the gemwin object
   
   

   ---

6. Create a gemhead object
   
   
   

   ---

7. Connect a color object to the outlet of your gemhead object
   
   

   ---

8. Connect a rotate object to the outlet of your color object
   
   

   ---

9. Connect a square object to the outlet of your rotate object
   
   

   ---

10. Connect two messages to the right inlet of your color object. One message should be one color and the other message should be white (1 1 1)
    
    

    ---

11. Connect a number to the center inlet of your rotate object
    
    

    ---

12. Connect two messages to the left inlet of your square object. One message should be the draw function with line as a parameter. The other message should be the draw function with fill as the parameter.
    
    

    ---

13. Connect two more messages to the left inlet of your square object. One message should be the width a small number as a parameter (try 1.5). The other message should be width with a larger number as the parameter.
    
    

    ---

You can both rotate and translate shapes:

Exercise

Draw each of the other primitives shapes. Add rotateXYZ and translateXYZ

• [rectangle a b]
• [triangle edge]
• [cube edge]
• [circle diameter slices]
• [disk outer-radius slices inner-radius]
• [cylinder diameter,height]
• [cone diameter,height]
• [sphere diameter]

Controlling color

There are Control-objects that allow Geos to be moved and colored. Generally these objects are called before Geos. Control objects tell the renderer to change the state of the object.

You have to pass the color you want for painting as sets of RGB values each ranging between 0 and 1. If you don't specify any arguments, the default is white.To chnage the color during runtime, you have to send a list of values to the second inlet:

Moving around

The default position for Geos is the center of the screen. Each Geo has a pivot point. When the renderer is told to draw a Geo, it will place the pivot-point of the Geo on the origin of the 3D-world. The camera is aligned to focus on the world-origin of the 3D-space. You can move Geos with [translateXYZ X Y Z] and [translate factor vectorX vectorY vectorZ]

You can also rotate a Geo around its pivot point.

[rotateXYZ rotX rotY rotZ]	Rotates the Geo around the X-axis, then around the Y=axis and then around the Z-axis
[rotate amount vecX vecY vecZ]	Rotates the Geo by amount around the vector specified by the direction-vector vecX vecY vecZ

The order of translating and rotating makes a difference. If you rotate first and translate after, you will end up with a GEO that rotates around the origin of the scene rather than around the pivot point

• Translate before rotate=rotate around own axis (purple)
  
• Rotate before translate=rotate around origin (green)

You can put multiple GEOs in a gemlist. When doing transforations, each object that is below the transformation will be affected by it. Objects above the transformation will not.


Lower objects are considered to be parts of the higher objects. Think of it this way: your fingers are parts of your hand. If you move your hand, the fingers move too.

Sometimes you might want partly independent control over a Gem-object. If you modeled a human body and wanted to be able to turn the head and lift an arm independently, you could split the gemlist into independent sub-gemlists by using the separator object. Everything below the separator object is decoupled from other sub-gemlists under other seperator objects.

Adding Light

To see the sides of the objects, you need to add some light-sources.world_light is used as a far away light source. You can rotate the light source.

For positioning a light source you can send a debug 1 message to the light source, which will display a small cone that represents the light source. Unfortunately, positioning the light source is not enough. You must explicitly turn on the source. This can be done by sending a lighting 1 or lighting 0 message to the gemwin.



Here's a sphere with a world_light and a light. Recreate this patch and play around with the settings:


Here's are patches that are animated:

Exercise

Build a solar system where your planets orbit at different speeds.

//Using GEM

1. Open Pd
   
   

   ---

2. Create a new patch and save it to a folder and gem1_images
   
   

   ---

3. When you are using an image, it must be stored in the same folder as the patch. Find an image, move it to where you saved your patch, and make a note of the name of the image.
   
   

   ---

4. Back in the patch, create a create message with a second message of 1.This will create the window and start rendering:
   
   
   

   ---

5. Next create a gemwin object and connect the inlet to the outlet of the create message
   
   In order to open up a window for drawing, you have to create an object called gemwin which is the render context and you send it the messages create to create the window and 1" to start the rendering.

   ---

6. Create a gemhead object
   
   
   

   ---

7. The simplest way to display an image is using pix_draw, which will plot the image in it's original size centered onto the Gem-window. If the image is bigger than the Gem-window, it won't be displayed completely.
   
   Create a pix_image object and pass it the name of your image as the argument. Connect the outlet of gemhead to the inlet of the pix_image:
   
   
   

   ---

8. GEM is an openGL-application. openGL-applications normally do not draw images directly. Instead images are used as textures on the Geos.
   
   pix_texture is used for applying a texture. This means that the image is uploaded onto the memory of the graphics card. Each Geo that is drawn after uploading the texture in the same gemlist will have this texture applied.
   
   Create a pix_texture object to store the image in memory. Connect outlet to inlet
   
   

   ---

9. Create a rectangle object and pass it 4 3 as an argument
   
   
   openGL only supports textures with widths and heights that are powers of 2 (512x256). If you try to apply a wrong sized imaged as a texture, you won't see anything. To deal with this, there is pix_texture2, with allows images of any size to be textured on Geos.
   
   

   ---

10. Add a destroy method to the patch
    
    

    ---

pix_objects and and 3D Shapes

The objects that deal with textures are called pix objects and have the prefix pix_ in their object name. The doorway into the shape world is called pix_texture. This object sends the image from your CPU memory to the graphics card memory from where it is mapped onto one or several shapes.

For any image processing you need a source image. This can be a file that you load from your harddrive or a live video feed.

pix_image

To load an image into pix_image either add the filename as an argument or send it the message open filename. If you want to select a file using a file browser you can use the object openpanel.

pix_film

You can load movie files by passing the pix_film object the file name as an argument, or by sending it an open message with the filename. The list of supported formats may vary depending on your installed movie codecs, but usually you should be able to play *.avi, *.mov and *.mpg files.

Be aware that sound is not supported by pix_film or pix_movie. If you want to sync a soundtrack of a video to your images, you have to first extract it using an external video editor.

pix film will play your movie automatically if you send it a message auto 1. The framerate at which your movie is played is dependent on the framerate that was set with gemwin. The message auto 0 will cause pix_film to just display the current frame. You can use the right inlet to scroll through your movie or jump to a certain frame. That also allows you to play movies at different speeds, even backwards.

The rightmost outlet of pix_film will output a bang everytime the end of the film is reached.

pix_video

pix_video will grab live input from your built in camera. Usually you can receive a video signal only once on your machine, so if another application or even another Pd patch is already using video input, pix_video will not be able to receive a signal. However, if you have several cameras attached to your system, you can use several pix_video objects and specify the the camera devices with messages like device 0 and device 1.

A dimen message will let you set the resolution of your video image. If you use a small resolution, your render engine will have less pixels to render and will be faster, but of course this will also decrease the image quality. To test different resolutions you might want to set the quality setting of pix_texture to 0. Not all resolutions are supported by your system. Just play around with dimen to figure out how high or low you can set dimensions.

// Video Mixer

For this to work, both movies or images must be the same size.


Note that the horizontal slider has been modified:

// Alpha Blending

1. The color object allows you to colorize the image
   
   
   

   ---

2. Transform the colorspace into rbga in order to enable the alpha blending.
   
   

   ---

3. Add the following
   
   
   

   ---

4. Adjust the scale properties
   
   
   

   ---

5. Add a pix_texture object
   
   
   

   ---

6. Add a rectangle object with arguments 4 and 3. Then connect two horizontal scrollbars to it. One will control the x-scale, the other the y-scale. Set the range to be between -5 and 5
   
   
7. Save and test
   
   

   ---

To just use the camera, this is all you need


To add pix_video to the gem string we make this:

You need to add the dimen object because to use pix_mix, both sources must be the same size.

By default, pix_video outputs 320x240. That is a decent resolution for realtime live video mixing.

//3-Chroma key

1. Start with a mixer patch
   
   
   The direction message selects which video to process.

   ---

2. You can use these standard blend modes like add, substract, multiply, difference. pix_add, pix_multipy,pix_diff. Look into the help of this objects before using it, otherwise, probably you will get only Absolutely white images or absolutely black
   This objects shows the difference between two pixes:
   
   

   ---

//pix_record

The patch might look a bit complicated but basically:

• The top quarter allows you to create a Gem window and import Gem and cyclone/counter
  
  
• The lower-right part will mix two pictures at a certain rate given by the metro object
  
  
• The lower-left part will trigger recoding through pix_record Note that pix_record can (at the time of writing) only record one pix chain - you cannot save a texture projected on a geometry

//Movement

pix_movement compares the previous frame to the current one. It takes a threshold value (0-1) as an argument in the second inlet:


This will result in all pixel information blacked out except the space where a difference is detected relative to the previous frame.

pix_background is the cousin of pix_movement with the difference that you can set the reference frame to which the current frame is compared. This is called background substraction


Click on the reset message to pix background to set the current frame to the reference frame.

A drawback of pix_background compared to pix_movement is, that if lightning situations change, we will have to re-reset the frame to compare the current frames to.

Whether you use movement or background you need another object—pix_blob. The monochrome information you receive of pix_movement/pix_background are called blobs. The pix_blob object will give you more infomations about those blobs, which you need if you want to

• trigger things if something moves in the field of view of the camera (pix_movement)
• something is in the field of view of the camera what wasn't there when you set the reference frame (pix_background).

Source:http://en.flossmanuals.net/PureData/