GLSL Shader Actor Tutorial Print

GLSL Shader Tutorial for Artists and Designers

This tutorial will introduce you to OpenGL Shader Language (GLSL) programs and how to make use of them within Isadora v2.5 and later. GLSL is a very advanced programming language; if you are an artist or designer, the source code for a GLSL program can be a bit intimidating on first glance. Nevertheless, if you consider yourself a maker or tinkerer, and can devote a bit of time to experimenting, this tutorial will show you how to make use of this very powerful image processing technology in Isadora.

What is OpenGL Shader Language?

GLSL is a programming language, similar in style to the C programing language, that tells your graphics card (GPU) how to manipulate or generate images. The power of GLSL shader programs is that, unlike programs that run in your computer's main processor (CPU) they run in parallel for every pixel of the image. This means they are incredibly fast, even with very high resolution images.

As of version 2.3, Isadora offers an actor called GLSL Shader that allows you to compile and run GLSL code.

Introduction

There are vast amounts of GLSL code on the Internet: you can find source code on sites like ShaderToy.com and GLSLSandbox, just to name two. But if you are not a programmer, the source code for a GLSL program may look like "gobbledygook" to you. This tutorial will offer strategies for non-programmers, that will allow you not only to compile and run GLSL programs in Isadora, but also to add custom parameters to allow real time interactive manipulation and/or modulation of those images.

Tutorial 1: Finding Shaders Online and Using Them in Isadora

The first step is to find some GLSL code to try in Isadora. ShaderToy.com is one of the most extensive repositories of GLSL shader code, and allows you to preview the video output of the code in any WebGL enabled browser. Isadora's GLSL compiler has been specifically designed to recognize code from the ShaderToy site so that, in most cases, using code found on ShaderToy is as simple as copying and pasting.

To start, go to http://shadertoy.com and enter this  in the search box

nebula - space

You will be led to a relatively simple shader called "nebula - space" submitted by a user called pazimor. The preview looks like this:

If you click on preview, you will be taken to a window that shows the preview on the left and source code on the right. (One nice thing about ShaderToy is that you can easily modify the source code on the right and immediately see the result in your browser.)

To try this code in Isadora, do the following

1. Click on the source code editor on the right and hit Command+A (Mac OS) or Control+A (Windows) to select the entire text. This selects all of the source code. Then, from the menu, choose Edit > Copy or Command/Control+C.
2. Switch to Isadora. Make sure the stage is visible by choosing Output > Show Stages or Output > Force Stage Preview.
3. Double-click in Isadora's Scene Editor to show the pop-up toolbox, and type "GLSL" (without the quotes.) You'll see the GLSL Shader actor listed in the pop-up toolbox. Click on the GLSL Shader item to deposit the actor into the Scene.
4. Double-click in the Scene Editor again, but this time type "Projector" in the pop-up Toolbox. Click on the Projector item to deposit the actor in the Scene.
5. Make a link between the video output of the GLSL Shader actor and the video input of the Projector actor.
6. Double click the GLSL Shader actor in Isadora. The first time you do this in a session, you'll see a warning dialog about using GLSL code. If you see this warning, Click OK to continue.
7. The GLSL Source code editor will appear. Choose Edit > Paste to paste the source code from ShaderToy into the GLSL Shader code editor.
8. Click the Compile button to compile the code. (Note that you don't need to close the editor window to see the results; as soon as you hit the Compile button the result of your changes will be visible at the GLSL Shader actor's output.) If all is well, should see no errors. If all is well, you'll see the same image you saw on the ShaderToy site appear on Isadora's stage.
9. Note that the 'time mod' input was automatically added to the GLSL Shader actor. You can adjust this value to change the rate of speed with which you fly through the nebula.

There are some common inputs that will be present on the GLSL Shader Actor. Some, like the size inputs will always be present. Others like the 'time mode' input will be present if they apply to that shader.

• 'horz size' and 'vert size' – set these inputs to non-zero values to specify the output resolution of the actor
• 'time mod' – If the shader generates a moving image, this input controls the overall speed of movement, with 1 being 'normal' speed, 0 being still, and 2 being twice normal speed.
• 'mouse horz' and 'mouse vert' – Some ShaderToy shaders expect a mouse input. In Isadora, these inputs can come from any source. If you wish to use the mouse, connect the 'horz pos' and 'vert pos' outputs of a Mouse Watcher actor to the 'mouse horz' and 'mouse vert' inputs of the GLSL Shader actor.

For many shader programs, it's as simple as that: copy the GLSL code and paste it into the code editor of Isadora's GLSL Shader actor. In the next tutorial, we'll handle a special case for ShaderToy shaders where the shader requires an input texture.

IMPORTANT NOTE: ShaderToy recently added a new feature called "Multipass Rendering." If you see tabs labeled "Buf A" or "Buf B" at the top of the source code editor will not yet be able to user this shader code in Isadora.

We'll be updating Isadora to allow multipass rendering in a future update.

Tutorial 2: Using ShaderToy Shaders that Require Textures

Sometimes the process above will either not render an image, or will not render the image as it appears on the ShaderToy preview. Often, this is because the shader requires an input texture to generate the correct image. To see what we mean, enter this text in the search box on ShaderToy

Clouds

and hit the search button. The first result should look like this. (If not, look through the previews to find this one.)

It is a fairly complex shader called Clouds by a user called "iq". Click on the preview to open the editor.

To begin this tutorial, repeat the entire process outlined in Tutorial 1 to copy and paste the source code into Isadora's GLSL Shader actor. After you've pasted it, close the editor dialog.

If you take a close look at the GLSL Shader Actor, you'll see that a video input labeled 'video in 1' has appeared as the first input. That means that the shader requires some kind of video or still image to function properly. You can find the reason by going back to page for the Clouds shader on Shader Toy

If you look under the code editor, you'll see the following.

ShaderToy offers the option to input predefined textures (images) as an input to the shader. You can see here that you can supply as many as four different texture inputs. For shaders that do not require any texture inputs, iChannel0 through iChannel3 will be black, indicating that there is no input required on that channel. But, if you do see an image in one of these preview panes, then you will need to provide the equivalent texture in Isadora.

In the example above, iChannel0 is not black; it has been set to a different source. To see the possible sources, click on the iChannel0 preview. This dialog will appear:

Unfortunately, this dialog does not show which input has been selected. (In this case, the actual texture being used is not even on the list. We're actually not quite sure why this is the case. perhaps this discrepancy was introduced because of a version change at ShaderToy?)

In any case, for the vast majority of shaders, the input texture is a noise source. So, choosing one of the noise pictures supplied later in this tutorial will generally give you the right result. Finding the right texture is simply a matter of experimentation.

Unfortunately we cannot give you an easy way to get those textures from ShaderToy, because the links for these pictures change with each session. But since the noise textures are the most commonly used, we've created our own versions below that you can download. Right-click (control-click) the pictures below and choose "Save Image to As..." to save these files to your hard drive so you can import them and use them in Isadora.

tex11.png

tex12.png

tex15.png

tex16.png

1. tex16.png texture as instructed above and import it into Isadora.
2. Create a Picture Player actor and set the picture the media index associated with the picture you just imported.
3. Connect the output of the picture player to the video input of the GLSLS Shader actor
4. The image should now render correctly.

Tutorial 3: Adding Real-Time Parameters to A Shader

The really exciting opportunity offered by the GLSL Shader actor is the ability to manipulate shader parameters in real time. Shaders accept real-time input through a mechanism known as uniform variables. For example, if you wanted the shader to receive a floating point number, you would add a line like this.

uniform float myVariableName;

This defines a floating point number called myVariableName as an input. Then, you would use that variable in your code as needed. The trick here is to make that variable available to you as an input to the GLSL Shader actor. This is achieved using a special comment line that you add to the shader like this

// ISADORA_FLOAT_PARAM(name, id, min, max, default, "help text");

In OpenGL Shader Language, any line that starts with "//" is a considered to be a comment; in other words, it is ignored by the compiler. But it is not ignored by Isadora. Using comments like the one above, you can create an input for the GLSL Shader actor that sends it's value to a uniform variable in the shader

Let's go though each part of this special comment in detail

• // ISADORA_FLOAT_PARAM — This portion simply identified what kind of parameter is being defined. In this case, single floating point number.
• name — This is where you define the name of the variable. This name must exactly correspond to the variable name given in the uniform variable — statement in the shader code. You cannot include spaces in this name; if you need a space, use the underscore (_) character instead.
• id — This identifier, which can be from 1 to 4 characters long, uniquely identifies the input. Should you re-arrange the order of the inputs, Isadora uses this identifier to ensure the links to that input are maintained. For each shader program, this identifier must be unique.
• min — For numeric parameters, this defines the minimum possible value for the input.
• max — For numeric parameters, this defines the maximum possible value for the input.
• default — For numeric parameters, this defines the the default value for this input when the actor is added to the scene. This input is only meaningful if you add the source code for this shader to the Isadora's GLSL Plugins file. (More on this feature later.)
• "help text" — This defines the help text that will appear in the information view for this input. If you share your shader code, you can help those who use it by providing useful, descriptive information about this input. Isadora's philosophy has always been to make it easy for the user to use the program; giving useful details in the help text supports that philosophy.

So how do you put all of this in practice? Here's a simple example.

// ISADORA_FLOAT_PARAM(red_color, red, 0.0, 1.0, 1.0, "Amount of red in the output color.");
// ISADORA_FLOAT_PARAM(green_color, gren, 0.0, 1.0, 1.0, "Amount of green in the output color.");
// ISADORA_FLOAT_PARAM(blue_color, blue, 0.0, 1.0, 1.0, "Amount of blue in the output color.");

uniform float red_color;// note: must match first parameter of ISADORA_FLOAT_PARAM
uniform float green_color;// note: must match first parameter of ISADORA_FLOAT_PARAM
uniform float blue_color;// note: must match first parameter of ISADORA_FLOAT_PARAM

void main(void) {

   // vec4 is contains four floating point values; it is 
   // often used to represent the red, green blue and alpha
   // channel components of a color. Here we create a color
   // using the red, green and blue uniform variables with 
   // full (1.0) alpha. We assign this to a vec4 called c0.

   vec4 c0 = vec4(red_color, green_color, blue_color, 1.0);

   // gl_FragColor is a predefined variable that receives the color
   // output of the shader. By setting gl_FragColor to the values
   // contained in c0, the output color will be controlled by the 
   // three uniform inputs.

   gl_FragColor = c0;

}

To see this code in action:

1. Add a GLSL Shader actor, as well as a Projector actor, to the Scene and link them together. Also, make sure to show the stage using Output > Show Stages
2. Double-click the GLSL Shader actor to open the editor.
3. Copy the source code above and paste it into the editor
4. Click OK in the editor dialog.
5. The stage output should be totally white. Now start changing the values of the 'red_color', 'green_color', and 'blue_color' inputs of the GLSL Shader actor. You'll see that the output color changes accordingly.

So the purpose of the ISADORA_FLOAT_PARAM comment is to make a link between the Isadora working environment and the internal values of the shader, allowing real-time, interactive manipulation of the shader values. While the output of this example is a not terribly exciting, it serves to show how this link is made. In the next Tutorial 6, we'll do something something that will get your blood-pumping a bit faster. ;-)

Tutorial 4: Adding Video Inputs

Most of the GLSL Shader code you'll find on the Internet is generative: it does not manipulate a video stream. But GLSL Shader code also allows you to modulate a video stream. Because shaders are so fast, you can process HD video streams with very little impact on performance. This tutorial gives a basic example of how to do this.

We're going to implement a variation of Isadora's MultiMix actor, which receives up to eight video streams and adds them together to produce one output. In our version, we will be able to modulate the brightness of each stream independently.

To specify that a shader has a video input, you need a statement like this:

uniform sampler2D tex0;

To see this in action, do the following:

1. Add a GLSL Shader actor to your patch and double-click it to open the editor.
2. Paste the following code in the the editor:

uniform sampler2D tex0;
void main(void)
{
   gl_FragColor = texture2D(tex0, gl_TexCoord[0].xy);
}

3. Then hit the OK button to compile the code and close the editor
4. You will see that a new video input labeled 'video in 1' has been added to the GLSL Shader actor. Connect a video source to this input, and then connect the 'video out' output to a Projector actor.
5. This is a very simple shader: the output is the same as the input. This shader simply passes the input input on to the output. But now, let's add a parameter to control the brightness. Update the source code inside the GLSL Shader to look like this:

uniform sampler2D tex0;

// ISADORA_FLOAT_PARAM(intensity_1, v1br, 0.0, 100.0, 100.0, "Intensity of the 'video in 1' video stream.");

uniform float intensity_1;

void main(void)
{
   float intensity = intensity_1 / 100.0);
   // texture2D -> means sample a pixel from an input texture
   // tex0 -> indicates the specific texture to sample
   // gl_TexCoord[0].xy -> determines which pixel to sample
   // multiplying by 'intensity' modulates the alpha, red, green
   // and blue components of the pixel
   gl_FragColor =  texture2D(tex0, gl_TexCoord[0].xy) * intensity;
}

6. Click OK to compile the shader and closet the editor.

As you can see, we've added a new floating point input parameter called 'intensity_1'. In the code, we divide this value by 100.0 to give us a range of 0.0 to 1.0, storing the the result in a variable called intensity. We then multiply the ARGB pixel retrieved by the texture2D statement by this value. If the intensity is 0.0, then the color is black because all the components (ARGB) are 0.0. If intensity value is 0.5 then the video will be at half intensity, because all the color components have been multiplied by one-half. If intensity is 1.0, then the input color is unchanged.

To add more inputs is a relatively simple matter. We've written the code to handle three inputs below. Try the following:

1. Replace the code in the shader with the code below

uniform sampler2D tex0;// video input 1
uniform sampler2D tex1;// video input 2
uniform sampler2D tex2;// video input 3

// INTENSITY MODULATION INPUTS
// ISADORA_FLOAT_PARAM(intensity_1, v1br, 0.0, 100.0, 100.0, "Intensity of the 'video in 1' video stream.");
// ISADORA_FLOAT_PARAM(intensity_2, v2br, 0.0, 100.0, 100.0, "Intensity of the 'video in 2' video stream.");
// ISADORA_FLOAT_PARAM(intensity_3, v3br, 0.0, 100.0, 100.0, "Intensity of the 'video in 3' video stream.");

uniform float intensity_1;// intensity for video stream 1, from 0 to 100
uniform float intensity_2;// intensity for video stream 2, from 0 to 100
uniform float intensity_3;// intensity for video stream 3, from 0 to 100

void main(void)
{
   vec4 outputColor;

   // modulate the pixel from input 1 by intensity_1 and store the result in outputColor
   outputColor = texture2D(tex0, gl_TexCoord[0].xy) * (intensity_1/100.0);

   // modulate the pixel from input 2 by intensity_2 and accumulate the result in outputColor
   outputColor += texture2D(tex1, gl_TexCoord[0].xy) * (intensity_2/100.0);

   // modulate the pixel from input 3 by intensity_3 and accumulate the result in outputColor
   outputColor += texture2D(tex2, gl_TexCoord[0].xy) * (intensity_3/100.0);

   // set the output (gl_FragColor) with the result in outputColor
   gl_FragColor = outputColor;
}

2. Click the OK button to compile the code and close the editor.
3. Connect three video sources to the three video inputs, and try changing the values of the three intensity inputs. You'll see that you can vary the brightness of each image independently.

Isadora offers a maximum of eight texture inputs to the GLSL Shader actor. So, with a bit of copying, pasting and editing, you could expand the code above to handle as many as eight video inputs.

Again, if you're not a coder, this tutorial might feel a bit intimidating. But if you've worked with a procedural language like Javascript or something similar, you're probably not too far away from creating simple shaders like the one above.

Tutorial 5: Putting it All Together: Real-Time Shader Manipulation

At this point, the design philosophy of Isadora -- make it easy for the user -- comes into conflict with the complexity of OpenGL Shader Language. In the end, there's no getting around the fact that some of the shader programs out there are extremely complex. Thus, to successfully change them would seem to require a great deal of skill. (If it offers you any comfort, the author of this tutorial is also the creator of Isadora. While he is a very skilled C++ programmer, even he finds code in many of these shaders daunting!) So, how in the world is a non-programmer supposed to modify code like this?

In this tutorial, we're going to do our best to offer some guidelines that will allow non-programmers to successfully add real-time control to GLSL Shaders. To be sure, it will require some time and effort. But, if you are the type that likes to tinker and experiment, we feel you'll be able to get meaningful results. Let's give it a try.

1. First let's get the code from this page on GLSL Sandbox: http://glslsandbox.com/e#22343.0. As before, you'll need a GLSL Shader actor connected to a Projector actor. Double-click the GLSL Shader actor and paste this code into the editor and hit Compile. You should see a slightly bumpy green sphere.
2. What you want to do is to start looking for constant values in this code. For starters, scroll down until you find the line that starts
   

   void main(void) {

   The main() function is a good place to start looking, because it is the main function that does the work to render the image. (There are other functions in this code, e.g., float sphere() and float sinusoidBumps(). The names of these functions give you a clue that they probably have to do with a) rendering the sphere, and b) the bumps on the sphere. Things like function names and variable names may help you discern good places to start tinkering.

3. Look for some constant values, like 1.0, or 22.5.
4. Change the value of one of those numbers and hit Compile. At first, don't change the value by a lot, e.g., if the value is 3 then change it to 4 or 5. Does something interesting happen? If not, choose File > Undo to restore the value. Then try a bigger change, e.g., if the value is 3, make it 300.
5. Eventually, you'll change some value that does something interesting to the resulting image. In the case of the source code given above, we discovered that the value 0.1 in the function sinusoidBumps() led to a cool effect, making the surface of the sphere more or less bumpy.
   

   return sphere(p, vec3(0., 0. , 2.), 1.) + 0.1 * sinusoidBumps(p);

6. Once you find an interesting value like this, you should experiment to find the minimum and maximum values that are useful. This simply requires some experimentation and guess work. In this example case, we found that a minimum value of 0.0 and a maximum value of 1.0 yielded good results. (You can go further than 1.0 if you like; the results are kind of crazy, but maybe you'll like them.)

So, we've found a parameter that one we want to modulate in real time. We will need to add a few statements to give us an input port on the GLSL Shader actor that will allow us to access this value. Continuing now in the editor:

1. Create an ISADORA_FLOAT_PARAM statement to give us an input port like this:
   

   // ISADORA_FLOAT_PARAM(bumpiness, bump, 0.0, 1.0, 0.04, "Bumpiness of the ball's surface.");

   and insert it after the uniform vec2 mouse; statement.

2. Just below the ISADORA_FLOAT_PARAM statement we add a uniform float statement so that the shader will recognize this parameter:
   

   uniform float bumpiness;

3. And finally we modify the function inside of the sinusoidBumps() function.
   

   return sphere(p, vec3(0., 0. , 2.), 1.) + bumpiness * sinusoidBumps(p);

4. Now hit the Compile button and close the editor. You'll see that a new parameter has appeared at the bottom of the GLSL Shader actor called 'bumpiness.' Apply some real-time input like the output of the Sound Level Watcher to watch the fun begin.

Tutorial 6: Adding GLSL Shaders to the Toolbox

If you find or create a shader that you find especially useful, you may want to add it to Isadora's toolbox for easy access in the future. This tutorial explains how to do that, as well as introducing some further comment statements that allow you to personalize your shader.

To add a shader to Isadora's toolbox, do the following:

1. Copy the text from the GLSL Shader actor's editor into any text editor. Then save the file, giving it a meaningful name that ends with a .txt extension, e.g. "my_shader.txt"
2. Navigate to the following folder:
   

   Mac OS: /Library/Application\ Support/TroikaTronix

   Windows: C:\Program Files (x86)\Common Files\TroikaTronix\

3. If you don't find a folder called GLSL Plugins in the TroikaTronix directory, then create it.
4. Place the text file you created above in the GLSL Plugins folder.
5. Quit and restart Isadora.
6. In the Toolbox, click "GLSL Shaders"
7. You will see the new plugin listed there; the name will be the same as the file name of the .txt file. Click the plugin to add it to the scene as you would with any Isadora actor.

After the shader source code to the GLSL Plugins folder, using this new actor is as simple as clicking it in the toolbox or searching for it using the popup toolbox.

You can further customize your shader by adding additional comment statements recognized by Isadora.

• // ISADORA_PLUGIN_NAME("Plugin Name Here") — Specifies the name that will be displayed in Isadora's Toolbox for this shader
• // ISADORA_PLUGIN_DESC("Description of what the plugin does here") — This is the help text that will be displayed in the information window if you hover the mouse over this GLSL Shader actor
• // ISADORA_PLUGIN_COPYRIGHT("(c) 2015 My Name Here") — Allows you to add a copyright statement to your plugin.
• // ISADORA_PLUGIN_LICENSE("License info here.") — If you would like to add a Creative Commons or other licensing information to this plugin, give the license information here.
• // ISADORA_INT_PARAM(name, id, min, max, default, "help text") — This comment works the same as the ISADORA_FLOAT_PARAM comment, but instead of adding a floating point number input, this version adds an integer input. The corresponding uniform would be uniform int variableNameHere;

To make your plugin easy for other to use, we hope that you will add the ISADORA_PLUGIN_NAME and ISADORA_PLUGIN_DESC statements.

Sharing your new GLSL Plugin is as simple as posting the source code. They don't have to be Isadora users — any GLSL compiler/program can use them. Keep in mind that programs other than Isadora will not recognize real-time input statements (like ISADORA_FLOAT_PARAM.) Because of this, the plugins probably won't function as expected in other software.

Addendum: Real-Time Input for GLSL Types Other Than Float & Int?

You may need to input real-time values to types other than float and int, but Isadora's comment format does not allow for types other than this. Here's how to solve that problem.

For example, let's say that you wanted a real-time input to a mat2. (mat2 is actually a two-dimensional matrix consisting of four floats: two columns across and two columns down. (You can learn more about GLSL types on this page.)

To provide like input to a mat2, you'd need to define four separate inputs, giving both the Isadora Param statement and the accompanying uniform float;

// ISADORA_FLOAT_PARAM(complexity00, cp00, 9.0, 900.0, 10.0, "mat[0][0]"); 
// ISADORA_FLOAT_PARAM(complexity01, cp01, 9.0, 900.0, 10.0, "mat[0][1]"); 
// ISADORA_FLOAT_PARAM(complexity10, cp10, 9.0, 900.0, 10.0, "mat[1][0]"); 
// ISADORA_FLOAT_PARAM(complexity11, cp11, 9.0, 900.0, 10.0, "mat[1][1]");
uniform float complexity00; 
uniform float complexity01; 
uniform float complexity10; 
uniform float complexity11;

Then, when you need the mat2 in your function, you'd set the values like this

mat2 complexity;
complexity[0][0] = complexity00; 
complexity[0][1] = complexity01; 
complexity[1][0] = complexity10; 
complexity[1][1] = complexity11;

Addendum: Performance Issues

Even though shaders are incredibly fast, they are not an unlimited resource. Remember, the program you see in the editor is going to be executed for every pixel in the image. For a 1920x1080 image, the GLSL shader program will be executed over 2 million times!

If you are experiencing performance issues (e.g., low frame rates) take a look at the shader code. If you see statements that start with "for" then the shader has a "for/next" loop, and are likely to be inefficient. Less critical, but nevertheless inefficient, are statements that being with "if" or "else". These indicate an "if/then" construct, which will also slow the shader down.

If you are a non-coder, you probably won't be able to modify those for, if or else statements to improve performance. Your only option in this case is to lower the resolution of the image to reduce the load on the graphics card.

If you want to try to tweak the code, the easiest thing to do is to reduce the number of repetitions in a "for" statement. These statements will generally be in the form

for (x=0; x<256; x++)

In this case, a variable called 'x' will count up from 0 to 255, with x being incremented by one after each repetition. That's 256 repetitions in all, and over 530 million calculations for a 1920x1080 image.

To reduce the number of repetitions, change the upper limit of the for/next loop to a smaller value. (In the example above, that number is shown in bold.) This will change the way the image looks, but you can decide if the result is acceptable or not.