Using the Roto Brush Tool

Getting the Project Files

1. Download the project files
2. After the download completes, unzip the file if your system hasn't done so automatically. You should find an After Effects Audio Spectrum folder containing all necessary assets for this tutorial.

Project Overview

In this comprehensive tutorial, we'll master rotoscoping using After Effects' powerful Roto Brush tool—a technique that has become indispensable in modern post-production workflows. Rotoscoping involves creating frame-by-frame selections of footage elements, and professionals use it for two primary purposes.

First, rotoscoping can create stylized cartoon effects, as seen in acclaimed films like A Scanner Darkly and Waking Life, where the entire footage is traced to achieve a distinctive animated aesthetic. Second—and more commonly in commercial work—rotoscoping isolates specific elements within your footage, enabling precise compositing, selective color correction, or the integration of graphics and effects.

For this tutorial, we'll focus on the latter application, demonstrating how to isolate a bird from its background so we can seamlessly place animated text graphics behind it. This technique is fundamental for creating professional-quality compositions where foreground elements interact naturally with added graphics.

The skills you'll learn here translate directly to high-end production work, from advertising campaigns to feature films. Let's dive into the technical process.

Step-by-Step Guide: Making the Roto Brush Selection

1. Import your footage by dragging it from the Project Panel directly onto the Timeline—this ensures proper layer organization from the start.
2. Press Return (Mac) or Enter (PC) to rename the layer with a descriptive name that reflects its purpose in your composition.
3. Double-click the footage layer in the Timeline to open it in the Layer Panel. This is crucial: the Roto Brush tool functions exclusively within the Layer Panel, not the Composition Panel.
4. Select the Roto Brush tool from the toolbar—look for the distinctive brush icon overlaying a human silhouette. This tool uses advanced edge-detection algorithms to predict and extend your initial selections.
5. Navigate to Windows > Workspace > Paint to establish an optimal workspace layout. This configuration provides side-by-side panels showing both your detailed layer work and the main composition, streamlining your workflow significantly.
6. Begin your initial selection by clicking and dragging around your subject. The Roto Brush uses intelligent sampling, so focus on capturing the general shape rather than pixel-perfect precision at this stage.
7. Refine your selection by holding Option (Mac) or Alt (PC) to subtract unwanted areas. This subtractive mode is essential for cleaning up the algorithm's initial interpretation of your subject boundaries.
8. Access the Refine Edge tool by clicking and holding the Roto Brush icon. This specialized tool excels at handling complex organic edges like hair, fur, feathers, and fabric textures that standard selection tools struggle with.
9. Apply the Refine Edge tool by dragging it across problematic edge areas. The tool analyzes texture patterns and edge characteristics to create more natural, believable cutouts.
10. Press Spacebar to initiate the preview process, allowing After Effects to analyze and process your selection across multiple frames. Processing time varies significantly based on your system's RAM, CPU power, and the complexity of your footage.

Advanced Techniques: Adding Professional Background Effects

Once your rotoscoping work is complete, you can leverage the isolation to create sophisticated layered effects that would be impossible with traditional masking techniques.

1. Create a duplicate background layer by dragging another instance of your original footage from the Project Panel to the Timeline. Position this layer below your rotoscoped element to establish proper layer hierarchy.
2. Rename this layer using Return (Mac) or Enter (PC) to maintain organized project structure—professional workflows demand clear layer naming conventions.
3. In the Effects and Presets panel, locate the Tritone effect. This powerful color-grading tool allows precise control over highlights, midtones, and shadows using three distinct color values.
4. Apply Gaussian Blur from the Effects and Presets panel to the same background layer. This creates depth separation between your isolated subject and the background elements.
5. In the Effect Controls panel, adjust the Tritone Midtones to a neutral grey value. This establishes the foundation for your color transition effect.
6. Set the Tritone "Blend With Original" parameter to 100%, temporarily neutralizing the effect while you establish your keyframe structure.
7. Position your playhead at the composition start and activate keyframing by clicking the stopwatch icons next to both Gaussian Blur and Tritone parameters.
8. Advance the playhead several seconds to establish your end keyframe position.
9. Increase the Gaussian Blur value to create the desired background softness—typically 15-25 pixels depending on your footage resolution and artistic intent.
10. Reduce the Tritone "Blend With Original" setting to 0%, allowing the full color effect to manifest.
11. Any additional elements—text layers, graphics, or secondary footage—placed below the rotoscoped layer will interact naturally with your isolated subject, creating convincing depth and integration.

Video Transcription

Hello, this is Tziporah Zions for Noble Desktop, and in this tutorial, I'm going to show you how to use the Roto Brush in Adobe After Effects. We'll be learning how to use the Roto Brush to easily separate elements from our footage. This allows you to place individual effects on each piece or place objects underneath video footage. Here's what our finished video is going to look like.

You can see how this parrot is separated out from the rest of the footage here. Rotoscoping is a great technique for separating foreground elements from background elements. The Roto Brush is cool tool in After Effects, that near automates and otherwise tedious manual process. By painting and adjusting with this brush, we can separate out our elements much faster and easier than if we would if we do the old school frame by frame method.

We're going to be using just this one piece of video footage over here and this precomp animation I've made for you as well. So not much to sift through in the project. The file will be available for download in the video description below.

So let's get started. First off, let's drag and drop our footage into the empty timeline over here. Thank you. Click on the footage layer in the layer stack over here. And if you have a Mac, you're going to hit Return and a PC like mine will hit ENTER and we're going to name this, which is called Roto Brushed Parrot. Now double click on the footage to go inside the layer.

This is where we're going to be doing most of our work with the Roto Brush. Now here's something of the cool. Head up to Window, Workspace. Look for Paint, and the reason why we're going to be doing this is because this is going to rearrange the interface to give us all the painting tools like brush sizes, side by side comparison, view our footage and so on and so forth. And you could check it out.

Now we can simultaneously work on a piece of footage and also see update on the main comp over here. So I'm actually going to shrink it a little bit. We get a bit more of my layer footage over here, right? Check out this little icon up here. You see, it's like a little person with the brush on it.

So this is the Roto Brush. So with our Playhead at the origin right here at the start we are going to click on the Roto Brush and now let's click and kind of like paint an outline, so to speak around the parrot. It really doesn't have to be exact. You know, just it's all about getting it done at this stage. So as you can see, the selection tools are pretty generous, so we're going to refine it. So hold on AKT on a PC, CMND on a Mac. And so the Roto Brush is red, we're actually going to slice up these chunks from a selection.

You know, bit by bit until we get more of an exact selection. All right. Once that's done, I take a look down here. You can see it's pretty small, but there's like a little green line over here. And what this bar is, is that this defines how many frames the Roto Brush is going to process. We're going to want to constrain the amount so as not to overwhelm the program.

So grab the end point over here and drag it to the left, drag to the left. We're going to put it at around three seconds. Yeah. About three seconds. That's what we're going to do now, we are going to refine your selection so click and hold on the Roto Brush tool here, and this is Refine Edge and it's really what it sounds like, you refine. Some edges are pretty useful for things like fur, hair feathers, you know, fine organic edges.

And with that, I'm just going to click and drag and paint over some of the more hairy edges, so to speak. All right now, we are going to be working with the effect controls. Now they should be docked in here on next to composition. And we're, you know, composition window, we are going to actually grab and I want to pull them out so I can work with them a little more easily. So I'm going to grab this hamburger and undock it.

Let's see. I will float you over here. I think that's all right. First thing that we're going to do. Make sure that the version is up to 2.0. And then we're going to be heading through these settings now if yours is not open, toggle open. Open Roto Brush Matte. And let's be heading through our selections. So Feather over here refers to not the feathers on the bird, but, you know, kind of like soft and blurry you want edges to be

We're going to put that about nine, you know your project might be a little different. This is what's going to work for mine personally. Contrast is how much of a, you know, a hard edge you want between the values of your selection and what's around it. So it's going to make it a little more defined.

I'm going to put mine at around 75. Let's see and then shift edge really again, what it sounds like, it moves the edges of selection in and out. I'm going to try with negative 95 for this. All right. All right, we're getting there. Let's see what else. See a little bit of cleanup we're going to do over here, but for the most part, this is what I want to see.

All right now, we are going to hit spacebar or whatever you have set for preview on your machine. And we're going to have the program process our selection, give the program a minute to get through all those frames. And you should note that the speed at the Roto Brush runs at is entirely based on the computer set up RAM, processor, video card, you know, so on, so forth. All right, you can see that our parrot now isolated from the background.

So what we're going to do, we're going to finish this up with some effects. So let's set up a background as well as some cool transition effects next. So let's see. Let's go over to our project one over here. Grab another copy of the original footage. We're going to put it right underneath our Roto Brush parrot right here. And let's see. Let's, you know, hit ENTER on a PC or Return on a Mac, and we're going to call this Background Video.

All right. Let's have it line up a little more with the footage. There we go. All right, then next up is the transition effects with blurring and grayscale, so we're going to head up to Window, Effects and Presets. Here it is. So mine pops into the left hand side over here. If yours appears different, you know, just dock it wherever it feeks comfortable. All right. We are going to be looking for Gaussian Blur.

Here we go. Just let's see. Let's use this Guassian Blur, drag and drop it onto our background video footage. Nothing's going to happen just yet. And let's see. So the first thing that we're going to do with this one, we're going to be hitting Repeat Edge Pixels over here.

This option here, you could see the whole sentence over there to start with, and that's going to have the edge pixels of this blurred out effect. Also be included in the blurred out effect, but you'll see it when we get to that. And then the next effect, let's see. Let's head back to effects and presets. I'm actually going to pull this out and going to undock and drag it over here so I could have access to multiple panels.

Let's put in Tritone. Drag that over to our backgrounds footage, great. We've got two of these things. And the next thing that we're gonna be doing is click that brown box over there. Drag it over to the grey because we don't want this to have a sepia tone. We want it to really just have more of a grayscale look. And then we're going to keyfram this thing. Now with your Playhead at the origin hit the stopwatches next to Blend with Original and Blurriness.

All right, and then Blend with Original should be all the way to 100%. So basically, we're not really seeing any effect just yet. So the next thing we're going to grab our Playhead, get it to be like one second in about ish, you know, and the next thing that we're going to do is we're going to change blurriness to 20, Blend with Original to 0. And once the machine catches up we're going to be seeing those effects. All right, and then finally go over to project panel, grab this text precomp over here, drag and drop it in between those layers so it's a bit of a sandwich.

And as you can see now, the Rotoscoped parrot is hovering above the animation right below it, and it retains its own colors on, you know, own values, while the other effects go on all around it. So let's preview that.

All right. As you can see, when I scroll through the playback, I could see that the parrot is moving, the backgrounds moving and text moving. But they are all segmented and separate from each other, with different effects applied to each one. Now we're all done. So as you can see, rotoscoping is really cool. The technique can be used not only on video footage, but on still images as well. It can be repeated on a single piece of video as well to separate multiple objects through the background and all sorts of different effects can be applied individually to each rotoscripted element.

You can try combining text with totally different footage or video or still images to the back or foreground of any future rotoscoped project. All right. That's all for this tutorial. I hope you've enjoyed learning how to use the Roto Brush in Adobe After Effects. This has been Tziporah Zions from Noble Desktop.