Using Photoshop's Refine Edge to Create a Hair Selection

Photo courtesy of istockphoto, LL28, Image #23058113

NOTE: While Adobe replaced the beloved Refine Edge feature with Select and Mask in Photoshop CC 2015.5, many professionals still prefer the original tool's precision and workflow. The newer version lacks certain features that make hair selection more intuitive, but there's a hidden method to access the classic Refine Edge interface in modern versions. This technique remains relevant even in today's AI-enhanced Photoshop landscape, as understanding manual selection fundamentals makes you a more versatile designer.

Setting Up Your Workspace and Images

Professional hair masking requires methodical preparation. We'll start by organizing our workspace and importing the necessary files to create a non-destructive workflow that preserves your original images.

1. Launch Photoshop.

NOTE: This workflow has been tested extensively with Photoshop CC 2017 through current versions and remains compatible with modern releases, though interface elements may vary slightly.

2. Navigate to File > Open and:

• Browse to Desktop > Class Files > Hair the Seminar.
• Open Hair1_backgrounds.tif.

3. Examine the Layers panel (Window > Layers). You'll notice multiple background layers that simulate real-world client scenarios where you need to test your subject against various environments. This multi-background approach is standard practice in professional studios.

4. Select the top layer (Bricks) before importing your subject. This ensures proper layer stacking order, which becomes crucial when managing complex compositions with multiple elements.

5. Import your subject by going to File > Open and:

• Navigate to Desktop > Class Files > Hair the Seminar.
• Open dark_curly_iStock_000023058113_LL28.jpg.

6. Since this file contains only a Background layer (which has certain limitations), choose Layer > Duplicate Layer to create a more flexible working copy.

7. In the Document dropdown menu, select Hair1_backgrounds.tif to transfer the layer to your main working document.

8. Click OK to complete the transfer.

9. Close the original subject file (dark_curly_iStock_000023058113_LL28.jpg) to maintain a clean workspace.

10. You should now be working in Hair1_backgrounds.tif with your subject properly positioned.

11. In the Layers panel, rename the top layer from "Background" to "original" by double-clicking the layer name. This naming convention helps maintain organization in professional workflows where you might handle dozens of layers.

Creating the Initial Selection

The key to successful hair masking lies in understanding that high contrast between subject and background significantly improves your results. In this case, the dark curly hair against the light background provides excellent conditions for Photoshop's edge detection algorithms.

While Select and Mask is Adobe's current standard, many seasoned professionals still prefer Refine Edge for its superior handling of complex hair textures and more intuitive preview options. We'll show you the workaround to access this powerful legacy feature.

The initial selection doesn't need to be perfect—think of it as providing rough coordinates that Refine Edge will intelligently analyze and improve. This two-stage approach (rough selection + refinement) is more efficient than attempting pixel-perfect manual selection.

1. Select the Quick Selection tool from the toolbar. If it's not visible, click and hold the Magic Wand tool to reveal the flyout menu.

2. The Quick Selection tool functions like an intelligent paintbrush that analyzes color and texture similarities as you drag. Unlike traditional selection tools, each stroke automatically adds to your selection based on Photoshop's content-aware algorithms. Focus on selecting the woman's body, face, and the bulk of her hair. Don't stress about perfect hair edge accuracy—that's what we'll refine next.

Pay special attention to her striped shirt, ensuring you capture all the fabric details. These high-contrast stripes can sometimes confuse automatic selection tools, so make additional passes over any missed areas.

Pro Tips for Quick Selection mastery:

• To subtract from your selection, hold Option (Mac) or ALT (Windows) while dragging. Watch for the minus symbol in your cursor to confirm subtraction mode.
• Use Cmd–Z (Mac) or CTRL–Z (Windows) to undo individual selection strokes if the tool selects unwanted areas.
• Vary your brush size using the bracket keys—smaller brushes for detailed areas, larger ones for broad selections.

3. Here's the hidden technique to access classic Refine Edge: Hold Shift while choosing Select > Select and Mask. This bypasses the newer interface and opens the familiar Refine Edge dialog that many professionals prefer. (Users of pre-2015 Photoshop versions will see Refine Edge directly in the Select menu without needing the Shift key.)

4. Establish a non-destructive workflow by setting the Output To menu at the bottom of the dialog to New Layer with Layer Mask. This preserves your original image while creating a separate masked layer—essential for client work where revisions are common. Don't click OK yet; we have significant refinement work ahead.

5. Click the thumbnail beside View at the top of the dialog to access preview options.

6. Double-click On White (W) for optimal hair visibility during the refinement process. The white background provides maximum contrast for evaluating fine hair details, unlike the complex brick texture in our working file.

Mastering the Refine Edge Technique

The real power of Refine Edge lies in its ability to analyze transition zones between subject and background. By defining these areas, you're essentially teaching Photoshop where to look for complex edge details that would take hours to select manually.

7. Set your brush Size to approximately 125 pixels in the Options bar. This size works well for most hair textures, but you'll adjust it based on the fineness of individual hair strands.

8. Position the Refine Edge dialog so you can see your entire subject while keeping the controls accessible. This dialog must remain open throughout the refinement process.

9. The current preview shows only your selected area. Press X to toggle this view off and see the complete original image, which is essential for accurate edge painting.

10. Now comes the critical step: paint over all hair edges where hair meets background. As you drag, you'll see a green highlight that defines Photoshop's analysis zone. This green area should encompass both hair strands and background pixels—it's telling the software "figure out what belongs to the subject within this highlighted region."

The highlight disappears when you stop brushing, but the analysis continues in the background. Press X to toggle back to the white preview and observe the results of your edge definition.

11. Continue alternating between edge highlighting (original view) and result preview (white background view) by pressing X. This iterative approach ensures comprehensive coverage of all hair edges. Key areas to address:

• Paint over yellow color contamination areas within the hair—these occur where background color has "bled" into the hair selection and need special attention.
• Don't overlook the subtle hair wisps around the jawline and chin area, which are often missed but crucial for realism.
• Enable Show Radius in the dialog periodically to visualize all painted areas as a red overlay, ensuring complete edge coverage without gaps.

12. Fine-tune the Radius setting, which determines how far from your selection edge Photoshop searches for refinements. Start with small adjustments and observe changes in areas you didn't manually paint, such as the shirt and arms. The algorithm needs a moment to process changes, especially at higher radius values.

Higher radius values cast a wider net for edge detection but risk including unwanted background elements or creating soft, unrealistic edges. Lower values keep changes localized but may miss fine details. The sweet spot varies by image—this is where professional experience becomes invaluable.

13. Set the Radius to 3 pixels—our testing shows this value provides optimal results for this particular hair texture and background contrast.

14. Enable Smart Radius, which allows Photoshop to automatically vary the radius amount based on local edge complexity. Hair typically benefits from this adaptive approach since strand thickness varies significantly across the head.

15. Switch to On Layers (L) preview mode by clicking the View thumbnail and selecting this option. This shows your selection against the actual background layers, providing the most accurate representation of your final result.

16. Evaluate your work at 100% magnification (View menu > 100% or Actual Pixels) for accurate quality assessment. Screen resolution can hide selection problems that become obvious at full size.

17. Use the Spacebar + drag combination to navigate around the image at high magnification—essential for detailed inspection of your mask quality.

18. Address the visible yellow color contamination by increasing Contrast to 15%. This setting sharpens the transition between selected and unselected areas, reducing color bleeding and creating cleaner edges.

19. Apply a Shift Edge value of -25% to contract the selection slightly, which removes more background contamination. Negative values pull the edge inward, while positive values expand it outward.

20. Add a subtle Feather of 1 pixel to soften the edge transition. This prevents the harsh, cut-out appearance that screams "digital manipulation" to viewers.

21. Verify your settings match these tested values:

22. These settings represent the optimal balance for this image type, but remember that every photo requires individual assessment. Hair color, background contrast, lighting conditions, and even camera settings all influence the ideal Refine Edge parameters.

23. Click OK to apply your refinements and generate the masked layer.

Note on Decontaminate Colors: While this feature promises to remove color contamination automatically, it often produces unnatural-looking results, particularly with hair. The manual technique we'll demonstrate next provides superior control and more realistic outcomes—a key reason why professionals often avoid automated solutions in favor of hands-on methods.

24. In the Layers panel, rename the new "original copy" layer to "silo"—industry terminology for a subject isolated from its background. Consistent naming conventions prevent confusion in complex projects.

Perfecting the Layer Mask

Even the best automatic selection tools occasionally miss spots or create inconsistencies. Professional workflows always include a manual inspection and touch-up phase to ensure flawless results.

1. Examine your mask in detail by holding Option (Mac) or ALT (Windows) and clicking the black-and-white layer mask thumbnail in the Layers panel.

2. The document window now displays the full-sized mask, where white areas reveal the subject and black areas hide the background. Look for any unwanted black or gray spots within the subject area—these represent gaps in your selection that need filling.

Depending on your initial selection quality and the subject's complexity, you may find minimal cleanup needed. However, it's always worth checking, as small mask imperfections become glaringly obvious when the subject is placed on contrasting backgrounds.

If cleanup is needed:

• Select the Brush tool from the toolbar.
• Press D to reset colors, then X if needed to set white as the foreground color. In mask editing, white reveals and black conceals—this fundamental principle governs all mask work.
• Adjust brush size using the bracket keys ([ to decrease, ] to increase) and modify hardness by holding Shift + bracket keys. Use harder brushes for defined edges, softer brushes for gradual transitions.
• Paint over any black spots that should be white, effectively "filling in" gaps in your subject.

3. When mask editing is complete, click the silo layer thumbnail (not the mask) to return to normal image viewing mode.

Eliminating Color Contamination

Color fringing—where background colors contaminate subject edges—is one of the most obvious signs of poor masking work. This yellow contamination around our subject's hair would immediately mark the composite as amateur work. Professional retouchers address this systematically using targeted color correction techniques.

1. Work at 100% magnification to accurately see and address color contamination. What looks acceptable at lower zoom levels often reveals problems when viewed at full resolution.

2. Ensure the silo layer thumbnail is selected (not the mask) since we're modifying the actual image pixels, not the mask.

3. Select the Brush tool and configure it for color correction work.

4. Set the brush Mode to Color in the Options bar. This blend mode affects only the color information of pixels, preserving the underlying luminosity and texture—crucial for maintaining natural hair appearance.

5. Reduce Opacity to 35% for subtle, buildable corrections. Professional retouching relies on multiple gentle applications rather than heavy-handed single strokes.

6. Choose a soft brush around 125 pixels to start, adjusting size as needed for different areas.

7. Sample clean hair color by holding Option (Mac) or ALT (Windows) and clicking on an uncontaminated dark hair section. This sets your "target" color for correction.

8. Gently brush over yellow-contaminated hair areas. Each stroke gradually replaces the yellow cast with the sampled natural hair color. Some areas may need multiple passes, while others require just a single stroke. This gradual approach maintains the hair's natural color variation.

Professional insight: Using 100% opacity would create unnaturally uniform color, destroying the subtle variations that make hair look real. The 35% opacity allows you to build up corrections gradually while preserving natural color fluctuations.

9. Adjust brush size as you work—use the bracket keys for quick size changes. Smaller areas like those near the chin require smaller brushes for precision.

10. Address skin contamination as well. Yellow highlights on the nose, lips, neck, and arm also need correction to sell the illusion of natural lighting on the new background.

11. Sample appropriate skin tones for each area—hold Option/ALT and click on uncontaminated skin nearby. For facial areas, sample from the same region (nose color for nose correction) to maintain consistent skin tone mapping.

12. Carefully paint over contaminated skin areas:

• Nose highlights
• Lip edges
• Neck transitions
• Arm boundaries

13. Reset the Brush mode to Normal when finished to avoid confusion in future tool use. This housekeeping step prevents accidental color-mode painting in subsequent work.

Advanced Mask Refinement

Even after careful color correction, a subtle halo effect often remains around complex selections. This requires targeted mask adjustment using Curves—a technique that sharpens the transition between subject and background without affecting the subject itself.

1. Click the silo layer's mask thumbnail to target the mask for adjustment.

2. Apply a Curves adjustment directly to the mask using Cmd–M (Mac) or CTRL–M (Windows). This opens the same Curves dialog used for image adjustments, but now it affects mask density rather than image tonality.

3. Create a curve that darkens the mask's gray areas while preserving the white and black extremes. Drag the curve downward in the shadow-to-midtone region as shown:

This adjustment converts gray "maybe" pixels to black "definitely background" pixels, tightening the edge transition and eliminating the soft halo effect.

4. Toggle the Preview checkbox to compare before and after results. The halo reduction should be immediately apparent, especially around fine hair details.

5. Click OK to apply the mask refinement.

6. Test your work against different backgrounds by hiding the Bricks layer (click its eye icon) to reveal the beach background underneath.

7. Evaluate the composite quality. Professional-level hair masking should look convincing across multiple background types—this versatility separates amateur from professional work.

Final Touches and Optimization

Professional compositing often requires additional refinements based on the specific background and intended use. These final adjustments can make the difference between a good composite and an exceptional one.

Optional refinements to consider:

• Selective flyaway removal: Select the silo layer mask and paint black over distracting stray hairs that don't contribute to the overall hair volume. Not all flyaways need removal—keep those that add to realism while eliminating obviously problematic ones.
• Edge transparency effects: Create subtle transparency around hair perimeters using a large, soft brush at 10% opacity with black paint on the mask. This technique mimics the natural light transmission through fine hairs and prevents the "cut-out" appearance common in amateur work.
• Mask edge softening: Apply the Blur tool directly to mask edges for a more organic transition, particularly effective with very fine hair details.

8. For this particular composition, flip the subject horizontally using Edit > Transform > Flip Horizontal to create better visual flow with the background.

9. View the complete composition using View > Fit on Screen to evaluate the overall impact and balance.

Professional Reality Check: Remember that Refine Edge performance varies significantly based on source material quality. Factors like image resolution, compression artifacts, lighting conditions, hair texture, and background complexity all influence results. This technique excels with high-contrast scenarios like our example but may struggle with low-contrast situations (blonde hair on white backgrounds) or extremely complex backgrounds.

The hair mask we've created works well on brick and beach backgrounds but would show limitations against black backgrounds due to the dark hair color. Professional workflows often require multiple masking approaches depending on intended use—this is why mastering various selection techniques is crucial for serious retouchers.

Expanding Your Creative Toolkit

Hair masking represents just one facet of professional image manipulation. Today's creative professionals leverage Photoshop alongside other industry-standard applications to create comprehensive visual solutions for clients across industries.

Modern graphic designers integrate Photoshop's image editing capabilities with InDesign's layout prowess and Illustrator's vector precision to deliver complete brand experiences, from digital campaigns to print materials.

The creative industry offers substantial opportunities for skilled professionals who master these interconnected tools. Graphic design roles command competitive salaries and offer diverse career paths, from agency work to in-house creative teams to freelance consulting.

Our comprehensive training programs reflect current industry practices, with instruction from active professionals who understand today's creative marketplace. Small class sizes ensure personalized attention, while project-based curricula help students build portfolios that demonstrate real-world capabilities. Flexible scheduling accommodates working professionals looking to advance their skills:

• Advanced Photoshop training NYC
• Professional InDesign courses NYC
• Expert Illustrator instruction NYC
• Comprehensive graphic design programs NYC