Skip to content

Chroma-Key

Title:

Creating Chroma-Key Effects with OpenCV and Java

Take Chroma-Key Science With You!

Abstract:

This project demonstrates how Chroma-Key technology works, commonly known as “green screen,” by developing a Java-based app using OpenCV to replace green backgrounds with custom images. The goal is to understand the technical process behind Chroma-Keying and apply it through a program capable of handling video frames in real time.

Question:

How can we use Chroma-Key technology to replace a green screen background in real-time using OpenCV and Java?

Hypothesis:

If we can effectively identify and mask the green screen pixels in a video feed, then we can replace the background with high-quality images, enhancing the visual appeal and practical applications of the footage.

Background Research:

Chroma-Keying is a special effects technique widely used in media production, where a solid color (usually green or blue) background is replaced by another image or video. OpenCV, an open-source computer vision library, allows us to perform image and video processing, making it ideal for experimenting with Chroma-Keying effects in Java. This project explores color detection, masking, and compositing to achieve the effect.

Additional Research Necessary for Implementation

How HSV Represents Color

HSV stands for Hue, Saturation, and Value, which are three characteristics that define color:

  • Hue (H): This is the main color component, represented by an angle on a color wheel (0–360 degrees). It determines the color type—e.g., red, blue, green. For example: 
    1
2
3
    Red ≈ 0° or 360°
Green ≈ 120°
Blue ≈ 240°

  • Saturation (S): Saturation measures the intensity or purity of the color, from 0 to 100%. A low saturation results in a more grayish color, while higher saturation yields vivid colors.

  • Value (V): Value represents the brightness of the color, from 0% (black) to 100% (full brightness). This aspect determines how light or dark the color appears.

In computer vision, HSV is often preferred over RGB for color filtering because it better separates chromatic content (i.e. color) from intensity (brightness), making it easier to isolate colors in various lighting conditions.

HSV Range in Java
  • To define a color range in HSV, we establish the lower and upper bounds for hue, saturation, and value.
  • In OpenCV, the range is typically set like shown below: 
    1
2
3
    // Define the lower and upper bounds for the color range
Scalar lowerBound = new Scalar(35, 100, 100); // Lower HSV bounds for green
Scalar upperBound = new Scalar(85, 255, 255); // Upper HSV bounds for green
  • Images then need to be converted from BGR to HSV for comparison 
    1
    Imgproc.cvtColor(image, hsvImage, Imgproc.COLOR_BGR2HSV);
  • Finally a inRange comparison can be made to mask the HSV range. 
    1
    Core.inRange(hsvImage, lowerBound, upperBound, mask);
Why a Bluish or Greenish Range is Ideal for Mask Creation

Green and blue are ideal for creating Chroma-Key masks for several reasons:

  • Distinct Hue: Green and blue hues are distinct from most natural skin tones and colors found in clothing, making it easier to isolate these colors without accidentally masking people or other foreground objects.
  • High Intensity and Saturation: Green and blue tend to provide high saturation values without losing brightness, resulting in a clean and easily distinguishable color even in varying lighting.
  • Lower Noise in Channels: Since RGB cameras often capture more data in the green and blue channels, they produce less noise in these colors compared to others, allowing for cleaner edges when creating the mask.

Materials:

  • Computer with Java and IntelliJ installed
  • OpenCV library
  • Web camera or video source with green screen backdrop
  • Sample images for background replacement
  • Good lighting setup for consistent background color

Procedure:

  1. Setup OpenCV and Java Environment: Integrate OpenCV into a Java project in IntelliJ.
  2. Capture Video Feed: Use OpenCV to access the webcam and capture video frames. 
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
    configuration = Configuration.getInstance();
capture = new VideoCapture();
if(configuration.getConfig(ConfigKey.CAMERA) != null) {
    cameraIndex = (int)configuration.getConfig(ConfigKey.CAMERA).value();
}
desiredSize = new Size(
        (int)configuration.getConfig(ConfigKey.FRAME_WIDTH).value(),
        (int)configuration.getConfig(ConfigKey.FRAME_HEIGHT).value()
);
/*...This happens in separate method...*/
capture.open(cameraIndex);
// Set the resolution of the captured video
capture.set(Videoio.CAP_PROP_FRAME_WIDTH, desiredSize.width);
capture.set(Videoio.CAP_PROP_FRAME_HEIGHT, desiredSize.height);
/*...Video processing starts in its own thread of execution...*/
// Launch processing and displaying frames in a separate thread
cameraThread = new Thread(this::processAndDisplayFrames);
cameraThread.start();
/*...Capture occurrs in Thread...*/
// Capture a frame from the webcam
Mat frame = new Mat();
capture.read(frame);
  3. Green Color Detection: Configure OpenCV to detect green pixels using lowerBound and upperBound HSV color thresholds. 
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
    // Define lower and upper bounds for green color in HSV (Hue, Saturation, Value)
int hue = (int)configuration.getConfig(ConfigKey.LOWER_HUE).value();
int sat = (int)configuration.getConfig(ConfigKey.LOWER_SATURATION).value();
int val = (int)configuration.getConfig(ConfigKey.LOWER_VALUE).value();
//System.out.printf("L-H:%d,S:%d,V:%d%n", hue, sat, val);
Scalar lowerBound = new Scalar(hue, sat, val);  // Lower Green
hue = (int)configuration.getConfig(ConfigKey.UPPER_HUE).value();
sat = (int)configuration.getConfig(ConfigKey.UPPER_SATURATION).value();
val = (int)configuration.getConfig(ConfigKey.UPPER_VALUE).value();
//System.out.printf("U-H:%d,S:%d,V:%d%n", hue, sat, val);
Scalar upperBound = new Scalar(hue, sat, val);  // Upper Green
  4. Mask Creation: Create two mask by first finding all pixels in HSV detection range mask, then invert the mask to extract everything NOT in the color range maskI. 
    1
2
3
4
5
6
    Mat mask = new Mat();
Mat maskI = new Mat();
// Create a mask based on the specified color range
Core.inRange(hsvFrame, lowerBound, upperBound, mask);
// Invert the mask (to select everything except the specified color)
Core.bitwise_not(mask, maskI);
  5. Apply Background Replacement: Use maskI and mask to create a new image, but overlaying a chosen background image on each pixel of mask. 
    1
2
3
4
5
6
7
8
9
    String background = (String)configuration.getConfig(ConfigKey.BACKGROUND).value();
// Load the replacement background image.
replacementMat = Utils.loadImage(background, desiredSize);
/*...Another part of code...*/
// Apply the inverted mask to the original frame
Core.bitwise_and(frame, frame, result, maskI);
Mat currentResult = matBuffer.swap(result);
// Apply the replacement image to the original frame where the mask is applied
Core.bitwise_or(replacementMat, replacementMat, currentResult, mask);
  6. Real-Time Processing: Ensure the replacement happens quickly enough for real-time playback. 
    1
2
3
4
    // Convert the modified frame to a JavaFX Image
Image imageToShow = Utils.mat2Image(currentResult);
// Update the image buffer with the new image
imageBuffer.updateImage(imageToShow);
  7. Test and Refine: Adjust thresholds and lighting to achieve clean edges and minimal color spill. Chroma-Key App Adjustments

Results:

  • Real-time Chroma-Key effect successfully replaces the green screen with the chosen images.
  • Key challenges include handling shadows and lighting inconsistencies, which affect green color detection. Chroma-Key Result

Conclusion:

Using OpenCV with Java effectively implements a Chroma-Key application, where proper masking and background replacement create professional-looking results. This project demonstrates how color-based keying and masking can simulate effects typically seen in media productions.

Future Direction:

  • Experiment with other background colors (e.g., blue) to understand color sensitivity.
  • Explore improvements in edge detection and shadow handling using advanced OpenCV filters.
  • Add video as a replacement background for dynamic scene changes.