The Boombox Incident: removing bald people from photos

In Seinfeld episode #163 , “The Slicer”, George has just landed a cushy job at Kruger Industrial Smoothing, when he sees himself in the background of a family photo on his new boss’s desk.

George is instantly reminded of the “boombox incident”, in which he had, years earlier, embarrassed himself in front of the Kruger family.

Later, upon hearing about the photo, Kramer suggests that George sneak the photo off to have himself airbrushed out of the picture, so that Kruger doesn’t remember the incident and fire George.

George enacts the plan and things are going fine until he receives the touched-up photo: the clerk has removed Kruger from the family photo instead of George.

The clerk mistook Kruger in the photo for George, since in the picture George had hair but Kruger was bald. Removing the only bald person from the photo was a pretty reasonable thing for the photo store clerk to do. I figured this is something that photo editors have to do frequently, so I decided to automate it.

The process for removing bald people from photos is as follows:

• detect faces in an image using off-the-shelf tools
• for each face
  • roughly locate the forehead/hair region
  • get the dominant color of the face and of the top of the head
  • compare the two colors
  • consider a bald subject to be one where the two colors are very close, i.e. the top of the head is the same color as the face (skin tone)
  • attempt to inpaint the region containing the bald individual

Face detection

Face detection in OpenCV can be accomplished with a cascade classifier . A pre-trained model from the OpenCV data repository is made available. The underlying algorithm at play is the Viola-Jones object detection framework , which uses a coarse-to-fine cascade of Haar-feature matching to identify human faces.

face_cascade = cv2.CascadeClassifier('./haarcascade_frontalface_default.xml')

def detect_faces(image):
    h, w, _ = image.shape
    gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    min_size = int(w*0.12)
    
    faces = face_cascade.detectMultiScale(
        gray,
        scaleFactor=1.1,
        minNeighbors=1,
        minSize=(min_size, min_size),
        flags = cv2.cv.CV_HAAR_SCALE_IMAGE
    )
    
    return faces

g_and_k_img = cv2.imread('./input_images/george-and-kruger.jpg')
tmp = np.copy(g_and_k_img)
for (x, y, w, h) in detect_faces(g_and_k_img):
    cv2.rectangle(tmp, (x, y), (x+w, y+h), (0, 255, 0), 2)

Forehead region

Once the faces in a photo have been located, we need to examine just above each face region to determine whether the person is bald or has hair. This approach assumes, of course, that the faces are oriented vertically. A better method might be to locate the eyes and/or mouth using other cascade classifiers and then approximate the forehead position from there.

def forehead_region(img, face_loc):
    img_h, img_w = img.shape[:2]
    x, y, w, h = face_loc
    fore_w = int(w * 0.33)
    fore_h = int(h * 0.25)
    fore_x = min(x + ((w - fore_w) // 2), img_w)
    fore_y = max(y - fore_h, 0)
    return img[fore_y:fore_y + fore_h, 
               fore_x:fore_x + fore_w,
               :]

Dominant color

The algorithm presented here compares the color in the forehead/top-of-head region to that of the face region. To find the dominant color, we can use k-means clustering . This quantizes all pixel BGR values to be one of k colors, and we chose the most frequent. Simply averaging all colors in the image can also work well (and is faster) if the region is tightly bound.

def dominant_color(img):
    # via https://docs.opencv.org/3.0-beta/doc/py_tutorials/py_ml/py_kmeans/py_kmeans_opencv/py_kmeans_opencv.html
    Z = img.reshape((-1,3))
    Z = np.float32(Z)
    
    # define criteria, number of clusters(K) and apply kmeans()
    criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 200, 1.0)
    K = 3
    _, labels, palette = cv2.kmeans(
        Z,
        K,
        criteria,
        10,
        cv2.KMEANS_RANDOM_CENTERS
    )
   
    # via https://stackoverflow.com/a/43111221
    _, counts = np.unique(labels, return_counts=True)
    dominant = palette[np.argmax(counts)]
    return np.uint8(dominant)

Bald?

With the dominant color for each subject’s face and hair regions obtained, we can compare the values to determine baldness. Comparing colors programatically is not as simple as taking the Euclidean distance of the BGR color vectors, because the resultant differences don’t correspond well to human color difference perception. A better distance metric is ΔE* in the CIE Lab color space . I used the colormath package to perform those distance calculations.

We compare the color difference to a threshold, and this heuristic is the baldness detection. It probably gives false positives for subjects with hair color close to their skin tone.

# via http://hanzratech.in/2015/01/16/color-difference-between-2-colors-using-python.html
from colormath.color_objects import sRGBColor, LabColor
from colormath.color_conversions import convert_color
from colormath.color_diff import delta_e_cie2000

def is_bald(img, face, thresh=40):
    face_img = face_region(img, face)
    forehead_img = forehead_region(img, face)
    face_color = dominant_color(face_img)
    forehead_color = dominant_color(forehead_img)
    face_rgb = sRGBColor(face_color[2], face_color[1], face_color[0])
    fore_rgb = sRGBColor(forehead_color[2], forehead_color[1], forehead_color[0])
    delta = delta_e_cie2000(
        convert_color(face_rgb, LabColor),
        convert_color(fore_rgb, LabColor)
    )
    return delta < thresh

Inpainting

Now that bald individuals have been identified, they can be removed from the image. OpenCV has an inpainting function, which is really only meant for removing small strokes from an image. The results of applying it here are…not ideal.

def rm_bald(img):
    tmp = np.copy(img)
    img_h, img_w = img.shape[:2]
    
    for b in bald_locs(img):
        x, y, h, w = b
        img_h, img_w = tmp.shape[:2]
        mask = np.zeros(img[:,:,0].shape, np.uint8)
        mask[max(0, y-50):min(img_h, y+h), max(0, x-10):min(img_w, x+w+10)] = 1
        mask[max(0, y+h):min(img_h, y+int(2.5*h)), max(0, x-10-w//2):min(img_w, x + w + int(w/2) + 10) ] = 1
        tmp = cv2.inpaint(tmp, mask, 10, cv2.INPAINT_TELEA)
    return tmp

References:

• https://docs.opencv.org/3.0-beta/doc/py_tutorials/py_ml/py_kmeans/py_kmeans_opencv/py_kmeans_opencv.html
• https://docs.opencv.org/3.4.3/d7/d8b/tutorial_py_face_detection.html
• http://kramersapartment.com/characters/mr-kruger/
• https://en.wikipedia.org/wiki/The_Slicer
• https://www.youtube.com/watch?v=pSB3HdmLcY4
• http://hanzratech.in/2015/01/16/color-difference-between-2-colors-using-python.html
• https://realpython.com/face-recognition-with-python/
• https://github.com/opencv/opencv/tree/master/data/haarcascades