Using OpenCV to detect face key points with C++

This post is a follow-up on my first post about building a face detector with OpenCV in C++. In this post we will build on the existing code and detect face key points. The result will look like this.

Since we will work with a relatively new version of OpenCV (4.2.0), you might want to go back to the previous post to read more on how to install the necessary packages.

The code is on my github.

Let’s get going!

Detecting face key points

After detecting faces in the last post, we now want to detect face key points. We use the cv::face::FacemarkLBF model to find key points in the face rectangles we identified in the last tutorial.

Adding the key point detection model file

As for the face detection model, we have to add a model file for the LBF model. I put the model file in the assets folder of this posts git repo, so you can go there and download it.

For passing the location of this model file to our code, we will use the same CMake target_compile_definitions trick we used in the first post. So make sure you have the model file in the right place and add the following lines to your CMakeLists.txt.

# Introduce preprocessor variables to keep paths of asset files
...
set(KEY_POINT_DETECTION_MODEL
    "${PROJECT_SOURCE_DIR}/assets/lbfmodel.yaml")
...
target_compile_definitions(${PROJECT_NAME}
    PRIVATE KEY_POINT_DETECTION_MODEL="${KEY_POINT_DETECTION_MODEL}")

A class for the key point detector

We start by adding a class for the key point detector. This way we have the code for initializing and calling the model in one place.

KeyPointDetector.h

In the include folder, we create a file KeyPointDetector.h. This will be the header file for the key point detector.

The KeyPointDetector will have two public methods. The first is a constructor. We will use the constructor to initialize the underlying LBF model.

The second method detect_key_points detects face key points within a given rectangle in an image. As the key points for each face are of type std::vector<cv::Point2f>, this function will return a vector of std::vector<cv::Point2f>.

The header file for the key point detector will look like this.

#ifndef KEYPOINTDETECTOR_H
#define KEYPOINTDETECTOR_H

#include <opencv4/opencv2/face.hpp>

class KeyPointDetector {
public:
    /// Constructor
    explicit KeyPointDetector();

    /// Detect face key points within a rectangle inside an image
    /// \param face_rectangles Rectangles that contain faces
    /// \param image Image in which we want to detect key points
    /// \return List of face keypoints for each face rectangle
    std::vector<std::vector<cv::Point2f>>
    detect_key_points(const std::vector<cv::Rect> &face_rectangles,
                      const cv::Mat &image) const;

private:
    cv::Ptr<cv::face::Facemark> facemark_;
};


#endif //KEYPOINTDETECTOR_H

KeyPointDetector.cpp

Next, we implement those methods in src/KeyPointDetector.cpp.

First, let’s look at the constructor. We create a new cv::face::FacemarkLBF model. Then we load the model configuration from the KEY_POINT_DETECTION_MODEL variable we passed in via CMake.

KeyPointDetector::KeyPointDetector() {
    facemark_ = cv::face::FacemarkLBF::create();
    facemark_->loadModel(KEY_POINT_DETECTION_MODEL);
}

Following, we implement detect_key_points.

To adhere to the API of cv::face::Facemark::fit(), we transform our input to a cv::InputArray. Then we call the models fit function and return the detected points.

std::vector<std::vector<cv::Point2f>>
KeyPointDetector::detect_key_points(
        const std::vector<cv::Rect> &face_rectangles, 
        const cv::Mat &image) const 
{

    cv::InputArray faces_as_input_array(face_rectangles);
    std::vector<std::vector<cv::Point2f> > key_points;
    facemark_->fit(image, 
            faces_as_input_array, 
            key_points);

    return key_points;
}

Using the key point detector

Now we jump to our main.cpp to use the key point detector we defined. We use the face detector from the previous post. Then we feed the detected rectangles to our key point detector.

#include <opencv4/opencv2/opencv.hpp>
#include "FaceDetector.h"
#include "KeyPointDetector.h"

int main(int argc, char **argv) {

    cv::VideoCapture video_capture;
    if (!video_capture.open(0)) {
        return 0;
    }

    FaceDetector face_detector;
    KeyPointDetector keypoint_detector;

    cv::Mat frame;
    while (true) {
        video_capture >> frame;
        auto rectangles = face_detector
                .detect_face_rectangles(frame);

        auto keypoint_faces = keypoint_detector
                .detect_key_points(rectangles, frame);

Instead of displaying the rectangles, we display the detected points.

        const auto red = cv::Scalar(0, 0, 255);
        for (const auto &face :keypoint_faces) {
            for (const cv::Point2f &keypoint : face) {
                cv::circle(frame, keypoint,
                           8, red, -1);
            }
        }

        imshow("Image", frame);
        const int esc_keycode = 27;
        if (cv::waitKey(10) == esc_keycode) {
            break;
        }
    }
    cv::destroyAllWindows();
    video_capture.release();
    return 0;
}

You should see a result similar to the image below.

Conclusion

In this post we used a face detection model to find faces in an image. Then we found key points in those images using OpenCV.

I hope this helps you to build interesting stuff! Here is a link to the code. Let me know if you run into any errors!

Follow me on twitter (@bewagner_) for more content on C++ and machine learning!