Just recently when looking at the papers presented at the 2019 CVPR (Conference on Computer Vision and Pattern Recognition) I fell in love since the first sight with one named “Expressive Body Capture: 3D Hands, Face, and Body from a Single Image”. Published mainly by the Max Planck Institute for Intelligent Systems, what it accomplish can quickly be summarized by the heading image.

That’s it. The paper presents SMPLify-X, a method to extract the complete 3D human body pose from a single RGB image. Thanks to Deep Learning, there has been recently rapid progress on human body pose detection and if you haven’t seen about it, then you should check OpenPose first. But while impressive, OpenPose is limited to only detect the 2D position of certain keypoints, and that may not be enough information about the body if we are trying to understand more complicated things like human interaction and communication. If we are pursing to read body language, facial expressions or detecting subtle movements, then we need a complex, expressive 3D model of the body, and a method to extract this model from a image.

The body model used by SMPLify-X is SMPL-X (SMPL eXpressive), an extension of another existing model SMPL, modified to make it more expressive.

To install SMPLify-X, first you need to download SMPLX, the body model used, from the project website, registering and accepting the terms for its use license. Once confirmed the mail and logged into the site, on the Downloads section you will find the “SMPL-X Model” and the “VPoser: Variational Human Pose Prior”, download and extract them somewhere on your disk. These are going to be your $MODEL_FOLDER and $VPOSER_FOLDER respectively. Then, clone the SMPLify-X repository and install the required libraries on the requirements.txt and optional-requirements.txt.

To run SMPLify-X you need to create a folder, place there the images on which you want to detect the bodies, and create another folder containing the OpenPose detection outputs for those images. This is because SMPLify-X relies on OpenPose’s detections to generate its own 3D body detection, so you need to run it on the images first and store there the JSON files outputs. Also, you need to point the configuration file for the body model you want to use, which contains all the parameters needed for the optimization.

The big amount of configuration available on SMPLify-X results in long pieces of code, and can generate some confusions, useless if we only want to run the code with the standard configuration and see the results quickly. On this experimentation scenario, we just want to provide minimal customization, perform the body detection on a single image and see the results. To be able to do this, I wrote some changes on the original code, and made them available on this repository: playing_smplifyx. It’s a fork from the original repo, where you can easily get the code I run to generate the examples run on this post.

Checking out the first commit you will get the modified SMPLify-X to get an easy run without much configuring. It uses the OpenPose Python wrapper, so you need to have it properly installed. The modified code is on three files:

• easy_configuration.py : Loads the standard configuration and arguments parsing.
• openpose_wrapper.py: Loads OpenPose, and performs the 2D keypoint detection on a image. CONFIGURE YOUR params[“model_folder”] HERE!!
• easy_run.py: Performs the body detection, calling the optimizer for the given image.

Download the repository and do git checkout d8e16257. To execute, run the following changing $MODEL_FOLDER and $VPOSER_FOLDER to where you extracted the downloaded files, and $IMAGE_PATH to the path you want to analyze:

  
    mkdir run
    mkdir run/images
    mkdir run/keypoints
    python3 smplifyx/easy_run.py --config cfg_files/fit_smplx.yaml  \
        --data_folder run \
        --output_folder out \
        --visualize="True" \
        --model_folder $MODEL_FOLDER/models \
        --vposer_ckpt $VPOSER_FOLDER \
        --input_media $IMAGE_PATH
  

This creates some required folders to trick SMPLify-X, don't pay attention on them, and then runs the detector with a standard configuration: using the SMPL-X model and a a neutral gender body type. Running it with this image provided on OpenPose source, we get the following detection:

SMPLify-X behind the scenes

Let’s see how does it work. Most of the action happens on fit_single_frame.py where it again loads a bunch of configuration and:

1. Makes a first estimation of the camera position, placing a body on the scene and moving the camera so that the rendered pose is close to the target body.
2. Optimizes the full body model, leaving the camera fixed and modifying the body model parameters so that it suits better the 2D keypoints.

To better understand this, I wrote a new Monitor Viewer, the object used to visualize the progress on the detection. The original one just renders the body model while optimizing it, I changed it so it renders the model on top of the original image, so we can actually see how the optimization is trying to fit the 3D model to the body.

On the playing_smplifyx repository, do git checkout ec3cd9082d and run again the detector, this time you will get something like:

It can bee seen how each body is first placed on a standard “stand-up” position, and then the body model starts to move trying to find which position better suits the image.

You can try SMPLify-X on your own images, and dive in the code to understand what is it doing. Meanwhile, I’m going to carry on the investigation and wrote another post soon!