Faceswap & state-of-the-art (SOTA)

This post is pending

Intro: faceswap & deepfake

Face swap refers to an activity in which a person's face is swapped with the face of another person.

Deepface = deeplearning + fake. Deepfake is an advanced technology, but 96% ò Deepfakes videos have triolous content. Deepfake also targets political purposes, fraud and market manipulation.

Traditional swap face - using opencv-dlib Facial landmark Detection

Facial landmarks are used to localize and represent salient regions of the face. The pre-trained facial landmark detector inside the dlib library is used to estimate the location of 68(x,y)-coordinates that map to facial structures on the face.

Each frame of the animation shows a Delaunay trianulation of the four points. Halfway through, the trianglulating edge flips showing that the Delaunay trianglulation maximizes the minimum angle, not the edge-length of the triangles.

The final steps of face alignment to to consider corresponding triangles between the source face and the targret face, and affine warp the source face triangle onto the target face. However, aligning the face and slapping one face on top of the other hardly looks unnatural. The seams are isible because of lighting and skin tone differences between the two images.

Seamless Cloning is a feature that you can copy an object form one image, and paste it into another image making a compsition that looks seamless and natural.

The central insight in the paper is that working with imagge gradients instead of image intensities can produce much more realistic results. After samless cloning the intensity of the result imagge in the masked region is NOT the same as the intensity of the source region in the masked region. Instead, the gradient of the result image in the masked region is about the same as the gradient of the source region in the masked region. Additionally, the intensity of the result image at the bounndary of the masked region is the same as the intensity of the destination image.

Deepfakes - Intro

When applied correctly, this technique is uncannili good at swapping faces. But it has a major disadvantage: it only works on pre-existing pictures. It cannot, for instance, morph Donald Trump;s face to match the expression of Ted Cruz.

That has changed in late 2017, when a new approach to face-swap has appeared on Reddit. Such a breakthough relies on neural networks, computationsal models that are loosely inspired by the way real brains process information. This novel technique allows generating so-called deepfakes, which actually morph a person's face to mimic someone else's features, although preserving the original facial expression.

Deepfakes - Training

It refers to the process which allows a neural network to convert a face into another. Although it takes several hours, the training phase needs to be done only once. Once completed, it can convert a face from person A into person B.

The diagram above shows an face being fed to an encode. Its result is a lower dimensional representation of that very same face, which is sometimes, reerred to as base vector or latent face. The latent face is then reconstructed. Autoencoders are lossy, hence the reconstructed face is unlikely to have the same level of detail that was originally present.

When the training process is complete, we can pass a laten face generated from Subject A to the Decode B. The Decoder B will try to reconstruct Subject B, from the information relative to Subject A.

If the network has generalized well enough wwhat makes a face, the latent space will represent facial expressions and orientations. This means generating a face for Subject B with the same expression and orientation of Subject A.

Continued