What will we do when the video clips we rely on for news are routinely and effortlessly faked – when it becomes nearly impossible to differentiate genuine statements from malicious imitations without a thorough investigation? Will you continue to believe every word uttered by US media organizations like CNN? Or will you slowly learn to question every piece of information you receive?
We’ve reported in the past about studies showing how effortlessly even a novice hacker could learn to edit videos of people speaking to make it seem as if their words were coming out of the mouth of a famous celebrity like, say, former US President Barack Obama.
Last year, a paper by the Stanford lab of Matthias Niessner titled "Face2Face: Real-time Face Capture and Reenactment of RGB Videos” shows how disturbingly easy it is to take a surrogate actor and, in real time, using everyday available tools, create the illusion that someone else, notably someone famous or important, is speaking. Even more disturbing: one doesn't need sophisticated equipment to create a "talking" clone - a commodity webcam and some software is all one needs to create the greatest of sensory manipulations.
And now, the Telegraph is reporting that a computer program has been created that can edit videos of people speaking to realistically make it look like they said something else, raising fears of clips being tampered with online.
As a reminder, here’s an excerpt from the Stanford paper:
We present a novel approach for real-time facial reenactment of a monocular target video sequence (e.g., Youtube video). The source sequence is also a monocular video stream, captured live with a commodity webcam. Our goal is to animate the facial expressions of the target video by a source actor and re-render the manipulated output video in a photo-realistic fashion. To this end, we first address the under-constrained problem of facial identity recovery from monocular video by non-rigid model-based bundling. At run time, we track facial expressions of both source and target video using a dense photometric consistency measure. Reenactment is then achieved by fast and efficient deformation transfer between source and target. The mouth interior that best matches the re-targeted expression is retrieved from the target sequence and warped to produce an accurate fit. Finally, we convincingly re-render the synthesized target face on top of the corresponding video stream such that it seamlessly blends with the real-world illumination. We demonstrate our method in a live setup, where Youtube videos are reenacted in real time.
In simple English: famous "talking heads" can be practically anyone masquerading as said celebrity, and due to the real time conversion, they can talk, react, answer questions and generally emote so that the deception is flawless and totally convincing.
Now, researchers at the University of Washington have lip-synced a video of former US President Barack Obama using the program to superimpose new audio onto the clip. The realistic results put words in Obama's mouth by converting audio sounds into mouth movements and blending them onto an existing video of speech.
//player.ooyala.com/static/v4/stable/4.7.9/skin-plugin/iframe.html
Researchers say the software could be used in special effects, or to improve the quality of video calls.
"When you watch Skype or Google Hangouts, often the connection is stuttery and low-resolution and really unpleasant, but the audio is pretty good," said Steve Seitz, co-author of the research and professor at the University of Washington.
"So if you could use the audio to produce much higher-quality video, that would be terrific."
Until recently, video lip-syncing involved hours of filming and editing. But the computer program can create a clip with new audio after analyzing one hour of speech rather than 14. So striking is the real time effect of the conversion, the creators of this algorithm felt the need to clarify their intentions:
This demo video is purely research-focused and we would like to clarify the goals and intent of our work. Our aim is to demonstrate the capabilities of modern computer vision and graphics technology, and convey it in an approachable and fun way. We want to emphasize that computer-generated videos have been part of feature-film movies for over 30 years. Virtually every high-end movie production contains a significant percentage of synthetically-generated content (from Lord of the Rings to Benjamin Button). These results are hard to distinguish from reality and it often goes unnoticed that the content is not real. The novelty and contribution of our work is that we can edit pre-recorded videos in real-time on a commodity PC. Please also note that our efforts include the detection of edits in video footage in order to verify a clip’s authenticity. For additional information, we refer to our project website (see above). Hopefully, you enjoyed watching our video, and we hope to provide a positive takeaway.
Stanford expands upon its methodology in the video below: