Google Neural Network Can Isolate Individual Voices in Videos

The bleeding edge of computer science these days is all about making computers more like humans. We’re using neural networks to help machines recognize objects, play games, and even speak in a more realistic way. In a new feat of machine learning magic, Google Research has developed a system that can replicate the “cocktail party effect,” where your brain focuses on a single audio source in a crowded room. The results are impressive — almost worryingly so.

Google calls this technique “Looking to Listen” because it watches videos with multiple speakers to split up the audio—it uses both auditory and visual signals, just like your brain does. There’s nothing special about these videos, either. They’re just videos with a single audio track consisting of more than one person.

To build a tool capable of this, Google started with 100,000 samples of high-quality lectures and talks from YouTube. Engineers chopped up the videos to get segments of clean speech with clearly visible speakers and no background noise. That left Google Research with 2,000 hours of video consisting of a single person speaking (they call this the AVSpeech data set). The trick was using these clean samples to create “fake” cocktail parties. The researchers combined the videos, so multiple people were speaking. That’s the data Google used to train its neural network.

Like many other Google Research breakthroughs, this one used a convolutional neural network. The input to the network consists of visual features of the speakers as well as the spectrogram of the video’s soundtrack. By processing the video, the network learns how to separate the video into a “time frequency mask” for each speaker. The output mask is matched up with the audio input spectrogram to generate separate audio tracks.

With the training done, Google unleashed the network on new videos. As you can see in Google’s examples, this works surprisingly well. The Looking to Listen model can identify what audio is coming from a speaker, and filter out everything else. This technology could have applications in video conferencing, hearing aids, and video surveillance.

On that last point, this technology could be so powerful that it’s not hard to imagine scenarios where it’s abused. With future speed and accuracy improvements, an observer could pick out your voice on a crowded street to find out what you said. There’s no indication Google has any intention of doing that, but it’s not alone in doing neural network research.

Continue reading

NASA Releases Incredible Perseverance Rover Landing Video

NASA's Perseverance rover has been on the surface of Mars for several days, giving the team here on Earth time to run system checks and download preliminary data from the robot. The agency has now released the first large batch of media from the mission, including hundreds of still images and the first video and audio ever recorded on Mars.

Will Microsoft Mesh Replace Video Calls?

In the latest attempt to jumpstart the somewhat moribund market for a VR future, Microsoft has un-veiled an ambitious new mixed-reality platform, Microsoft Mesh. Running on its Azure services platform, it initially supports the company's own Hololens devices, and its Altspace VR application, but the plan is to make it work with other VR, AR, and MR devices as well.

Boutique PC Builder Launches ‘No GPU’ Boxes to Cope With Video Card Shortage

At least one boutique vendor is now shipping gaming PCs without GPUs as a way of moving hardware in the midst of the ongoing shortage.

YouTube Designed Its Own Video Transcoding Hardware

YouTube has developed its own video transcoding chips to break bottlenecks imposed by the slowing of Moore's Law.