Hands On With Adobe’s New Content-Aware Tools for Video, Other NAB Updates

The annual NAB (National Association of Broadcasters) event is traditionally a showcase for new high-end video and audio editing features, and this year is no exception. Adobe has rolled out a host of new features and performance tweaks to its Creative Suite tools — particularly Premiere Pro, After Effects, Audition, and Character Animator. Not to be left out Cyberlink is also rolling out updates to PhotoDirector and PowerDirector for subscribers to its 365 Suite. For those who still think of video production as a niche market, Adobe projects that there will be nearly 605 billion minutes of video created every week by 2021. Of course, as anyone on YouTube knows, most of that will unfortunately never be edited, but it still leaves a huge amount that will be.

Content-Aware Fill for Video

The ability to plausibly fill in erased areas by using content-aware tools has been one of the most-impressive new photo editing technologies of the last decade. Now, Adobe has brought that same capability to video. Intuitively, that is a harder problem and certainly requires more processing power. However, video does have one natural advantage. Since there are typically many frames of the same scene, if you are trying to remove an offending object that is moving across the scene, the application can look at a variety of frames to see what the occluded background should be in other frames. Of course, that still means there is a need for plenty of advanced algorithms to get the color blending and lighting right.

There isn’t a lot of documentation out yet on how to use the new Content-Aware Fill in After Effects, but basically, it takes a few steps. First, you create a mask around the element you want to remove and then Track it as the object you want to remove moves, making sure the mask moves to follow the object. If the Mask doesn’t need to change shape, then AE’s automatic mask tracking works pretty well. Finally, you can choose to Content-Aware Fill using one of three modes. Object is the most sophisticated, and attempts to isolate and replace an object within the mask. Edge Blending gives you the sort of blur you see when faces or license plates are obscured in videos. Surface is the simplest and is designed to fill from solid surfaces under the section being replaced (like removing logos from the back of a laptop). The only other setting is choosing whether you want to fill the entire comp or just the work area.

Once I got the hang of the workflow, the process was pretty straightforward. It does lean much more on the CPU than the GPU, which is a little surprising given how good GPUs are at most of the needed tasks, so it can take a while. The fill is accomplished with an overlay of PNG files, so it can also consume quite a bit of disk space if you have a high-resolution video or are filling a long clip. As a simple example, here is some 4K drone footage I edited to remove our tracking vehicle for part of the shot (from about 7 seconds in to 18 seconds):

Here is a screenshot of what that looks like in After Effects:

It did take me a little while to learn the limitations of Content-Aware Fill. I was a little over-optimistic in my first attempts to use the feature, hoping it would work magic. It’s powerful, but not magic. Below is a quick screen recording of a few seconds of a clip where I masked out a woman walking along a path and then used the CAF Object setting to replace her. You can see that the Mask Tracking starts to fail, making her left arm visible after she starts swinging it. That’s not surprising and follows the documented behavior of mask tracking, but means that there are limits to what you can remove automatically. Of course, I could have manually adjusted the mask as the clip went along. More surprisingly, AE puts foliage on the path it fills, rather than analyzing other frames and realizing that the path is solid dirt. However, the overall effect is certainly impressive given that it takes almost no time to do the editing:

Finally, a Project Panel for Storyboarding in Premiere Pro

If you’re just assembling a few clips, the traditional grid or lists layouts for flipping through media clips is fine, but for larger projects, they don’t allow for much organization. Now Premiere Pro has added a Freeform option for arranging your assets. Even as an individual user, this should be a helpful tool for organizing story ideas prior to assembly. For teams, there will be additional value, as collaboration will be a lot easier with the visual cues provided by the custom layout of assets.

Less exciting, but also useful, Adobe has added the Rulers and Guides familiar to those who use some of its other tools to Premiere Pro. That should certainly make it easier to have consistent title and graphic placement throughout a project.

Auto Ducking and Punch and Roll, in Audition and Premiere Pro

Adobe’s apps have always allowed some automated audio leveling, but they are now extended to provide automated ducking — lowering the volume level — of ambient noise tracks when there is spoken dialog overlaid. The capability can be tuned with keyframes and should be a decent timesaver for those who are creating projects with complex soundtracks.

Those editing narrations will also be pleased to learn that Adobe has added direct support for Punch and Roll to Audition. That should make it much easier to patch in corrections to glitches and errors in the original recording.

Performance Improvements

Cyberlink Upgrades Its 365 Applications

Cyberlink has continued to improve on its easy-to-use video assembly interface with dual preview windows and enhanced support in PowerDirector 365. It’s also released a variety of additional add-on content for download by subscribers and improved support for multiple displays. Laying out elements is also easier with the ability to snap objects to guides or align with other elements. Cyberlink’s image editing application, PhotoDirector 365, also receives an upgrade to the company’s FaceMe facial recognition engine, for faster and better image management.

Pricing and Availability

Updated Creative Suite applications with the new features are available for download now, as part of existing Creative Cloud subscriptions. Unfortunately, there wasn’t any word on when we’d see the impressive capabilities of Premiere Rush brought to Android, one of the biggies I’m waiting for.

Image Credit: [David Cardinal]