Deep Space Nine Project Update: Why MakeMKV-Derived Files Don’t Work

Note: There is a difference between having an MKV file with media content in it and specifically using MakeMKV to create a file. We are discussing the latter scenario; the DS9 Project Update’s default output is in MKV. Also, as a reminder, the DS9 Upscale Project is a fan effort, unofficial, and not affiliated with Paramount or ViacomCBS.

I returned to the question this weekend after several readers raised it to me in email. Instead of attempting another round of fruitless filter passes, I decided to take a MakeMKV file apart and examine the underlying frames. What I found will shock you explains a few things as to why MakeMKV source turns to hash no matter how you treat it.

Much is typically made of the fact that MakeMKV creates an exact duplicate of the frames inside a DVD VOB file. On the one hand, this is true. In addition to extracting all of the frames from a MakeMKV-created file, I also extracted the frames from a VOB file directly. The frame-by-frame output between a VOB file and a MakeMKV file is identical.

This is actually part of the problem.

To be clear: Part of the reason why you can’t use a MakeMKV file in this process without hosing your motion is that MakeMKV creates variable frame rate files and AviSynth is not designed to edit variable-length files. Thus, AviSynth forces MakeMKV-derived output into a non-standard 24.6 fps rate in an attempt to harmonize the two. This destroys any chance of synchronizing audio. But the problem goes a little deeper than that.

DVDs: What You See Isn’t What You’ve Got

There are aspects to this situation that I am still piecing together, so I apologize if I’m a little vague in places. The first point I want to make is that there is a difference between what you see in a DVD stream when you play the file back — even if you go frame by frame — and the actual frame sequences stored on disk.

All of the content written to an NTSC DVD intended for playback in a DVD player is “conceptually interlaced”, to borrow a term from HomeTheaterHiFi. To quote:

It’s important to understand at the outset that DVDs are designed for interlaced displays. There’s a persistent myth that DVDs are inherently progressive, and all a DVD player needs to do to display a progressive signal is to grab the progressive frames off the disc and show them. This is not exactly true…

DVDs are based on MPEG-2 encoding, which allows for either progressive or interlaced sequences. However, very few discs use progressive sequences, because the players are specifically designed for interlaced output. Interestingly, while the sequences (i.e., the films and videos) are seldom stored progressive, there’s nothing wrong with using individual progressive frames in an interlaced sequence…

The encoder can mix and match interlaced fields and progressive frames as long as each second of MPEG-2 data contains 60 fields, no more, no less (or 50 fields per second for PAL discs)… In short, the content on a DVD is interlaced conceptually. (Emphasis added)

Displaying perfect 23.976 fps video out of 29.97 fps video requires an inverse telecine. Telecine is the process of converting 24 fps film to be displayed on 29.97 fps video. One typical pattern of telecine in the NTSC universe is 3:2 pulldown, in which interlaced frames are created in a specific sequence to convert ~24 progressive frames into 60 fields a second for broadcasting frames. It is possible to unwind this type of content back into perfectly progressive frames if the DVD is mastered properly, which is where the idea that DVD footage is progressive comes from. DVD footage can display as fully progressive, but the standard assumes 59.94 fields per second, which we translate to 29.97 frames, which can be displayed at 23.976 fps progressive thanks to the use of flags telling the player which fields to show during playback.

Here’s where we get to the critical difference between using DVD Decrypter and DGIndex versus attempting to edit a MakeMKV-derived source file. When I dumped the contents of both types of sources to disk, I discovered a significant difference between them. The D2V-derived source contained the familiar 3:2 pattern we’d expect from progressive content that’s been put through telecine: Three progressive frames, followed by two interlaced frames. When you create a D2V file, it will specify 29.97 fps playback unless your content is 100 percent film (meaning a perfectly progressive stream of 23.976 fps data encoded interlaced in 29.97 fps fields).

The MakeMKV-created file contained a very different frame sequence. Instead of 3:2, it contained entirely progressive frames encoded in the following pattern: 1-2-3-4-4 (frame 4 repeats) or 1-2-3-4-5-5 (frame 5 repeats). Internally, the frame rate of the MakeMKV-encoded file remains 29.97 fps. MakeMKV-created footage only returns to the standard 3:2 format during scenes that were expected to play back at 29.97 fps. The frame-to-frame contents of the two video streams are different in the 23.976 fps progressive sections.

Here’s how I know that’s true: By default, the output from a D2V-derived file is 29.97 fps, unless you change it in some way. A MakeMKV file and a D2V-created file at 29.97 fps track each other frame for frame. That doesn’t mean frame output between the two is always identical, however. Here’s Frame 327 from a 29.97 fps D2V-derived file broken down into its constituent frames, versus a Frame 327 from a MakeMKV-derived source.

I can’t embed the slider, so I’ll also embed a screenshot below:

In the MakeMKV file, Frame 327 is a shot of Sisko. In the D2V-derived file, Frame 327 is an interlaced frame containing data from the frame before (the two admirals sitting) and the frame to come (Sisko talking). Sisko’s mouth position in Frame 328 of the MakeMKV source is different than his mouth position in Frame 328 of the D2V source. This continues through Frame 329, but the output synchronizes again at Frame 330.

TDecimate might be able to theoretically scan the video stream and remove the duplicate images, but AviSynth doesn’t understand what the frame rate of the MakeMKV-derived source is supposed to be in the first place. The constant 29.97 fps D2V file DGIndex creates, in contrast, presents AviSynth with a data stream in a format it understands and can work with. There are ways to use a timecodes file to work with VFR in AviSynth, but the point is moot as soon as you get to Topaz Video Enhance AI, which does not support VFR input.

The reason why MakeMKV files don’t work in this project is that MakeMKV files do not represent the video data contained within the VOB in a manner that AviSynth can work with. The progressive 1-2-3-4-4 and 1-2-3-4-5-5 pattern encoded into MakeMKV files plays back smoothly because the file stream contains a plethora of hints telling the player software how to play it and which fields/frames should be displayed. Turning that stream into something that other applications can process, however, is anything but easy.

I cannot speak to the exact reason why DGIndex can look at a VOB and create a D2V project file that outputs 3:2 pulldown, while the VOB and MakeMKV-created sources do not. But it’s this difference in output that causes problems. And given that Topaz doesn’t support VFR ingestion, the point is largely moot. The DVD set and DVD Decrypter are, as of this writing, both required for this process.

Continue reading

Time to Update: Google Patches 2 Severe Zero-Day Chrome Vulnerabilities

Unlike the last few zero-days, Google didn't find these security holes itself. Instead, it was tipped by anonymous third-parties, and the problems are severe enough that it hasn't released full details. Suffice it to say, you should stop putting off that update.

Samsung Starts Rolling Out Galaxy S20 Android 11 Update on Verizon

Not only does this include the Googley Android 11 enhancements, but it also has numerous Samsung-specific changes as part of the One UI 3.0 revamp.

Apple Urges Immediate iPhone Update to Block Active Online Hacks

There's a new version of Apple's iOS software for iPhone and iPad devices, and as usual, Apple is going to start pestering users to update. This time, the nagging for iOS 14.4 comes with a little more urgency.

Samsung Promises to Update Its Android Phones Even Longer Than Google

Smartphone updates have been a mess for as long as the modern smartphone has existed, but Samsung just took a big step in the right direction: The company has decided to extend security update support to a full four years.