9 Replies Latest reply: Feb 25, 2012 8:04 AM by jfaughnan
jfaughnan Level 3 Level 3 (780 points)

I'm following up on a 2008 discussion on archival video formats:

 

https://discussions.apple.com/message/6840163#6840163

 

That discussion contrasted ProRes to DV. I'm still in learning mode, but my understanding is ...

 

The ProRes family is Apple FCP specific - no other software can read or write it.

ProRes is lossless, DV is lossy. There's some expansion going from DV to ProRes (amount?).

Because DV is lossy, a DV project produced from a DV original will lose some quality. A ProRes file created from a ProRes original would not have quality loss.

DV is a label for a set of formats that have evolved since the 1990s. So a DV file from 2000 is not necessarily the same as a DV file from 2010. As media evolves over time software may stop rendering early generation DV correctly even as it renders late generation.

The Library of Congress doesn't consider any archival video format acceptable, but considers MJPEG the best of the bad (but there are probably audio sync issues).

 

I wonder if, given the state of video formats (volatile and patent laden) there's something to be said for using a minimally compressed h264 video format as an 'archival' format. In other words, take a set of DV media of different generations, and norm it to a h264/mpeg video with minimal compression (25+MB/sec data, mininal frame loss, etc). A 30GB DV file, for example, might become a 25GB h264 file. There'd be some quality loss, but at least everything would be normed to a 2012 baseline. In ten years that video would then be (probably) convertible to a lossless archival format assuming 1000 TB drives and some resolution of the IP issues.

 

Any thoughts from the experts? Is there any guidance on compression settings to produce "archival" h264?


i5 iMac 27, iPhone OS 3.1.2, MacBook Core-2 Duo, G5 iMac, G3 iBook, Mac mini
  • 1. Re: Archival video formats 2012
    David Harbsmeier Level 7 Level 7 (29,650 points)

    I have to strongly disagree with some of your statements:

     

    >Because DV is lossy, a DV project produced from a DV original will lose some quality

    Unless you do something to actually degrade the image during the edit, it's pretty much the same coming out of an NLE as it was going in.  While DV is compressed at about a 5:1 ratio and uses a 4:1:1 color sampling scheme, it is somewhat lossy as an acquisition format, but once recorded, it's all 1s and 0s.  Keep in mind that the ProRess formats are also compressed ... almost all video formats are.

     

    >DV is a label for a set of formats that have evolved since the 1990s. So a DV file from 2000 is not necessarily the same as a DV file from 2010. As media evolves over time software may stop rendering early generation DV correctly even as it renders late generation.

    The term "DV" essentially stands for the DV25 codec family.  That includes DV, DVCam, DVCPro and Digital8.  But the actual codecs don't "evolve" as you suggest.  A DVCam recording from 2000 uses the exact same codec as a DVcam recording from 2012.  Generally speaking, codecs don't evolve once released ... however, variations of certain codecs are developed and released continually.  Those variations don't necessarily replace the old ones; they're just used in newer equipment and have a different (although sometimes similar) name.  For example, H.264 is an MPEG-4 variant ... MPEG-4, part 10 as I recall and is also one of the AVC family of formats.

     

    >there's something to be said for using a minimally compressed h264 video format as an 'archival' format. In other words, take a set of DV media of different generations, and norm it to a h264/mpeg video with minimal compression (25+MB/sec data, mininal frame loss, etc). A 30GB DV file, for example, might become a 25GB h264 file. There'd be some quality loss, but at least everything would be normed to a 2012 baseline.

    H.264 is much more highly compressed format than DV.  Quality and quality loss isn't just about file size or even data rate.  It's more about color sampling and frames.  Most MPEG formats delete a lot of pixel information from non-I frames and that's where the file size savings comes into play.

     

    Bottom line, I would much rather have a DV original for archiving than a compressed version, be it H.264 or MJPEG.  Every conversion will take its hit on quality so the fewer times you convert, the better off you are.  What may come into play though is the playback availability of older formats.  If DV VTRs suddenly all stopped working, a library of DV tapes wouldn't do you any good at all.  Remember the old 5.5" floppy discs?  There's a ton of them out there, all with perfectly good data on them, but few people still have the ability to access that data.

     

    -DH

  • 2. Re: Archival video formats 2012
    Shane Ross Level 8 Level 8 (41,895 points)

    Have DV tapes?  Keep the tapes.  That is what you archive.

     

    Converting to anything else will cause quality loss.  Converting to H.264 will be SEVERE quality loss.

  • 3. Re: Archival video formats 2012
    BoBo1 Level 1 Level 1 (10 points)

    I have been keeping my original mini DV tapes produced on a Canon 1080i Vixia HV30 for the last few years.

     

    My (archival) concern is being able to playback these tapes in the future with this camera, .....if for some reason I accidentally break the camera or the loading mechanism fails, .......and Canon or anyone else no longer manufactures these mini DV tape cameras.

     

    I have a friend who kept his Hi8 tapes but can no longer find a camcorder to play them back.

     

    I always thought capturing video from my HV30 camera in a ProRes 422 format and storing as a QT file on an external hard drive, along with retaining the original mini DV tape would be the best approach.  I agree that H.264 is NOT the way to retain original quality.

     

    Sending them to the "Cloud" I suppose is another form of backup.

     

    BoBo

  • 4. Re: Archival video formats 2012
    Shane Ross Level 8 Level 8 (41,895 points)

    Ahhh...HDV.  You didn't say that.  HDV is an entirely different matter than DV.  Completely different formats that use the same tape.

     

    HDV, captured with FCP, is in a format that only computers with FCP can read.  Give the footage to someone else who doesn't have HDV...and they cannot view the files.  That's bad.  ProRes, on the other hand, has a decoder built into QT7...so any computer, PCs too, with QT7, can view ProRes.  ProRes is a great archival format.

     

    But sending ProRes to the cloud isn't practical.  LARGE file sizes.

  • 5. Re: Archival video formats 2012
    jfaughnan Level 3 Level 3 (780 points)

    Thanks David, this is helpful.

     

    I did a little experiment with interesting results. I selected a 5 min .DV clip then exported it two ways -- as a .DV and as .MP4.

     

    The quality, on playback, was almost identical. The only way I could tell the .mp4 from the .DV was because the aspect ratio was different. Both were slightly inferior to the original .DV file

     

    How could this be? In my experiment I set the .mp4 data rate to 50Mbps (DV is 25 Mpbs) and the key frame to 2! In other words, almost pathologic settings. The .DV file was 845MB, the .mp4 was 886MB.

     

    It @ 6 minutes to produce the .DV export and about @20 to produce the .mp4.

     

    What I see (others may disagree) is that .DV is a lossy form of compression. So producing a .DV export from a .DV clip causes a small amount of quality loss. .MP4 is lossy too, but not necessarily more lossy. It's simply that we usually configure .MP4 for compression, not for quality.

     

    it's a familiar story with JPEG -- so-called 99% JPEG export is half the size of RAW but the line resolution differences are very modest.

     

    On the other hand you've reassured me that the DV25 files from my 2007 camcorder are likely to be readable for at least a decade, at which point I'll convert them into a lossless format (hopefully, by then we have an archival standard!)

     

    For the records, if you convert a .DV to ProRes, how much does the file grow?

  • 6. Re: Archival video formats 2012
    jfaughnan Level 3 Level 3 (780 points)

    Ooops. That should read 846MB for .DV, 556MB for .MP4. The .MP4 was smaller, but of identical quality. Again, aiming for archival function, not compression. The .DV export was slightly inferior to the .DV original - as expected for a lossy format.

     

    The point is that I think you can create an .MP4 export of equal or better quality to a .DV export. The .MP4 will be slightly smaller, and may be more standard/archival. It will take considerably longer to produce.

  • 7. Re: Archival video formats 2012
    David Harbsmeier Level 7 Level 7 (29,650 points)

    Keep in mind that .dv is a DV stream; video multiplexed with audio.  It is somewhat different than a QuickTime DV file.  .DV files are used natively by iMovie, while FCP uses QuickTime DV natively.

     

    -DH

  • 8. Re: Archival video formats 2012
    Shane Ross Level 8 Level 8 (41,895 points)

    OK...do what you want. Ignore what we say and do what you think is best.  We obviously have no clue what we are talking about.

     

    You shot HDV.  Converting that to .DV will cause loss of quality.  A LOT.  Because you are taking HD footage, and capturing it as SD.  And .DV is DV Stream...which is slighly lossy as opposed to Quicktime  (.mov) using the DV/NTSC capture option from FCP.  .DV is what you get when you capture from iMovie.  And iMovie captures ABSOLUTELY cause loss of quality.  .DV...Apple Intermediate Codec that it captures HDV as...lossy. 

     

    .MP4 is a delivery codec, not an archival one.  Archival means that you someday plan on using that footage to edit again.  .MP4 is not an editable codec, and if you convert to it, you are doing a lossy conversion. You will lose quality. You can't see it?  Fine...but quality has been lost. So wanting to make it an archival format means that you would need to convert the footage again. BAMMO...another level of compression!  More loss. 

     

    >For the records, if you convert a .DV to ProRes, how much does the file grow?

     

    Yes.  Because it is less compressed.  The smaller the file size, the more compression is done to the file, the more quality is lost. 

  • 9. Re: Archival video formats 2012
    jfaughnan Level 3 Level 3 (780 points)

    Shane, Another source sent me a discussion on h.264 vs. MPEG-2 encoding for digital archiving of home video. The discussion told me how deep these waters are, though I knew that. It sounds like at least some video geeks feel that MPEG-4/h.264 is a competitive archival format for old-school analog-to-dv or dv output ...

     

    http://forum.doom9.org/showthread.php?t=158407

     

    Here is a synthesis I made for a future blog post I'm working on. I made some minimal edits ...

     

    • [asking if] MPEG-2 is better suited for encoding DV video then H.264 since MPEG-2 can encode interlaced content without de-interlacing or other motion estimation enhancements.
    • ... deinterlace it with tgmc and encode it progressively. The deinterlacing will be far better than what any television could do...
    • ... I am looking for the best [picture quality]. So whatever would give me that for my highly interlaced home videos, whether that be MPEG-2, de-interlaced H.264 encode, or interlaced H.264 encode. If H.264 could offer very comparable quality to MPEG-2, yet save on file space, then that would be a benefit also. However, I am hearing that H.264 encoded with interlacing may be similar in file size to MPEG-2 anyway...  If I did [deinterlace to get] comparable PQ in H.264 to that of MPEG-2 interlaced, then I would go with the slower but more quality: yadmifmod()+nnedi() or mcbob()+nnedi() de-interlace filters for AVIsynth...
    • ...  the encoder is not the "limiting" step for "quality" here, "Quality" is bound by "whatever" is doing the "deinterlacing" ...
      If your DVDs are played back on a 1080p HDMI upscaling DVD player, then it is doing the deinterlacing (on the fly) ... if a standard DVD player is feeding a HDTV panel via RCA, then the HDTV is doing the deinterlacing (on the fly) ... a PC with 2-4 cores of dedicated computing power running at 5 FPS can do a much better job at deinterlacing than a weak single core APU in a HDTV/DVD player that is forced to do the conversion on-the-fly (at ~ 30 FPS) ...
      So while MPEG-2 might reproduce the interlacing better than H.264 (which I actually doubt anyway), the device responsible for deinterlacing will just trash the detail anyway (so why not burn your CPU power doing the best deinterlacing job that you can and encode to H.264, the most efficient format on the planet, while burning CPU cycles anyway?!?)
    • For older tapes ... filter chain ...
      • Video Denoise (default)
      • Dynamic Noise Reduction (7)
      • hue/saturation (1, .095, 1)
      • Smart Smoother HQ (3, 40, 160, 0)
      • brightness/contrast (+7%, 102%)
      • Flaxen VHS (default; horizotal +6)
      • Null Transform (cut off black and noisy edges)
      • MSharpen (80, 20)Sharpen (8)
    • For archival purposes, I'd make very high-quality encodes without VBV limits. "Constant-quality" encoding at CRF=12 or less would be good, considering the HDD prices. Then you can make multiple lower-quality encodes for different devices by using either the high quality files or some lossless intermediate file to avoid filtering again:
      1. source -> filtered lossless encode -> high-quality x264 encode to archive
      2. lossless encode -> Blu-ray encode
      3. lossless encode -> web encode
    • ... The big deal with deinterlacing before, and encoding at 60p is that you can use OMG_AWESOME deinterlacers like QTGMC. This adds to the encoding time, but overall picture quality will be better. Try encoding interlaced, and letting your GPU deinterlace on playback to 60p. If you're ok with its hardware deinterlacing, then just do that. If you prefer the QTGMC look enough to spend the time, then do that...

     

    Their conclusion was that h.264 was a better archival choice even considering interlacing when playback will be on modern displays, but of course these are high end encoding discussons; I'm not sure what QT is capable of.

     

    As you stated, the ultimate archive is the tapes -- some of which have relatively long lifespans (so far my 15 yo analog tapes don't seem too bad). So I certainly will keep those too.

     

    Practically speaking, for now I'm staying with .DV anyway. That's what's streaming from my capture source. I'm deferring any further archival processing for a few years. Maybe in a few years there will be better technology that will take the video output from analog sources and encode it in a lossless format that will make good use of our Petabyte drives to come. Who knows, maybe 10 years from now my analog tapes will still be readable, and 2022 video post-processing will give better results than the originals even allowing for aging effects.