Description of the test data :

• JFK is a "digitaly-remastered" collector version of the famous and controversial Oliver Stone film. The image is very clean, there's nearly no noise. The choosen scene include some flash-backs in black and white, where the camera is moving very fast, very nervously. Those parts lasts perhaps one minute, the rest of the scene is rather calm, features long travelings. Some flash-backs are slightly tinted to looks like old, past times.

• Pale Rider is an old (1985), classic and very good western, played and directed by Clint Eastwood. The image is noisy, grainy. There's probably too much contrast on this copy. The choosen scene includes (maybe 10s) a view of continuously projected water and rain, something libavcodec doesn't like very much. On the DVD it looks like a dense cloud of water with a very small but still visible rain. Other than that, the scene start in a dark room with very little action. Then the camera switch to another place, outdoor, with lots of contrast and some travelings.

• The Dead Zone is a rather unknown film, made by David Cronenberg from a Stephen King story. This DVD was very badly encoded, it looks like a good VHS but the image is very far from the quality of a standard DVD. For example sometimes you see very clearly waves of chroma components and overall the image is way too noisy. It comes in the 2-field, PAL-interlaced format, although I've deinterlaced it before running the tests (note: PAL isn't NTSC, deinterlacing is very easy, in fact mencoder does it automatically). The choosen scene starts slowly and then the central character walk through a demonstration, looking for someone. There's really a lot of people crossing the camera, moving everywhere and the scene is very complex. This will probably be the hardest to properly encode, out of the six in this test.

• Dancer In The Dark is the famous masterpiece of Lars Von Trier. This movie was shot with a handheld DV camera. This results in a continuously moving film, with a rather unique image grain. It simply doesn't look like any other film but feelings are hard to express, specialy in a foreign language :-) I choosed this DVD because the image might be more or less like the one you could have with a consumer DV cam.

• The Great Dictator is the ethernal Charles Chaplin's chef-d'oeuvre. Since the movie itself is quite old (1940) the image is obviously grainy and noisy but not that much. The difficulty is the 4:3 aspect ratio which means there's a lot of pixels to compress on each frame, if you don't want to reduce the resolution too much. Since it's a black and white movie, we won't have to encode chroma. The choosen scene doesn't have anything particular, despite being one of the most famous (the dictator Hinkle playing / dancing with the globe).

• Princess Mononoke / Mononoke Hime is the well-known anime from Hayao Miyazaki. The image itself is fine and there isn't anything particular to say. The choosen scene is a few minutes from the opening, where the demon attacks Ashitaka's village. The bitrate & resolution were choosed to backup this DVD on two CDs.

Here's the technical data for these scenes :

movie	resolution	target bitrate	bits per pixel	length
JFK	576 x 240	800 kbits/s	0.231	6576 frames, 4 minutes 23 seconds
Pale Rider	576 x 240	800 kbits/s	0.231	5656 frames, 3 minutes 46 seconds
Dead Zone	576 x 352	1100 kbits/s	0.217	4500 frames, 3 minutes
Dancer In The Dark	608 x 256	800 kbits/s	0.206	4163 frames, 2 minutes 46 seconds
The Great Dictator	576 x 432	1450 kbits/s	0.233	3401 frames, 2 minutes 16 seconds
Princess Mononoke	688 x 384	1300 kbits/s	0.220	4476 frames, 2 minutes 59 seconds

Methodology

Description of the fields :

• PSNR : peak signal to noise ratio, as given by the ve_lavc.c module from mplayer / mencoder. For "The Great Dictator", the PSNR displayed is obvisously the Y one, since Cb & Cr doesn't matter.
• FPS : this is the encoding speed, in frames per second. Since I'm doing 2 pass, this column display the result of (number of frames in the scene / (time to do the first pass + time to do the second pass) ). Note that I took the user time, not the clock time to try to avoid any noise from foreground processes on my machine.
• avQ : average quantizer.

• The individual Y, Cb and Cr PSNR values
• The average quantizer for I, P and B frames
• The quantizer graph for I, P and B frames
• The bitrate accuracy : I asked an X bitrate and libavcodec gave me a Y bitrate file. By how much Y is away from X ?

Hardware used

Some final notes

• Yes, I will include some tests with XViD sometimes in the future ;
• No, I don't have any VHS / TV source (ie a TV grabbing device) ;
• No, these tests aren't probably meaningful for low bitrates encodings (for streaming stamp-sized images)
  
• Yes, the whole thing is scripted (a big part in Java, a small part in shell), and I may publish the tools but I don't know when or even if I will ;
• No, I'm not (by far) a native english speaker ;-)
• Yes, if you find some spelling / grammar mistakes or poorly written parts, please drop me a line ;

Damn, give me the results !

1. vhq, v4mv
2. diamond size
3. ME comparison functions
4. bframes
5. bframes, comparison functions
6. bframes, diamond size, comparison functions
7. qpel
8. qpel, bframes
   
9. qpel, comparison functions
10. qpel, comparison functions, bframes
11. trellis quantization, comparison functions
12. bframes, trellis quantization, comparison functions
13. qpel, bframes, trellis quantization, comparison functions
    
14. luma & chroma single coefficient elimination
15. lumi & darkness masking
    
16. spacial & temporal complexity masking
17. ...
    

TODO list, in no particular order :

• add "qpel:precmp=N,cmp=N,subcmp=N-1" tests
  
• store all results (including per-frame statistics) in a database
• add low-bandwidth tests (400 kbits/s and 100 kbits/s)
• add fixed quantizer tests (testing file size instead of psnr or other quality indicators)
  
• add another anime scene from something else than P.M.
  (waiting for Ghost In The Shell)
• use graphics to display a large number of results on one page
• display (base compression PSNR - current PSNR) instead
  
• add xvid tests
• compute & display psnr variance instead of avQ
• publish the java controller source code
• design dynamic pages where the user would be able to select the desired options, sources & fields
• design a distributed test engine (my athlon is smoking)