It has been pointed out to me that these tests are probably invalid because On2 calculated PSNR in RGP rather than YUV. The rest of the page is still below, anyway.

PSNR comparison between libavcodec mpeg-4 and On2's VP6/WMV9 results

On2 Corporation recently released the VP6 codec, along with some sample videos and PSNR graphs comparing it to WMV9. I was curious as to how it compared to FFmpeg lavc mpeg-4. I did some quick tests and found that, at least with respect to PSNR, lavc performs much better. Unfortunately, I currently have no way to play the VP6-encoded samples, so I can't compare the actual visual quality.

Notes:

As I mentioned above, I cannot compare the actual visual quality; however, the graphs posted at On2 show only PSNR to demonstrate that VP6 performs better than WMV9, so that is the only dataset I can extend.

I couldn't find the source numbers used to create the graphs at On2, so I had to "eyeball" them to derive the values to plot on my own graphs. I probably made mistakes. Tell me: bugfood[dash]ml[at]fatooh[dot]org

Methods:

I used MEncoder, version "dev-CVS-031102-12:11-3.3.2" to encode. Here's a sample command for lavc-default:

mencoder tkal_reaching_raw.avi -ovc lavc -lavcopts vcodec=mpeg4:vratetol=100:\
vbitrate=400:vpass=1:psnr
(and then again with vpass=2)

lavc-mbd2_trell has mbd=2:trell added to the lavcopts.
In my experience, these two are rather safe always-a-benefit options for increasing quality (though, there is an encoding fps hit).

lavc-experimental has some options I wanted to include for testing, out of personal curiosity:
mbd=2:mv0:trell:cbp:precmp=2:cmp=2:subcmp=2:vmax_b_frames=1:predia=2:dia=2:preme=2:vme=5:naq:v4mv

I used 2-pass encoding because lavc always manages ratecontrol better on the second pass. The VP6 white paper(pdf) (on page 16) mentions that using 2-pass with WMV9 produces worse results, and with VP6 makes little difference, so I think it's still fair to compare 2-pass lavc to the other codecs at 1-pass.

These clips are very short, which is why I set vratetol=100. This makes lavc obey the requested bitrate more strictly, rather than the default tolerance of 8000 kbits.

Graphs:

All the source data used to create the graphs, and the graphs in postscript, are available here. Remember that the data for VP6 and WMV9 are estimated from On2's graphs.

"Mobile" Source 89MB
I encoded at 200, 300, 400, 500, and 600 kbit/sec. The ratecontrol has trouble below 300, but does well otherwise. The experimental set does very well here, but not in the other tests.

"Reaching" Source 111MB
I encoded with bitrate from 250 to 2000, at intervals of 250 kbit/sec. The experimental set gets weird above ~800, but the other two lavc encodes perform quite nicely.

"Fence" Source 50MB
This one is from 200 to 2300, at intervals of 300 kbit/sec. Again, the experimental set gets weird, so don't pay attention to it. The other two lavc sets really shine here too.

"Football" Source 290MB
I encoded bitrates from 400 to 1500, at intervals of 100 kbit/sec. The source is very difficult (688x480, 24bpp, 29.970 fps, and lots of high detail and fast motion), and lavc just can't really handle below 700kbit/s. In practice, encoding such a clip at these rates would result in unreasonably low quality anyway.