Background

When I was preparing a set of parameters for Doom9 to use, I had a feeling that might be the case. At the time, though, I didn't know how to fix it other than by increasing vqcomp from the default value of 0.5 to 0.6 or 0.7. This seemed to help, though not very much. After hearing mixed opinions with regard to how effective vqcomp is, I ended up being conservative and choosing 0.6.

Since the competition, Michael Niedermayer has committed several improvements to lavc. vb_strategy=2, in particular, reduces the usage of B-frames in high-motion scenes, which gives a substantial improvement (the older vb_strategy=1 does this too, but with a net quality loss). My own small contribution of sc_factor (specifically, sc_factor=6) makes some high-motion scenes look a bit better by being more likely to use an I-frame. Still, none of these improvements make lavc look nearly as good as XviD in high-motion scenes. Rate control is still the main underlying factor.

Analysis

Take a look at these two pages. The first is for lavc, using the best mencoder command I have at this point (and the default ratecontrol equation).

for i in 1:turbo:vb_strategy=2 2 ; do
  mencoder matrix.vob -aid 128 -oac copy -vf \
  crop=718:356:0:60,scale=640:272 -sws 9 -ovc lavc -lavcopts \
  vcodec=mpeg4:vbitrate=581:psnr:vpass=$i:mbd=2:mv0:trell:cbp:\
precmp=2:cmp=2:subcmp=2:predia=2:dia=2:preme=2:vme=5:v4mv:\
last_pred=2:vqcomp=0.6:qpel:sc_factor=6:vmax_b_frames=2 \
  -ofps 24000/1001 -o lavc-default.avi
done

for i in 1:turbo 2 ; do
  mencoder ~/dumpstream/matrix.vob -aid 128 -oac copy -vf \
  crop=718:356:0:60,scale=640:272 -sws 9 -ovc xvid -xvidencopts \
  bitrate=581:psnr:qpel:nopacked:pass=$i -ofps 24000/1001 -o xvid.avi
done

• The statistics section has various numeric information for the entire file, broken down by type of frame.
• The big graph contains bitrate variation (indexed on the left vertical axis) and quantizer variation (indexed on the right vertical axis). Each data point used to generate the graph is an average of the 120 preceding frames, which reduces the spikiness and makes the graph more readable.
• The QP histogram shows the frequency of the various QPs used.
• The frame size histogram shows the frequency of various frame sizes, in bytes and split into 1000-byte ranges.

There are several notable differences between the lavc information and the XviD information.

• The XviD graph shows much more variation in bitrate, including several high spikes. Instead, the lavc graph shows much more variation in qp, including several high spikes where XviD used higher bitrate. This visually confirms what I had suspected before: XviD has more variable bitrate.
• Although XviD has more qp=6 and qp=7 frames, it doesn't use any QPs higher than 8 (no, the histogram didn't get truncated). Also, XviD has very few qp=2 frames.
• The frame size histograms have the same shape, but XviD has much higher numbers of frames at each extreme.

Failed Attempts

Increasing vqblur didn't have the desired effect; there wasn't much of a change, but some of the QP spikes on the graph got higher and others got lower. Visually, I could find a few places that had lower quality and none really looked better. Also, PSNR dropped from 43.03 to 43.02.
vqblur=0.6
vqblur=0.7
vqblur=0.8

Increasing vqcomp even more looks like it has the desired effect, but only according to the statistics. The bitrate spikes more, the quantizer spikes less, and the PSNR even increased from 43.03 to 43.13. Unfortunately I couldn't see a noticeable improvement in most high-motion scenes, and, at higher values of vqcomp, low-motion scenes did indeed look worse.
vqcomp=0.65
vqcomp=0.70
vqcomp=0.75
vqcomp=0.80
vqcomp=0.85
vqcomp=0.90

I started a vqmax test before I went to bed, and when I got up I found some surprisingly bad results. Even though less than 0.5% of the frames in the original encode had a qp higher than 9, removing those qps dropped the PSNR to 42.87 and made the visual quality much worse.
vqmax=9

Success

vrc_eq=(tex+100000000*mcVar)^qComp

Note that if you try to use that equation you may have to use quotes or escapes so the shell doesn't try to interpret the parenthesis. Also, that big number (100000000, or 10^8) isn't highly tuned, but I'm confident I like it better than the two surrounding orders of magnitude; smaller variations don't make much of a difference.

Although this new ratecontrol results in a qp/bitrate graph that still doesn't look much like the XviD one, the two encodes are actually quite similar looking. At this point, I cannot honestly say which encode is better. There are very slight variations, but they both look quite good in both high-motion scenes and low-motion scenes.

Also, this equation raises PSNR from 43.03 to 43.08.

Future