Security Suites Tested: AV-Test vs AV-Comparatives

I sometimes forget what has been reported here; if this is old news, I apologize.

I'm not a particular fan of  commercial single-vendor security suites, but polls do suggest they are the commonest security solution chosen by most users, hence this post FYI for those looking at suites.

In the last few weeks 2 independent and respected testing organizations have released results from tests of these suites conducted over the summer of 2010:

AV-Test:  http://www.av-test.org/certifications.php
AV-Comparatives:  http://www.av-comparatives.org/images/stories/test/dyn/wholedynamic2010.pdf

They are similar in that they evaluate the protection afforded by the major commercial security suites available, using the latest versions available at the time, over roughly the same time period. They both use XP/sp3 as their test platform. Both include dynamic tests which mimic real life protection of the entire product (as opposed to the static tests of "on-demand " scans only of the stand-alone anti-virus, which is yesterday's testing standard). And both are ongoing projects over time.

Where they differ significantly is in their resultant rankings. Not surprising, given their different methodologies.

When reviewing their results, consider that:
1) AV-Comparative seems to only test the pertcentage of malicious URLs and malware succesfully blocked, but they do use a much larger number of test cases to date (>1,000).
2) AV-Test evaluated not only protection, but also removal of malware. The do however use a far fewer number of malware test samples. They also evaluate product "usability", wich includes the impact of the suites on computer slowdown, and the frequency of false positive  scan detections and false warnings during legitimate program installations.
3) AV-Test does give good individual product summaries in pdf format explaining their scoring result.
4) These tests are of the whole suite only, and do not apply to the stand-alone anti-viruses from the same vendor. And particularly not to any free AV versions from these vendors (the exception being Microsoft Security Essentials, which was only reported by AV-Test).
5) Neither report considers the "grief" factor that I rate importantly: the bugs and glitches that cause one to curse one's security  program (update errors, alerts, conflicts, system-crashing false positives, piggybacked and unwanted software, ease of uninstallation, etc).

I draw no conclusions here, but found both reports interesting.