In addition to the difference in gaming performance, there's certainly a perceived speed difference because of Aero Glass vs. the much simpler UI in XP.

 

Personally, I've found the extras in Vista (system search, file stacking in Windows Explorer, etc.) to be well worth any reduction in speed. And, I really don't notice that difference unless I happen to go straight from XP to Vista, which is to say, Vista is plenty fast enough on all of my (albeit fairly recent) systems.