That won't work. You need a socket 478 CPU, not the LGA775 socket. The GX260 needs a CPU with a FSB of 400MHz or 533MHz. 800MHz won't work. I also don't think Prescott CPUs will work.
The best CPU you can get willbe a socket478 Pentium 4, 3.06GHz, 512k L2 cache, 533MHz FSB. Since it's the best CPU for the 533FSB systems, it's likely to be rare, expensive, or both. If that becomes a problem, look for CPUs of the same specs but at slower GHz speeds.
As per the graphics card, the installed memory has little to do with how good it is. It's a common misconception people make to think that. What matters is just about everything else, namely what GPU it has. What card(s) are you thinking about? For the GX260, it needs to be AGP.
It depends on where you're shopping on how much that will get you. If you're going new, Newegg.com has the 7600GS hovering just above that price point, but the 7600GT is a fair bit beyond that. The 7600GS falls between the 6600GT and 6800 cards in terms of performance. Since you linked to a CPU on eBay, if you search there, you may be able to find them cheaper, but without the warranties and all that.
I'd recommend a 7600GT, a 6800 GS/GT (avoid the XT and LE versions of the 6800), a 7600GS, or a 6600GT, in that order from best to worst. I got my 6800GS on eBay a few months back for about $100 (not including shipping). The ATi equivalents of these cards (X1600, X1650, etc.) have been reported to have trouble working in alot of the older Dell computers, let alone they're a bit worse in performance than these listed nVidia cards, so I'd avoid those.
No, but the error is not on your part. The listing says AGP, but by looking at the connector in that picture, I notice that's actually a PCI Express card. Here's a page that shows some different interfaces.