Robert Olsson is doing very insightful high-performance networking research on Linux-based machines for many years. Little people know his huge collection of ASCII-snippets at http://robur.slu.se/Linux/net-development/experiments/. It's a real pity that he's basically doing all this research in his spare time, being a systems administrator at university. Intel and others should actually look at that and fund his invaluable research!
Recently he achieved 2.1Mpps aggregated packet forwarding rate over four Gigabit Ethernet ports using a Dual 2.4GHz Opteron 250 machine with a specially optimized NAPI driver patch.
Another interesting graph (almost one year old) compares the memory latency on Xeon vs. Opteron. Looking at the results, you will understand that really want to get Opteron CPU's with integrated memory controller if you care about network forwarding performance :)
Please note that this number is under very synthetic conditions only. This is single-flow UDP performance, so any routing cache misses / fib lookups are not yet in the picture. Also, due to the stupid nature of _all_ Ethernet cards, we have to do IRQ affinity and thus only achieve highest performance on the two interface pairs that are bound to the same IRQ.