The last two days I was at a network performance lab in Stralsund, Germany. We were testing dual Opteron 250 (2,4GHz) machines with e1000 cards and Linux.
One of the interesting results was that ip_conntrack [again] scales better as the load generators. The generators couldn't establish more than 25,000 new TCP connections per second and no more than 1 million total concurrent connections ;)
Thus I'm now pretty much convinced that ip_conntrack scales quite reasonable, and we should concentrate optimizations to other areas of netfilter/iptables.