Hi Karen,
A lot will depend on the exact configuration you need, especially in terms of extras like storage, a rack, etc. Ignoring those extra costs right now, you could probably get 8 nodes (each 8 cores / 16 GB of RAM) with IB for under $40K, and that'd be my suggestion. If you're mainly running CAM and CCSM, you might also wish to consider the AMD processors if you're not already doing so - they're miserable on most serial codes, but for parallel applications they tend to be quite good, especially those like CAM and CCSM which need lots of bandwidth.
The above assumes 'cheap' SDR infiniband, which is still quite a bit better than gigabit ethernet. Take a look at:
http://www.clustermonkey.net//content/view/222/1/
I'll respectfully disagree with Mark's comment, since his own numbers on only 16 processors show IB scaling much better - at 16 processors (8 nodes), he was getting 3.55 years/day on IB and only 2.6 days/year on gigE. That's a difference of 1.365x for what should be (these days) a difference in price of only 1.1x if you can get the cheap SDR cards / network listed above. Also, bear in mind that Mark's data is using gigE with a 2:1 core to interconnect ratio whereas these days it'll more likely be 8:1 with dual-socket nodes, which will make gigE scale even worse. Also, the processors are faster these days, so for the same model, communication will occur more frequently... all these things say IB is probably better, in my opinion.
Finally, just bear in mind that it's close - you're pretty much at the cross-over point for your applications. If the difference were between buying 5 nodes with gigE or 4 nodes with IB, it's hard to say which would be best. Hope that's all clear, and if you aren't buying for another 5-6 weeks, I have some hardware coming in soon that'll have both IB and gigE links, and I can do an apples-to-apples comparison for you. Just let me know.
Cheers,
- Brian
(PS. Heh, sorry for the length of this!)
