Just some feedback on the need for IB interconnects in certain clustering applications.

http://www.theregister.co.uk/2009/06/22/mellanox_infiniband_switches/pag...

"While generalizing a bit, Monson says that the increase in speed and bandwidth of the InfiniBand protocol has not only allowed parallel supercomputer clusters to scale out, it has also allowed them to run more efficiently at the processor level. Monson put together some numbers from the past couple of Top 500 supercomputer rankings, based on the Linpack Fortran benchmark tests that are used to rank the machines, which suggested that the typical cluster using Gigabit Ethernet was able to run at about 52 per cent efficiency. (That percentage is the result of dividing the maximum performance of a machine, in gigaflops, on the Linpack test by its peak theoretical floating point performance.)

For machines linked by 10 Gigabit Ethernet switches, the efficiency rose to 61 percent. But machines lashed together using 20 Gb/sec InfiniBand are hitting 74 per cent efficiency, and Mellanox will demonstrate this week on some new machines added to the June 2009 Top 500 list that comes out on Tuesday that it can drive 92 per cent efficiency with 40 Gb/sec InfiniBand. The choice of protocol and bandwidth certainly does make a difference at the system level (as you would expect)."