Networking

Hosts in my cluster all have two 10GbE links to a switch bonded with LACP. That gives them an effective link of 20GbE normally. But really, even 5Gbps would be enough for most hosts. The kernel in most boxes combined with drivers and such isn't going to be able to push 20GbE, much less 40GbE. So faster isn't worth it. 

Note: Don't bother with IPoIB. You're better off with 10GbE x2 via LACP. Although if it's a choice between 40Gbps IB and IPoIB or a couple of 1GbE lines, go IPoIB. 

The network side of my rack redo is almost done. Once the two SRX650s on top are removed it'll be good to go. 

Gear

• 4x SRX650s w/ 10GbE
• 2x RackSwitch G8264
• 2x RackSwitch G8000 w/ 10GbE
• 1x Digi console serve
• 1x Avocent KVM
• 2x Voltaire Infiniband 4036 - 4xQDR (aka 40Gbps)

IB Network

The IB network will be used for Lustre.