So as to contribute what I can, I uploaded a few photos and annotated them as best I could in a gallery I call "A Photographic Tour of Gordon and Trestles." Here are the thumbnails:
For those who may not be familiar, SDSC is the home to two "national systems," or supercomputers which are available for anyone in the U.S. to use through the National Science Foundation's XSEDE program. These two systems were both integrated by Cray (formerly Appro) and are:
- Trestles, a 324-node, 10,363-core, 100 TF, AMD Opteron (Magny Cours) cluster outfitted with a fat tree QDR InfiniBand interconnect designed to accommodate the computational needs of the common man. This machine was deployed in 2011.
- Gordon, a 1024-node, 16,384-core, 341 TF, Intel Xeon (Sandy Bridge) cluster outfitted with two (dual-rail) hybrid 3D torus QDR InfiniBand fabric designed to tackle data-intensive problems with an architecture very rich in IO capabilities.
Most of the gallery details Gordon, since it has a very unique architecture based around a relatively complex basic building block:
Sixteen compute nodes in an 8U subrack share a common 36-port QDR IB switch, and an IO node with sixteen SSDs also hangs off that switch. The IO node is connected directly to Lustre via a 2×10Gbe bonded link, and it also provides the SSDs to the compute nodes over InfiniBand. This entire building block represents one torus node on Gordon's overall 4×4×4 torus, and since the whole fabric is really dual-rail, each node (compute and IO) is connected to two of these 36-port switches.
This all makes for a very pretty machine.