NAK Technical Information

NAK is a cluster supercomputer built primarily as a technology demonstration and experimental evaluation platform for the concepts surrounding the use of hosts as nothing but power and ground for GPU-resident computations. However, the technologies used for the system span a much wider range.

NAK's GPU Software Environment

More about this will appear here shortly. For now, think about MOG (MIMD On GPU) and the other GPU software technologies discussed at our SC09 Research Exhibit.

NAK's Network Design

NAK's network is an FNN (Flat Neighborhood Network). Although FNNs seem new and strange to most people, we've been using them ever since we invented the concept and supporting technologies a decade ago. FNNs give single-hop latency and excellent bisection bandwidth by using a clever topology. In NAK's case, the design is simply:

 0:  0  1  8  9 11 13 16 17 20 21 24 25 26 28 29 32 36 37 44 45 48 49 52 53
 1:  2  3  4  5  6  8  9 10 11 12 14 18 20 24 25 26 27 28 29 30 31 34 35 46
 2:  3  6  7  8 10 11 12 14 15 19 22 23 25 26 28 30 35 38 39 43 50 51 54 55
 3:  8 11 18 20 24 25 26 27 28 33 40 41 42 43 47 56 57 58 59 60 61 62 63
 4:  1  7 17 18 19 22 27 31 32 33 37 39 40 44 45 48 49 50 51 52 53 54 55
 5:  7  9 19 22 29 34 39 41 42 46 47 50 51 54 55 56 57 58 59 60 61 62 63
 6:  9 13 14 15 16 17 21 23 29 30 31 33 34 38 40 41 42 43 45 46 47 48 53
 7:  0  2  4  5  7 13 15 16 18 19 20 21 22 23 24 27 36 38 39 50 51 54 55
 8:  2  3  4  5  6 10 12 13 16 17 21 35 45 48 53 56 57 58 59 60 61 62 63
 9:  0  1  2  3  4  5  6 10 12 32 33 34 35 36 37 40 41 42 43 44 47 49 52
10:  0  1 14 15 23 30 31 32 36 37 38 44 46 49 52 56 57 58 59 60 61 62 63

The lines starting with a number followed by ":" specify the actual wiring pattern; the first number is the switch number, the remaining numbers on each line are the node numbers connected to that switch. Notice that we are using 24-port switches; 8 of them have a port free for an uplink to the outside world.

Another way to look at that wiring pattern is the actual wiring:

Scary, eh? Well, it's not really that bad. If you look closely, you can see the wiring color code tabs stuck on the nodes next to the PCI card slots. Notice that, as on KASY0, we made more cable colors by adding a stripe to some (i.e., green and black). In total, the labels look like:

The wiring is pretty random, literally evolved using a GA (Genetic Algorithm), but physically wiring the system is just a matter of connecting NICs and switches of the same color. It took three of us (one partially colorblind) a total of about 4 hours on Friday, April 23, 2010 to wire the network. One trick to speed it up was that we wired a color at a time. However, it would have been a much faster process if we didn't have all the switches clumped together in the center of the cluster... that makes wiring density too high.

So, what's the point of this network design?

It cost nothing, because it is built using a subset of KASY0's 100Mb/s network hardware; even using original purchase prices, it's only about $2,700 for all the switches, NICs, and cables
It has an effective bisection bandwidth approaching 25.6Gb/s with a node-to-node worst-case latency of about 30us
The network design gives at least 100Mb/s (in each direction) conflict-free bandwidth between each node pair, with additional pairwise bandwidth between nodes whose numbers differ by a single bit or by +/-1
Four 100Mb/s NICs give 800Mb/s (total in both directions) per node, essentially saturating the PCI bus into which both the NICs and GPU are plugged

The only thing set in stone is our name.