[naffrancois]

Hello,

I am planning to build a cluster of 1 master + 4 nodes. I have some questions about the configuration for the master node. My first idea was a dell poweredge but I am also considering setting it up myself based on commercial components:

Here is what I have in mind
- ASUS Z10PE-D8 WS
- 2*Intel Xeon E5-2609 v4 (1.7 GHz)
- 8*8Go Kingston ValueRAM DDR4 2400 MHz ECC Registered CL17 SR X8
- ZOTAC GeForce GTX 1080 Ti Founders Edition
- Seasonic P-1050 Platinum 80PLUS Platinum
- Phanteks Enthoo Luxe Tempered Glass (Anthracite)
- LSI MegaRAID SAS 9240-8i SGL
- 2*Samsung SSD 850 PRO 128 Go
- 4*Samsung SSD 850 PRO 1 To

This workstation would be used as a master node for 4 nodes having each 2*xeon E5-2697 v4 (2*18 cores) and may also be used for postprocessing/visualisation (paraview). The first group of 2 SSD would be raid 1 for the OS and the second group of 4 SSD in raid 10 shared via a nfs partition to the compute nodes.

My first big doubt in on the raid 10 for the shared partition, given that for budget reasons the network will be 1GBe at first. We run transient simulations with periodic writes, so will the performance of raid SSD be bottlenecked by the 1Gbe connection ? I would say so but would it however be better than a simple raid 5 HDD (configuration I have on an other machine) ? When the machine is used for transient visualisation then I guess the gain would be very good, file loading takes ages on the raid 5, am I right ?

My second doubt is on the choice of cpus. I think but I may be wrong that I do not need much cpu-power for what the master node is meant for. On the post-processing stage, the GPU should make most of the work ?

An other doubt that comes to me is about the general compatibility of the aforementionned components in a Linux environment (Lubuntu). The raid controller is supposed to work with SSD's, any experience with it or other recommendation ?

I am not sure if the ram is of good quality as it is ValueRam.

Many thanks for your comments and help.

[flotus1]

Your current setup is pretty far from ideal. Here is why.

My second doubt is on the choice of cpus. I think but I may be wrong that I do not need much cpu-power for what the master node is meant for. On the post-processing stage, the GPU should make most of the work ?

First of all: If you think that you do not need much CPU performance, a dual-CPU setup is pretty much the opposite of that. Then again using some of the slowest CPUs available for that setup is not a good idea at all.
You need CPU performance for your master node if you want to use it for pre- and postprocessing. But instead of going for a large amount of very slow cores, use less faster cores. Many workloads in pre- and postprocessing are serial or do not scale well on many cores. My recommendation here is a single-CPU setup with one of the following CPUs, depending on your budget: Core-I7 6800k, Core-I7 6900k, Xeon E5-1650v4, Xeon E5-1660v4.
The GPU has nothing to do with this. It needs enough memory to fit your visualizations and some performance to manipulate the views smoothly. A GTX 1070 would be a good compromise if you are on a budget. If money is not an issue at all, a GTX 1080TI is even better.

My first big doubt in on the raid 10 for the shared partition, given that for budget reasons the network will be 1GBe at first. We run transient simulations with periodic writes, so will the performance of raid SSD be bottlenecked by the 1Gbe connection ?

Gigabit Ethernet has a bandwidth of ~120MB/s. Even a single SATA3-SSD achieves read and write speeds of nearly 500MB/s. So yes, Gigabit Ethernet might be a bottleneck here provided your solver produces data at a faster rate.
I would say scrap the idea of SSD-RAID. If you want a fast storage, use a PCIe-SSD like the Samsung 960 Pro 2TB. If you want cheap fast storage, use single SATA3-SSDs like the Samsung 850 EVO/PRO 2TB. It is still much faster than Gigabit Ethernet. Redundancy for your fastest drive is not necessary in my opinion. You save the data there, analyze it and when you are done you move it to a HDD-based storage.
Instead of 2*128GB for the system SSDs, I would recommend 2*250GB Samsung 850 Evo. Same price, twice the storage capacity.

A few comments on the rest of your setup:

ASUS Z10PE-D8 WS

Like I mentioned before, use a single-CPU setup. Which type of motherboard depends on your choice of CPU and the features you want (10G Ethernet perhaps...)

8*8Go Kingston ValueRAM DDR4 2400 MHz ECC Registered CL17

The CPUs you had in your initial setup officially support DDR4-1866. Now with your single-CPU setup I would recommend you start out with 4*16GB DDR4-2400. If you want ECC memory or not is up to you and affects your CPU choice as well. Non-ECC -> I7; ECC -> Xeon. You can use the cheapest RAM you can find from any reputable brand.

Seasonic P-1050 Platinum 80PLUS Platinum

A little bit over the top. A BeQuiet Dark power pro P11 550W is more than sufficient.

Are you planning to use Infiniband to connect your nodes? Then you might as well use that instead of an Ethernet connection to the head node.
A last comment on the CPU selection for your compute nodes: you might be better off with more nodes that have CPUs with less cores. Consider the performance and price/performance estimate I made here: Socket 2011-3 processors - an overwiew

[naffrancois]

First of all, many thanks for the time spent in answering all my questions. I understand and agree most of your comments.

I agree a better option would be a single core faster cpu machine with I7 6800k or better, 4 channel memory so 4*16Go ram. Plus it has the advantage to work with cheaper non ECC memory and cheaper motherboard.

About the choice of the disks. Redundancy has a price but maintenance and disponibility as well, I will think about it. Raid 0 was in case of a future upgrade to infiniband as FDR thoretical throughput is about 6.8 GB/s. But maybe for this cluster size communication load will never be that big ?

About the power supply, perfect. I thought 250w consumption was for the gpu only, but apparently not:

http://www.anandtech.com/show/11180/...0-ti-review/16

About the cpu choice for the compute nodes, there were two reasons to cluster many cores on each nodes. The first reason is that although the ratio $/core may not in favour of this particular cpu, increasing the number of nodes is pricy unless if one builds the computer itself (2k€ as you mention in your study). A dell 7910 with 2 modest E5-2603 v3 and 64go is already 4k€. In the end, with Dell, a node with 2*xeon E5-2697 v4 and 8x8go ddr4 2400 gives 224€/core while it gives 265€/core with a E5-2620 v4, assuming I did not make a mistake

The second reason was about ethernet connection in order to limit inter-nodes connection.

In the end, I think I will configure a single socket master node based on i7 and reconsider the raid configuration. Many thanks for your valuable help

[flotus1]

You are right, the 2000$ for the rest of the workstation are pretty cheap if you buy from DELL, HP or whatever brand. But since they also charge more for the CPU than the MSRP, the relative price/performance ratio should still be meaningful.
The "$/core" metric may be right, but it assumes 100% scaling which is impossible for more than 2-4 cores.

You are right, a GTX 1080 TI FE can draw up to 250W all by itself. But a decent quality power supply with 550W is still enough for the whole system. Add 140W for the CPU and a very conservative 100W for the rest of the system and you still have more than enough headroom even under extreme load.

maybe for this cluster size communication load will never be that big ?

Maybe...you can give it a try.
But considering how cheap used Infiniband hardware is, you should at least not spend extra money on any other interconnect in the first place. Give Gigabit Ethernet a try, and if you see insufficient scaling between the nodes upgrade to Infiniband directly.

[naffrancois]

No I think you were right about the power consumption, in the link under full load (crysis) they state that the power is for the full system. So yes 550w from a good manufacturer should be more than enough.

About infiniband, as you say it is hard to evaluate the necessity before testing. Our code scaled well up to 150cores some time ago on 1Gbe but maybe with extra physics or another code it would not. Do you know specific places where to find used infiniband ? I estimated 6k€ for a new equipment with 5 adapters (550€ each) + 1 switch (3k€) + 5 cables (60€ each), it is quite significant.

Many thanks

[flotus1]

Ebay...

Concerning a comparison between various interconnects: http://www.hpcadvisorycouncil.com/pd...igE_in_HPC.pdf
This is pretty old by now so I would expect the interconnect to have a higher influence by now since the processing speed is higher.

[naffrancois]

Many thanks for the link, it gives a good idea !

I'll have a look on ebay to get an estimate, thanks again for all your advice