[Andrew Norfolk]

Hi,

I'm about to place an order for the parts needed to build a new CFD workstation.

Here is a previous thread describing the requirements.

http://www.cfd-online.com/Forums/har...rkstation.html

Parts List:

CPU: 2x Intel Xeon E5-2637 v3

http://www.lambda-tek.com/Intel-CM80...01~sh/B1935313

Motherboard: Supermicro X10DRL-i

http://www.lambda-tek.com/shop/?
region=GB&searchString=evo+212&go=go

Memory: 2x Crucial 32GB 2133MHz DDR4 ECC Reg

http://www.ebuyer.com/655315-crucial...ct4k8g4rfs4213

CPU Heatsink: 2x Cooler Master Hyper 212 Evo

http://www.lambda-tek.com/shop/?regi...=evo+212&go=go

Storage: Samsung 512GB 850 PRO

http://www.lambda-tek.com/Samsung-MZ...BW~sh/B1915316

Power Supply:Corsair CS850M

http://www.lambda-tek.com/Corsair-CP...UK~sh/B1954079

PC Case: Fractal R5 Black

http://www.ebuyer.com/675950-fractal...d-ca-def-r5-bk

Total Price: £2987

I have a few questions regarding compatibility.

The motherboard is ATX form factor however it is a server based board. Will it fit into a consumer based ATX case like the fractal R5? I've had a warning that it may be awkward but I do not understand why this would be the case.

I am also concerned that the two CPU coolers I have chosen might overlap due to the proximity of the CPU sockets on the motherboard?

Finally, when it comes to RAM i'm totally lost. I want a total of 64GB ECC DDR4 at 2133MHZ but do I do this as 4x16GB? 8x8GB? Do I use single, dual or quad ranked modules?

[flotus1]

Before going into detail: Did your requirements change or why did you switch to a dual-CPU setup?

[Andrew Norfolk]

Hi Flotus,

No the requirements did not change. I had a chat with the ANSYS tech support team. I suggested the 8 core 2667v3 as that offered the faster RAM and larger cache you recommended. Apparently because of the memory bandwidth limitations of CFX, you can actually get better performance running on dual quad cores than a single eight core. This set up kept me nicely under a £3000 budget and I think offers better performance. Do you think a single 8 core would be better?

[flotus1]

That makes sense. You might need to check the processor affinity so your 4 threads are distributed correctly among the 2 processors

The mainboard you picked is slightly wider than a standard ATX mainboard. It should still fit into most ATX cases. To be on the safe side here choose a case that can hold E-ATX or XL-ATX boards.
You might consider board with 16 DIMM slots so you can easily upgrade RAM later if necessary.

The CPU coolers wont interfere. However, there are 2 components you should not cheap out on when building a workstation: Power supply and CPU cooling. My choice would be the BeQuiet dark power pro 11 550W and CPU coolers from Noctua. Which one depends if your mainboard has square- or narrow-ILM sockets.

RAM: you need 8 DIMMs to fill all 4 memory channels of both CPUs. For 64GB total 8GB DIMMs are your only option. Dual-rank DIMMs can offer slightly better performance than single-rank DIMMs due to rank interleaving.

Remember that you still need a graphics card.

[RobertB]

We have a cluster built around this CPU running CENTOS. We use CCM+, I have two suggestions,

1) you turn off hyperthreading
2) you bind the processes to the CPU (I assume CFX can do this, if it is not standard)

On our previous cluster built around some 2.4GHZ Xeons from probably 2 generations earlier cpu binding did not seem to matter. With these faster chips I was somewhat disappointed initially as I had estimated they would be 2x per core faster. Running initially without binding they weren't, but they picked up about 20% speed (iirc) when binding was turned on.

[Elgaard]

Hi,

We are looking at the same setup with dual E5-2637 for running simulations with ANSYS CFX.

But why not choose RAM with a a frequency of 2400 Mhz? Both the CPU and motherboard support this. Is it just waste of RAM bandwidth due to the low core count, or?

We would like to also use the machine for solid simulations with ANSYS Mechanical. Other than the amount of RAM (choosing 128 GB instead) is there anything else we should be aware of? Is it worth going with a M.2 PCIe SSD instead?

[RobertB]

The processor above is a V3 which use DDR4 2133, the newer V4s use 2400.

In general you can't have too much bandwidth. You will still need 8 DIMMs to achieve full memory bandwidth.

Are you license limited, you can get more throughput with the 8 or 12 core CPUs.

One issue with SSDs is that they do wear out. If you are running reasonably big CFD models and saving them regularly I would check out the expected endurance.

40GB a day is a lot of writing for an office desktop but can be less than 1 save for a large CFD model (probably bigger than you would run on a single workstation but if you save a smaller one every hour of run time it all adds up).

[Elgaard]

Originally Posted by RobertB

The processor above is a V3 which use DDR4 2133, the newer V4s use 2400.

In general you can't have too much bandwidth. You will still need 8 DIMMs to achieve full memory bandwidth.

Thanks!

Originally Posted by RobertB

Are you license limited, you can get more throughput with the 8 or 12 core CPUs.

We are limited to 1 HPC pack with 8 cores - Unfortunately.

Originally Posted by RobertB

One issue with SSDs is that they do wear out. If you are running reasonably big CFD models and saving them regularly I would check out the expected endurance.

40GB a day is a lot of writing for an office desktop but can be less than 1 save for a large CFD model (probably bigger than you would run on a single workstation but if you save a smaller one every hour of run time it all adds up).

Regarding storange, i was interested in knowing if there where any gains going with multiple M.2 PCIe NVMe drives in RAID 0, with the posibility of write speeds arround 2500 Mb/s compared to the 600 MB/s with SATA III SSD.

[RobertB]

On our cluster we have conventional disks in a RAID 6 setup. They write at ~6-700MB/s.

We have big models >40GB and it's certainly not bad doing a write. even there it takes a minute or so which in the scheme of things is not that long.

At a certain (speed) point I would go for capacity over speed.

As an aside I would try locking the CPU affinity, if CFX allows it - it is typically in the mpirun commands. This helped significantly on CCM+ on the V3 version of these processors.

[Andrew Norfolk]

Originally Posted by RobertB

On our cluster we have conventional disks in a RAID 6 setup. They write at ~6-700MB/s.

We have big models >40GB and it's certainly not bad doing a write. even there it takes a minute or so which in the scheme of things is not that long.

At a certain (speed) point I would go for capacity over speed.

As an aside I would try locking the CPU affinity, if CFX allows it - it is typically in the mpirun commands. This helped significantly on CCM+ on the V3 version of these processors.

I can't find anything about setting the process affinity... when I load the solver I have the option to set the number of cores but that's all. Could you give me a bit more detail about core binding?

[RobertB]

Here is a link to the mpirun commands

https://www.ibm.com/support/knowledg..._cpu_bind.html

On CCM+ you can either use the cpubind option or it allows you to specify extra mpirun commands to use when the server process is used.

I have not used CFX so don't know how they do it, you could ping support.

[bindesboll]

If you are limited to 4 cores, I dont see the point of spending money on dual quad core CPU - you will max use half of it.
Go for i7-5820K and overclocking + matching RAM speed, that will give you max performance on 4 cores (and still having 2 cores for other tasks while solving).

The system you describe will be suitable for a single ANSYS HPC Pack licence (8 cores), which you should consider taking your management into, as it might be a more cost efficient solution than 4 single HPC licenses.

Sure you will need to have 4 memory modules per CPU to utilize the 4 memory lanes. I agree that dual rank RAM is slightly better than single rank.