[KalleT]

Greetings fellow CFD enthusiasts,

We are running CFD on ANSYS CFX. We have licenses to utilize 32 cores. The most common use is to have four solvers running each using 8 cores. From time to time we also run larger simulations utilizing all 32 cores for one solve. We currently have a cluster with two nodes having 16 cores each (2 * Xeon E5-2670 @ 2.6GHz) connected with gigabit ethernet. We are now planning to replace this hardware as it is now quite old.

Would it make sense of buying a workstation instead of a new cluster server? It seems that I could get 32 cores into one machine using two E5-2683 v4 processors (16 cores each). Will this be slower than the current cluster as the clock speed is smaller (2.1 GHz vs. 2.6 GHz) and memory bandwidth/core is also lower. If you think that we should stick to updating to a new cluster what would be recommendation of processor bearing in mind that we are limited to 32 cores by CFX licenses.

[flotus1]

There are quite a few possible options to replace your current setup. They differ in cost and performance.
Let me list a few, ranked by a rough performance estimate for CFX:
1) Four nodes with dual-CPUs, e.g. Xeon E5-2637v4, 4 cores
2) A quad-socket node e.g. with Xeon E5-4655v4, 8 cores
3) Two nodes with dual-socket CPUs, e.g. Xeon E5-2667v4, 8 Cores
3) Four nodes with single CPUs, e.g. Xeon E5-1660v4 or I7-6900k (8 cores each)
5) A single workstation with dual-CPUs, e.g. Xeon E5-2697Av4 or E5-2683v4 (16 cores each)

You can connect the nodes with Gigabit or 10G-Ethernet, but I would recommend Infiniband if performance is an important factor for the larger simulations -at least if you connect 4 nodes. Of course you don't need to buy several workstations, you can also get the multinode-setups in a server.
The reason why number 5 might even be slower than your current setup is not so much the lower clock speed, this is compensated by the newer CPU architecture. It is the lack of memory bandwidth. 8 channels DDR4-2400 deliver less bandwidth than 16 channels DDR3-1600, provided your current setup has properly populated memory channels. I can not recommend this option.
Which of the other options is suitable for you mostly depends on your budget.

Edit: keep in mind that AMD Naples will be released within the next few months, as well as Intels Skylake-EP. If you are not in a hurry with buying the new system you might want to wait since they will offer a pretty substantial increase in CFD performance thanks to more and faster memory channels.

[evcelica]

Your current setup is pretty good, You won't gain too much increase except with flotus1's option #1, since that will more than double your memory bandwidth. Options 2-4 are pretty much just the same thing as what you have, with a little bit faster memory.
Your suggestion and option 5 will be slower than what you have
I said won't update my cluster until there is a big change like the hex channel memory that was rumored to come with the Skylake-EP.
I second the Infiniband. You do not need a switch with only 2 nodes, but will with 4.

I see that having 1 memory channel for each core is about the limit of linear scaling (when using double precision) once you start using 2 and 4 cores per memory channel, you are really bottlenecked by the memory bandwidth and won't see much for gains.

Option 1 has 32 cores and 32 memory channels = Beautiful.

[KalleT]

Thank you guys for your insight. I will ask quotes from our hardware partner as per your suggestions. I will come back to you when I have new setup compared against the current one.

[Christophe]

I would get an Infiniband switch, and hit ebay and get more of that exact same node. Stuff that's a few years old is about 1/6th the cost of new. You'll probably spend $25k on a new workstation. You can get the IB switch, used, and the IB cards and prob 6-8 more nodes for that price. That's what I did. I've benchmarked core for cor e some of the older E5s vs somewhat newer E5s and it was dead even.


Sent from my iPhone using CFD Online Forum mobile app

[KalleT]

Dear all,

Our hardware vendor has now started to offer machines with the latest Intel CPU:s. We also got new licensing scheme for CFD enabling us to use 48 cores for CFX (typically running 4 simulations on 12 cores each).

I am considering to go with 2 compute nodes each having two 12-core Xeon Gold 6146 processors. Do you think if there would be real benefit in having more nodes with less cores per processor instead? I suppose that having 6 memory channels instead of four allows more cores per processor to be efficiently utilized but I haven't found any official recommendations for CFD regarding these new processors.

[flotus1]

I hereby officially recommend Skylake-SP processors over their Broadwell-EP predecessors

Do you think if there would be real benefit in having more nodes with less cores per processor instead?

More nodes with less cores per node will be faster, although it is difficult to estimate the difference. I guess it comes down to the question if you can draw any benefit from shorter solution times. If this is the case, I would definitely recommend 4 smaller nodes because it will pay off very soon even if the performance benefit is small. And I don't know how much Ansys charges you for their licenses, I can only guess that over the course of 3 years it is several times more than the cost for 4 cluster nodes. By the way, do you have separate pre/postprocessing workstations or will you use the cluster nodes?

[KalleT]

Than you for your reply. I waas thinking about having a separate head node for pre- and post-processing. This I think is question of whether it-department will support us to have Linux on this server. My understanding is that CFD Post for example is not supported for interactive session under Windows Server OS.