[Kaskade]

As previously mentioned I am looking into getting a Skylake-SP system to carry out simulations in Ansys Fluent.

Apparenty there is a large difference in price between two 8 GB modules and one 16 GB module. So instead of using 12 ram modules per socket in dual channel mode, the vendor suggests using 6 modules (one per memory channel). Am I mistaken in assuming that 12 modules would be preferable for CFD?


[Moderator note: Changed title from "Dual-channel Memory" to "Why are double-ranked DIMMs better than single-ranked?"]

[flotus1]

The thread title is misleading. Your question revolves around the number of DIMMs per memory channel. Using 2 DIMMs per channel is not dual-channel, it's just 2 DPC.
However, there is no performance increase when using two or more DIMMs per channel instead of one. Rather the opposite is true: with more than one DIMM per channel, the maximum supported memory frequency is usually lower. And even if your platform can handle two DIMMs per channel with the same frequency, don't expect any performance gains.
You get the best memory performance using one dual-rank DIMM per memory channel.

[Kaskade]

It was not trying to be misleading. But you are right, the thread title (and much of the question) is wrong. All the quad-, dual- and double-somethings in regards to memory got me confused for a moment.

Why are double-ranked DIMMs better than single-ranked? I tried to research it, but the explanations did not help.

[flotus1]

My apologies, the first sentence really should have read "I think you mixed some things up".

Why are double-ranked DIMMs better than single-ranked?

It is generally attributed to a feature called "rank interleaving". But please don't ask me anything more specific about it, I am just a mechanical engineer
When measuring memory throughput, dual-rank DIMMs can perform 5-10% better than single-rank DIMMs. Not that much, but every little bit helps and it is basically free.

[Kaskade]

Thank you for the link. That explanation is much better than my what search turned up.

My biggest issue currently is convincing vendors that one 4-socket system should not cost more than two 2-socket systems. I mean, the CPUs are the same and you only need half of most components (e.g. Infiniband-, Ethernet- and RAID-controllers). If anything it should cost less.

[flotus1]

Good luck with that. There are quite a few reasons why quad-socket systems are more expensive than two dual-socket systems.
The motherboards for quad-socket systems are way more expensive, and so are the power supplies and the cases. Additionally, they are not as common as dual-socket workstations. So the vendor has to split the cost for development and testing among fewer customers. All of this costs more than a few Infiniband cards and RAID controllers. The CPUs specifically for quad-socket systems are also more expensive. You might want to consider using different CPUs for dual- and quad-socket respectively.

[Kaskade]

Originally Posted by flotus1

The CPUs specifically for quad-socket systems are also more expensive.

That point is not true for Skylake-SP. The Gold CPUs can be installed in 2- and 4-socket systems. They shifted the price jump to 8-socket-systems for which you need Xeon Platinum.

The Xeon Silver CPUs are definitely cheaper, but have a lower frequency, memory bandwidth, cache size and just one FMA-Pipeline.

I got one offer were one 4-socket-system was slightly cheaper than two equally equipped 2-socket-systems. So it is possible. Plus I wouldn't need an expensive switch to use 8 sockets in total. That would be my preferred solution in terms of budget.

[flotus1]

You are right. I recalled reading somewhere that only the platinum models had 3 UPI links. Apparently my information was wrong.

[Kaskade]

We are both right and wrong. Xeon Gold 51xx only has 2 UPI-Links, which would probably slow down the performance in a 4-socket-scenario. Xeon Gold 61xx has 3 UPI-Links. Also Xeon Gold 51xx only has one FMA-Pipeline, except for the Intel Xeon Gold 5122 which has two.

I think they used a random number generator to create their processor range and then just sorted the results into 4,5 price categories.

[Kaskade]

Hi Alex,

is there some way to monitor if the calculation is limited by the memory bandwidth besides doing benchmarks? Can I look at the CPU load or will the CPU run at 100% when it's waiting for data?



P.S.: I found a second manufacturer (Supermicro) offering 4-socket-machine for slightly less than the price of two 2-socket-machines. So there is a shift.

[flotus1]

None of the monitoring tools I used so far was able to indicate a memory bandwidth bound workload. CPU usage remains near 100% even when a lack of memory bandwidth is slowing things down. Let me know if you find a way to to monitor this.