[Mike_Hissje]

Hello,

we are currently in the process of configuring a CFD server. CFD simulations with up to 10 million cells and FEM simulations are to be carried out on this. The question now arises as to which system is to be preferred. For example, there are two systems to choose from:



1. 2x Intel Xeon Silver 4210R (10 cores, 6 memory channels per CPU)


2. 1x AMD EPYC 7352 (24 cores, 8 memory channels)


Both systems are said to work with 8x16GB RAM. What is the best way to choose between the two systems and what are the major weaknesses in the systems? In addition, both systems are to be equipped with an RTXA2000. Many thanks for your help!

[evcelica]

First off, you don't want to run only 8 RAM DIMMS on the dual XEON system. You want 12 so that you use all the memory channels. so 12x16GB = 192GB.

Those 2 systems will be pretty close in performance.
CPU clock speeds are the same XEON = 2.4-3.2 GHz and EPYC = 2.3-3.2GHz

CFD is often memory bandwidth limited:
The XEON has 12 memory channels @ 2400 MHz max
The EPYC has 8 @ 3200 MHz max
Only a 12.5% advantage for the Xeons.

24 vs 20 cores, but I don't think you will scale all the way to that many cores on these machines.

With Dual CPU systems, I have seen my windows operating system not splitting the load very evenly between the two CPUs. But that is software and MPI dependent.

I can't say for sure which one would be better, they will be very close, so perhaps cost or the larger RAM capacity of the XEONs (128 vs 192GB) can help you decide.

[Mike_Hissje]

Hi and thank you for your comments!

Regarding the RAM DIMMS you are absolutely right! I swapped the CPU and didnt reconfigure the needed RAM DIMMS!

The calculation involving the memory channels and the frequency is very helpfull!
With that i have the oportunity to compare Systems.

So as I see it, the advantage of the dual CPU-System is the opportunity to put way more RAM in my System (if i want to) and a higher Memory Bandwidth but in may be more expensive and the communication between the two cpus und the splitting of load may be problematic.

As a conclusion both systems will work (assuming i use the correct amount of DIMMS) and it is a matter of taste, am i right?

Thank you for your Help!

[flotus1]

So as I see it, the advantage of the dual CPU-System is the opportunity to put way more RAM in my System

A single Epyc Rome CPU supports up to 4TB of RAM. Two Xeon Silver 4210R CPUs support up to 2TB or RAM. So it is safe to assume that memory capacity is not a limiting factor here.

The calculation involving the memory channels and the frequency is very helpfull!

To an extent, yes. But CPU core frequency can not be compared 1:1 across completely different CPUs. And the 10-core Xeons won't be able to saturate available memory bandwidth entirely.
The Epyc CPU also has larger L3 caches, which does help a bit with FV and FEA.

the communication between the two cpus und the splitting of load may be problematic

In terms of NUMA and potentially slow inter-core communication, both setups have this issue. The Xeons due to two physical sockets, and the Epyc because it consists of 4 separate CPU dies in a single package.
There are nuances to how each of these solutions behave in some edge-cases, but the gist of it: you can't avoid it entirely with this class of CPUs.

The systems are probably close in terms of peak performance for your applications. If I had to guess, the Epyc CPU might have a slight advantage. And it will definitely be faster for single- and lightly-threaded workloads like mesh generation.
I often recommend buying one CPU, but putting it on a dual-socket board anyway. This allows for an easy drop-in upgrade later to effectively double the performance.

[Mike_Hissje]

Thank you for your input!

These are some Informations i didnt know. These help a lot konfigurating a System who fits my needs.

Thank you!
Since the configuration and ordering process is still taking some time due to the holidays, I will probably have more questions in a week

[flotus1]

May I suggest answering a few of the questions from chapter 0 here: General recommendations for CFD hardware [WIP]
They are intended to narrow down possible configurations, and avoid going into various rabbit holes.

And another question that needs answering in your particular case: any reason why you are shopping for older hardware generations? Both Intel and AMD have newer generations available, with some benefits and modest price increases.
E.g. Xeon Silver 4310, which is objectively better than the 4210R, at the same list price.
On the AMD side, if you have decided that a single CPU is all you ever need, you should take a look at CPUs with a "P" suffix. They can't be used in dual-socket configurations, but are usually a bit faster than their non-P counterparts. E.g. 7402P.

[Mike_Hissje]

Hi, thank again

Here i tried to answer your question of Chapter 0

1. Which Software --> Ansys fluent
2. Limited Licences --> I dont think so. Never experienced a limitation so far

3. Fluid Simulations of flow in heat exchanger, around airfoils, simualtions of moving wind-turbines. Estimated Cell-Count: Up to 10 Mio peak
4. Budget: 5500€ (including Taxes)

5. Setting: Academic research/ engineer
6. Source of Computer: New Parts. We are tied to several providers
7. Part of the World- Germany
8. Anything else: Mechanical Simulations from time to time. Several users must be able to access the server at the same time. Graphic processing necessary. My old workgroups server didnt have any graphics processing in the form of a graphics card. Post-processing was almost impossible with this system.


I hope the informations provided are helpful.


Regarding your second querstion:
The processors mentioned werde taken from two offers that i have. The budget that was given to the different providers was different. Ee have agreed on a budget of 5500 (including taxes). In the meantime I have also had offers with two 4310s

[flotus1]

2. Limited Licences --> I dont think so. Never experienced a limitation so far

You really need to get to the bottom of this as a priority.
Ask your IT department, or whoever is in charge of purchasing these licenses.
No point in buying a 32-core workstation if you are limited to 8 threads.
Maybe your university actually has access to lots of HPC packs? Maybe you are using the student edition? We need to know.

The processors mentioned werde taken from two offers that i have. The budget that was given to the different providers was different.

This might be a pet peeve of mine, but I would avoid the sellers that offered you older hardware generations. Doing so is fine in parts of the world where the latest generations are hard to source. But if it happens in Germany, they are trying to offload their old inventory to paying customers.

[Mike_Hissje]

According to my system administrator I am able to use one core. Since I just simulated a minor case with 10 Processes (without hyperthreading) and one with 19 processes (with hyperthreading) (Just to check the license limitation), i dont think that he is correct at all. I am waiting for the answer of the person who is responsible for the licenses itself. Both simulations worked with the correspondig workload of the cpus.

Looking into the license manager you can read "The maximum number of HPC Pack licenses that can be requestet per user for a solve ist 14"

Assuming, what i read here (Another Thread in CDP-Online Forum) is correct, i dont think, that i should have any problems regarding licenses. But i am sorry, i am not experienced regarding Ansys licences.
What i am sure about is, that i dont use the student edition, its the educational edition!

Regarding the provided components, the newest offers include:

1 CPU-Systems:
a: AMD Epyc 7452
b: AMD Epyc Milan 7443P

2CPU-Systems:
a: Intel Xeon Silver 4314 (2x)
b: Intel Xeon Silver 4210R (2x) or Intel Xeon Gold 6226R (2x) (+1250€)

All Systems use 128GB RAM. Since some providers cant provide 8GB DIMMS (to adress the amount of memory channels of a two cpu system) i think about concentrating on a one cpu system. But have in mind that these components and providers arent safe at all. I'm not at all familiar with the price-performance ratio of the CPUs and so far I've only compared the CPUs on offer via benchmarks.


If the etiquette of this forum allows it, we can also continue to write in German. If this is not welcomed, we will of course remain in English.

Nonetheless, thank you very much for the help and suggestions.

[flotus1]

Let's keep the discussion in English.

Looking into the license manager you can read "The maximum number of HPC Pack licenses that can be requestet per user for a solve ist 14"

That sounds more like the absolute limit for the license manager, not how many HPC pack licenses you are allowed to use. HPC packs allow running on 2x4^N threads, which is half a billion threads for 14 packs.
I think there are still regular HPC licenses (without "pack") that add a single thread for each license.
Personally, I would want a definitive answer to how many threads I can use before buying anything.

Let's leave that aside for now. From your selection of CPUs on offer, I would rule out the Intel Xeon Gold 6226R. It is a sidegrade at best, offering slightly higher single-threaded performance compared to the more modern Xeon Silver 4314. Not worth the extra money in my opinion.
Another important issue is matching the amount of DIMMs to the total amount of memory channels. 128GB doesn't add up with 2nd gen scalable CPUs like the 6226R or 4210R. They have 6 memory channels each, so you really want 12 DIMMs total. Same old story: OEMs or SIs really should not offer these CPUs with unbalanbced memory configurations, especially if you told them you are doing CFD and FEA. It happens far too often.

Anyway, we can narrow it down further: if your sellers can't provide 8GB DIMMs, and you want 128GB or RAM, a single CPU with 8 memory channels is the best you can do.
That narrows it down to the AMD Epyc 7443P. I recommend pairing it with 8x16GB DDR4-3200 reg ECC memory. The modules should be dual-rank "2Rx8" for the best performance. Single-rank "1Rx4" or "1Rx8" is a few percent slower.
So that is my recommendation for now.

[evcelica]

To figure out number of cores, there should be an Ansys program called "Client Anslic Admin Utility" Open that and go to "display license status". This will show you the number of licenses you have and if they are being used or not. Check for HPC Pack or ANSYSHPC, and see the number you have available to you. FLOTUS1 already described how additional cores each provides per license.

[Mike_Hissje]

Hi
In the meantime I have received the following information:
Each user uses an "Main License" witch which he can use 8 cores by default. Each additional core requires an HPC license, of which 100 are available for the entire university.
The amount of freely available HPC licenses fluctuates throughout the year depending on the demands of lectures and projects.
In the ideal case, I can count on up to 100 cores, while in the worst case only 8 cores can be used.
Regarding these Informations and my budget, i dont think that the licenses are a problem. I dont think i have to worry about building a system that is limited by the amount of available licenses

@vcelica thank you
There are two Ansys versions on my computer. Version 20 R2 and 21 R2. Under version 20 I find said application (version 21 R2 is the version I work with). According to this, the lmgr is not running on the license server. However, the system administrator has already been contacted about this.