[Sharku]

Hello,
When selecting the model of Intel Xeon CPU for a workstation for FEA using Ansys, I have noticed that the included CPU cache memory tends to increase with the number of cores of the model. Let's assume I have a license that does not allow to use more than 8 cores for the FEA. Would it be reasonable to choose a CPU with more than 8 cores just to get more cache memory?

[flotus1]

Maybe not with the way Intel currently sells their last level cache. I.e. keeping it a constant value per core. The benefits of larger caches are small. So it's "go big or go home"
They have Xeon "MAX" CPUs with a ton of HBM inside the CPU package. But the prices are a bit excessive IMO.

The AMD Epyc 9174F is a 16-core CPU, with 265MB of L3 cache, and high core clock frequency.
And then there are the "X3D" variants, which were specifically designed for workloads that benefit from large caches. Like FEA and CFD. AMD Epyc 9184X, 16 cores, 768 MB of L3 cache.

[Sharku]

Thank you, flotus1. Assuming only 8 cores will be used, do you think AMD Epyc 9174F (4.4 GHz, 265MB cache) would perform better for FEA than Intel Xeon W7-2495X (4.8 GHz, 45 MB cache), because of the much higher cache of the former?

[flotus1]

Using anything but Intel has been a gamble with FEA solvers in the past.
But not any more it seems. Ansys is providing slides where AMD CPUs come out on top:
https://www.ansys.com/content/dam/am...tion-brief.pdf
So I would probably go with the AMD option if it was my money.

Is this "RAID 0" question still relevant here? Because for out-of-core execution, getting the scratch space fast enough matters more than the CPU. And maximizing memory capacity of course.

[Sharku]

Originally Posted by flotus1

Using anything but Intel has been a gamble with FEA solvers in the past.
But not any more it seems. Ansys is providing slides where AMD CPUs come out on top:
https://www.ansys.com/content/dam/am...tion-brief.pdf
So I would probably go with the AMD option if it was my money.

Thank you for your helpful suggestions.

Originally Posted by flotus1

Is this "RAID 0" question still relevant here? Because for out-of-core execution, getting the scratch space fast enough matters more than the CPU. And maximizing memory capacity of course.

Sorry, but I'm afraid I don't understand what you mean here. Unfortunately, I still have much to learn about hardware concepts.

[flotus1]

This thread right here, also started by you today: Workstation with Raid 0

If running the solver is heavily bottle-necked by getting the data from the SSDs, faster CPUs do not help much. The first priority then would be to get the scratch space as fast as possible.
It boils down to this: do your simulations fit into physical memory, or do you want to run out-of-core. I.e. using additional space on the SSD(s) to make up for a lack of memory capacity.
Modern NVMe SSDs can be fast. But they are still at least an order of magnitude slower than RAM.
Memory bandwidth of an AMD Epyc Genoa CPU: 460.8 GB/s
Sequential read speed of an NVMe SSD with 4 PCIe 4.0 lanes: 7000 MB/s. That's 7 GB/s

[Sharku]

Originally Posted by flotus1

This thread right here, also started by you today: Workstation with Raid 0

If running the solver is heavily bottle-necked by getting the data from the SSDs, faster CPUs do not help much. The first priority then would be to get the scratch space as fast as possible.
It boils down to this: do your simulations fit into physical memory, or do you want to run out-of-core. I.e. using additional space on the SSD(s) to make up for a lack of memory capacity.
Modern NVMe SSDs can be fast. But they are still at least an order of magnitude slower than RAM.

For an AMD Epyc 9174F with 12 memory channels, I would get 12*32=384 Gb of RAM. I'm not sure yet if that will be enough RAM to avoid requiring significant scratch space in the SSDs for my simulation cases, though... I don't have much experience with that yet. I'm planning to perform multiphysics simulation of electric machines, including electromagnetic and thermal behavior, using Ansys Maxwell and Fluent.

[Sharku]

Is the memory cache of the CPU per core very important for complex FEA simulations?

I can see that the Epyc 9174F has 256MB L3 cache and 4.1GHz-4.4GHz clock frequency, whereas the Epyc 9184X has
768MB L3 cache and 3.55GHz-4.2GHz clock frequency. That is, the former is better in clock frequency, whereas the latter is better in memory cache. Which option could be expected to perform better for demanding FEA simulations?

[flotus1]

The latter will be faster most of the time.
These CPUs with extra large caches are specifically designed and marketed for this type of workload. Base clock speeds are just that: a baseline. Both CPUs will run at higher boost clock speeds anyway, even with all cores busy.
There used to be tables for which boost clock speed to expect, for which type of code, at different amounts of cores loaded. But I haven't seen those around for any modern CPU. I guess the logic for boost clocks just became too complicated to put it into a table.

[Sharku]

Thank you again, your responses are very helpful.

Regarding the EPYC 9174F (https://www.amd.com/en/products/cpu/amd-epyc-9174f), I can see that the EPYC 9274F (https://www.amd.com/en/products/cpu/amd-epyc-9274f) has 8 more cores, practically the same clock frequency, and yet it is about 25% cheaper. Doesn't the latter seem much preferable over the former?

[flotus1]

Oh yeah, they have some weird scaling on prices going on with the "F"-SKUs.
These are for maximum per-core performance in traditional CPU-heavy workloads.
AMD designed their prices around software licensing costs. There must be an extremely expensive, and widely-used software floating around, that charges based on cores per socket. Mixing together all these costs, it turns out that money is to be saved with a 16-core CPU, even if that costs more.

Nothing that concerns us though. If the 16-core variant is actually more expensive on the retail market, the 24-core variant is the better option.

[Sharku]

Excellent. And what about the graphic card when using Ansys Fluent, assuming the CPU is the EPYC 9274F and there is 12*64GB=768GB RAM? Is it worth going for a powerful graphic card such as "4 GB PNY NVIDIA T400 GDDR6 384 cores CUDA", or even "12 GB PNY NVIDIA RTX A2000 GDDR6 3328 Cores CUDA"?

[flotus1]

General recommendations for CFD hardware [WIP]
Chapter 3

the tl;dr of that: no, probably not worth spending serious cash on a GPU for compute.
And for just rendering the image on the screen, anything remotely recent with at least 8GB of VRAM should work.

[Sharku]

Thank you!

I have one additional question, which hopefully may be the last one about this subject. Regarding the Epyc 9274F, it seems very difficult to find it mounted on a tower case instead of a server rack case. I asked the vendor about this problem (PCSpecialist) and they said that after some searches they were able to find a provider who would be able to mount the Epyc 9274F on a tower case, as I asked for. Do you think this may mean any problem, like decreased performance or reliability?

EDIT: I have now talked again with the vendor and they discourage using a tower for the processor because a different motherboard would have to be used. Now, the question I have is, would it mean any decrease in performance using the Epyc as a server with which my personal PC would communicate to run the FEA simulations, compared with having the Epyc directly in the same PC that I would use? Intuitively, I tend to think the communications between the PC and server would imply a greater latency, am I wrong?

[naffrancois]

You clearly do not want an air-cooled rack sitting next to you, it will be loud as hell. Racks are compact form-factors with small fans spinning fast and are meant to sit in a dedicated room.

What your vendor said is not very clear, you should maybe contact an other one. Some time ago a vendor made me an estimate for a water-cooled dual genoa within a tower, did not seem to be that hard for them.

[Sharku]

Originally Posted by naffrancois

You clearly do not want an air-cooled rack sitting next to you, it will be loud as hell. Racks are compact form-factors with small fans spinning fast and are meant to sit in a dedicated room.

What your vendor said is not very clear, you should maybe contact an other one. Some time ago a vendor made me an estimate for a water-cooled dual genoa within a tower, did not seem to be that hard for them.

Thank you naffrancois for your helpful comments, and for sending me the information about that vendor. I have contacted them and I am awaiting their response. Based on your comments, I have discarded the idea of a server rack, and I am looking for alternatives that implement an Epyc CPU of the Zen4 generation in a tower case for use as a workstation. The vendor you suggested is one promising possibility, but after some search I have also found some European vendors that offer specific workstations on their websites for Zen4 Epyc in tower cases, like this one:
https://www.rect.coreto-europe.com/e...-96-cores.html

Does it look got to you? Note that my budget is 7800€ so it would have just one Epyc CPU, not two. Unfortunately, in principle it does not include the option of liquid cooling. Do you think this particular option would also be too noisy?

[naffrancois]

Hello,

I am a bit out of the loop concerning prices, sorry I cannot tell if it is a good deal or not. What surprises me is that the cpu list mixes single (P suffix) and dual socket processors. But I am not up to date, maybe all these genoa cpus can be set up on a single socket motherboard. Anyhow I doubt you would fit a dual socket config within your budget.

Still you would have to ask their cooling solution as it is not specified.

If you can make an estimate on the RAM you need beforehand you could save some money, 384Gb may (or may not) be a lot more than needed. See if half is a viable option, keeping of course 12 sticks to feed the all the channels.

If it were my money I wouldn't go for less than 2Tb nvme, the price difference is not that high.

Do not forget to include a graphics card, nothing fancy needed as long as it has enough vram to display smoothly, does not need to be in the professional line (see the flotus1 sticky post, general recommendations for cfd hardware)

[Sharku]

Thank you again, naffrancois.

Please, note that the link I posted was for the default settings of this workstation, not exactly for the specific settings I would select. I would indeed add a graphic card (I was thinking about the PNY Nvidia T1000 with 8 GB GDDR6-RAM or the RTXA2000 with 12 GB GDRR6 ECCE-RAM) and a greater SSD (I was thinking about a couple of 4 TB M.2 NVMe SSD SN850X with 1,200,000 IOPS). My concern was mainly about whether the system would be as noisy as a dual-Epyc workstation. I will ask the vendor about the cooling, as you suggested.

Many thanks!

[Sharku]

I have now found another European vendor who offers Epyc 9004 CPUs in tower servers. I have set this configuration:

https://server-konfigurieren.de/prod...453552764/4206

In contrast to the previous German vendor, this one does specify the cooler:

"Supermicro SNK-P0084AP4 4U Active CPU Heat Sink for AMD Socket SP5 Processors"

I have looked the cooler code on the Internet and apparently it only produces 42 dB, which seems reasonable for an office, doesn't it?

[naffrancois]

This is hard to tell, reasonable for me may not be reasonable for you and vice versa as noise is highly subjective. 42 dB/5000 rpm is not what I would call silent but there may not be better alternatives with air cooling. I am already 3 generations of EPYC late, at that times Noctua coolers were enough to cool EPYC cpus and really silent ~20dB/2000 rpm. Since then TDP has increased a lot. I hope other people with more recent EPYC config can give you more insight on that topic.