[otokemo]

Originally Posted by flotus1

I see no point in buying these CPUs in "new" condition.
Just look at the seller rating. You would be better off just buying them used on ebay for half the price.
The CPU coolers are for servers, and rated for only 180W. They might work for these CPUs, just be aware that they are loud. What is this going to be? A workstation or a server? Which chassis do you intend to use?
Nemix RAM should be avoided at all costs, especially for AMD systems. Get Samsung/Micron/Hynix

Noted. University is refusing to buy used CPUs, if it was my own money, I wouldn't hesitate.
This is going to be a workstation. It's just from my previous experience: I bought several CPU coolers for Dell PT7810 workstation and they didn't fit.. So I had to buy smaller ones.
Also, it is a big problem to buy Extended ATX case in Georgia, so I have to order it online as well (transportation costs will be huge).
I'm thinkg something like lian Li Lancool III, which will fit bigger coolers.

[otokemo]

Originally Posted by flotus1

We are getting closer. You can buy that bundle, but be
You could upgrade to Milan-X CPUs at some point with a new H12DSi or MZ72-HB0.
And of course you would get a new board, with warranty. Motherboards and PSUs are the components most likely to fail.
Whether that's worth the 800$+ price tag is up to you.

Since Milan CPU prices has dropped on Ebay and Aliexpress, what would be your first go to CPUs from Milan family over 7532?

https://www.techradar.com/news/hundr...bay-aliexpress

[flotus1]

Meh...unless you have some other workloads in mind that can make use of the additional cores, I don't think the step up in price is worth it. You are looking at a performance difference in the order of 15-20% for CFD stuff.

Epyc Naples: OpenFOAM benchmarks on various hardware
Epyc Rome with 256MB L3: OpenFOAM benchmarks on various hardware
Epyc Milan with 256MB L3: OpenFOAM benchmarks on various hardware

-Solver run time in seconds-
nthreads   Naples    Rome   Milan
      01    907.7   643.8   471.9
      64     27.7    16.0    13.8

Rome result is not mine, so I can not vouch that the same level of optimization went into it. But the big generational leap was definitely between 1st and 2nd gen. 3rd gen is more of an incremental improvement.
Also look at the scaling in the last link for the 32-core Milan CPUs. It really tapers off towards 64 threads, there is not much to be gained from higher core counts.

We have all been there: "just one more slightly more expensive component to to bump performance a bit". And suddenly, the whole thing costs twice as much as you originally intended.
I am currently toying with the idea of upgrading my personal workstation to Genoa. I'm still on 1st gen Naples CPUs. And it's suddenly more than 10k for all the components I "need".

[otokemo]

Originally Posted by flotus1

Meh...unless you have some other workloads in mind that can make use of the additional cores, I don't think the step up in price is worth it. You are looking at a performance difference in the order of 15-20% for CFD stuff.

Epyc Naples: OpenFOAM benchmarks on various hardware
Epyc Rome with 256MB L3: OpenFOAM benchmarks on various hardware
Epyc Milan with 256MB L3: OpenFOAM benchmarks on various hardware

-Solver run time in seconds-
nthreads   Naples    Rome   Milan
      01    907.7   643.8   471.9
      64     27.7    16.0    13.8

Rome result is not mine, so I can not vouch that the same level of optimization went into it. But the big generational leap was definitely between 1st and 2nd gen. 3rd gen is more of an incremental improvement.
Also look at the scaling in the last link for the 32-core Milan CPUs. It really tapers off towards 64 threads, there is not much to be gained from higher core counts.

We have all been there: "just one more slightly more expensive component to to bump performance a bit". And suddenly, the whole thing costs twice as much as you originally intended.
I am currently toying with the idea of upgrading my personal workstation to Genoa. I'm still on 1st gen Naples CPUs. And it's suddenly more than 10k for all the components I "need".

I got the parts (2x7532). I have couple of questions, should I install windows server? Or windows 10 pro N will work?

[flotus1]

I have run windows 10 pro bare metal on similar machines for testing purposes. It should work.
Installation can be tricky though, you need to play around with bios settings. Mainly IOMMU.
See also: https://learn.microsoft.com/en-US/tr...mily-processor

Disable the SMT settings (such as the logical processors setting) in the BIOS.
Disable the x2APIC settings and IOMMU in the BIOS.
Use the OS media to install Windows Server.
Install the latest Windows Server updates.
Restart the system and enable the SMT, x2APIC and IOMMU settings for Windows Server 2019 in the BIOS.

This is for server with 128 cores total. But it is probably best to follow the same procedure for regular windows 10. And then leave SMT disabled anyway

[otokemo]

Originally Posted by flotus1

I have run windows 10 pro bare metal on similar machines for testing purposes. It should work.
Installation can be tricky though, you need to play around with bios settings. Mainly IOMMU.
See also: https://learn.microsoft.com/en-US/tr...mily-processor

This is for server with 128 cores total. But it is probably best to follow the same procedure for regular windows 10. And then leave SMT disabled anyway

Thanks! One more question, as you predicted, Dynatron server coolers are very noisy, what would you suggest as a replacement? ARCTIC Freezer 4U looks massive and I'm not sure if H12DSI can fit two of them.

[flotus1]

Don't know about those, but since they are for a 4U form factor with only 92mm fans, there should be plenty of clearance for two of them side-by-side. They are front to back orientation though. Which means the cooler towards the I/O is sucking in hot air from the other CPU cooler.

The Noctua NH-U14S TR4-SP3 have a neat feature: the position of the heatsinks relative to the mounting position can be adjusted slightly. Just enough to make two of them fit.
If they are too tall for your case, there is always the Noctua NH-U9 TR4-SP3.

Depends a bit which case you are using, and how quiet you want it. The Noctuas are good for regular desktop cases, with air exhaust at the top.
The ARCTIC Freezer 4U is better for more restricted cases, where you can only exhaust air towards the rear. It will be louder though.

[otokemo]

Originally Posted by flotus1

Don't know about those, but since they are for a 4U form factor with only 92mm fans, there should be plenty of clearance for two of them side-by-side. They are front to back orientation though. Which means the cooler towards the I/O is sucking in hot air from the other CPU cooler.

The Noctua NH-U14S TR4-SP3 have a neat feature: the position of the heatsinks relative to the mounting position can be adjusted slightly. Just enough to make two of them fit.
If they are too tall for your case, there is always the Noctua NH-U9 TR4-SP3.

Depends a bit which case you are using, and how quiet you want it. The Noctuas are good for regular desktop cases, with air exhaust at the top.
The ARCTIC Freezer 4U is better for more restricted cases, where you can only exhaust air towards the rear. It will be louder though.

Got it.

I disabled SMT. I only had a chance to test the setup with a small model ~ 1mil elements and the power consumption of both CPUs doesn't go beyond 140W, is it normal? It's possible, that the performance is bottlenecked because of the temperature? (As it does for 7950x) Which goes up to 54c for one and 67c for other. And as for performace, the simulation was done in
589 seconds (Total wall-clock time), which is almost Identical to what 5800x was capable of.

[flotus1]

Could be a lot going on here.
Why don't we start by laying out the necessary details:
1) Which hardware are we running now? Please be specific. Also about memory population
2) Which software? How many threads?
3) What's the CPU core clock speed while running the benchmark
4) What is this benchmark measuring exactly? Does it include single-threaded setup/meshing times? Is there a lot of file-I/O?
5) How is power consumption measured for the CPUs? Is that 140W for each of them, or 140W for both combined?

You could also run a few well-established benchmarks, in order to compare to similar hardware.
Before doing that though, make sure windows updates were completed.

[otokemo]

Originally Posted by flotus1

Could be a lot going on here.
Why don't we start by laying out the necessary details:
1) Which hardware are we running now? Please be specific. Also about memory population
2) Which software? How many threads?
3) What's the CPU core clock speed while running the benchmark
4) What is this benchmark measuring exactly? Does it include single-threaded setup/meshing times? Is there a lot of file-I/O?
5) How is power consumption measured for the CPUs? Is that 140W for each of them, or 140W for both combined?

You could also run a few well-established benchmarks, in order to compare to similar hardware.
Before doing that though, make sure windows updates were completed.

Ok, let's start
1) All the hardware is listed here 5965WX vs 7950X3D You said not to use Nemix, but it was too late, the company had already bought them. I used all 16 identical Nemix DDR4-3200 s.
2) Ansys Fluent, 64 cores.
3) It goes up to 3.29 GHz
4) The benchmark only calculates the Fluent setup, which consists of a quadratic tube and PCBs inside them, the Argon gas flows inside the tube, the PCBs produce heat and the gas is cooling them down.
5) Each of them use 145W seperately.

I've run Passmark, and got 64000 score. A single 7532 scores 53000 based on the Passmark ranking. Is it ok? I want to try OPENFoam benchmark, but I don't really want to go throught the struggle of installing virtual linux on my system. I will do it though, if it's really necessary.

[flotus1]

A scaling test could answer a lot of questions.
I.e. run the case on 1,2,4,8...threads and report the run times.

Also maybe relevant: https://forum.ansys.com/forums/topic...ow-bandwitdth/
I have not used Fluent in years, so I can't comment on this.

[otokemo]

Originally Posted by flotus1

A scaling test could answer a lot of questions.
I.e. run the case on 1,2,4,8...threads and report the run times.

Also maybe relevant: https://forum.ansys.com/forums/topic...ow-bandwitdth/
I have not used Fluent in years, so I can't comment on this.

I will try different version of Ansys, but is there a chance the problem is with Ram? or H12DSi-N6-O has a problem with M.2 nvme ssd?

[flotus1]

Originally Posted by otokemo

is there a chance the problem is with Ram? or H12DSi-N6-O has a problem with M.2 nvme ssd?

There are individual benchmarks for each of these components if you are curious.
Though if storage should turn out to be the bottleneck, the fluent case you are currently using would not be a good test for a 64-core workstation
That's why I asked if file-I/O might be holding you back.

[otokemo]

Originally Posted by flotus1

There are individual benchmarks for each of these components if you are curious.
Though if storage should turn out to be the bottleneck, the fluent case you are currently using would not be a good test for a 64-core workstation
That's why I asked if file-I/O might be holding you back.

Ok, this is very strange, I will open a different thread for this in Fluent sub-forum.
The more cores I use, the slower is the calculation speed.

[otokemo]

Updated plots
I tested the same Fluent setup on 3 different machines. Both AMD cpus 5800x and 2x Epyc 7532 are experiencing declining performance as the used core number increases. While, 2xE5-2697A has rising performance up to 8 cores (since the Fluent setup itself isn't very heavy).

[flotus1]

That's definitely something you need to contact Ansys support about.

In the Fluent launcher, can you select any other MPI implementations? What's the default setting?

[otokemo]

Originally Posted by flotus1

That's definitely something you need to contact Ansys support about.

In the Fluent launcher, can you select any other MPI implementations? What's the default setting?

There has been found a solution, which works well!
"The default setting for MPI doesn't work the best with AMD Threadripper. So we need some fine-tuning as follows:

In the launcher, set MPI to MSMPI

and use the following syntaxes in the environmental tab.

FLUENT_AFFINITY=0

MPIEXEC_AFFINITY=3:P



Using this setting, Fluent 2023R1 and Fluent 2020R2 perform similarly with comparable computational time. Please update me if you achieve the same improvement."