[Simbelmynë]

Hey hardware experts with EPYC systems. Before hitting the purchase button, , I would like to know if there might be any potential problems with the following setup. For instance, I am a bit worried that the motherboard might buckle under the torque from the stress of two rather heavy Noctua heat sinks.


16 x Kingston Server Premier - DDR4 - 16 GB - DIMM 288-pin - 2666 MHz / PC4-21300 - CL19 - 1.2 V - reg. - ECC

Seasonic FOCUS Plus 850 Platinum - ATX12V - 80 PLUS Platinum - AC 100-240 V - 850 Watt - aktive PFC

Fractal Design Define Series R6 - Tower - extended ATX

2 x AMD EPYC 7301 - 2.2 GHz - 16-core - 32 threads - 64 MB cache - Socket SP3

Samsung 850 EVO MZ-75E500 - Solid state drive - 500 GB - 2.5" - SATA 6Gb/s - buffert: 512 MB - Self-Encrypting Drive (SED), TCG Opal Encryption 2.0

2 x Noctua NH-U9 TR4-SP3 - (for: TR4, SP3) - 92 mm

Supermicro H11DSi - AMD SP3 socket - DDR4 RAM - Extended ATX

[flotus1]

The CPU coolers are relatively small and light, I expect no problems here.
SSD: Samsung 860 Evo instead of 850 Evo
Case: that motherboard won't fit inside a Define R6 without modifications. The board has SSI-EEB form factor, The case is only suitable for a slightly wider variant of ATX - often falsely referred to as E-ATX. I can highly recommend Phanteks Enthoo Pro if you swap out the fans.
Memory: this is dual-rank memory? It is still not officially supported to run at 2666MT/s on Epyc. Supermicro has some DR modules listed with that speed. Maybe stick to those if you want to be on he safe side.
No graphics card required?

[Simbelmynë]

Perfect, thanks Alex!


So I was under the impression that 16 GB modules were always dual rank (and 8 GB were a toss of a coin dual/non-dual rank). And in this case I found a really good deal of about 110 Euro per module (most other 16 GB ECC 2666MHz memories start at 180 Euro here in Sweden).


However, you are right, I should stick with the recommended memory from Supermicro.


I thought 2666 MHz memory was supported since you recommended to buy that in your benchmark of EPYC.


Finally, with regards to the GPU. I have not yet decided if we will go with a quadro card for some nice VR post-processing (Star-CCM+). Perhaps a regular high-end GPU will work anyways?

[flotus1]

16GB DDR4 UDIMM modules are still exclusively dual-rank. But with DDR4 RDIMM, 16GB modules can be single-rank or dual-rank.
If you can get them for this price, go for it. 2400MT/s dual-rank won't be slower than 2666MT/s single-rank. And if you are lucky, you can run them even faster. With one DIMM per channel, Epyc officially supports 2666MT/s SR and 2400MT/s DR. But there have been reports of 2666MT/s DR running with later bios versions.
I recommend 2666MT/s because it usually does not cost much more. And with more mature bios versions, memory support should improve over time. Mine still only runs at 2133MT/s. To be fair, this is what it is rated for. But I had the same modules running in an Intel system with 2666MT/s without any issues.
GPU-wise...If CCM+ does not lock consumer GPUs with a whitelist like Ansys does for some of their products, you should get more bang for the buck with a consumer graphics card. I lack insight here, better check with their customer support.

[Simbelmynë]

So I have made the changes according to your suggestions. I had to change the memory (it was indeed only 1R in the original setup). I have searched for lower capacity memory with 2R, but it seems only 16 GB modules are available.


Anyways my final (?) 7301 EPYC setup have 16 x 16 GB Kingston KSM26RD8/16HAI memory modules that are 2R.


With the changes to SSD and case and also with a placeholder 1050 4GB GPU, I land on approximately 5000 Euro. Only the SSD and PSU are in stock so I have no idea on how long it will take for delivery.


--



The other option that I am considering is to buy 2 systems with:


2 x Intel Xeon Silver 4114 and 48 GB of RAM (2666 MHz, only 1R), X11DAI-N



Similar components as the EPYC system in the other slots.


The price for each of those systems land on about 2500 Euro and everything is in stock except the case.



A dual card mellanox 40 Gbps including cable is approximately 100 Euro on Ebay.


What is the most sensible choice here? We normally run cases with fewer than 1 million cells (usually takes 1-2 weeks on a 7940X computer). Will the 32 core / 16 memory channel EPYC beat the 40 core / 24 memory channel Xeon mini cluster?

Edit: Come to think of it, I could also go down to 64 GB 1R memory on the EPYC system. That would decrease the cost by approx. 1800 Euro, perhaps making it the most cost efficient option?

[flotus1]

I didn't know that the models you run have such a small cell size. Buying 256GB of RAM just to get dual-rank would be something worth questioning.
I mean we are talking about a performance difference in the order of 10% between dual- and single rank. So 16x4GB instead would be the more cost-efficient option.
Caveat: With only 8GB of RAM per die, performance for lightly-threaded applications that use more than 8GB of RAM will suffer. Not sure if that affects your typical workload.
Btw, there are 8GB DDR4 RDIMMs with 2 ranks: https://geizhals.eu/?cat=ramddr3&sor...red+ECC+(RDIMM)

If you want to get an idea how well your typical simulations perform on dual 7301 you can send me a sample case I can test along with some instructions. I won't get to it before next week though.

[Simbelmynë]

If you could run one typical case that would be perfect.



I think that our cases are most close to the dam break tutorials in OpenFOAM. I'll see if I can set it up in a similar manner that the previous benchmarks were laid out.


I might end up with the 256 GB option anyway since we are also planning some non-CFD calculations that actually do require a large amount of memory.


Btw, you are a Fluent user right? This is what most of our PhD students use in their research, so a Fluent test would be fantastic.

[flotus1]

I would not consider myself a Fluent user, but I am familiar with it and used it occasionally in the past.
Benchmarks are here: AMD Epyc CFD benchmarks with Ansys Fluent
I can not run more due to lack of licenses.

[flopiflopi]

Hi, I would also like to build myself a workstation for CFD work with OpenFOAM.
I am thinking about following configuration:

-2 x AMD Ryzen Threadripper 2990WX, 32x 3.00GHz
-Supermicro H11DSi-NT bulk
-16 x Kingston Server Premier DIMM 16GB, DDR4-2666
-SanDisk Ultra 3D 1TB
-Chieftec LBX-02B-U3 black
-Seasonic Focus Plus Platinum 850W ATX 2.4
-Noctua NH-U14S TR4-SP3
-MSI GeForce GTX 1060

do you might think there will be any problems with this configuration?

[flotus1]

Yes, there will be problems.
TR 2990WX is not dual-socket capable. And even a single one won't work on that motherboard. Plus it is a horrible choice for CFD/OpenFoam: It only has 4 memory channels and half of the cores have no direct path to memory. They have to access memory over infinity fabric which imposes a huge penalty on bandwidth and latency.
For parallel CFD, even a single Epyc 7301 easily beats a TR2990WX. So use 2x Epyc 7301, 7351 or 7401 depending on your budget. The rest of the system seems ok, though I have no idea i the case is wide enough for these coolers.

[flopiflopi]

Thank you very much flotus1 for your reply. I was so impressed by the number of cores per €, I overlooked everything else.

[flotus1]

I cannot help but thinking this might have been one of the motivations for AMD to throw this CPU on the market. Bragging rights for highest core count.

[edomalley1]

This thread has been super-helpful. I've been doing RANS OpenFOAM simulations on my HP Specter x360 convertible notebook with an i7 8550u (4 cores) and 16GB of memory with no problems (90 minutes to run) but now I have to do LES and am running the motorbike tutorial as a test run. 20 hours in and I think I have at least 10 hours to go before I am done!!!

I've built a NewEgg wish list based on Simbelmynë's machine and flotus1's modifaction suggestions. But NewEgg is pricing it out at almost $6k! I'm sure I would get a pretty big performance boost using only one AMD Epyc 7301 16 core and 8x16MB of RAM, right? Any gut feeling on speed increase from my little laptop vs. Simbelmynë's machine vs. just 1 processor and 8x16?

Thanks!

[edomalley1]

Oh - one other dumb question. My notebook is windows 10 on which I just run an Ubuntu terminal. I assume with a machine like this it's best just to make it an Ubuntu machine?

[flotus1]

As an estimated ballpark figure, a single Epyc 7301 should run faster by a factor of ~3. For bare metal performance comparisons.
I am not familiar with the kind of virtualization you are using, but it does sound sketchy. https://www.linux.com/learn/howdy-ub...ws-how-fast-it outlines that especially memory performance can suffer. If you want the best performance, use the operating system on bare metal.

[edomalley1]

No 20 pin USB 3.0 header on H11DSi?

Would you suggest to connect front panel USB 3.0 ports to get an adapter to plug into the back panel of the MB? Would have to do something janky to get the cable out of the case.... Or do you just leave those ports inactive?

[flotus1]

If I ever need front USB3.0 I would probably use a PCIe-card to enable it. There are not really many USB ports on the board itself, I would not like to lose any of them.
My plan is still to replace the motherboard if something more "workstation" becomes available from any other brand than Supermicro.