[roenby]

Dear hardware experts

I am about to buy a workstation for OpenFOAM simulations and have tried to learn from the threads in this forum. It is all quite confusing for a non-hardware expert like me, but as far as I have understood, memory band width is the main bottleneck with CFD.

The type of simulations I will do is transient interFoam (interfacial flows, with a VOF field, a pressure field and a velocity field to be solved for) and I will probably rarely go over 2-5M cells (if I need more, my plan is to look for a rented cluster).

I am considering buying a workstation with the new EPYC 7301 processor which gives me 16 cores, 8 memory channels (DDR4-2666), 64 MB L3 cache [1] at a reasonable price and with possibility to double all this at a later time.

I have found a similar configuration at a price of around 3.650 USD [2].

Here are the specs:

System: Supermicro 4023S-TRT Tower
CPU: 1 x AMD EPYC 7301 Processor
Memory: 8 x 4GB DDR4 2666MHz RAM
HDD 1: 256 SSD disk (unspecified)
HDD 2: 1T HDD (unspecified)
Power: 1280W Redundant High-efficiency Digital Power Supplies w/ PMBus 1.2

Now, can any of you give me an educated guess on how risky this configuration is, given that the EPYC processors are brand new and there are still no proper OpenFOAM CFD benchmarks available yet.

Could some architectural detail make the apparently very promising EPYC CPU specs disappoint with regards to CFD?

Would it be safer to make a similarly priced configuration with one of the new Intel SP processors?

Any advice or guidance will be much appreciated.

Kind regards,
Johan

[1] https://en.wikipedia.org/wiki/Epyc
[2] https://www.nextron.no/en/main.php3?...info&CID=40382

[flotus1]

I am currently waiting until Epyc CPUs and motherbards are finally available to build a similar setup. However, I am building it myself. So if I should be disappointed by the performance all I have to do is swap the motherboard and CPU(s).
But I guess the risk of Epyc performing much worse than anticipated is rather small. All benchmarks I have seen so far indicate that it meets the expectations.
What I would change is using at least 8GB DIMMs. They they should offer a better price/GB ratio

A word of warning: the workstation in the second link you posted will be loud as hell. Those supermicro cases and fans are not designed to be particularly silent. But the biggest problem is the power supply. In order to cram it into a small standard form factor, it is equipped with extremely noisy 40mm fans that will drive you crazy if this machine sits under your desk.

[roenby]

Thanks a lot for your quick and informative reply, flotus1!

I plan to stow the computer away in a technical installation shaft in my home but would still prefer that it is not ridiculously noisy.

I had a look again at you post here, and if I remove 1 CPU and its RAM and fan, then I get down to 2940$:

AMD Epyc:
E-ATX Case (~100$)
Power supply, e.g. Bequiet Dark Power Pro P11 550W (~130$)
Dual-socket board, e.g. Supermicro H11DSi (~550$)
1x AMD Epyc 7301, 16 cores (~1x 920$)
1x Noctua NH-U14S TR4-SP3 (~1x 80$)
8x8GB DDR4-2666 dual-rank (~800$)
GTX 1060 6GB (~270$)
SSD 250GB, e.g. Samsung 850 Evo (~90$)
+as much storage as you need, maybe even an m.2 SSD with 1-2 TB if you need fast file I/O for your transient simulations

With my current max budget 4000-4500$ this looks quite interesting with the option of adding a cpu at a later time.

I wouldn't mind building the thing myself. I had fun building my first computer many years ago and I played with Lego as a kid, so I guess it is doable :-)

In my communication with Velocity Micro, I was told that they are "against 2666MHz RAM, because in testing it is not as stable as the 2400MHz".

Any comment on that?

Kind regards,
Johan

[flotus1]

If you already built PCs yourself and you don't need support and warranty for the whole system you can build it yourself for a lower price.
The only caveat is that you will have to wait a few more weeks until boards and CPUs become available from retailers.

For the memory...
You can have a look at the motherboard manuals at supermicro: they officially support RDIMM DDR4-2400 dual-rank and RDIMM DDR4-2666 single-rank. This is identical to the official memory support of AMD Ryzen and Threadripper with one DIMM per channel. I am still waiting for a response from their support people to clear up a few of the questions I had.
But then again, dual-rank DIMMs have proven to deliver better performance than single-rank DIMMs with higher speeds, especially for AMDs new Zen architecture.
So based on the little information I have my recommendation is: if you can get dual-rank DDR4-2400 DIMMs significantly cheaper than dual-rank DDR4-2666 you can buy them. Otherwise just get DDR4-2666 dual-rank. Running it at lower speeds until memory support catches up is always possible.
Speaking of memory support: if you absolutely want to avoid any issues you can stick to the memory QVL of your motherboard. Other modules might work as well, but there is no guarantee.

[roenby]

I can wait a few weeks - no problem.

I had a look at the tested memory blocks here and there are only 16 and 32 GB blocks with 2666 MHz which is overkill if I am to fill out all the 8 slots (around 185$ per Micron 16 GB block). With 2400 MHz they have a Micron 8GB block but it is only single rank. Similar story with the other Super Micro EPYC 7000 series rmotherboards here.

Please let us know if you hear anything regarding supported dual-rank 2666 MHz memory from the support people while we wait for the boards to become available.

Thanks so much,
Johan

[roenby]

Finally, the new AMD EPYC 7301 CPU's and the compatible SuperMicro H11DSi motherboard are available in a shop near me (Denmark).

So I am ready to buy!

The only problem is the ram configuration. To populate all 8 ram slots per CPU, I need 8Gb ram modules. As far as I have understood it is important that it is 2666MHz and dual rank to get the most out of the system in terms of memory bandwidth. There are, however, no 8GB 2666MHz dual rank ram modules on the "Tested memory list" on SuperMicro's webpage.

Question:
Does someone have an educated guess of the chances of running into trouble if I just choose a 8GB 2666MHz dual rank ram module which is not on the "Tested memory list"? For instance this one?

Kind regards,
Johan

[flotus1]

Yes, with this memory type you will definitely run into trouble. The CPUs and boards require RDIMM aka registered ECC.
Unfortunately, I still haven't found any performance comparison between the maximum supported DDR4-2400 dual-rank and DDR4-2666 single-rank. However, if you buy DDR4-2666 dual-rank you can always run it at DDR4-2400 if the board is not happy with the higher frequency.

[roenby]

Thanks again for the quick reply, flotus1!

Just to make sure I understand you correctly: The trouble is because I accidentally linked to a non-ECC (EDIT: Non-RDIMM) type of ram. Not because it is not on the "Tested Memroy List" on SuperMicro's webpage. Right?

So this one for instance should be OK? Or would you say it is high risk to choose ram that is not on the "Tested memory list"?

If I have to buy 8x16 GB ram modules pr CPU this system is simply not an option for me since the price blows out of my budget.

Kind regards,
Johan

[flotus1]

The problem was not that you linked non-ECC memory, but "unbuffered" UDIMMs. I am currently running Samsung memory that is not on their tested memory list.
UDIMMs exist in ECC and non-ECC variants. Yet all RDIMMs also have ECC.

You might want to use a search engine to get better prices for your RAM
https://geizhals.eu/?cat=ramddr3&xf=...red+ECC+(RDIMM

Edit: the risk of buying memory that is not on the QVL is rather small, at least when you purchase as a private individual inside the EU. You can always return it within 14 days if it should not work at all.

[ErikAdr]

It is very difficult to find the right memory available anywhere. Though, at crucial.com, use the memory selector to find the right memory. The Supermicro motherboards H11DSi and H11DSi-NT are found under: Supermicro, Servers (not motherboards) in the memory selector. For Supermicro H11DSi-NT I got:
http://eu.crucial.com/eur/en/compati...icro/h11dsi-nt
A 8 GB dual rank stick that crucial guarantee works is
http://eu.crucial.com/eur/en/h11dsi-nt/CT10622691

Samsung have M393A1G43EB1 CTD (2666 Mhz) and M393A1G43EB1 CRC (2400) that also are dual rank 8 GB SDRAM that I believe(!) should work. They are not available any where at the moment.

[roenby]

Thanks, flotus1 and Erik

Using Erik's tip for the crucial.com ram selector, I end up with this choice for the SuperMicro's H11DSi board:

http://eu.crucial.com/eur/en/h11dsi/CT10622791

Same price: 103 EUR ex. vat and seems to be available.

Best,
Johan

[flotus1]

Just to make my point once more: As long as you stay within the specifications and choose a reputable brand (Samsung, Kingston, Crucial...) there is virtually zero chance that the memory will not work. No matter if Supermicro tested it or not.
By the way, being recommended and compatible does not necessarily imply that the memory will run at the rated speeds. With one DIMM per channel, AMD Epyc officially supports DDR4-2400 dual-rank and DDR4-2666 single-rank. Everything beyond that might not work, even if the module has been tested. Supermicro does not disclose the frequency at which the modules were tested. Running DDR4-2666 dual-rank is technically overclocking.

[roenby]

Thanks for clarifying.

So if the safe choice is between DDR4-2400 dual-rank and DDR4-2666 single-rank, do you have any recommendation for which one to choose?

[flotus1]

Unfortunately not.
Judging from the benchmarks for Ryzen CPUs, dual-rank yields higher performance than slightly higher clocked single-rank. Now I am tempted to extrapolate these results to Epyc because it is basically four Ryzen dies in one CPU. But since the inter-die communication speed is linked to the memory speed (something that is not present in Ryzen CPUs because it is only one die) this extrapolation might fail for some workloads.
If I had to buy now, I would still go for DDR4-2666 dual-rank and see if it works. If it doesn't, I would run it at DDR4-2400 and wait for memory support to catch up with future bios updates. Reminds me that I need to try the latest bios version 1.0b for my motherboard, it has a new agesa version...
In the end, we are probably talking about less than 5% performance difference DDR4-2400 DR vs DDR4-2666 SR. So dont't overthink, it will still be much faster than anything Intel has to offer in this price range.

[roenby]

Hi guys

Thought I'd pass on some of my experiences with building an EPYC based workstation.

I ended up with this configuration:

- 1 x AMD EPYC 7351 / 2.4 GHz Processor CPU
- Supermicro H11DSi motherboard
- 8 x Samsung 8GB Module DDR4 2666MHz ECC Reg_ (M393A1K43BB1-CTD)
- ASUS GeForce GTX 1060 6GB DUAL graphics card
- Corsair Obsidian 750D Airflow - Chassis - Fulltower
- Noctua NH-U14S TR4-SP3 slim CPU cooler (came with thermal paste so no need to buy separately)
- 2xSeagate BarraCuda 3.5" ST2000DM006 Harddisk - 2 TB
- 1xSamsung 850 EVO SSD - 500GB
- Be Quiet Dark Power Pro 11 550W (BN250) power supply
- Ubuntu 16.04 LTS
- No overclocking
- SMT (mutlithreading) turnd off in bios


Here are some experiences from my build process:

1. The screw hole positions in the SuperMicro H11DSi motherboard are apparently non-standard, which meant that I had to carefully drill holes in my new chassis. Not having build a new PC for 15 years, this was quite far out of my comfort zone. But it eventually worked out well. Be careful to keep all electronic parts away to avoid while drilling in order to protect them from the generated metal grains.

2. I used the Noctua NH-U14S TR4-SP3 slim CPU cooler for my CPU. Apparently this is a low RPM cooler design for low noise levels. But the motherboard kept flashing a red LED2 periodically (couple of secs period) with the fan speed going up and down periodically. This problem was solved by changing the fan speed thresholds like described here.

3. My motherboard came without an I/O shield. So now I have an air gap at the back of my chassis around the I/O ports where dust can enter. Not optimal. Will buy one soon.

4. My motherboard came without proper cabling for harddrives. Had to buy a Supermicro SMH CBL-0188l Cable separately.

5. To get proper speed up in this benchmark, I had to turn off SMT (multithreading) in bios.

6. Currently I am facing problems with annoying lagging especially with Firefox when typing in google docs and scrolling webpages etc. I have tried with different graphics drivers, but have not really found a solution to this annoying problem yet.

7. I really cannot recommend the Samsung UD590 Series U28E590D 28" display. It is cheap plastic and makes an annoying (low but noticeable) buzzing/beeping sound when in sleep mode.

8. I initially bought the Phanteks Enthoo Pro M Tempered Glass - Black - chassis which is officially E-ATX. But it turned out that the SuperMicro H11DSi motherboard was too wide. So had to return this chassis :-|

9. If you only have a single CPU, be aware that you might have to place it in the CPU1 slot (not the CPU2). Also you might not be able to use all the harddrive connectors on the motherboard as the ports on the motherboard might be associated with a particular CPU slot (see motherboard manual).

I hope this info will help someone wanting to build an AMD EPYC based workstation.

Kind regards,
Johan

[roenby]

Hello hardware experts

I am generally happy with my new AMD Epyc 7351 based workstation build. Due to budget constraints I initially only bought 1 CPU and 8x8Gb ram although my SuperMicro motherboard supports two CPU's.

Now I want to buy an extra Epyc 7351 CPU and another 8x8Gb ram. My question is now, if it is vital that I buy exactly the same brand and type of ram that I already have for my first CPU? Or do I have freedom to buy something else as long as the specs are as good or better than my existing ram?

Hope you can help me out here.

Kind regards,
Johan

[flotus1]

Concerning your issues with the motherboards physical dimensions and screw holes: I had the same issues with SM boards in the past. Except when I was using the Phanteks Enthoo Pro case which is large enough to for this motherboard and also has standoff holes for Supermicros weird standoff hole pattern.
Unfortunately, "E-ATX" is not the same physical dimension for all case manufacturers. For some it is just a slightly wider variant of ATX. Look out for the actual supported dimensions of the board which should be 12"x13" for full E-ATX aka SSI-EEB.

I think even the bulk version of the motherboard should come with an I/O shield. Ask your vendor to ship one.
Additional cables for HDDs are only part of the retail version of the board. I had ordered retail but mine came without these cables. The seller admitted that because there is a supply shortage, they send out retail versions with the content of the bulk version - and hope that no one notices. They agreed to send the cables free of charge.

Concerning RAM I would not take any risks. Sure, it will run with other DIMMs. But considering how picky AMD CPUs are when it comes to RAM and the impact it has on performance, I would avoid using two different types whenever possible.

[roenby]

Thanks for the quick response, flotus1!

Concerning RAM type and brand you previously wrote:

Just to make my point once more: As long as you stay within the specifications and choose a reputable brand (Samsung, Kingston, Crucial...) there is virtually zero chance that the memory will not work. No matter if Supermicro tested it or not.

Have you learned anything new since you wrote this regarding pickiness of AMD CPU's? Any links or references here?

Best,
Johan

[ErikAdr]

I just saw that computersalg.dk have the exact same type of memory in stock as you already are using. (689 DKR + moms)


Best Erik

[roenby]

Originally Posted by ErikAdr

I just saw that computersalg.dk have the exact same type of memory in stock as you already are using. (689 DKR + moms)


Best Erik

That is 100 kroner more per block since my last purchase. Also this ram is only single rank. I was considering if I could find dual rank 2666 MHz ram instead for the 2nd CPU. But will follow flutus1's advise and stick to what I have.