[hsnyder]

Hi everyone. I'm considering a workstation build, since the commodity hardware that I currently use (i7 etc) can take weeks to finish some of the heavier simulations I do (typically combustion simulations). Most of the simulations I do involve heat transfer in one way or another, and some also include chemical reactions. Right now, I exclusively use OpenFOAM (mainly rhoPimpleFoam and rhoReactingFoam) but I'm interested in expanding (for certain applications) to Nek5000 (spectral element method) and various Lattice-Boltzmann solvers. So, I'd like to make the right hardware choices with those factors in mind. I have no plans to use commercial software at this point.

Let's start with the CPU(s). I'm thinking a two-socket Epyc workstation. Here are the processors that I'm considering right now, but I am totally open to other options.

Epyc 7352 (24c, 2.3-3.2 GHz)
Epyc 7302 (16c, 3.0-3.3 GHz)

Basically I'm weighing the cost and base clock advantage of the 7302 against the higher core count of the 7352. I understand that the conventional wisdom is that most OpenFOAM work is limited by memory bandwidth, and so Epycs in the 16-24 core range tend to be the sweet spot. Does the fact that I sometimes do rhoReactingFoam simulations, or the fact that I'm interested spectral element methods / Lattice Boltzmann methods influence that at all? For example, does clock become more important given any of those considerations?

I'd be curious for any and all advice regarding CPU selection. Thanks.

[flotus1]

I would say that with your applications in mind, the 7352 is hands down the better choice compared to the 7302. And when factoring in the total system cost, the price difference between the CPUs becomes less pronounced.
While base clock makes it look like the 7352 has a clock speed disadvantage, you have to keep in mind that these CPUs never really operate at base frequency. When running 16 cores, both CPUs will turbo to very similar frequencies. And the 7352 still has 50% more cores for situations where cores are more important than frequency.

[hsnyder]

Hey, thanks for the quick reply / the advice. What's your reasoning, just so I understand what the relevant factors are? I'm guessing it has something to do with the increased workload per iteration compared with incompressible/isothermal flow, but I'm not well versed in these sorts of assessments... Would going to a 32 core processor be a good idea?

[flotus1]

My reasoning for higher core count CPUs when using open source parallel CFD codes: workloads are rarely 100% memory bandwidth limited at 2 cores per memory channel. Scaling might get worse, but there is still some more performance to be had with higher core counts. The more computationally expensive your models get, the better.

If I had to buy a new AMD Epyc setup now, and I had to buy new retail CPUs, my choice would probably be the 7452. With the 7352 coming in at a close second place. At least if your budget allows the 32-core CPUs without compromising on other aspects of the machine.

[hsnyder]

Ah I see. I don't have to buy epyc, nor new retail CPUs - I'm open to alternatives if there are better ones. I've just been hearing good things about the epyc rome chips. Is something else more advisable?



I can upgrade to a 7452 if I start with one socket filled and fill up the second one a few months down the road.

[flotus1]

Don't get me wrong, Epyc Rome is the best choice. I just meant that you don't necessarily have to pay new retail prices for these CPUs, when you are building a system yourself https://www.ebay.com/itm/100-0000000...cAAOSwnTBeZjF3

[hsnyder]

Excellent point. So...

CPU: 2x Epyc 7452

Motherboard: the only appropriate dual socket epyc board that I know of is the H11DSi from supermicro, so I guess it'll be that.

RAM: What I understand from reading elsewhere on the forum is that I should be looking for 3200MHz dual rank memory... perhaps this: https://www.newegg.com/p/1X5-003Z-017J5 (I would of course need one set per CPU).

CPU Coolers: I've seen the Noctua NH-U14s recommended for epyc processors in this TDP range elsewhere on the forum, so I figured I'd go with that for CPU cooling.

GPU: Nvidia GTX 1070 (I already have it).

Storage: Various SATA SSDs I have lying around.


Case and case fans: Unsure, still looking into what fits.



I'm still drawn to the idea of filling one socket and upgrading to two down the road, from a budget standpoint. If I were to do that, I'm curious what the power supply situation would be... Should I just get a ~750 watt power supply right out of the gate, and run it at fairly low load until I fill the second socket, or should I use a lower power unit until I get the second CPU? I have a 500W 80+ bronze supply lying around, would that be acceptable for 1 CPU, or is that too sketchy?

[flotus1]

Sounds like a plan. Though I would be a bit sketched out by the memory manufacturer. I know the situation seems dire on newegg. Maybe you can find a more reputable brand on a different site. 16GB DDR4-3200 reg ECC should cost in the range of 100$ per DIMM
https://geizhals.eu/?cat=ramddr3&xf=...=p#productlist

Starting with only one CPU and half the memory should be possible. You probably could use your existing power supply in the beginning, but with all the savings from memory and CPU, I personally would not bother with that. Get a proper power supply, with two 8-pin EPS connectors.

Noctua NH-U14s TR4/SP3 CPU coolers are fine for this.
For the case, my default recommendation is Phanteks Enthoo Pro. It is one of the very few PC cases that can fit SSI-EEB motherboards without any hassle, and doesn't look too gamer-y. For the fans, Noctua is always a good choice. Arctic has some good value alternatives https://www.arctic.ac/de_en/f14-pwm.html

Edit: be sure to get a rev. 2 of the H11DSi. Apparently, there are ways to make Epyc Rome work on rev. 1, but you can easily avoid that hassle.

[hsnyder]

Thanks again Alex. I'll keep all of that in mind.



The case seems like a great pick, and affordable too.


Regarding the memory, keep in mind what I linked was for a set of 8 DIMMs not 16, so it's about $96 per DIMM... Still sketchy? Either way, I'll look around for alternatives as per your suggestion.

[flotus1]

It wasn't about the price. That's about what you would expect.
The thing is, Nemix just puts whatever ICs they want on memory modules. Which is why they will never end up on a supported memory list for server boards. Or a list of "reputable" memory manufacturers for that matter.
If you can return the memory in case it is not compatible, you can give it a try. And if you have an easy way to handle a return in case of premature failure.

[hsnyder]

Got it. I'll avoid them. Thank you.

[hsnyder]

I've been sitting on this for a few months hoping the Canadian Dollar would go up, but I could use the upgrade so I'm getting ready to pull the trigger. I ended up going with 2x 7302, because I found a great deal on them, but I haven't bought the rest of the system yet.


The rest of the build is pretty much planned out, but I just had this thought... now that we have some more actual performance data for the Rome chips in the OpenFOAM hardware thread, it looks like 2 cores per memory channel does not totally saturate the memory bandwidth of these CPUs. That being the case, and since 128 GB would be enough RAM for me, I'm considering going with single rank memory (8GB sticks). I figure that if the full bandwidth of 16x dual rank sticks isn't being used at 32 cores, then I'm unlikely to see the ~10-15% performance improvement that dual rank brings and there's little reason to go dual rank. Does that sound reasonable or am I missing something?

[Simbelmynë]

There are 8 GB modules with dual rank. They are more difficult to find, but the price should not be very different compared to single rank so I see no reason to opt for single rank.

[hsnyder]

Hey thanks for the response. Do you happen to know of a manufacturer for them? I haven't been able to find them so far.

[Simbelmynë]

No, it was a few years back when I ordered an EPYC with (only) 128 GB of ram. I remember it took quite a lot of Sherlock Holmes-ing in order to find it. The 7301 was also a bit worse with the memory compatibility so it was a bit of a cross-fingers-and-hope situation, that in the end worked perfectly.

https://geizhals.eu/?cat=ramddr3&xf=...al%7E5015_3200

Perhaps the Micron memory is ok?

Terrible latency, but this is perhaps where we stand @ 3200 MT/s, dual rank and Reg. ECC memory.