[ifil]

Hello everyone!,

I wish any of you could give me his/her opinion on how to choose the right workstation configuration. The machine will be mostly used for CFD computation (starCCM+) but also FEM from time to time. The simulations types will be mostly external and internal non-compressible flow (marine applications including 6DOF) and not very often though, heat transfer analysis. I do not expect to have mesh sizes larger than 40M (usually smaller) cells. I have received some configurations options and the quotations along. I am wondering if any of you could let me know which option is better or if you could suggest even a better one according to the job that the machine will be performing.

The budget for the workstation is max 12k USD

So far the options received (within the budget) are:

MAIN SPECS

Option 1)

Motherboard:Supermicro X11DAi-N - Intel C621 Chipset - Socket
LGA 3647 Workstation Motherboard
CPU / Processor:2x Intel Xeon Gold 6138 SP 2.0GHz (3.7GHz
Turbo) 125W 27.5MB Cache (40 Coress (80 Threads)
Memory:128GB (8 x 16GB) 288-Pin DDR4 SDRAM 2666
(PC4-21300) ECC Server Memory
Video Card:NVIDIA Quadro P4000 8GB GDDR5 PCIe 3.0 x16
Workstation Video Card
M.2 / U.2 NVMe Drive:WD Black 512GB M.2 NVMe PCIe 3.0 x4
Solid State Drive - read/write 2050/800 MBps
SATA Drive 1:Seagate 2TB 7200 RPM 64MB Cache SATA 6.0Gb/s
3.5" Internal Hard Drive

Option 2)

Motherboard:Supermicro X11DAi-N - Intel C621 Chipset - Socket
LGA 3647 Workstation Motherboard
CPU / Processor:2x Intel Xeon Gold 6140 SP 2.3GHz (3.7GHz
Turbo) 125W 27.5MB Cache (36 Cores / 72 Threads)
Memory:256GB (8 x 32GB) 288-Pin DDR4 SDRAM 2666
(PC4-21300) ECC Server Memory
Video Card:NVIDIA Quadro P4000 8GB GDDR5 PCIe 3.0 x16
Workstation Video Card
M.2 / U.2 NVMe Drive:WD Black 512GB M.2 NVMe PCIe 3.0 x4
Solid State Drive - read/write 2050/800 MBps
SATA Drive 1:Seagate 2TB 7200 RPM 64MB Cache SATA 6.0Gb/s
3.5" Internal Hard Drive

The option 2 has a slightly higher price however, still within the budget.

Has anyone experience with the above mentioned CPUs?

Thank you very much!

[flotus1]

Which retailer put together these configurations?
Skylake-SP CPUs have 6 memory channels each. By using only 8 DIMMs total, a large portion of the performance is wasted. Rendering the choice of CPU more or less irrelevant.
Both workstations should be equipped with 12 identical DIMMs (dual-rank if possible) instead of 8. And after they have been shipped I would check if they put them in the right slots.

[ifil]

Hi flotus1,
thank you for your prompt reply, well, I have to admit that am not a super expert in hardware architecture. I believe they may have just made a mistake then, I can still ask to rectify.
Nevertheless if the config can be corrected, what is your opinion on the two options? Do you have a better suggestion that would still fall withing that budget?

Thank you

[flotus1]

The most versatile choice would probably be a Xeon Gold 6146 thanks to its very high clock speeds. It is a little on the expensive side though, so settling for a 6136 instead yields a better price/performance ratio.
Here is a very useful article that also lists all-core turbo frequencies for these CPUs wich are more relevant than base clock and single-core turbo https://www.pugetsystems.com/labs/hp...rs-Guide-1077/

AMD Epyc is also an option. I have not seen any benchmarks with CCM+ directly, but results for other CFD solvers look very promising.

I believe they may have just made a mistake then

In my opinion, this is not a simple mistake from the retailer. When I buy directly from a retailer and pay the premium they charge, I expect them to know what they are doing and not cripple performance to last-gen levels. Otherwise I could just build the system myself and save a few thousand euros.

[ifil]

Thank you very much again,

I will investigate with the retailer the issue you mentioned for sure then.

Besides that, I see that also the 6148 comes out to be a good option from the article you linked (very interesting btw), but still rather expensive...

I have been going through the benchmark between intel and AMD for fluent and it shows that AMD is a very good choice too! I don't know though how much difference there might be between fluent and CCM+...this may be a tricky.

Do you happen to have an idea of what could be a good choice with AMD Epyc then?

thank you

[flotus1]

The 16-core models offer the best price/performance ratio for CFD. More cores do not increase performance very much, they mostly use more licenses.
With your budget Epyc 7351 seems to be the only reasonable choice since it is the fastest CPU with 16 cores and only costs around 1100$ each.
Notice to myself: consider selling reasonably configured workstations as a side job

[ifil]

Thank you very much Flotus1!
I will have some dealer preparing some quotations comparing the Epyc procs. I am very curious with what they will come out with.

Thank you!

[LuckyTran]

Wow, I had to verify and see it for myself. Sure enough, a Supermicro X11DAi-N really only has 2 banks of 4 DIMMS.

I would dump both setups and find a budget single node Epyc with 8 DIMMS or a better motherboard that isn't crippled. A $12k budget is respectable and you deserve much more than this.

[flotus1]

Are you sure about that? The images and descriptions on SMs homepage suggest it has enough slots https://www.supermicro.com/products/...0/X11DAi-N.cfm
AFAIK, the only dual-socket boards for socket 3647 that only have 8 DIMM slots total are Supermicro X11DPL and Asus Z11PA-D8.

[LuckyTran]

Ahh I only counted and did not notice the black bars. So it supports 2 banks of quad channel ram or 1 bank of six channel ram. Well that is a lot better in terms of specs. But I still have to call out the retailer for not offering an option that exploits the six memory channels when there's plenty of room.

So is it possible to negotiate a setup with
192GB (12 x 16GB) 288-Pin DDR4 SDRAM 2666 ??

Or you could buy either, find 4 more RAM sticks and throw them in.

The real difference between OPTION 1 and 2 is the overall RAM capacity. I generally say 4GB for every million cells if your cases are simple, more if you plan to run a lot of models. So I would decide based on planned usage. If you plan on doing a lot of big cases, go for option 2. Otherwise option 1 will save you some money. In either case, you could always add more RAM later if you find you need it.

[trampoCFD]

I can do https://trampocfd.com/collections/wo...es-workstation
with 2x AMD EPYC 7351 processors for your budget.

gui@trampo

[ifil]

Hi Alex,

few months have passed but we finally got the budget to finally go ahead with the workstation.

We managed to find these main specs falling within our budget:

Supermicro 4U CHASSIS 8X3.5 3X5.25 SAS3 1280W (1+1) PLATINUM 7FP EATX (CSE-745BAC-R1K28B2)
Supermicro H11DSI-NT EPYC7000 DDR4 M2 VGA 2X10GBE 10XSATA RETAIL (CSE-745BAC-R1K28B2)
8 x Kingston Ram KSM26RD8/16HAI (16GB 2666MHz DDR4 ECC Reg CL19 DIMM 2Rx8 Hynix A IDT)
2 x AMD EPYC 24-CORE 7451 3.2GHZ SKT SP3 64MB CACHE 180W WOF
RADEON PRO DUO 32GB GDDR5 PCIE 3.0 16X 3X DP HDMI RETAIL
2 x Supermicro 4U Active CPU Heat Sink Socket OLGA4094 (SNK-P0064AP4)
Seagate ST2000NM0125 3.5 2TB, SATA 6Gb/s, 128MB, 512E
Samsung 1TB NVMe SSD 970 Pro Series M.2 PCI-Express [MZ-V7P1T0BW]



Any comment?

Do you think this may be a good option?

Thank you in Advance!

[flotus1]

Yes, I have a few comments.

8 x Kingston Ram KSM26RD8/16HAI (16GB 2666MHz DDR4 ECC Reg CL19 DIMM 2Rx8 Hynix A IDT)

With 2 AMD Epyc CPUs, you need 16 DIMMs to populate all memory channels. So either double the amount of memory to 16x16GB or step down to 16x8GB.

RADEON PRO DUO 32GB GDDR5 PCIE 3.0 16X 3X DP HDMI RETAIL

Are you 100% certain that any of your software benefits from such an exotic dual GPU? And that it plays nicely with the rest of the software you use?
Otherwise I would play it safe and chose a midrange Quadro card, e.g. P4000. Or at least a single-GPU from AMD.

Seagate ST2000NM0125 3.5 2TB, SATA 6Gb/s, 128MB, 512E

No need for more local storage? A single 2TB HDD seems rather small.

Be warned that these machines can be awfully loud. The supermicro chassis uses small fans with up to 9400rpm, the power supplies have an even smaller fan with even higher rpm and their CPU coolers are not on the quiet side either. Not an issue if it is sitting in a separate room, but you probably would not want this sitting under your desk if you are sensitive to noise while working

[ifil]

Hi Alex,

thank you for your prompt reply! Well regarding the Graphic card: we are planning to use the workstation also for some graphic rendering every now and then so this is the reason why we decided to go for such a exotic one.
Regarding the memory I will contact the dealer and see what he can do.

Do you think that the DIMMs population may heavily affect the computing performance? I am sorry for asking this question but I am not an expert on this specific matter.

Thank you again.

Cheers

[ifil]

Sorry I almost forgot,

regarding the memory, regarding the storage we will use an external server to store the executed jobs. So this is the reason why we didn't go for a very large one.

Thanks

[flotus1]

Originally Posted by ifil

Do you think that the DIMMs population may heavily affect the computing performance?

Using all 16 channels vs. only 8 will effectively double the performance of the machine for bandwidth-limited applications like CFD or FEM. With 48 cores total you can easily max out the memory bandwidth of all 16 channels.