Suggestions for GPU/CPU ("Shared") Workstation

JM92 · May 30, 2022, 09:49

Hi all,

I got my hands on a high-performance nVidia GPU, which will be used for SPH simulations. As I will be able to build a workstation for roughly 3k€ (excluding GPU) around it, I'm currently thinking about setting this up for a second "use case", which would be simulations with openFoam. For this I'd currently aim at one Epyc 7313P or 7343 with (at least) 64 GB of RAM.

Generally, I think the information given in this forum would be sufficient to build a CPU-only workstation without further help, but I just wanted to ask if anyone has experience with this combination of two use cases (SPH on GPU, openFoam on CPU [of course both simulations will not be run at the same time)] and knows about any pitfalls to avoid.

E.g., I'm unsure about including a motherboard with onboard GPU to minimize the "additional" load on the nVidia card while running SPH simulations.

Here's the checklist (focusing on the non-GPU part)

Which software do you intend to use? --> openFoam
Are you limited by license constraints? I.e. does your software license only allow you to run on N threads? --> No license constraints
What type of simulations do you want to run? And what's the maximum cell count? --> 2D and 3D, mainly two-phase flow (interFoam), max. No. of cells 10-20 million (but a bit unsure what the future may hold in that matter)
If there is a budget, how high is it? --> 3k €
What kind of setting are you in? Hobbyist? Student? Academic research? Engineer? --> Academic researcher
Where can you source your new computer? Buying a complete package from a large OEM? Assemble it yourself from parts? Are used parts an option? --> Planning to assemble it myself
Which part of the world are you from? It's cool if you don't want to tell, but since prices and availability vary depending on the region, this can sometimes be relevant. Particularly if it's not North America or Europe. --> Europe
Anything else that people should know to help you better?

Kind regards

flotus1 · May 30, 2022, 10:51

Onboard graphics is not something you need to worry about here. As far as I know, all motherboards compatible with Epyc 7000 CPUs come with some basic Aspeed graphics processor on board. The connector is only VGA though, and of course the processing power is severely limited. Displaying some console windows works fine, anything beyond that doesn't.
It might be a good idea to add a small Nvidia graphics card for good measure. Prices are still high, but not as obscene as they used to be. 290€ for a GTX 1660 for example.
That will allow you to use all the available memory of your highend GPU for computations, and generally simplify the setup.

Speaking of pitfalls and unnamed "high-performance nVidia GPU"
If it's PCIe 4.0, make sure your motherboard supports that. And beware of cooling for the GPU. Most of the Nvidia datacenter cards are passively cooled, relying on airflow from the chassis fans for cooling. They can still be used in a workstation, but require some thought and modding effort put into a custom cooling solution.
And if it's just a regular cooling solution with axial fans, beware that all of the heat gets dumped into the case, so plan for that with plenty of airflow.
And obviously, the power supply has to match the highend components.

As for the plan itself, you will probably get better answers in the OpenFOAM sub-forum.

JM92 · August 1, 2022, 08:21

Dear flotus1,

thanks for your reply, I continued the "design" of the workstation and would like to ask for one final opinion on the parts before buying them:

Mainboard: Supermicro H12SSL-NT
CPU: AMD Epyc 7313P
GPU: nVidia RTX A6000 (which is actively cooled)
RAM: 8 x 16 GB Kingston Server Premier RDIMM 16GB, DDR4-3200, reg ECC, dual rank, x8
SSDs: Crucial P5 Plus 1 TB + Crucial P5 Plus 2 TB
CPU Fan: Noctua NH-U14S TR4-SP3
Case: be quiet! Silent Base 601 schwarz, schallgedämmt
Power supply: be quiet! Straight Power 11 Platinum 850W ATX 2.51

Especially the dimensioning of the power supply gave me some headaches, but as CPU and GPU won't be running on full throttle at the same time, 850 W should be sufficient in my opinion.
Additionally, I'm thinking about following your advice of adding another small GPU to the setup, but no decision was made on that yet. Maybe a small nVidia T400 or T600 would be sufficient.

Kind regards

flotus1 · August 1, 2022, 12:10

Looks fine to me. The RTX A6000 doesn't have regular PCIe power connectors, but requires an 8-pin EPS connector. The PSU you picked has two of those, one for the motherboard, and one for the GPU. And the GPU should come with an adapter 2x8pin PCIe -> 8pin EPS. 850W are enough for these components.
I guess you are aware that the A6000 has rather limited FP64 performance.

May 30, 2022, 09:49	Suggestions for GPU/CPU ("Shared") Workstation	#1
JM92 New Member Join Date: May 2022 Posts: 4 Rep Power: 4	Hi all, I got my hands on a high-performance nVidia GPU, which will be used for SPH simulations. As I will be able to build a workstation for roughly 3k€ (excluding GPU) around it, I'm currently thinking about setting this up for a second "use case", which would be simulations with openFoam. For this I'd currently aim at one Epyc 7313P or 7343 with (at least) 64 GB of RAM. Generally, I think the information given in this forum would be sufficient to build a CPU-only workstation without further help, but I just wanted to ask if anyone has experience with this combination of two use cases (SPH on GPU, openFoam on CPU [of course both simulations will not be run at the same time)] and knows about any pitfalls to avoid. E.g., I'm unsure about including a motherboard with onboard GPU to minimize the "additional" load on the nVidia card while running SPH simulations. Here's the checklist (focusing on the non-GPU part) Which software do you intend to use? --> openFoam Are you limited by license constraints? I.e. does your software license only allow you to run on N threads? --> No license constraints What type of simulations do you want to run? And what's the maximum cell count? --> 2D and 3D, mainly two-phase flow (interFoam), max. No. of cells 10-20 million (but a bit unsure what the future may hold in that matter) If there is a budget, how high is it? --> 3k € What kind of setting are you in? Hobbyist? Student? Academic research? Engineer? --> Academic researcher Where can you source your new computer? Buying a complete package from a large OEM? Assemble it yourself from parts? Are used parts an option? --> Planning to assemble it myself Which part of the world are you from? It's cool if you don't want to tell, but since prices and availability vary depending on the region, this can sometimes be relevant. Particularly if it's not North America or Europe. --> Europe Anything else that people should know to help you better? Kind regards

August 1, 2022, 08:21		#3
JM92 New Member Join Date: May 2022 Posts: 4 Rep Power: 4	Dear flotus1, thanks for your reply, I continued the "design" of the workstation and would like to ask for one final opinion on the parts before buying them: Mainboard: Supermicro H12SSL-NT CPU: AMD Epyc 7313P GPU: nVidia RTX A6000 (which is actively cooled) RAM: 8 x 16 GB Kingston Server Premier RDIMM 16GB, DDR4-3200, reg ECC, dual rank, x8 SSDs: Crucial P5 Plus 1 TB + Crucial P5 Plus 2 TB CPU Fan: Noctua NH-U14S TR4-SP3 Case: be quiet! Silent Base 601 schwarz, schallgedämmt Power supply: be quiet! Straight Power 11 Platinum 850W ATX 2.51 Especially the dimensioning of the power supply gave me some headaches, but as CPU and GPU won't be running on full throttle at the same time, 850 W should be sufficient in my opinion. Additionally, I'm thinking about following your advice of adding another small GPU to the setup, but no decision was made on that yet. Maybe a small nVidia T400 or T600 would be sufficient. Kind regards

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Suggestions for StarCCM+ Workstation configuration	ifil	Hardware	15	October 30, 2018 06:09
Another Workstation spec questions....	jeanhyuk	Hardware	4	July 10, 2018 06:37
Abysmal performance of 64 cores opteron based workstation for CFD	Fauster	Hardware	8	June 4, 2018 11:51
Low budget Workstation . Second Hand suggestions well received.	Fer_Arus	Hardware	4	April 10, 2017 12:20
Xeon workstation: suggestions needed	Blanco	Hardware	23	September 2, 2016 17:29

May 30, 2022, 10:51		#2
flotus1 Super Moderator Alex Join Date: Jun 2012 Location: Germany Posts: 3,427 Rep Power: 49	Onboard graphics is not something you need to worry about here. As far as I know, all motherboards compatible with Epyc 7000 CPUs come with some basic Aspeed graphics processor on board. The connector is only VGA though, and of course the processing power is severely limited. Displaying some console windows works fine, anything beyond that doesn't. It might be a good idea to add a small Nvidia graphics card for good measure. Prices are still high, but not as obscene as they used to be. 290€ for a GTX 1660 for example. That will allow you to use all the available memory of your highend GPU for computations, and generally simplify the setup. Speaking of pitfalls and unnamed "high-performance nVidia GPU" If it's PCIe 4.0, make sure your motherboard supports that. And beware of cooling for the GPU. Most of the Nvidia datacenter cards are passively cooled, relying on airflow from the chassis fans for cooling. They can still be used in a workstation, but require some thought and modding effort put into a custom cooling solution. And if it's just a regular cooling solution with axial fans, beware that all of the heat gets dumped into the case, so plan for that with plenty of airflow. And obviously, the power supply has to match the highend components. As for the plan itself, you will probably get better answers in the OpenFOAM sub-forum.

August 1, 2022, 12:10		#4
flotus1 Super Moderator Alex Join Date: Jun 2012 Location: Germany Posts: 3,427 Rep Power: 49	Looks fine to me. The RTX A6000 doesn't have regular PCIe power connectors, but requires an 8-pin EPS connector. The PSU you picked has two of those, one for the motherboard, and one for the GPU. And the GPU should come with an adapter 2x8pin PCIe -> 8pin EPS. 850W are enough for these components. I guess you are aware that the A6000 has rather limited FP64 performance.