[Anna Tian]

Hi,

We are going to purchase hardware for a CFD department. Fluent and ICEM are used. We will have 32 CPUs (128 cores) to support our simulation. And 2 CFD people will use it. I'm wondering shall we buy a workstation (Large workstation shall contain 32 CPUs. And I guess there should be a way for 2 people to access the computational resource of one workstation at the same time. If not, please correct me.) or we'd better build a cluster? Any suggestions?

[Kaskade]

I don't understand what you mean by "32 CPUs (120 cores)". A 32 core workstation would be based on a board with 4 cpu sockets and 8-core cpus. Although it should be noted that only certain Xeons work on mainboards with 4 sockets.

Most straight forward setup would consist of 3 machines.
1 machine for Fluent (doesn't even need monitor, keyboard etc)
2 machines for Pre/Postprocessing
File-Transfer using SMB or SFTP

Possibly cheaper
1 big machine: run Fluent on fewer cores than available during daytime and use the computer for pre-/post-processing
1 smaller machine for engineer number 2

Are you planning on using Windows or Linux?

Edit: Apparently AMD offers a 16 core cpu (consisting on 4 quad cores), which is capable of running on an 4 socket board -> 64 cores on one mainboard.

[Anna Tian]

Originally Posted by Kaskade

I don't understand what you mean by "32 CPUs (120 cores)". A 32 core workstation would be based on a board with 4 cpu sockets and 8-core cpus. Although it should be noted that only certain Xeons work on mainboards with 4 sockets.

Most straight forward setup would consist of 3 machines.
1 machine for Fluent (doesn't even need monitor, keyboard etc)
2 machines for Pre/Postprocessing
File-Transfer using SMB or SFTP

Possibly cheaper
1 big machine: run Fluent on fewer cores than available during daytime and use the computer for pre-/post-processing
1 smaller machine for engineer number 2

Are you planning on using Windows or Linux?

Edit: Apparently AMD offers a 16 core cpu (consisting on 4 quad cores), which is capable of running on an 4 socket board -> 64 cores on one mainboard.

Hi,

I'm planing on a workstation with 32 CPUs and each of them have 4 cores. So there are 128 cores in total. Sorry that 120 is a tempo. I have corrected that.

I'm planning on having at least the large workstation on Linux.

Regarding the two setup ways you mentioned, will they have any performance difference? I guess the first way you mentioned (3 machines one) could be more complex to build. I'm really wondering is this big workstation normal? And will the large size of the machine, large number of CPUs and cores cause any problem?

[Kaskade]

How are you going to fit 32 cpus into a single machine?

Ansys is restrictive when it comes to Linux distributions: http://www.ansys.com/staticassets/AN....0_11-2011.pdf
I doubt Fluent wouldn't work on a free distribution, but Ansys might not give you support if something doesn't function properly.

BTW: StarCCM+ is cheaper for that many cores and CD-Adapco allows the use of CentOS.

[Anna Tian]

Originally Posted by Kaskade

How are you going to fit 32 cpus into a single machine?

Ansys is restrictive when it comes to Linux distributions: http://www.ansys.com/staticassets/AN....0_11-2011.pdf
I doubt Fluent wouldn't work on a free distribution, but Ansys might not give you support if something doesn't function properly.

BTW: StarCCM+ is cheaper for that many cores and CD-Adapco allows the use of CentOS.

No that much big machine on the market? I once heard about workstation with 32 CPUs but I never checked it online. Intuitively I think it could be quite big and as heavy as me.

I don't understand what do you mean by 'free distribution' here. I'm reading the link you provide me hardly. Thanks for your information.

We don't have a problem of the fluent license. We can easily rent or borrow licenses from our supplier company at the neighbor.

Btw, what about having win XP on the large machine? I read somewhere in this forum that Fluent runs much faster on Win XP. Fluent 15 still supports win XP. If I have Windows on my big machine, how do we set up so that the CFD engineer using the small machine can run the simulation directly on the big machine while another engineer is using the big machine at the same time?

[Kaskade]

I only found mainboards/workstations with 4 cpu sockets. If you want to use more cpus, you need to connect multiple machines using something like infiniband or ethernet -> cluster.

Free as in "for free". RHEL and SLES require a fee: https://www.suse.com/products/server/how-to-buy/ (You can set the language at the top.)
They offer support and a long distribution life time.

XP support ends in April. And I doubt it is faster.

[Anna Tian]

Originally Posted by Kaskade

I only found mainboards/workstations with 4 cpu sockets. If you want to use more cpus, you need to connect multiple machines using something like infiniband or ethernet -> cluster.

Free as in "for free". RHEL and SLES require a fee: https://www.suse.com/products/server/how-to-buy/ (You can set the language at the top.)
They offer support and a long distribution life time.

XP support ends in April. And I doubt it is faster.

I will check the size on the market with the workstation stores. Maybe we will be running into building a cluster. Is there any tutorial on how to build a cluster? We're totally fresh on this.

[Kaskade]

On the plus side: a cluster is easier to upgrade.

Most tutorials for setting up a cluster are pretty ancient. But it should be easy if you only want run one Fluent session across multiple machines.

If you want to use a job scheduler and ressource manager it gets trickier. Possible Google Searches: Platform LSF / Torque maui / Hadoop

We are in the process of setting up a 32 core cluster ourselves. Our hardware is virtual, but the software setup should be similar.

[macfly]

Edit : nevermind, I was suggesting to go read the hardware forum, and I just saw you did

[Anna Tian]

Originally Posted by Kaskade

On the plus side: a cluster is easier to upgrade.

Most tutorials for setting up a cluster are pretty ancient. But it should be easy if you only want run one Fluent session across multiple machines.

If you want to use a job scheduler and ressource manager it gets trickier. Possible Google Searches: Platform LSF / Torque maui / Hadoop

We are in the process of setting up a 32 core cluster ourselves. Our hardware is virtual, but the software setup should be similar.

I just checked the other threads on this forum. It seems that it will be much more expensive if you want to build a cluster with good data exchange speed between CPUs. Why don't you just choose to buy a 4 sockets, 128 cores workstation? In this way, the maintenance and management cost shall be also a little lower. Later on, if you want to upgrade it (like adding more CPUs), you can also change the workstation into a cluster.

[gfoam]

Check out the SGI workstations I think they have the machine you need. Regards.
Gonzalo

[Anna Tian]

Originally Posted by gfoam

Check out the SGI workstations I think they have the machine you need. Regards.
Gonzalo

Yes. I found hardware companies may sell it, including HP. But they are mainly for the server, not for scientific calculation, like CFD. I never saw anyone mentioned there is a this large workstation in this forum. So may I ask are they okay for CFD?

I'm also wondering what is the advantage of cluster over workstation in this case? And how much more expensive and complex would the cluster choose be for my case?

[Anna Tian]

Originally Posted by Kaskade

How are you going to fit 32 cpus into a single machine?

Ansys is restrictive when it comes to Linux distributions: http://www.ansys.com/staticassets/AN....0_11-2011.pdf
I doubt Fluent wouldn't work on a free distribution, but Ansys might not give you support if something doesn't function properly.

BTW: StarCCM+ is cheaper for that many cores and CD-Adapco allows the use of CentOS.

I'm quite fresh to the enterprise linux area. What I understand from the link you provide is as below.

1. Linux is open for one computer. But if it is connected to another computer (no matter it is linux or windows), we need to pay for the connection which is called SUSE Linux Enterprise Server.

2. Fluent also restrict file distribution in Linux.

Is this correct? Windows doesn't have this restriction? We already have Windows system.

[ghost82]

Hi,
I never seen these type of worstations/servers managing more than 4 cpus.

To have an idea of the price, if you want to build a cluster based on 8 workstations connected to 1 infiniband switch, with 32gb ram 1600Mhz ecc, 2x Xeon e5-2687w (which are not the latest cpus on the market, but they're very good 8 real cores), 1000W psu gold, asus motherboard, liquid cooled, infiniband 40gb/s, based on USA costs, you are going to spend more or less about 50.000 USD (installation costs not included).

What will be the cost of the single server?

PS: from my experience I would go with windows

[Kaskade]

@ghost82: Why would you build a cluster from workstation? I think some rack mounted machines would be far easier to handle.

@Anna Tian: The problem with large amounts of cores in a single machine is probably heat. The heat production of core/CPU rises linearly with the clock rate, additionaly the voltages need to be raised to keep the system stable, which increases the heat production exponentially. So if you want a lot of cores, you will end up with slower cores.

Have you considered asking Ansys for advice?

[ghost82]

Originally Posted by Kaskade

@ghost82: Why would you build a cluster from workstation? I think some rack mounted machines would be far easier to handle.

Yes you're right, however I think the total cost will not be too much different; also 8 machines are not too many, they can be installed in asmall space, and can be easily controlled/maintained in the form of workstations.

[Kaskade]

For commercial applications the sheer computing power is not enough. You'll need a decent RAID system plus backup and an UPS (Uninterruptible Power Supply). And a rack will make handling the cables easier.

The next step should probably be getting quotes for different system.

[Anna Tian]

Originally Posted by ghost82

Hi,
I never seen these type of worstations/servers managing more than 4 cpus.

To have an idea of the price, if you want to build a cluster based on 8 workstations connected to 1 infiniband switch, with 32gb ram 1600Mhz ecc, 2x Xeon e5-2687w (which are not the latest cpus on the market, but they're very good 8 real cores), 1000W psu gold, asus motherboard, liquid cooled, infiniband 40gb/s, based on USA costs, you are going to spend more or less about 50.000 USD (installation costs not included).

What will be the cost of the single server?

PS: from my experience I would go with windows

You go with Windows only for the price reason? I read on this forum that linux is better on the memory usage.

[ghost82]

Not for the price: I tried several linux distribution, not certified by ansys (Centos, ubuntu, kubuntu) and the Sles distribution; in all cases I had problems not with fluent itself, but with the license manager: sometimes starting 2 times and not able to connect, sometimes not starting; I found (at least for my system) that windows 7 64 bit is better since everithing was and is smooth.

Daniele

[Anna Tian]

Originally Posted by ghost82

Not for the price: I tried several linux distribution, not certified by ansys (Centos, ubuntu, kubuntu) and the Sles distribution; in all cases I had problems not with fluent itself, but with the license manager: sometimes starting 2 times and not able to connect, sometimes not starting; I found (at least for my system) that windows 7 64 bit is better since everithing was and is smooth.

Daniele

Regarding the liquid cooling, is liquid cooling important? Is it expensive? We can place it in another room so that we won't be disturbed by the noise from the machine if air cooling is used. Any other concerns? Will that give additional requirements like the quality of the water?