[flotus1]

Just dont.
Water doesnt go together with electronics, no matter what the retailers of water cooling systems say.
They are expensive compared to air cooling, need maintenance and are just as good in terms of noise and cooling performance as a good air cooling.
Unless you are planning to overclock the CPUs to a point where they will reach lifetime cycle after 6 months, you dont need it.

[ghost82]

I chosed liquid coolers because I have to dissipate near 150W x2 at full load.
Air coolers exist, but they are too big for my motherboard and the dissipators interfered with the ram slots.
I chosed liquid coolers from corsair, as they are small and give good performance: at 100% load I have a maximum temperature of 54 ºC.
These coolers are closed systems, composed by a block to install above the cpu, which comprises the pump, 2 flexible tubes, connected to a radiator and 2 fans attached to the radiator to have a push-pull configuration.
Liquid circulates in this system.
Each cooler is connected to the motherboard or to an external usb port to control the fans speed and to monitor temperatures.
The 2 fans on the radiator are also connected by a y cable to the pump block.
The last connection is from the pump block to the motherboard through the fan connector.
The cost is near the same a good air cooler.
Corsair warranty is 5 years for these units.

[Kaskade]

@ghost82: Are you using one machine or several?

Regarding liquid cooling: You can use certain non-conductive oils or demineralized water. But I really don't think liquid cooling is a good idea. In a well ventilated case good air coolers are more than sufficient. If a fan breaks, the CPU clocks down to keep from overheating. A fan can be replaced quickly, but if something goes wrong with your liquid cooling, you need a mop.

Get some quotes for professional systems. This starts to read like some Overclockers Forum.

[ghost82]

Only one machine.
Unfortunately the rule is the higher the heat to be dissipated by the cpu the bigger the dissipator: and if the dissipator is "big" you need more fans attached to it to cool it down.
I looked at several professional/non professional air dissipators but all interefere with the slots of the ram (ASUS z9pe-d8 ws populated with 8 sticks of ram).
There are some air coolers which not interfere with the slots, but able to dissipate near 130 W (lower than my 150 W), and they have the fan mounted above the cpu, with air flux perpendicular to it; not a good solution at all.
You are right about the maintenance: a fan could be replaced in a few minutes, and liquid coolers have intrinsic risks..but scheduled inspections to the workstation can really low down the risks.
I'm happy with my coolers, I have very low temperature at full load, but if there were air coolers for my case I would have chosen them, to set the risks to zero.

[Anna Tian]

Have anyone computed the power cost of the cluster? Is the electricity cost a significant part of the overall cost of hardware? Of course this depends on the electricity price. But how many watt? I'd like to just have a basic feeling on that.

[ghost82]

it depends on the chosen hardware, but I think it will absorb about 4-5 kW at full load.

[Anna Tian]

Originally Posted by ghost82

it depends on the chosen hardware, but I think it will absorb about 4-5 kW at full load.

Is it mainly driven by the number of CPUs? Or cooling device?

[Kaskade]

A very rough estimate would proably be number_of_cpus*TDP*1,25. If you use the same cpus as ghost82, that would mean 8cpu*150w/cpu*1,25=3000W.

Most cpus use less power than their TDP, due to the way Intel sorts them.

A typical cpu fan uses less than 5 W.

[Kaskade]

Sorry, my brain was on lunch break. 8*8 is 64, not 128. So the power usage is around 6KW. Or 4 electric kettles.

I don't know where you live, but an AC unit might be in order. Which further increases the power usage.

[Anna Tian]

Originally Posted by Kaskade

Sorry, my brain was on lunch break. 8*8 is 64, not 128. So the power usage is around 6KW. Or 4 electric kettles.

I don't know where you live, but an AC unit might be in order. Which further increases the power usage.

Thanks for your information. I gain a basic feeling on that. It is quite some cost per year if we keep the machine running.

[Eakarach]

Dear Anna

Before you consider to buy your new workstation. Plese check your fluid flow problems that fluent hpc can be employed. Regarding to my knowledge, only a few case (radiation case) is confirm. I hope this information can help you.


Goodluck
Eakarach

[Anna Tian]

Originally Posted by Eakarach

Dear Anna

Before you consider to buy your new workstation. Plese check your fluid flow problems that fluent hpc can be employed. Regarding to my knowledge, only a few case (radiation case) is confirm. I hope this information can help you.


Goodluck
Eakarach

Thanks for your information. What do you mean? You mean the license problem?

[Kokemoor]

I'd guess that Eakarach is referring to GPUs in Fluent, which have limited applicability in versions earlier than 15. In 15, the AMG solver can use the GPU(s), but I haven't seen much attesting to the real speed up.

[Eakarach]

In order to speed up the computation, there is two ways including workstation with more cpus and GPU computer. For the second case, there is only a few successful case (radiation). The first one is another choice to accelerate the computation. However, if you search the topics about the acceleration of computation. The results revealed that an increasing in cpu cores can be reduced the computed time. However, the calculation time is not decreased linearly with incracing the cpu core. In order words, if you solved the problems with 2 cores workstation. The caculation time is 2 hrs. When you employ 4 cores workstation. The caculation time is not 1hr. It should be 1.5 hrs. Thus, you should to search more information about the acceleration topics.

Best regards,
Eakarach

[Kaskade]

No one was talking about GPUs. (FVM doesn't really benefit and LBM still has a long way to go.)

@Eakarach: Of course they don't scale lineary, but it is not nearly as bad as you describe. And what else are you going to do if you need to solve a complex problem? Scale back to 100.000 cells or wait half a year? Although you are right if you are suggesting, she should do a cost/benefit analysis depending on her use cases. Maybe two fast workstations with a case each would be more effective, than a small cluster. Really depends on the number of simultaneous cases.

[Anna Tian]

Originally Posted by Kaskade

@ghost82: Why would you build a cluster from workstation? I think some rack mounted machines would be far easier to handle.

@Anna Tian: The problem with large amounts of cores in a single machine is probably heat. The heat production of core/CPU rises linearly with the clock rate, additionaly the voltages need to be raised to keep the system stable, which increases the heat production exponentially. So if you want a lot of cores, you will end up with slower cores.

Have you considered asking Ansys for advice?

Hi Kaskade,

If i follow ghost82's advice and build cluster by connecting 8 workstation, I guess in this way the cooling will be quite good per CPU. Then I can choose CPUs with many strong cores, right?

[Kaskade]

If the fast CPUs spend half the waiting for data from other machines, they are not a good investment. Depending on your budget and the number of parallel jobs, it might be good idea to buy cheaper CPUs and invest in an Infiniband Interconnect.

And I wouldn't buy workstations (invidual towers including graphics card) but servers/blades in a rack. Easier to maintain and cooler.

[ghost82]

Just for reference, having a workstation doesn't mean you have to buy individual graphic cards: for example the asus z9pe d8 ws has a jumper to set onboard vga; you haven't any vga port output, the system will start without keyboard and by setting correctly that jumper, and you can control it by remote management.
I think that if you have "few" cores you can go for indivual towers: you can use each individually for "small" cases (<=16-14 cores, on newer xeon v2 cpus), but if your cases are "big" and you always need to run them on a lot of cores (>48 cores) then go for a rack mounted system.

[Anna Tian]

Originally Posted by Kaskade

If the fast CPUs spend half the waiting for data from other machines, they are not a good investment. Depending on your budget and the number of parallel jobs, it might be good idea to buy cheaper CPUs and invest in an Infiniband Interconnect.

And I wouldn't buy workstations (invidual towers including graphics card) but servers/blades in a rack. Easier to maintain and cooler.

Yes. It is usually not a good investment to fast CPUs for CFD applications. But I just realized that the CPU Intel Xeon E5-1620 v2 with 4 3.7 GHz cores and max memory bandwidth 59.7 GB/s only sells 370 US dollars in U.S. and less than 300 US dollars in China. I think its price in Europe won't be too different. Even it is for 2.5 GHz, I still didn't see a obvious better choice than it. And it is 3.7 GHz. So I go for it. This is the reason I choose fast CPU. I will connect 8 or 16 E5-1620 v2 CPUs to have 32 cores. This is the best maximum memory bandwidth (GB/s) per dollar I found.

There is another advantage of using a fast CPU. If I use a slow CPU (e.g. 2.5 GHz), I will always need to go for 2.5 GHz when I want to expand the cluster. 2.5 GHz CPUs might be two slow in the future and a main factor which limit the CFD calculation speed. In the future, the price difference between 3.7 GHz CPU and 2.5 GHz won't be as large as right now.

Btw, I will anyway need to invest in Infiniband for CFD Cluster.

[Anna Tian]

Originally Posted by ghost82

Just for reference, having a workstation doesn't mean you have to buy individual graphic cards: for example the asus z9pe d8 ws has a jumper to set onboard vga; you haven't any vga port output, the system will start without keyboard and by setting correctly that jumper, and you can control it by remote management.
I think that if you have "few" cores you can go for indivual towers: you can use each individually for "small" cases (<=16-14 cores, on newer xeon v2 cpus), but if your cases are "big" and you always need to run them on a lot of cores (>48 cores) then go for a rack mounted system.

I'm around the middle of the range you provide. I will have around 32 cores.

I think there is another advantage of using the '8 connecting individual tower workstations cluster'. This is just my speculation. I don't have any practical experience on that. Correct me if it is not correct.

If I build a rack mounted cluster, i can't increase the box size to contain more motherboards easily. The space in the rack mounted box is fixed. But if I go for individual towers, one day if I find i need another CPU, I can just add it by adding another individual tower workstation. This is much easier, right?