[Noco]

Hello!

I have 1 new computer:

1 CPU i9 7980XE with Kraken X62
- ASUS S2066 PRIME X299-A RTL
- GeForce PCI-E 11264Mb 1080 Ti InnoVision (not helps actually)
- 64Gb DDR4 2133 MHz (4x16)
- 512 SSD for system
- 2x8 SATA RAID0 for storage
- Windows 10
- ANSYS CFX R17

And I can buy for good price 5 old one from 2010:

2 CPU Xeon X5660
- Aquarius S5520SC
- Nvidia Quadro FX 1800
- DDR 12x8192 + SSD (1 of 5) DDR 12x4096 (4 of 5)
- Windows 7 Pro

I also want to buy 1 new 2 CPU Xeon Gold / AMD Epyc in 2018 (it will be 7 computer)

Questions:
1. I can not find normal instruction how to organise parallel computing (using ansys parallel solver) in CFX R17 (I use old instruction for CFX 10 + ansys manual - it works somehow, but as for now 3% of time is calculation, 97% of time - writing to sata disk). Never did this before. Where can I find good instruction for CFX R17 with parallel solver for 6-7 and more computers?

2. Is it make sense to buy this old computers, install ethernet cards, buy extra licenses, and use them as slaves with new computer? I will have 5*12+18= 78 cores system, but what will be bottlenecks for this system?

Thank you in advance!

[ghorrocks]

For distributed parallel computing, all nodes need to be the same computer. Mixing computer speeds will not give you an effective speedup.

The parallel setup has not changed in CFX for a long time, across many versions. Your comment that your cluster does 3% calculation and 97% disk access probably means one of the following:
1) At least one node has run out of memory
2) You are writing too many results files

[Noco]

How can I check that one or more nodes run out of the memory?

I run 3 old computers as a test:

1 master:

2 CPU Xeon X5660
- Aquarius S5520SC
- Nvidia Quadro FX 1800
- DDR 12x8192
- SSD
- Windows 7 Pro

2 slaves:

2 CPU Xeon X5660
- Aquarius S5520SC
- Nvidia Quadro FX 1800
- SATA
- DDR 12x4096
- Windows 7 Pro

Regarding the 'writing too many results files' - maybe, but it was first time like this, the master was writing during about 7 hours. Can it be related with SSD/HD difference writing speed?

[ghorrocks]

To check memory - log onto the slaves as they are running and look at taskmanager (windows) or top (linux) and check the memory allocated.

If your results files are huge it could take 7 hours to write. But that would require seriously huge files. It could also be your network is very slow, or your virus checker is killing it (or some other background process).

[Noco]

Ok, I will try to check this.

Other issue:

When I run:
- 2 CPU Xeon X5660
- Aquarius S5520SC
- Nvidia Quadro FX 1800
- DDR 12x4096
- Windows 7 Pro
- ANSYS CFX R17

During iteration I have up to DDR 30Gb load from 48 Gb. All 12 cores are 80-100% load.

But with the same task with new one:
- 1 CPU i9 7980XE with Kraken X62
- ASUS S2066 PRIME X299-A RTL
- GeForce PCI-E 11264Mb 1080 Ti InnoVision (not helps actually)
- 64Gb DDR4 2133 MHz (4x16)
- 512 SSD for system
- 2x8 SATA RAID0 for storage
- Windows 10
- ANSYS CFX R17

During iteration I have only up to DDR4 8 Gb load from 64 Gb. All 18 cores are 90-100% load. Why 18 cores do not use all 64 Gb? There is official and not official recommendation that for CFD we need 8 Gb per core, so 18*8= 144 (128 maximum support for this motherboard), I have 64, but it is nor fully load.

[urosgrivc]

It will only use as much memory as your simulation needs.
How much memory is used is mostly dependant on the mesh size.
More elements you have more ram you need.
And this cannot be because of the diferences betveen the computers
Are you sure that the simulation is exactly the same in both tests?

and The i9 has 4 memory chanels so you dont need more than 4 sticks of ram.
and the dual xeon has 6 total memory chanels, 3 per procesor, so 6 or 12 sticks of ram are prefered for that case.
so you use up as much memory bandwidth as posible.

[Rec]

When distributed calculations, the amount of memory will be equally used in 1,2,3 computers?
Or memory is distributed as well as calculations?

Thank you.

Originally Posted by urosgrivc

It will only use as much memory as your simulation needs.
How much memory is used is mostly dependant on the mesh size.
More elements you have more ram you need.
And this cannot be because of the diferences betveen the computers
Are you sure that the simulation is exactly the same in both tests.

and The i9 has 4 memory chanels so you dont need more than 4 sticks of ram

[ghorrocks]

In distributed simulations the mesh is divided into segments and each process does one segment. So each process sees a smaller mesh.