|
[Sponsors] |
![]() |
![]() |
#1 |
Member
Ivan
Join Date: Oct 2017
Location: 3rd planet
Posts: 34
Rep Power: 8 ![]() |
Hello!
I have 1 new computer: 1 CPU i9 7980XE with Kraken X62 - ASUS S2066 PRIME X299-A RTL - GeForce PCI-E 11264Mb 1080 Ti InnoVision (not helps actually) - 64Gb DDR4 2133 MHz (4x16) - 512 SSD for system - 2x8 SATA RAID0 for storage - Windows 10 - ANSYS CFX R17 And I can buy for good price 5 old one from 2010: 2 CPU Xeon X5660 - Aquarius S5520SC - Nvidia Quadro FX 1800 - DDR 12x8192 + SSD (1 of 5) DDR 12x4096 (4 of 5) - Windows 7 Pro I also want to buy 1 new 2 CPU Xeon Gold / AMD Epyc in 2018 (it will be 7 computer) Questions: 1. I can not find normal instruction how to organise parallel computing (using ansys parallel solver) in CFX R17 (I use old instruction for CFX 10 + ansys manual - it works somehow, but as for now 3% of time is calculation, 97% of time - writing to sata disk). Never did this before. Where can I find good instruction for CFX R17 with parallel solver for 6-7 and more computers? 2. Is it make sense to buy this old computers, install ethernet cards, buy extra licenses, and use them as slaves with new computer? I will have 5*12+18= 78 cores system, but what will be bottlenecks for this system? Thank you in advance! Last edited by Noco; January 14, 2018 at 08:12. |
|
![]() |
![]() |
![]() |
![]() |
#2 |
Super Moderator
Glenn Horrocks
Join Date: Mar 2009
Location: Sydney, Australia
Posts: 17,816
Rep Power: 144 ![]() ![]() ![]() ![]() |
For distributed parallel computing, all nodes need to be the same computer. Mixing computer speeds will not give you an effective speedup.
The parallel setup has not changed in CFX for a long time, across many versions. Your comment that your cluster does 3% calculation and 97% disk access probably means one of the following: 1) At least one node has run out of memory 2) You are writing too many results files |
|
![]() |
![]() |
![]() |
![]() |
#3 |
Member
Ivan
Join Date: Oct 2017
Location: 3rd planet
Posts: 34
Rep Power: 8 ![]() |
How can I check that one or more nodes run out of the memory?
I run 3 old computers as a test: 1 master: 2 CPU Xeon X5660 - Aquarius S5520SC - Nvidia Quadro FX 1800 - DDR 12x8192 - SSD - Windows 7 Pro 2 slaves: 2 CPU Xeon X5660 - Aquarius S5520SC - Nvidia Quadro FX 1800 - SATA - DDR 12x4096 - Windows 7 Pro Regarding the 'writing too many results files' - maybe, but it was first time like this, the master was writing during about 7 hours. Can it be related with SSD/HD difference writing speed? |
|
![]() |
![]() |
![]() |
![]() |
#4 |
Super Moderator
Glenn Horrocks
Join Date: Mar 2009
Location: Sydney, Australia
Posts: 17,816
Rep Power: 144 ![]() ![]() ![]() ![]() |
To check memory - log onto the slaves as they are running and look at taskmanager (windows) or top (linux) and check the memory allocated.
If your results files are huge it could take 7 hours to write. But that would require seriously huge files. It could also be your network is very slow, or your virus checker is killing it (or some other background process). |
|
![]() |
![]() |
![]() |
![]() |
#5 |
Member
Ivan
Join Date: Oct 2017
Location: 3rd planet
Posts: 34
Rep Power: 8 ![]() |
Ok, I will try to check this.
Other issue: When I run: - 2 CPU Xeon X5660 - Aquarius S5520SC - Nvidia Quadro FX 1800 - DDR 12x4096 - Windows 7 Pro - ANSYS CFX R17 During iteration I have up to DDR 30Gb load from 48 Gb. All 12 cores are 80-100% load. But with the same task with new one: - 1 CPU i9 7980XE with Kraken X62 - ASUS S2066 PRIME X299-A RTL - GeForce PCI-E 11264Mb 1080 Ti InnoVision (not helps actually) - 64Gb DDR4 2133 MHz (4x16) - 512 SSD for system - 2x8 SATA RAID0 for storage - Windows 10 - ANSYS CFX R17 During iteration I have only up to DDR4 8 Gb load from 64 Gb. All 18 cores are 90-100% load. Why 18 cores do not use all 64 Gb? There is official and not official recommendation that for CFD we need 8 Gb per core, so 18*8= 144 (128 maximum support for this motherboard), I have 64, but it is nor fully load. |
|
![]() |
![]() |
![]() |
![]() |
#6 |
Senior Member
urosgrivc
Join Date: Dec 2015
Location: Slovenija
Posts: 365
Rep Power: 11 ![]() |
It will only use as much memory as your simulation needs.
How much memory is used is mostly dependant on the mesh size. More elements you have more ram you need. And this cannot be because of the diferences betveen the computers Are you sure that the simulation is exactly the same in both tests? and The i9 has 4 memory chanels so you dont need more than 4 sticks of ram. and the dual xeon has 6 total memory chanels, 3 per procesor, so 6 or 12 sticks of ram are prefered for that case. so you use up as much memory bandwidth as posible. Last edited by urosgrivc; January 17, 2018 at 07:47. |
|
![]() |
![]() |
![]() |
![]() |
#7 | |
New Member
Sergey
Join Date: Jan 2018
Posts: 18
Rep Power: 8 ![]() |
When distributed calculations, the amount of memory will be equally used in 1,2,3 computers?
Or memory is distributed as well as calculations? Thank you. Quote:
|
||
![]() |
![]() |
![]() |
![]() |
#8 |
Super Moderator
Glenn Horrocks
Join Date: Mar 2009
Location: Sydney, Australia
Posts: 17,816
Rep Power: 144 ![]() ![]() ![]() ![]() |
In distributed simulations the mesh is divided into segments and each process does one segment. So each process sees a smaller mesh.
|
|
![]() |
![]() |
![]() |
|
|
![]() |
||||
Thread | Thread Starter | Forum | Replies | Last Post |
Remote cluster parallel solve without master, Ansys CFX 14.5 | aaguirre | CFX | 4 | January 27, 2015 09:15 |
CFX Parallel | ihsb | CFX | 1 | October 15, 2014 17:12 |
correction number and parallel computing | psosnows | OpenFOAM Bugs | 5 | June 17, 2013 20:56 |
RSH problem for parallel running in CFX | Nicola | CFX | 5 | June 18, 2012 18:31 |
CFX, NT parallel, Linux, best platform | Heiko Gerhauser | CFX | 1 | August 21, 2001 09:46 |