[gnwt4a]

Hello,
Has anyone an opinion on the following?

My current front end system (hp ml350 g9, 2x 2690v4, ddr4 2400) seems to be struggling computationally holding back gpu (4x mi50/60) occupancy to less than 50% (75%-80% of the fp64 operations are done on gpus.) buying it was a mistake. i can only stretch it to ~800M nodes (spectral) even though the gpus are underused. this holds things back quite a bit.

while looking for a cheap upgrade, it looks as if an hp dl580 g9 with 4x E7-8880 v4 and 16x 16GB - ddr4 2400MHz which will default to 1800MHz, may be a viable option, with a cost of ~1.5K euro (incl vat) - from the uk.

Forgetting the gpus, for general cfd computations, is this a 'worthwhile' upgrade?

Another possible option is an asus esc4000 g4 with 2x xeon 6148 and 12x 16GB ddr4 @2666. This looks like hovering at ~2.2K euro - if components are bought separatelly.

is it possible to say which one system (asus/dl580) would be the better option?
Any alternatives?
Thanks.
==

[flotus1]

Are we absolutely certain that it is the CPUs FP64 performance that is holding back the GPUs?
From the docs, it looks like the hp ml350 g9 only has three PCIe3.0 x16 slots. There is a fourth mechanical x16 slot, but the bus width on that is only x8. Maybe it's a bandwidth or locality issue?

[gnwt4a]

Originally Posted by flotus1

Are we absolutely certain that it is the CPUs FP64 performance that is holding back the GPUs?
From the docs, it looks like the hp ml350 g9 only has three PCIe3.0 x16 slots. There is a fourth mechanical x16 slot, but the bus width on that is only x8. Maybe it's a bandwidth or locality issue?

good point. i did not want to write an overlong post, so not every detail was given.


read/write between the gpu and host ram is controlled by the card itself. so u can get a good idea where the calc is spending most time by simply watching the output of htop for a few minutes. also, an implicit check is monitoring the temperatures of the cpus and gpus.


more to the point, i use the gpu dma asynchronous data transfer hardware (with pinned memory on the host) which enable data exchanges while calculations are being carried out. only gpu priming entails idling hardware but its effect is quite small.


i went to the intel fortran forum and asked if sections of data can be allocated to specific socket ram. the short answer was: not directly. the distribution of data between socket rams is auto-determined by the system when first use of an array is made and not during declaration. the only control u have is to experiment with different first-use methods of access and hope that u will hit a favourable set up which u can reuse.

==

[flotus1]

There is not too much crossing fingers involved with data placement. Unless you instruct the OS to mess things up, first touch data placement is fairly reliable.
Provided there is enough free memory on every NUMA node at startup. numactl -H is our friend
Dropping caches before running a code that is sensitive to data placement is a must (requires root privileges): echo 3 > /proc/sys/vm/drop_caches
With that, data placement should be 100% repeatable.

These issues will get more pronounced with more complicated NUMA topology. Like having 4 sockets in a shared memory system.

But yeah, the hp dl580 g9 should give you roughly twice the CPU performance. And full PCIe bandwidth for each of the 4 GPUs. As a nice bonus, each GPU can be attached to one of the 4 sockets. That might make some pinning easier/more straightforward.
1500€ sounds a bit expensive, this stuff is really old by now. Nobody really wants it any more, outside of very niche use cases. Power draw and heat generation will also be pretty spectacular

[wkernkamp]

Originally Posted by gnwt4a

Hello,
Has anyone an opinion on the following?
...
while looking for a cheap upgrade, it looks as if an hp dl580 g9 with 4x E7-8880 v4 and 16x 16GB - ddr4 2400MHz which will default to 1800MHz, may be a viable option, with a cost of ~1.5K euro (incl vat) - from the uk.

==

I thought these E7's had best memory performance with 8 dimms per socket. The memory is on cards that have a special processor for comms with the cpu. That doubles the bandwidth to 8-channel equivalent. (If I remember correctly)

Will

[gnwt4a]

Thanks all.


https://forums.servethehome.com/inde...le-core.36891/