[naffrancois]

Dear forumers,

We are about to receive a dual-socket workstation based on amd epyc. Memory should be populated correctly with 16*8go dual-rank DDR4-2666. I was wondering if there were some bios settings to tune (like HT for intel) for use with our in-house parallel unstructured solver, or I can use it out of the box ?

Thanks

[flotus1]

I disabled SMT in the bios for CFD use, that was about it. Played around with some of the other settings that I thought might affect performance, but to no avail. I would recommend leaving the default setting as they are until you actually encounter a problem. With the exception of SMT of course

[naffrancois]

Thank you, good to know that you already tried to customize things. I will disable SMT then.

[naveedgt]

Any recommended settings for running on windows? MPI issues?

[naffrancois]

Dear flotus or other forumer having an epyc-based workstation,

Would you tell me the temperatures you get on full load ?

We have a dual 7501 on which I installed the supermicro tool SuperDoctor and CPU1 is at 65°C whereas CPU2 is at 50°C when all cores are on heavy work. I am a bit surprised by the difference between the two sockets. Also I find the CPU1 temperature a quite high, which tends to increase the dynatron A26 fans to an unacceptable noise level for daily use in an office room (typically at 5500rpm, honestly it becomes very annoying, and it can hypothetically go up to 8kRPM, I cant imagine).

Would you tell me your opinion about it ? Bad fan setup, thermal paste ? The retailer explained the difference between the cpus by the fact that CPU1 is heated by the airflow coming from CPU2, they did not seem shocked by the 65°C temperature neither.

Ambient temperature is about 22-23°C.

Are you using higher diameter noctua fans ? Did you monitor temperatures ?

Thanks a lot

[flotus1]

Would you tell me your opinion about it ? Bad fan setup, thermal paste ? The retailer explained the difference between the cpus by the fact that CPU1 is heated by the airflow coming from CPU2, they did not seem shocked by the 65°C temperature neither.

The retailer might be correct about CPU1 sitting in the hot exhaust air of CPU2, depending on the orientation of the CPU coolers. Mine are both pointing upwards so I don't have this issue. It is a common problem with shadowed layouts.
65°C is perfectly fine for these CPUs, nothing to worry about. Mine are sitting in the low 50s under full load.
Those Dynatron coolers are a royal pita. Retailers using these for normal workstations instead of servers should not be allowed to sell workstations in my opinion. I used Noctua NH-U14s TR4-SP3, the largest they had. There are two smaller variants if you can not fit them into your case: NH-U12s TR4-SP3 and NH-U9 TR4-SP3. I would recommend swapping out the coolers and sending the invoice to your retailer

Here is how mine looked in the old case. I am using a slightly different case now, but the layout is similar.
20171213_095728.jpg

[naffrancois]

Thank you for your answer and the picture which explains very well your comment. Your setup is clever, the cpu coolers seem to extract heat in the vertical direction which is evacuated to the top with the big case's fans, it is nice . I should definitely have paid more attention to the characteristics of the dynatron before accepting, given the overall price it would not have made a big difference, somehow it is my fault even if I expected more clever choices from the retailer.

By the way you mentionned these dynatron are more suited to servers, why is that ? Compactness ? Noise level not that much important because usually in dedicated room ? I do not see the point versus noctua fans, even at top speed 8kRPM the theoretical airflow does not seem impressive at all

[flotus1]

These Dynatron coolers are designed to fit in a 2U server rack. Hence they have to be small and compensate the lack of mass and surface area with a high-rpm fan. The noise this produces is not an issue in server applications because the server room is loud anyway. Only if you use it as a workstation sitting in the same room with you. The high-rpm fans also help with static pressure which is needed to force enough air through servers with restricted airflow. Slow-spinning larger fans would be worse here despite their higher maximum volume flow rate. It's all fluid dynamics
Workstations need to be less compact and can accommodate larger coolers with slower spinning fans.

[naffrancois]

Thanks you flotus1 for this great explanation

[CFDfan]

Originally Posted by flotus1

The retailer might be correct about CPU1 sitting in the hot exhaust air of CPU2, depending on the orientation of the CPU coolers. Mine are both pointing upwards so I don't have this issue. It is a common problem with shadowed layouts.
65°C is perfectly fine for these CPUs, nothing to worry about. Mine are sitting in the low 50s under full load.
Those Dynatron coolers are a royal pita. Retailers using these for normal workstations instead of servers should not be allowed to sell workstations in my opinion. I used Noctua NH-U14s TR4-SP3, the largest they had. There are two smaller variants if you can not fit them into your case: NH-U12s TR4-SP3 and NH-U9 TR4-SP3. I would recommend swapping out the coolers and sending the invoice to your retailer

Here is how mine looked in the old case. I am using a slightly different case now, but the layout is similar.
Attachment 62607

I built a PC with Asus wrx80E-sage motherboard and Threadripper 5975wx CPU for CFD simulations. I installed Noctua NH-U14s, because that seems to be the "largest" aircooler for SP3 socket. At full CPU load its temperatures went to 85C. I am an electronic Engineer and know that 85C is not good long term. Tried with EVGA liquid cooler (two, 120mm fans) , but its pump base covers only 50%-60% of the 5975wx surface, so the temperature went even higher (with 10C). I wander if you know of a good air or water cooler for the SP3 socket and TDP of about 300W. I did a lot of googling about it and the best I found was NH-U14s (for air cooling).
Also should I disable AMD's SMT in the bios for better CFD simulation performance?

[flotus1]

The CPU is rated for a maximum temperature of 95°C https://www.amd.com/de/products/cpu/...per-pro-5975wx

There are a few air coolers available that might do better than a Noctua NH-U14s.
be quiet! Dark Rock Pro TR4: it's a dual-tower cooler with 2 fans. Don't expect huge improvements.
IceGiant ProSiphon Elite: allegedly does well with power consumption 200W and beyond. Beware of the size though, and it's not particularly quiet.

There are a few AIO liquid coolers with cold plates specifically designed for these larger CPUs. Don't know which ones on the top of my head.
Edit: Enermax Liqtech 360 OC TR4 is one example. They had some severe quality issues with these, don't know how that turned out.

Or you can bite the bullet and do a custom water loop. EK water blocks has plenty of water blocks that completely cover these CPUs.

[CFDfan]

Originally Posted by flotus1

The CPU is rated for a maximum temperature of 95°C https://www.amd.com/de/products/cpu/...per-pro-5975wx

There are a few air coolers available that might do better than a Noctua NH-U14s.
be quiet! Dark Rock Pro TR4: it's a dual-tower cooler with 2 fans. Don't expect huge improvements.
IceGiant ProSiphon Elite: allegedly does well with power consumption 200W and beyond. Beware of the size though, and it's not particularly quiet.

There are a few AIO liquid coolers with cold plates specifically designed for these larger CPUs. Don't know which ones on the top of my head.
Edit: Enermax Liqtech 360 OC TR4 is one example. They had some severe quality issues with these, don't know how that turned out.

Or you can bite the bullet and do a custom water loop. EK water blocks has plenty of water blocks that completely cover these CPUs.

Here is a real life comparison I found between different air-coolers for 5950x.
You were exactly right - ProSiphon and Dark rock you recommended were performing best in terms of temperature and noise and better than corsair's liquid coolers. Thank you again. Another comparison on the bottom is also in favor of ProSiphon.

https://www.cowcotland.com/articles/...r-ton-cpu.html
https://www.youtube.com/watch?v=U-BWEDfrE9c

[cegan09]

You can also try the Arctic 4U SP3 cooler, it makes that noctua look like a baby. It will fit threadripper chips just fine, and based on the pictures I'm looking at for that motherboard it would actually orient the fans to blow front/back instead of up/down.


I have one installed on an EPYC 7532 and at full load I've never seen it exceed 53°C with stock fan settings. Tomorrow I'm installing my second CPU so I'll see what happens to temps when one cooler is blowing into the other.

Only problem is that they seem to be out of stock everywhere right now. I asked Arctic directly and they said they expect more in mid September.

[CFDfan]

Originally Posted by cegan09

You can also try the Arctic 4U SP3 cooler, it makes that noctua look like a baby. It will fit threadripper chips just fine, and based on the pictures I'm looking at for that motherboard it would actually orient the fans to blow front/back instead of up/down.


I have one installed on an EPYC 7532 and at full load I've never seen it exceed 53°C with stock fan settings. Tomorrow I'm installing my second CPU so I'll see what happens to temps when one cooler is blowing into the other.

Only problem is that they seem to be out of stock everywhere right now. I asked Arctic directly and they said they expect more in mid September.

Thank you for your suggestion. Artic Freezer 4u SP3 has very good reviews but is out of stock anywhere I looked for, even on ebay. Also its heatsink sits quite low and am afraid it may block two of the adjacent RAM slots, which for CFD is No-No as you know. Another concern I have is that the lower diameter fans it has run probably at higher speeds and get noisier. Would be interesting to know your opinion about its (acoustic) noise at full power compared with say an AIO or air cooler (If you have such data).

The Prosyphon I bought was massive and seemed a very good choice but could only be installed vertically on my motherboard and I realized too late that it couldn't operate in that position at all. Returned it and got an expensive ASUS 360 AIO that claimed on the website of being compatible with the SP3/TP4 socket, but it was not. Returned it and am expecting Silverstone IceGem 360 AIO tomorrow. In the meantime running the PC with the Notctua having a second fan in push-pull configuration. The temperature reaches 80-85C when running the CPU with 100% load for say 10hours (an average CFD project). Not happy temperatures, will see how IceGem 360 AIO performs having a SP3 base. In short, it's been very disappointing 2 weeks of testing coolers for Thredripper 5975wx.

[cegan09]

Unless you're using RAM with monster heat spreaders there is plenty of clearance. Yesterday I was removing and installing RAM on my board and several of the slots sit under the heatsink. I was able to get sticks in and out without removing the heatsink. I'm using RAM with no heat spreaders because that's how most affordable ECC stuff comes. There is probably clearance for moderately sized heat spreaders, I can fit my finger between the top of my RAM and the bottom of the cooler where it overhangs.

As far as noise, it's so quiet. My system is built into a Sliger 4U rack mount chassis, so there are 3 fans across the front, and then the 4 fans over the two heatsinks. I don't have amazing fan control on my machine, there are just 4 preset modes I can pick from and I can't control individual fans. However, set to "optimal" mode the computer is very quiet, even with two of these heatsinks installed I think I’d be fine with this machine in the room with me (noise wise). My main workstation is a 5900x with a 240mm AIO, and that setup is louder than the Arctic when it spools up to handle a heavy load. This is mostly subjective data points, but I can say that even at full fan load I wouldn’t call the arctic loud.

Basically the Arctic is so massive it doesn't need to spin the fans much to keep cool. I set the system to full fan speed the other day and loaded the CPU to full load and it found steady state at 47°C. In "optimal" fan mode it evens out at 53°C. Even when I installed the second CPU, which has the two Arctic coolers feeding directly outlet to inlet, the chip that was getting fed warm air never got hotter than 63°C in "optimal" fan mode.

Now, I will point out that the 7532 I'm using is a 200W TDP chip, and your 5975WX is a 280W chip, so likely you will see slightly higher numbers with the same cooler. But based on the numbers I'm seeing, I don't think they'll be too much higher.

Hopefully the AIO works for you, but if for some reason it doesn’t, the Arctic is really good.

[CFDfan]

Originally Posted by cegan09

Unless you're using RAM with monster heat spreaders there is plenty of clearance. Yesterday I was removing and installing RAM on my board and several of the slots sit under the heatsink. I was able to get sticks in and out without removing the heatsink. I'm using RAM with no heat spreaders because that's how most affordable ECC stuff comes. There is probably clearance for moderately sized heat spreaders, I can fit my finger between the top of my RAM and the bottom of the cooler where it overhangs.

As far as noise, it's so quiet. My system is built into a Sliger 4U rack mount chassis, so there are 3 fans across the front, and then the 4 fans over the two heatsinks. I don't have amazing fan control on my machine, there are just 4 preset modes I can pick from and I can't control individual fans. However, set to "optimal" mode the computer is very quiet, even with two of these heatsinks installed I think I’d be fine with this machine in the room with me (noise wise). My main workstation is a 5900x with a 240mm AIO, and that setup is louder than the Arctic when it spools up to handle a heavy load. This is mostly subjective data points, but I can say that even at full fan load I wouldn’t call the arctic loud.

Basically the Arctic is so massive it doesn't need to spin the fans much to keep cool. I set the system to full fan speed the other day and loaded the CPU to full load and it found steady state at 47°C. In "optimal" fan mode it evens out at 53°C. Even when I installed the second CPU, which has the two Arctic coolers feeding directly outlet to inlet, the chip that was getting fed warm air never got hotter than 63°C in "optimal" fan mode.

Now, I will point out that the 7532 I'm using is a 200W TDP chip, and your 5975WX is a 280W chip, so likely you will see slightly higher numbers with the same cooler. But based on the numbers I'm seeing, I don't think they'll be too much higher.

Hopefully the AIO works for you, but if for some reason it doesn’t, the Arctic is really good.

Thank you for the information. Artic sounds very encouraging. The size of my RAM is L x W x H: ‎13.3 x 0.7 x 3.4 cm. Would be very grateful if you could tell me if the 3.4cm height of my RAM would interfere with the heatsink of the Artic

[cegan09]

Originally Posted by CFDfan

Thank you for the information. Artic sounds very encouraging. The size of my RAM is L x W x H: ‎13.3 x 0.7 x 3.4 cm. Would be very grateful if you could tell me if the 3.4cm height of my RAM would interfere with the heatsink of the Artic

I just measured one of the sticks I pulled out yesterday, and they are 3.12cm tall. There is more than 3mm of extra clearance, so it sounds like they should fit.

[CFDfan]

IceGem 360 AIO performed quite well with 5975wx. The temperatures dropped 5-7C compared with noctua U14s (with an extra fan). The acoustic noise also seemed to be a bit lower, so I will use IceGem 360 till Artic 4u becomes available to try.

[CFDfan]

Originally Posted by flotus1

The retailer might be correct about CPU1 sitting in the hot exhaust air of CPU2, depending on the orientation of the CPU coolers. Mine are both pointing upwards so I don't have this issue. It is a common problem with shadowed layouts.
65°C is perfectly fine for these CPUs, nothing to worry about. Mine are sitting in the low 50s under full load.
Those Dynatron coolers are a royal pita. Retailers using these for normal workstations instead of servers should not be allowed to sell workstations in my opinion. I used Noctua NH-U14s TR4-SP3, the largest they had. There are two smaller variants if you can not fit them into your case: NH-U12s TR4-SP3 and NH-U9 TR4-SP3. I would recommend swapping out the coolers and sending the invoice to your retailer

Here is how mine looked in the old case. I am using a slightly different case now, but the layout is similar.
Attachment 62607

Hi flotus1. Yes, the assembly layout you posted was nice. I wander what the temperatures are of the RAMs sitting on the top of the CPU fans and most importantly what tool you use to get the temperature readings. The HWMonitor tool I use would show the RAM temperatures of a Chinese, dual X99 motherboard, but doesn't read them on the 10 times more expensive Asus WRX80E-sage motherboard. The MSI monitoring tool you've suggested in some of the posts doesn't work with ASUS. Asus has its own monitoring tool that monitors a lot of rubbish, but not the RAM temperatures. So I wander if you know of something else to get the RAM temperatures.

[flotus1]

As far as I know, the only somewhat reliable way to get these non-standard temperature sensor readouts is with proprietary software. I.e. the stuff you can download from your motherboard manufacturers homepage. Supermicro for example has their "SuperDoctor" software. That's where I get memory temperature values.
Memory overheating is not necessarily a huge issue. It only happens with high density DDR4 RDIMMs. The image here was taken with 16GB modules that were unproblematic. 64 GB RDIMMs for example run much hotter.

If you are on Linux, you can try your luck with i2c and sensors. At your own risk
https://www.reddit.com/r/linux/comme...ture_on_linux/