Tuning the EPYC Rome for CFD

Habib-CFD · January 7, 2020, 12:52

Hi, after many effort on 2920x and 2970wx finally I built a new PC based on EPYC Rome.

A single CPU, 7302p installed on MBD-H11SSL-i Rev2 Bios update V2b. Due to short plan and the limited budget, I had to choose 8*2666MHz ECC registered (Dual) RAM. In addition to some common problems with VGA driver and CPU mounted direction (Cooler installing), I found another issue with this configuration. The simulation run time based on Flow-3d software only exhibit 15 percent enhancement in comparison to 2920x thread ripper!
I was expect at least a 50 percent improvement because of the 8 channel benefits. The result comes from 12 cores of 2920x and 16 cores of 7302p (smt disabled). I already tried to use some manual to tun up the CPU but this motherboard is really strange and not very user friend. The booting time takes too much even on the 970evo nvme. I also tried to overclocked the RAM frequency. The system is very stable on 3200MHz but the latency decreases the performance. The CPU itself work really cool. Between 15 to 20 degree less than 2920x using exactly same case and cooling fan. The Numa configuration in automatic mode of bios, completely disabled but it possible to change it manually with out positive effect. Currently, I am thinking about a solution to increase the clock of CPU up to 3.6GHz but still no way!!

http://developer.amd.com/wp-content/resources/56420.pdf

Could you please share your experience on the Bios customization of Epyc rome CPU for CFD purposes.

Habib-CFD · January 8, 2020, 02:04

The customized option so far:

SMT=disabled

Transparent Secure Memory Encryption TSME or SMEE = disabled

NUMA Node per Socket was checked but the default one NUMA domain per socket is working fine (different option NPS1 to NPS4)

Memory interleaving set Auto with 2666MHz

cTDP set to 280W,
APBDIS set to pP0: Highest-performing Infinity Fabric P-state

The temperature remains around 40 degrees and the "cpupower monitor" shows the 3200 frequency for all 16 cores.

One suspected item is L3 cash which is reporting 16MB by "lscpu" but the "sudo lshw -short -c memory" shows the correct value (128MB).

https://developer.amd.com/wp-content...56745_0.80.pdf

Suggested By AMD
https://www.amd.com/system/files/doc...SYS-FLUENT.pdf

BIOS Setting NPS = NPS4, SMT = Off, Boost = On, X2APIC = On, Determinism Slider = Performance, Preferred IO=Enabled
OS Settings Governor=Performance, CC6 = Disabled (GNU Compiler Collection: generate FMA4 and AVX instructions )

https://www.certdepot.net/rhel7-get-...-cpu-governor/

http://www.prace-ri.eu/best-practice-guide-amd-epyc

January 7, 2020, 12:52	Tuning the EPYC Rome for CFD	#1
Habib-CFD Member Join Date: Oct 2019 Posts: 65 Rep Power: 7	Hi, after many effort on 2920x and 2970wx finally I built a new PC based on EPYC Rome. A single CPU, 7302p installed on MBD-H11SSL-i Rev2 Bios update V2b. Due to short plan and the limited budget, I had to choose 8*2666MHz ECC registered (Dual) RAM. In addition to some common problems with VGA driver and CPU mounted direction (Cooler installing), I found another issue with this configuration. The simulation run time based on Flow-3d software only exhibit 15 percent enhancement in comparison to 2920x thread ripper! I was expect at least a 50 percent improvement because of the 8 channel benefits. The result comes from 12 cores of 2920x and 16 cores of 7302p (smt disabled). I already tried to use some manual to tun up the CPU but this motherboard is really strange and not very user friend. The booting time takes too much even on the 970evo nvme. I also tried to overclocked the RAM frequency. The system is very stable on 3200MHz but the latency decreases the performance. The CPU itself work really cool. Between 15 to 20 degree less than 2920x using exactly same case and cooling fan. The Numa configuration in automatic mode of bios, completely disabled but it possible to change it manually with out positive effect. Currently, I am thinking about a solution to increase the clock of CPU up to 3.6GHz but still no way!! http://developer.amd.com/wp-content/resources/56420.pdf Could you please share your experience on the Bios customization of Epyc rome CPU for CFD purposes. Freewill1 likes this.

January 8, 2020, 02:04		#2
Habib-CFD Member Join Date: Oct 2019 Posts: 65 Rep Power: 7	The customized option so far: SMT=disabled Transparent Secure Memory Encryption TSME or SMEE = disabled NUMA Node per Socket was checked but the default one NUMA domain per socket is working fine (different option NPS1 to NPS4) Memory interleaving set Auto with 2666MHz cTDP set to 280W, APBDIS set to pP0: Highest-performing Infinity Fabric P-state The temperature remains around 40 degrees and the "cpupower monitor" shows the 3200 frequency for all 16 cores. One suspected item is L3 cash which is reporting 16MB by "lscpu" but the "sudo lshw -short -c memory" shows the correct value (128MB). https://developer.amd.com/wp-content...56745_0.80.pdf Suggested By AMD https://www.amd.com/system/files/doc...SYS-FLUENT.pdf BIOS Setting NPS = NPS4, SMT = Off, Boost = On, X2APIC = On, Determinism Slider = Performance, Preferred IO=Enabled OS Settings Governor=Performance, CC6 = Disabled (GNU Compiler Collection: generate FMA4 and AVX instructions ) https://www.certdepot.net/rhel7-get-...-cpu-governor/ http://www.prace-ri.eu/best-practice-guide-amd-epyc Last edited by Habib-CFD; January 8, 2020 at 09:53.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Epyc 7551 vs 6850K; Ansys Mechanical Bench	Duke711	Hardware	24	March 26, 2020 11:16
New 128 mini cluster - Cascade Lake SP or EPYC Rome?	SLC	Hardware	8	December 16, 2019 17:25
Operating System for AMD Epyc Workstation	jakethejake	Hardware	14	November 19, 2019 06:52
Epyc 7551 vs 6850K; Fluent Bench	Duke711	Hardware	4	April 7, 2019 23:05
AMD Epyc CFD benchmarks with Ansys Fluent	flotus1	Hardware	55	November 12, 2018 06:33