OpenFOAM benchmarks on various hardware

wkernkamp · February 19, 2022, 22:53

Quote:

Originally Posted by ErikAdr

I'm considering to buy three more of these as an alternative to a dual EPYC 7313 system, since I have given up finding EPYC 7313 for sale. I think/hope the Ram prices will reduce in near future. The chipset/board is born with 2.5 Gb ethernet, and I think that could be sufficient for our need. A 5 port switch supporting 2.5 Gb ethernet is about Dkr 750 ~ 100 €.

I found the link in this thread to my r810 cluster of 2 with 1Gb ethernet. It scales perfectly, so you should be fine with 2.5Gb Ethernet.

Quote:

Originally Posted by wkernkamp

A single Dell Poweredge R810 4x(E7-4870 2.4 Ghz x 10 cores/20 threads)
1 2138.42
2 1213.28
4 454.14
6 329.9
8 243.29
10 166.17
12 182.94
16 160.47
20 149.23
24 144.08
30 148.82
32 139.42
36 138.79
40 151.01

Two Dell Poweredge R810 4x(E7-4870 2.4 Ghz x 10 cores/20 threads)
First half of processes on one node, the rest on the other.

8 263.8
16 155.51
24 122.77
32 105.56
40 102.47
48 86.83
64 78.25

The speed of the network is just 1Gb Ethernet right now (switch limited), so I might improve on this a bit. The difference for 8 nodes is 20 seconds.

wkernkamp · February 22, 2022, 00:05

supermicro H8QG6-F CSE-828 4x Opteron 6376 32x8MB DDR3-1600

1 2173.62
2 1456.88
4 512.48
8 252.12
12 194.95
16 157.38
24 123.78
32 96.17
48 90.1
64 86.31

flotus1 · February 22, 2022, 11:36

Finally, the elusive quad-Opteron.
Can you share any insight for how this result compares to the quad-Xeon system you posted? I'm a bit surprised that the Opterons are so much faster. Or more precisely, the quad-Xeon seems a bit too slow. Was there anything holding the Dell Poweredge R810 back?

wkernkamp · February 22, 2022, 20:42

The r810 with four processors has just 2 memory channels per processor (a really poor design choice). You can replace cpu 3 and 4 with a special blank that links their memory channels to cpu 1 and 2. So as a two processor machine it has four channels per processor.

linuxguy123 · February 23, 2022, 21:30

Great numbers, ErikAdr. Your data points clearly illustrate that memory speed is the deciding factor when running these benchmarks.

Best price/performance ?

Intel i5-12600 - C$359
Kingston Fury Beast 2x16 GB 6000 MHz - C$500
Asus TUF Gaming B660M-plus wifi - C$235
Total: $1,095
Execution time: 107 seconds.

Used EPYC 7601 processor C$525
New Supermicro H11SSi C$490
8x4GB 2666MT/s RAM C$260
Total: $1,275
Execution time: 65 seconds.

I included only the CPU, RAM and MB as the rest of these systems are pretty much the same.

Quote:

I'm considering to buy three more of these as an alternative to a dual EPYC 7313 system

Dual 7601 or 7551 systems are pretty affordable and will run the benchmark in sub 40 seconds. Probably 35 seconds. 7002 and 7003 processors are somewhat faster, but the cost goes through the roof.

You could probably build 2 dual processor 7601s and connect them with 10 GbE and have a run time of 25 seconds or so. Would be cheaper and faster than 4 12600 systems with fast RAM.

linuxguy123 · February 23, 2022, 21:45

Quote:

Originally Posted by ErikAdr

I have build a quite low end computer, but with quality components. It is using a i5-12600 cpu with two channel DDR5 Ram. I'm pleased with the performance.

12th Gen Intel(R) Core(TM) i5-12600 DDR5 @ 4800

# cores Wall time (s):
------------------------
1 427.45
2 234.12
4 149.91
6 125.75

12th Gen Intel(R) Core(TM) i5-12600 DDR5 @ 5600

# cores Wall time (s):
------------------------
1 410.79
2 219.95
4 137.39
6 112.58

12th Gen Intel(R) Core(TM) i5-12600 DDR5 @ 6000

# cores Wall time (s):
------------------------
1 399.94
2 213.75
4 131.87
6 107.09

Your processing time was still dropping with increases in cores used. I'm guessing the run time would drop further with processors with a higher core count, ie the 12700, 12800 or 12900.

107 seconds is no slouch, especially for a "desktop" processor. Personal CFD workstations just became a lot more available to people on budgets.

wkernkamp · February 23, 2022, 21:55

Quote:

Originally Posted by linuxguy123

Best price/performance ?

Intel i5-12600 - C$359
Kingston Fury Beast 2x16 GB 6000 MHz - C$500
Asus TUF Gaming B660M-plus wifi - C$235
Total: $1,095
Execution time: 107 seconds.

Used EPYC 7601 processor C$525
New Supermicro H11SSi C$490
8x4GB 2666MT/s RAM C$260
Total: $1,275
Execution time: 65 seconds.

The price for the supermicro QUAD opteron 6376 was $320
256 Gb DDR3-1600 RDIMM's about $480 (I had it already, but that is what you pay on Ebay)
The system came with a 40 Gb Infiniband card (working I checked) and dual Gigabit Ethernet
Total: $800
Execution time: 86 seconds.

linuxguy123 · February 23, 2022, 22:31

Quote:

Originally Posted by wkernkamp

The price for the supermicro QUAD opteron 6376 was $320
256 Gb DDR3-1600 RDIMM's about $480 (I had it already, but that is what you pay on Ebay)
The system came with a 40 Gb Infiniband card (working I checked) and dual Gigabit Ethernet
Total: $800
Execution time: 86 seconds.

Speed ratios:
12600: 107/107 = 1x
7601: 107/65 = 1.646x
Opteron: 107/86 = 1.244x

Cost/speed ratio
12600: C$1095 / 1x = $1096
7601: C$1275 / 1.646x = $774 / 1x
Opteron: US$800/.8 = C$1000 /1.244 = $803 / 1x

It would be really interesting to see how fast the Intel 12th gen would be with more cores.

The other interesting thing is that Zen4 is due out in April and Intel 13th gen later in 2022. DDR5 memory cost will fall like a rock at some point and the $/performance of these desktop systems will only rise.

wkernkamp · February 24, 2022, 00:17

Cost of memory is the issue with the newer systems versus the old servers.

ErikAdr · February 24, 2022, 16:15

[QUOTE=linuxguy123;822980]Your processing time was still dropping with increases in cores used. I'm guessing the run time would drop further with processors with a higher core count, ie the 12700, 12800 or 12900.

The i5-12600 has 6 power cores, whereas all the i7 Alder Lake cpus have 8 power cores, and a number of efficient cores. I don't think efficient cores are usefull for HPC. How does mpi work in a heterogeneous environment? I think it is possible to switch off efficient cores on motherboards with a Z690 chipset, but I don't know if that is the case with a B660. I took the safe solution, and bought a cpu without efficient cores.

linuxguy123 · February 24, 2022, 16:56

Quote:

Originally Posted by wkernkamp

Cost of memory is the issue with the newer systems versus the old servers.

Most of the modelling we'll do doesn't need much RAM. What openFOAM needs is lots of memory bandwidth. Fast DDR5 memory is expensive (right now), but 2 channel systems only need 2 sticks of it. Whereas a single EPYC needs 8 sticks.

I can buy 2x16GB sticks of fast DDR5 for about the same price as 8x8GB sticks of DDR4 3200 for an EPYC setup.

Don't get me wrong, I love my EPYC 7601 system. But it won't be long until desktop systems surpass it. The first architecture to use 4 channels of fast DDR5 memory with 12ish fast cores will make the 7001 Naples EPYC processors obsolete. Until then, I'll continue to use mine.

As the core count on desktop processors climbs, the memory subsystem needs to be faster and faster to feed the cores. Zen4 is slated to remain at 16 cores. Probably because that is the core limit to feed with 2 memory channels. Rumor has it that Zen5 will have more cores - 24 or maybe 32. I'm guessing that they'll use a 4 channel memory system on Zen 5 to feed the cores.

linuxguy123 · February 24, 2022, 17:00

Quote:

Originally Posted by ErikAdr

The i5-12600 has 6 power cores, whereas all the i7 Alder Lake cpus have 8 power cores, and a number of efficient cores. I don't think efficient cores are usefull for HPC.

I suspect you are right.

Quote:

How does mpi work in a heterogeneous environment? I think it is possible to switch off efficient cores on motherboards with a Z690 chipset, but I don't know if that is the case with a B660. I took the safe solution, and bought a cpu without efficient cores.

You can control the number of cores used when you call openFOAM. So just limit the cores being called to the performance cores.

I suspect that the Linux scheduler will get enhancements to deal with performance core and efficiency core scheduling in the near future.

Iuvatix · February 25, 2022, 07:13

The system of ErikAdr uses 2 single rank DDR5 sticks if I understand correctly. As far as I know for DDR4 you needed to have 2 dual rank or 4 single rank sticks to populate all memory channels of a 2 channel CPU. Does DDR5 RAM works the same way?

With this in mind, 8 performance cores of an i7 12700 and overclocking of the RAM, Alder Lake seems like a really good workstation option

flotus1 · February 25, 2022, 07:46

Quote:

The system of ErikAdr uses 2 single rank DDR5 sticks if I understand correctly

Yes, all 16GB DDR5 UDIMM modules currently sold are single rank.

Quote:

As far as I know for DDR4 you needed to have 2 dual rank or 4 single rank sticks to populate all memory channels of a 2 channel

Populating each channel with at least one DIMM -regardless of the DIMMs internal organization- was always enough to utilize all memory channels. You got a slight increase in performance with more than one rank per memory channel, assuming identical transfer rate.

Quote:

Does DDR5 RAM works the same way?

Seems like it: https://www.igorslab.de/single-vs-du...-alder-lake/2/

RolandS · March 1, 2022, 05:40

Quote:

Originally Posted by RolandS

Dear Will, Flotus, George,

Thanks for all your effort and advices.

I turned my computer into a dual boot machine: Win10 + LinuxMint.
The Linux results are drastically improved.

As I said, under Win10 I operate OpenFOAM-v2006 under the mingw cross-compiler (as prepared for the FreeCAD windows software).

The Linux results are:

# cores Wall time (s):
------------------------
6 - 163.36
10 - 109.74
14 - 89.72
18 - 81.18
22 - 77.7
24 - 76.53

In the attached graph you see the incredible speed-up of operating openFOAM directly under Linux.

Unless you see other issues to address, I am happy with this result.

Best regards,

Roland

Dear Forum,

I ran the benchmark 1 year ago, and optimized my machine.

Today I made some repairs to the memory. Tried to re-test its performance, but it appears that the benchmark test does not run anymore.

Since the successful tests some changes have occurred in the Linux Mint OS. Perhaps Python was renewed, etc.

Do you know if I should make edits to the benchmark code? Or perhaps should make other adaptations?

Here is my terminal report, showing various error messages:

Code:

@:~/Software/CFD/Benchmarking/bench_template_PostRepair$ bash run24.sh
Prepare case run_6...
surfaceFeatureExtract already run on /home/roland/Software/CFD/Benchmarking/bench_template_PostRepair/run_6: remove log file 'log.surfaceFeatureExtract' to re-run
blockMesh already run on /home/roland/Software/CFD/Benchmarking/bench_template_PostRepair/run_6: remove log file 'log.blockMesh' to re-run
decomposePar already run on /home/roland/Software/CFD/Benchmarking/bench_template_PostRepair/run_6: remove log file 'log.decomposePar' to re-run
Error getting 'numberOfSubdomains' from 'system/decomposeParDict'
snappyHexMesh already run on /home/roland/Software/CFD/Benchmarking/bench_template_PostRepair/run_6: remove log file 'log.snappyHexMesh' to re-run
ls: cannot access 'processor*': No such file or directory
ls: cannot access 'processor*': No such file or directory

real    0m0,034s
user    0m0,005s
sys    0m0,020s
Prepare case run_10...
surfaceFeatureExtract already run on /home/roland/Software/CFD/Benchmarking/bench_template_PostRepair/run_10: remove log file 'log.surfaceFeatureExtract' to re-run
blockMesh already run on /home/roland/Software/CFD/Benchmarking/bench_template_PostRepair/run_10: remove log file 'log.blockMesh' to re-run
decomposePar already run on /home/roland/Software/CFD/Benchmarking/bench_template_PostRepair/run_10: remove log file 'log.decomposePar' to re-run
Error getting 'numberOfSubdomains' from 'system/decomposeParDict'
snappyHexMesh already run on /home/roland/Software/CFD/Benchmarking/bench_template_PostRepair/run_10: remove log file 'log.snappyHexMesh' to re-run
ls: cannot access 'processor*': No such file or directory
ls: cannot access 'processor*': No such file or directory

real    0m0,024s
user    0m0,013s
sys    0m0,009s
Prepare case run_14...
surfaceFeatureExtract already run on /home/roland/Software/CFD/Benchmarking/bench_template_PostRepair/run_14: remove log file 'log.surfaceFeatureExtract' to re-run
blockMesh already run on /home/roland/Software/CFD/Benchmarking/bench_template_PostRepair/run_14: remove log file 'log.blockMesh' to re-run
decomposePar already run on /home/roland/Software/CFD/Benchmarking/bench_template_PostRepair/run_14: remove log file 'log.decomposePar' to re-run
Error getting 'numberOfSubdomains' from 'system/decomposeParDict'
snappyHexMesh already run on /home/roland/Software/CFD/Benchmarking/bench_template_PostRepair/run_14: remove log file 'log.snappyHexMesh' to re-run
ls: cannot access 'processor*': No such file or directory
ls: cannot access 'processor*': No such file or directory

real    0m0,017s
user    0m0,011s
sys    0m0,010s
Prepare case run_18...
surfaceFeatureExtract already run on /home/roland/Software/CFD/Benchmarking/bench_template_PostRepair/run_18: remove log file 'log.surfaceFeatureExtract' to re-run
blockMesh already run on /home/roland/Software/CFD/Benchmarking/bench_template_PostRepair/run_18: remove log file 'log.blockMesh' to re-run
decomposePar already run on /home/roland/Software/CFD/Benchmarking/bench_template_PostRepair/run_18: remove log file 'log.decomposePar' to re-run
Error getting 'numberOfSubdomains' from 'system/decomposeParDict'
snappyHexMesh already run on /home/roland/Software/CFD/Benchmarking/bench_template_PostRepair/run_18: remove log file 'log.snappyHexMesh' to re-run
ls: cannot access 'processor*': No such file or directory
ls: cannot access 'processor*': No such file or directory

real    0m0,017s
user    0m0,009s
sys    0m0,012s
Prepare case run_22...
surfaceFeatureExtract already run on /home/roland/Software/CFD/Benchmarking/bench_template_PostRepair/run_22: remove log file 'log.surfaceFeatureExtract' to re-run
blockMesh already run on /home/roland/Software/CFD/Benchmarking/bench_template_PostRepair/run_22: remove log file 'log.blockMesh' to re-run
decomposePar already run on /home/roland/Software/CFD/Benchmarking/bench_template_PostRepair/run_22: remove log file 'log.decomposePar' to re-run
Error getting 'numberOfSubdomains' from 'system/decomposeParDict'
snappyHexMesh already run on /home/roland/Software/CFD/Benchmarking/bench_template_PostRepair/run_22: remove log file 'log.snappyHexMesh' to re-run
ls: cannot access 'processor*': No such file or directory
ls: cannot access 'processor*': No such file or directory

real    0m0,018s
user    0m0,008s
sys    0m0,014s
Prepare case run_24...
surfaceFeatureExtract already run on /home/roland/Software/CFD/Benchmarking/bench_template_PostRepair/run_24: remove log file 'log.surfaceFeatureExtract' to re-run
blockMesh already run on /home/roland/Software/CFD/Benchmarking/bench_template_PostRepair/run_24: remove log file 'log.blockMesh' to re-run
decomposePar already run on /home/roland/Software/CFD/Benchmarking/bench_template_PostRepair/run_24: remove log file 'log.decomposePar' to re-run
Error getting 'numberOfSubdomains' from 'system/decomposeParDict'
snappyHexMesh already run on /home/roland/Software/CFD/Benchmarking/bench_template_PostRepair/run_24: remove log file 'log.snappyHexMesh' to re-run
ls: cannot access 'processor*': No such file or directory
ls: cannot access 'processor*': No such file or directory

real    0m0,017s
user    0m0,014s
sys    0m0,006s
Run for 6...
Run for 10...
Run for 14...
Run for 18...
Run for 22...
Run for 24...
# cores   Wall time (s):
------------------------
6
10
14
18
22
24
@:~/Software/CFD/Benchmarking/bench_template_PostRepair$

Here is my Benchmark Code:

Code:

#!/bin/bash

# Prepare cases
for i in 6 10 14 18 22 24; do
    d=run_$i
    echo "Prepare case ${d}..."
    cp -r basecase $d
    cd $d
    if [ $i -eq 1 ] 
    then
        mv Allmesh_serial Allmesh
    fi
    sed -i "s/method.*/method scotch;/" system/decomposeParDict
    sed -i "s/numberOfSubdomains.*/numberOfSubdomains ${i};/" system/decomposeParDict
    time ./Allmesh
    cd ..
done

# Run cases
for i in 6 10 14 18 22 24; do
    echo "Run for ${i}..."
    cd run_$i
    if [ $i -eq 1 ] 
    then
        simpleFoam > log.simpleFoam 2>&1
    else
        mpiexec -np ${i} simpleFoam -parallel > log.simpleFoam 2>&1
        # mpiexec -np 48 simpleFoam -parallel
    fi
    cd ..
done

# Extract times
echo "# cores   Wall time (s):"
echo "------------------------"
for i in 6 10 14 18 22 24; do
    echo $i `grep Execution run_${i}/log.simpleFoam | tail -n 1 | cut -d " " -f 3`
done

Greetz!

Roland

Simbelmynë · March 1, 2022, 06:01

@RolandS

Could you post the content of decomposeParDict?

Also, please check the OF version that you are running.

RolandS · March 1, 2022, 07:30

Quote:

Originally Posted by Simbelmynë

@RolandS

Could you post the content of decomposeParDict?

Also, please check the OF version that you are running.

Hi Simbelmynë,

This is OpenFOAM-v2012 (patch=210618)

decomposeParDict:

Code:

/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  4.x                                   |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    object      decomposeParDict;
}

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains 6;

method          hierarchical;
// method          ptscotch;

simpleCoeffs
{
    n               (4 1 1);
    delta           0.001;
}

hierarchicalCoeffs
{
    n               (3 2 1);
    delta           0.001;
    order           xyz;
}

manualCoeffs
{
    dataFile        "cellDecomposition";
}


// ************************************************************************* //

Simbelmynë · March 1, 2022, 07:53

Is that from the basecase or is it from some run_ folder?

Perhaps you also need to clear the case (remove all run_* folders).

RolandS · March 1, 2022, 08:12

Quote:

Originally Posted by Simbelmynë

Is that from the basecase or is it from some run_ folder?

Perhaps you also need to clear the case (remove all run_* folders).

Indeed, it was from a run_* folder. Here is the one from basecase:

Code:

/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  4.x                                   |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    object      decomposeParDict;
}

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains 6;

method          hierarchical;
// method          ptscotch;

simpleCoeffs
{
    n               (4 1 1);
    delta           0.001;
}

hierarchicalCoeffs
{
    n               (3 2 1);
    delta           0.001;
    order           xyz;
}

manualCoeffs
{
    dataFile        "cellDecomposition";
}


// ************************************************************************* //

Hoever, I did remove all de run_* folders before this test. So, that is not the issue.

Is the OF version Okay?

Simbelmynë · March 1, 2022, 10:11

As far as I can tell there is no difference between the basecase and the run_* decomposeParDict?

In that case it seems that the sed command fails. You should have scotch not hierarchical after the sed command.

The only suggestion I have left is to also look at the log.* files. In particular log.decomposePar, and try to find out why you execute the commands several times for each case.

February 22, 2022, 00:05	supermicro 4x opteron 6376	#462
wkernkamp Senior Member Will Kernkamp Join Date: Jun 2014 Posts: 372 Rep Power: 14	supermicro H8QG6-F CSE-828 4x Opteron 6376 32x8MB DDR3-1600 1 2173.62 2 1456.88 4 512.48 8 252.12 12 194.95 16 157.38 24 123.78 32 96.17 48 90.1 64 86.31 flotus1 likes this.

February 22, 2022, 20:42	r810 has just 2 channels per processor	#464
wkernkamp Senior Member Will Kernkamp Join Date: Jun 2014 Posts: 372 Rep Power: 14	The r810 with four processors has just 2 memory channels per processor (a really poor design choice). You can replace cpu 3 and 4 with a special blank that links their memory channels to cpu 1 and 2. So as a two processor machine it has four channels per processor. flotus1 and linuxguy123 like this.

February 24, 2022, 16:15		#470
ErikAdr Member Erik Andresen Join Date: Feb 2016 Location: Denmark Posts: 35 Rep Power: 10	[QUOTE=linuxguy123;822980]Your processing time was still dropping with increases in cores used. I'm guessing the run time would drop further with processors with a higher core count, ie the 12700, 12800 or 12900. The i5-12600 has 6 power cores, whereas all the i7 Alder Lake cpus have 8 power cores, and a number of efficient cores. I don't think efficient cores are usefull for HPC. How does mpi work in a heterogeneous environment? I think it is possible to switch off efficient cores on motherboards with a Z690 chipset, but I don't know if that is the case with a B660. I took the safe solution, and bought a cpu without efficient cores. linuxguy123 likes this.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
How to contribute to the community of OpenFOAM users and to the OpenFOAM technology	wyldckat	OpenFOAM	17	November 10, 2017 16:54
UNIGE February 13th-17th - 2107. OpenFOAM advaced training days	joegi.geo	OpenFOAM Announcements from Other Sources	0	October 1, 2016 20:20
OpenFOAM Training Beijing 22-26 Aug 2016	cfd.direct	OpenFOAM Announcements from Other Sources	0	May 3, 2016 05:57
New OpenFOAM Forum Structure	jola	OpenFOAM	2	October 19, 2011 07:55
Hardware for OpenFOAM LES	LijieNPIC	Hardware	0	November 8, 2010 10:54

February 22, 2022, 11:36		#463
flotus1 Super Moderator Alex Join Date: Jun 2012 Location: Germany Posts: 3,428 Rep Power: 49	Finally, the elusive quad-Opteron. Can you share any insight for how this result compares to the quad-Xeon system you posted? I'm a bit surprised that the Opterons are so much faster. Or more precisely, the quad-Xeon seems a bit too slow. Was there anything holding the Dell Poweredge R810 back?

February 24, 2022, 00:17		#469
wkernkamp Senior Member Will Kernkamp Join Date: Jun 2014 Posts: 372 Rep Power: 14	Cost of memory is the issue with the newer systems versus the old servers.

February 25, 2022, 07:13		#473
Iuvatix New Member Florian Join Date: May 2021 Posts: 8 Rep Power: 5	The system of ErikAdr uses 2 single rank DDR5 sticks if I understand correctly. As far as I know for DDR4 you needed to have 2 dual rank or 4 single rank sticks to populate all memory channels of a 2 channel CPU. Does DDR5 RAM works the same way? With this in mind, 8 performance cores of an i7 12700 and overclocking of the RAM, Alder Lake seems like a really good workstation option

March 1, 2022, 06:01		#476
Simbelmynë Senior Member Join Date: May 2012 Posts: 552 Rep Power: 16	@RolandS Could you post the content of decomposeParDict? Also, please check the OF version that you are running.

March 1, 2022, 07:53		#478
Simbelmynë Senior Member Join Date: May 2012 Posts: 552 Rep Power: 16	Is that from the basecase or is it from some run_ folder? Perhaps you also need to clear the case (remove all run_* folders).

March 1, 2022, 10:11		#480
Simbelmynë Senior Member Join Date: May 2012 Posts: 552 Rep Power: 16	As far as I can tell there is no difference between the basecase and the run_* decomposeParDict? In that case it seems that the sed command fails. You should have scotch not hierarchical after the sed command. The only suggestion I have left is to also look at the log.* files. In particular log.decomposePar, and try to find out why you execute the commands several times for each case.