[Hajart]

Greetings Everyone,

I have just started a PhD on renewable energy modelling which will require to couple near-field and far-field CFD models of an estuary. My supervisor gave me the task of building a workstation. I am quite relatively new to CFD modelling having a civil engineering background and after long hours of research on the web, I designed a workstation with the following spec (Budget around £2000):

Motherboard: ASRock X99 Extreme6 - ATX - LGA2011-V3 Socket - X99 (£169)
CPU: Intel Xeon E5-2630V4 / 2.2 GHz processor (£661)
GPU: ASUS NVIDIA GeForce GTX 960 Graphics Card - 4G (£216)
RAM: 8*16GB Kingston - DDR4 - 2133 - DIMM 288-pin (£75)
Sound Card: ASUS Xonar DS - sound card (£43)
SSD: Samsung 850 PRO MZ-7KE128BW - 128 GB - SATA 6Gb/S (£71)
Harddrive: WD Black 2TB Performance Desktop Hard Disk Drive 7200 RPM SATA 6 Gb/S (£118)
Case: Antec Performance One P100 (£85)
Power Supply: Antec VP700P (£65)
Fans: 2*Antec TrueQuiet 140 - Case fan - 140 mm (£23)
CPU Fan: This is where I need help because I don't know if I need to buy a square ILM or Narrow ILM CPU Cooler

Could someone confirm that all the components for this workstation are compatible for linux ubuntu and tell me if the component association is adequate to run smoothly CFD models using FVCOM and Hydro3D thanks!!

[flotus1]

This Workstation will get the job done, but it might not be the ideal choice.

Motherboard: ASRock X99 Extreme6 - ATX - LGA2011-V3 Socket - X99 (£169)
This Motherboard will support Broadwell-E CPUs after a bios update, but I would recommend getting a motherboard that was specifically designed for Broadwell-E Processors. For example Asus X99-A II.

CPU: Intel Xeon E5-2630V4 / 2.2 GHz processor (£661)
This CPU has 10 relatively slow cores, only supports slow memory and you pay a premium price for 2S scalability. The better option is the Core I7-6900k. It has a high clock speed and can use fast memory. Both the memory and the CPU can be overclocked. It is more expensive than your iniital choice but definitely worth the money. Get a "boxed" version although you won't use the stock cooler.

GPU: ASUS NVIDIA GeForce GTX 960 Graphics Card - 4G (£216)
Good choice with linux compatibility in mind. Though the price seems a bit high for a GTX 960.

RAM: 8*16GB Kingston - DDR4 - 2133 - DIMM 288-pin (£75)
Are you sure that you need 128GB of RAM in a single-cpu workstation? I would start with 4*16GB, you can still upgrade later if necessary.
To be on the safe side use memory from the QVL of your motherboard: http://dlcdnet.asus.com/pub/ASUS/mb/...274.1469304114
Use the fastest 16GB modules you can afford, I recommend DDR4-3000 or higher.

Sound Card: ASUS Xonar DS - sound card (£43)
Do you really need a separate sound card? Virtually all motherboards have onboard sound.

SSD: Samsung 850 PRO MZ-7KE128BW - 128 GB - SATA 6Gb/S (£71)
The 850 pro is not worth the money compared to a 850 Evo. You should be able to get the 250GB version for the same price.

Harddrive: WD Black 2TB Performance Desktop Hard Disk Drive 7200 RPM SATA 6 Gb/S (£118)
I personally think that HDDs with less than 4TB are a waste of space and would recommend the HGST MegaScale DC 4000.B 4TB. But that is just my personal opinion.

Case: Antec Performance One P100 (£85)
That should do. My choice would be Fractal Design Define R5 since it comes with 3 140mm fans for quiet operation. Antec Performance One P100 only has room for one 120mm fan in the back which is quite obsolete.

Power Supply: Antec VP700P (£65)
No. Get a decent quality power supply that does not have to be oversized, for example BeQuiet Straight power E10 400W.

Fans: 2*Antec TrueQuiet 140 - Case fan - 140 mm (£23)
Depends on the case...

CPU Fan:
The motherboard you had in mind and the one i recommended use square ILM layout.
I would recommend one of the these two: Noctua NH-D15 or BeQuiet Dark Rock Pro 3.

[Hajart]

Cheers Alex, really appreciate for the times you’ve put to make this detailed review of my spec.
I have taken some of your advice and recommendation on board:

Motherboard: I think I will stick with the ASRock as its price is relatively cheap and based on reviews I read the BIOS seems really sturdy and complete.

CPU + RAM: If I reduced the RAM to 64GB then it gives me the budget to invest in the i7-6900K. So I think I will go for this processor with 4*16 “Corsair Vengeance LPX - DDR4 - 64 GB : 4 x 16 GB - DIMM 288-pin”.

Sound Card: I have gotten rid of it, as it is not necessary.

SSD: I went for the 850 Evo 250Gb.

HDD: I kept the same one, I will expand if need be.

Case + Fan: I followed your advice and went for the Fractal Design Define R5.

Power supply: I took a 80glod certified power supply as I couldn’t find your specific one in my university provider website. I went for the “Fractal Design Edison M 550W - power supply - 550 Watt”

CPU Cooler: I went for the Noctua NH-U9DX i4 as yours were not available on the supplier website.

I have one question, I believe that the Xeon processors are appreciated in CFD Modelling as they support ECC-RAM that reduces significantly the occurrence of simulation crash. It appears that with the i7-6900K I trade computational power over safety and stability of the ECC-RAM and Xeon processor, was a novice in CFD Modelling is that trade worth doing ?

[flotus1]

The Edison PSU is an excellent choice aswell.
For the CPU cooler, we have the same in one of our workstations but only for one reason: there was not enough space for larger coolers and the CPUs have a TDP of only 95W.
If you choose the I7-6900k which has a TDP of 140W it is a bit flimsy and will be quite noisy. I would recommend some top-range air cooler with at least two 140mm fans for quiet operation.
I hope the RAM you got is specified for at least DDR4-3000. Remember, memory bandwidth is very important for most CFD workloads. Fast memory is the cheapest way to get more performance.

I did not mention the ECC issue because you initially did not have ECC memory in your build. So I assumed you had already decided against it.
While it is true that ECC can prevent some soft-errors, this is usually not relevant for single-CPU workstations with a moderate amount of memory because these kinds of errors are pretty rare. If you can not tolerate a single soft-error during the whole life-time of the workstation of course you should use ECC memory. It is a trade-off between preventing very rare errors on the one hand and about 40-50% more performance for the same price on the other hand.

One last thing: keep in mind that if your motherboard comes shipped with an old bios version, you will need a "spare" Haswell-E CPU to do the bios update. The PC will not boot at all with a Broadwell-E CPU in this case. That is one of the main reasons why I recommended a new motherboard. Of course this is not an issue you do not assemble the workstation yourself which is what I would recommend.

[hyugakojiro999]

hi flotus1,

-Could u pls explain why mhz of Ram's is very important? Are there any tests done for this comparison?

I'm looking for 2. hand 5960x cpu. Can u recommend motherboard for this cpu? I cant afford too much. In fact, i think msi raider would be OK enough.

[flotus1]

Higher memory frequency translates into higher memory bandwidth. This allows data to be exchanged faster between CPU and memory which translates into a considerable performance improvement for many CFD applications. Maximizing memory bandwidth is a cheap and easy way to improve the performance of CFD workstation.
If you are looking for a cost-efficient CPU, the I7-5960x might not be your best choice since the two additional cores cost a lot. An I7-5820k should offer more "bang for the buck".

[hyugakojiro999]

Originally Posted by flotus1

To start with you might want to read this thread, it covers most of your questions (and some you did not ask but should know about):
http://www.cfd-online.com/Forums/har...modelling.html

I would go with the I7-6800k. It will draw less power and has better RAM compatibility when it comes to large quantities and high frequency.
Never mind that these CPUs officially support only up to DDR4-2400. Using XMP profiles or manual adjustment you can usually use much faster RAM. Be aware that beyond DDR4-2800, the CPU strap might be increased to 125MHz automatically. You might have to reduce the CPU multiplier to compensate for this.
Use the least amount of DIMMs possible to start with. In your case since you are aiming for 64GB total that would be 4*16GB. This increases system stability and RAM can be upgraded later if necessary.

Whatever you do, do not buy a Nvidia GT 730. It is old and slow. You will have trouble post-processing larger simulations with such a flimsy graphics card. The minimum should be a GTX 750TI with 2GB GDDR5.
Edit: And get an SSD with at least 240GB. Smaller models are just not worth the money and you will run out of disk space rather sooner than later.

- Beyond DDR4-2800mhz, 125mhz CPU multiplier may occur u say. If so, lets say instead of 100*40=4Ghz, 125*32=4Ghz will be chosen. Will this stable and OK?

- Can you suggest motherboard for 5820K? (Planning Overclock), Do you suggest CPU cooler NH D15 Noctua?

- Are there any advantages (calculation time etc.) 64GB RAM over 32GB RAM ? (Lets say 5milyon meshes only for both cases)

[flotus1]

When the CPU strap is increased automatically to 125 MHz, many systems will not boot anymore with all other settings at their default value. I only mentioned this because it is the origin of a common misconception about the X99-platform: "Memory faster than DDR4-2800 is unstable". It is not. All you have to do is to decrease the multiplier or increase the CPU voltage.
The Noctua NH-D15 is about as good as it gets with air-cooling. A good choice if you want to overclock the CPU.
If a simulation fits within 32 GB of RAM, there is no benefit from more RAM. Other than the possibility of running other memory-intensive tasks at the same time. But of course this will increase simulation time to some extent.

[hyugakojiro999]

Thank You very much for the reply my friend,

Asrock Taichi,
Msi Sli Plus,
Asus X99A or Deluxe,
Evga Classified

and these are the motherboards that i will chose one of them. What do you think?

[flotus1]

I usually recommend the Asus X99-A II. It is relatively cheap and has all the features I consider useful, among others USB 3.1 and m.2 connector.
Unless you know a reason to spend more money on a specific motherboard, use this.

[hyugakojiro999]

"Asus X99-A II" is not compatible with 2011-v3 my friend.

[flotus1]

Of course it is. Where did you find this wrong information? And which CPU socket is it according to your source?

[hyugakojiro999]

I don't remember but you are right

https://www.asus.com/us/Motherboards...pecifications/

[hyugakojiro999]

flotus1,

Could you please explain how there is a bottleneck between the CPU and RAM? It is said that the bottleneck is at the RAM side and can not be at the CPU side.

[flotus1]

The term bottleneck does not really apply here. I am not sure if I used it in this context.
6 or 8 cores are a fairly good match for a DDR4 quad-channel memory interface. This means that both the CPU and the memory affect simulation time. You significantly gain performance using a faster CPU (or overclocking it) AND using faster memory.
A real bottleneck for CFD applications will occur if you use high core count Xeon CPUs with 14 or more cores that still only have a quad-channel memory interface. Their scope of application is obviously not CFD.

[alexj]

Dear Alex and all hardware enthusiasts,

I just want to join the conversation about the bottleneck and many core systems. I am faced with a similar decision as discussed in this thread, I have to choose between a workstation with either setup listed below. Both would cost about the same:

Workstation 1:
1x Intel Core i7-6950X Extreme Edition, 10x 3.00GHz
1x ASUS X99-A II Motherboard
8x 8GB DDR4-3600 RAM modules unbufferd (UDIMM)

Workstation 2:
2x Intel Xeon E5-2640 v4, 10x 2.40GHz (20 cores in total)
1x ASUS Z10PA-D8 Motherboard
8x 8GB DDR4-2400 RAM modules registered ECC (RDIMM)

For simplicity I leave out all the other parts of the workstation design. The purpose is to run OpenFoam simulations with 2 million cells and upward meshes. That would give at least 200k cells per core for workstation 1 and 100k cells per core for workstation 2.

Here are my questions and I would highly appreciate any help/comments on those :

1) Which one of the two workstations is more suitable for the job?
2) I figured workstation 2 can only handle DDR4-2400 RAM modules due to the specs of the ASUS MB.
3) Are the extra 10, slower cores with slower memory modules actually an advantage for CFD (OpenFoam, so no license restrictions)?

Kind regards,
Alex

[flotus1]

Originally Posted by alexj

1) Which one of the two workstations is more suitable for the job?
2) I figured workstation 2 can only handle DDR4-2400 RAM modules due to the specs of the ASUS MB.
3) Are the extra 10, slower cores with slower memory modules actually an advantage for CFD (OpenFoam, so no license restrictions)?

1) Number 2. A large amount of relatively slow cores is less of an issue if you don't have to pay for your licenses.
2) Memory will actually be running at DDR4-2133. The memory controller in the CPU is the limiting factor here. Only few Xeon processors are able to use DDR4-2400 https://www.microway.com/knowledge-c...ep-processors/
3) It will be faster than the I7. The total memory bandwidth is still higher despite lower frequency. Plus you get more cache and don't have to deal with overclocking or cooling issues. If you still want to buy the I7 workstation use 4 DIMMs with 16GB each.

[alexj]

Originally Posted by flotus1

1) Number 2. A large amount of relatively slow cores is less of an issue if you don't have to pay for your licenses.
2) Memory will actually be running at DDR4-2133. The memory controller in the CPU is the limiting factor here. Only few Xeon processors are able to use DDR4-2400 https://www.microway.com/knowledge-c...ep-processors/
3) It will be faster than the I7. The total memory bandwidth is still higher despite lower frequency. Plus you get more cache and don't have to deal with overclocking or cooling issues. If you still want to buy the I7 workstation use 4 DIMMs with 16GB each.

Thanks so much Alex.

Looking at the document you linked above at 2), I guess the E5-2650 V4 would be interesting too. It has more cores, supports DDR4-2400 RAM, more cache. It is slower per core however (2.2 GHz instead of 2.4 GHz), but more memory bandwidth (76.8 GB/s instead of 68.3 GB/s) and that counts in CFD, right?

At spec.org they compare like this at the SPECfp2006 benchmark (I looked both up for a Dell PowerEdge T430)

Intel Xeon E5-2650 v4, 2.20 GHz, total of 24 cores: 811
Intel Xeon E5-2640 v4, 2.40 GHz, total of 20 cores: 694

So there is a clear advantage I guess but there is also an extra ~300 Euro per processor. Maybe I can stretch my budget to get the 2650s, looks like worth it.

Kind regards,
Alex

[hyugakojiro999]

Hi flotus1,

Lets look another side; "Solid State Disk"s

What do you think about these;

-Samsung SSD 950 Pro
-Samsung SM951
-Samsung 850 EVO

Can we see any benefit from 32gb/s SSDs? or 850EVO is ok?

[flotus1]

While I consider a normal SATA3 SSD mandatory for any CFD workstation (in fact mandatory for any computer), the benefit of PCIe-SSDs depends on the workload. If you use Ansys software that is known to cause a lot of disk I/O during simulations a fast SSD might help a bit. If reading or writing files is the bottleneck of your workload, e.g. for some transient simulations or post-processing, a fast SSD can definitely help.
Apart from these special cases, the price for decent PCIe-SSDs is still a bit too high and SATA-SSDs or even the good old HDDs are fast enough to avoid an I/O bottleneck.
So there is no final answer to your question. If you see any benefit depends on what you do and how you do it.