[pmlynch]

Hi All

Sorry to bump an old thread, but has any further progress been made on developing AMIs for the EC2 cluster compute instances?

I'm particularly interested in testing an AMI for OpenFoam-ext 1.6, ideally in the EU west region.

cloudFlu looks interesting, but we may have QA issues with switching to OF 1.7.

[philippose]

Hello everyone,
Good Evening!

I didnt want to start a new thread because this topic already existed....

Last week I put together a CentOS 6.5 64-bit AMI with the intention of running OpenFOAM (specifically foam-extend-3.1) on Amazon EC2. The
basic setup is as follows:

1. I started with the official CentOS 6.5 64-bit AMI available on the Amazon servers which contains a minimal installation of CentOS 6.5

2. Updated the installation and installed the developer tools, libraries and dependencies required to compile foam-extend-3.1 and the Thirdparty libraries.... everything except Qt and Paraview since my intention was to create a text-based interface.

3. Then, I created a new EBS (Elastic Block Store) Volume and attached it to the Installation in [2]

4. In then split up this volume into two partitions, one which is mounted into "/opt/Simulation" and the other which gets mounted as "/home", in order to separate the User home directories from the base installation.

5. Now, in "/opt/Simulation", I created a folder called "site" in which I created a file called "sim_settings" to act as the main interface to the entire Simulation software installation.

6. In order to separate the whole Simulation setup from the base installation, I created a file in "/etc/profile.d" (which gets called automatically each time a terminal is opened) which simply calls the "sim_settings" file in [5].

7. In "/opt/Simulation", I created a folder "OpenFOAM" under which a new folder "foam-extend-3.1" was created containing a clone of the foam-extend-3.1 git repo. I then compiled foam-extend-3.1 and all the Thirdparty software (except for Qt and Paraview since my intention was to create a text interface).

8. Once all this was done, I unmounted and detached the additional EBS volume created for the simulation setup, and created a new AMI of the base installation of CentOS 6.5 with the modifications required to mount and call the simulation setup (if available... if not, it runs as a normal CentOS installation without the simulation stuff).

9. Finally I made a snapshot of the EBS Volume which now contains a fully compiled and working copy of foam-extend-3.1 and the Thirdparty utilities along with the scripts needed to source them all bundled into a single script called "sim_settings" which is called from "/etc/profile.d".

So basically, I ended up with one AMI (and the corresponding snapshot) of a modified CentOS 6.5 x86_64 installation, and one snapshot of an EBS Volume containing OpenFOAM and its dependencies.

In order to run the whole system, I need to launch an instance of the AMI after attaching the EBS volume snapshot to it.


What do you think of this workflow / concept for running OpenFOAM on Amazon EC2? Do you think this method of splitting the two parts is advantageous and eases maintenance? For example, such an approach could be used for testing, or maintaining multiple simulation setups, etc....

What other ideas have you all thought of and implemented for getting OpenFOAM up and running on Amazon EC2 keeping in mind a modular approach?

I would be interested to get some feedback :-) If anyone is interested in this AMI / EBS Volume, maybe there is some way of sharing it on Amazon (I am a complete newbie with regards to Amazon EC2..... :-)!)

Have a great day ahead!

Philippose

(P.S.... Most of the setup was done using a t1.micro instance type, except for the actual compilation... for which I switched to a c3.2xlarge which is an 8 core system with 15GB RAM using a Spot Instance for 8,5 cents/hour...)

[KateEisenhower]

Hello Philippose,

thank you for sharing this information.
I don't really understand what you want to achieve with the modular setup. Can you explain it in more detail? For example, how would such an approach ease maintenance?

Currently, I am also setting up a custom OpenFOAM AMI (still struggling) and I will share my experience here once it is done.

Any other experiences with OpenFOAM on EC2?

Best regards,

Kate

Edit:
@Paul: I got the impression the active development for cloudFlu has ceased some time ago, correct me if I'm wrong.

@all: I think the number one reason I am still struggling with ec2 is, that I am missunderstanding the whole concept:
1) Right now, I am installing OpenFOAM on a publicly available AMI
2) Then I plan to create an image from this running instance. This is then my custome AMI which contains OpenFOAM
3) Suppose I run my custom AMI and make a few simulations. How can I accomplish not to loose my simulation data when I am shutting down my instance? Do I even have to consider this problem since my custom AMI is EBS-backed?
4) Or do I have to create a new image from my running instance before every shutdown in order to not loose my simulation data?

I have many more questions, but the above one troubles me the most.

[KateEisenhower]

Hello guys,

after some time using AWS, it is running quite well now. But there is an issue I can't overcome.

Most of the times i am running small models (100,000 to 200,000 cells) on compute-optimized instance types. However these all use hyperthreading. Hence I only use a fraction of the physical processing power.

To summarize it, when I am calculating on a c4.large instance-type, I am only using half the vCPUs I am paying for. What am I missing here?

Best regards,

Kate

[gab_ventosa]

Originally Posted by KateEisenhower

Hello guys,

after some time using AWS, it is running quite well now. But there is an issue I can't overcome.

Most of the times i am running small models (100,000 to 200,000 cells) on compute-optimized instance types. However these all use hyperthreading. Hence I only use a fraction of the physical processing power.

To summarize it, when I am calculating on a c4.large instance-type, I am only using half the vCPUs I am paying for. What am I missing here?

Best regards,

Kate

Hi Kate,
Try penguin computing on demand, it is cheaper than amazon as you pay only for the time your solver is running, openfoam is already setup and configured and the solver runs on physical cores and not virtual machines, try it and you will love it, if you need any help using it just let me know.
Greetings



Sent from my iPhone using CFD Online Forum mobile app