[mcc12]

** AWS EC2 Instance:
- Ubuntu 18.04 LTS (HVM)
- SSD Volume Type
- c5a.8xlarge, powered by 2nd generation AMD EPYC processors: 32 vCPUs, 64 GiB memory, EBS-optimized bandwidth 3.170 Gbps, and network bandwidth 10 Gbps

** Note: only OpenFOAM 8 software was installed on this machine.

Initial steps taken:
1. Provisioned aforementioned EC2 instance with 100 GiB disk space in root volume
2. Connected to instance successfully
3. Installed OpenFOAM software, ran simulation
4. AMI created while instance was running, notes:
- Failed to launch NEW identical EC2 instance with AMI
- Failed to restart original EC2 instance

"Failed" meaning these instances both had state "Running" but only "1/2 checks passed" under the "Status check" column and became unreachable. After selecting each instance and viewing "Status checks", it was clear the "Instances status checks" failed. Notes on troubleshooting this:
- Viewed instance screenshot - no issue
- Viewed system log of instance - no useful or indicative information
- Ran EC2Rescue via Systems Manager while instance was running - failed to complete steps, never completed; 'oom-killer' module failed in diagnostics test
- Ran EC2Rescue again while instance was stopped - steps were completed successfully but no useful or indicative information was provided
- Tried restarting instance - no difference in result
- Tested & confirmed launching EC2 with identical configuration was not the issue
- Tested & confirmed creating AMI from new identical (empty) VM and launching another new VM was not the issue
- Extensive research on internet and some AWS guidance - no further clues found or determination reached

Initial conclusion & state:
- Submitted three separate reports to AWS, have not heard back yet
- Believe without further information that issue occurred after installing OpenFOAM software and/or running simulation, no definitive conclusion at this point
- Created new EC2 instance, re-installed software, re-ran simulation data for consideration of time


------------------------------------------


Further steps taken:
1. After several days of running simulation, the new instance previously mentioned ran out of disk space and expansion was requested (double, 200 GiB)
2. Created snapshot of volume on this EC2 instance for safety
3. Before increasing disk space, (requiring a restart of the EC2 instance, possibly running into same issue as before) I performed the following tests:
a. Created new EC2 instance of identical type & configuration, stopped, increased disk space, started instance - no issue with this process
b. Created image from snapshot, launched with same disk space as current instance (100 GiB) - failed to pass 'Instance status checks' and became unreachable
c. Created image from snapshot, launched with double disk space (200 GiB) - failed with same reason
d. Launched EC2 instance, connected, stopped, replaced root volume with volume created from snapshot - failed with same reason
e. Launched EC2 instance, connect, stopped, added volume created from snapshot with root volume - failed with same reason
4. Because of all above failures, I performed the following tests to ensure it was not an issue on AWS' side
a. Created EC2 instance of identical type & configuration, created snapshot, connected, stopped, attached volume created from snapshot, started & reachable without issue
b. Created EC2 instance of identical type & configuration, created snapshot, connected, stopped, replaced root volume with volume created from snapshot, started & reachable without issue
5. Because I was unable to verify any of the above tests, and do not have enough time to wait for AWS technical support on product side, I discussed with user our options to wait for their help or create a new instance and copy over all data previously generated; user chose to move forward with creating a new instance, re-installing software, copying data over from previous instance with less storage, and continuing simulation on new instance

Further conclusion & results:
- After user confirmed previous instance was no longer needed, I stopped it, increased disk space to 200 GiB, and started. No issue as anticipated after all...
- For my own testing, I also created an instance with identical type & configuration and installed OpenFOAM software, did not run any simulation, stopped the instance, took snapshot, launched new instance which failed 'Instance status checks' and became unreachable!
- I can conclude at this point without AWS support and current findings that since there seems to be no known issue with the AWS resources and services, there appears a bug when backing up (at least) Ubuntu bionic instances that have OpenFOAM software installed


------------------------------------------


TLDR:
Problem: after installing OpenFOAM software on an AWS EC2 Ubuntu bionic instance with above specs, taking a snapshot of the root volume or creating an AMI has either caused future instances provisioned from AMI and/or attaching volume created from snapshot to fail reachability checks, and in some cases causes original instance which AMI was derived from and restarted to fail reachability check as well, for reasons undiagnosable without technical support (haven't heard back yet).
Theory: OpenFOAM software installed and/or running simulation causes such issues.
Findings: AWS resources alone ruled out as the issue, when OpenFOAM software alone is installed, (no simulation done) such issues occur. Some sort of incompatibility appears to exist.

Has anyone experienced this or something similar? Any idea why this might be?

[s1291]

• I don't really think I fully understand your problem. I have tried in the past both AWS and MS Azure and never experienced any issues. While the simulations were running I have been "rsyncing" them to my machine.
  
  
• I don't see how is this related to OpenFOAM programming.