Preventing the deletion of EBS Volumes during the termination of AWS Spot Instances
I started learning about AWS (Amazon Web Services) a few weeks ago as part of the openSAP class. Although this blog is geared towards the openSAP Software Development on SAP HANA class, it may be useful for others who are new to AWS.
AWS Instance (HANA One Developer Edition):
Let us run through the process of creating an image (AMI — Amazon Machine Image) via the AWS Console:
The Delete on Termination flag controls the deletion of the EBS volume during termination of Spot Instances: If the flag is marked, the EBS volume is deleted and if it is unmarked, the EBS volume is not deleted.
EBS Volumes tab:
Here the non-root EBS volume devices of the Instance are named /dev/sdb1 through /dev/sdb6. If you take a look at the Device:/dev/.. dropdown to the left of the Delete on Termination: entry, you can only choose /dev/sdb through /dev/sdk. If our instance non-root devices were in the range /dev/sdb through /dev/sdk, we may be able to Remove the device with the true flag and Add a replacement entry with the false flag. This is the limitation of using the console to create an AMI for our Instance. An alternate method for creating an AMI with the Delete on Termination flag set to false for all the EBS volumes is described below.
Follow the steps in Installing the Amazon EC2 Command Line Interface Tools on Windows – Amazon Elastic Compute Cloud. You may skip step 7 since we will be accessing a Linux Instance.
Below are my Windows 7 environment variables and values:
JAVA_HOME = C:\Program Files\Java\jdk1.6.0_45
EC2_HOME = C:\AWS\CLI Tools\ec2-api-tools\ec2-api-tools-184.108.40.206
EC2_URL = https://ec2.us-east-1.amazonaws.com
AWS_ACCESS_KEY = <AWS account specific>
AWS_SECRET_KEY = <AWS account specific>
Let us issue a command from a Windows command prompt (CLI Commands List ==> Commands (CLI Tools) – Amazon Elastic Compute Cloud) and also save the output — we need this later.
ec2-describe-instance-attribute i-c88ef4ad –block-device-mapping
Below are the steps for creating and using a Spot Instance:
Step 1: Create Snapshots of your Instance (or terminated Spot Instance) EBS volumes:
You are free to choose any names for the Snapshots. I’m including the device name (from the CLI command output) in the Snapshot name for easy identification later. I have found it useful to name Snapshots as Base-Root-sda1, Base-sdb[1-6] for the initial base Snapshots, and Week[1-6]-Root-sda1, Week[1-6]-sdb[1-6] for Snapshots after Week [1-6].
Wait for the Snapshot to be created (green “Completed” status — should take less than a minute)
Repeat the above steps for the other 6 EBS volumes.
Below are the Snapshots for all the 7 EBS volumes:
Once you have created the Snapshots from EBS volumes, you may delete the EBS volumes (available state) if they are persisting from a terminated Spot Instance. Obviously , don’t delete the EBS volumes of the non-Spot Instance (in-use state) as shown in the screenshot above.
Step 2: Create an AMI from the Snapshots:
Obtain the Snapshot-to-Volume mapping as shown above. Obtain volume-to-device mapping from the CLI command (ec2-describe-instance-attribute) output and use it in the ec2-register command below.
Create a Windows .txt file with the lines below:
–name “Base-AMI-Rev56” ^
–description “Base AMI after upgrade to Rev 56” ^
–architecture x86_64 ^
–kernel aki-825ea7eb ^
–root-device-name /dev/sda1 ^
–region us-east-1 ^
–block-device-mapping “/dev/sda1=snap-f26356ac:10:false:standard” ^
–block-device-mapping “/dev/sdb1=snap-086b5e56:20:false:standard” ^
–block-device-mapping “/dev/sdb5=snap-a96a5ff7:20:false:standard” ^
–block-device-mapping “/dev/sdb4=snap-e95560b7:20:false:standard” ^
–block-device-mapping “/dev/sdb3=snap-6054613e:20:false:standard” ^
–block-device-mapping “/dev/sdb2=snap-db576285:20:false:standard” ^
ℹ ^ is the line continuation character for Windows Command line.
ℹ I obtained the kernel ID above from the Instance Description tab.
ℹ I obtained the architecture by creating an AMI via the AWS Console
❗ I found it safer to use region even if EC2_URL is defined as a Windows environment variable.
Copy and paste the above lines into a Windows command prompt and run it by hitting <Enter>:
❗ I could have created a *.cmd file above. The risk is that if you accidentally double-clicked on the *.cmd file, it will execute the commands in it.
❗ When you want to create the next Spot Instance, architecture, kernel, root-device-name, and region will remain the same as in the above CLI command. You may want to change the Name & Description (for example: –name “Week1-AMI-Rev56”, –description “AMI after Week 1”). The snapshot IDs will change.
Step 3: Request a Spot Instance:
Run through the other screens as you normally would.
Spot Instance EBS Volumes:
Save the volume-to-device mapping:
As soon as the Spot Instance starts running, issue the command below and save the output — we need this when we loop back to Step 2. Once the Spot Instance terminates, the EBS volumes change from in-use to available state at which point the volume-to-device mapping would be lost.
ec2-describe-instance-attribute i-ae0890cd –block-device-mapping
Step 4: Spot Instance Usage and termination:
You can now work with the Spot Instance .
The above 7 EBS volumes will remain (undeleted) during the Spot Instance termination regardless of whether the termination was user-initiated or AWS-initiated.
Below is the message you get when you terminate the Spot Instance — it indicates the EBS volumes will NOT be deleted.
Loop back to Step 1 when you need to create a new Spot Instance from the EBS volumes of the terminated Spot Instance.
You may delete the old AMIs/Snapshots or you may keep them around if you need to go back and create a Spot Instance from a previously saved AMI/Snapshots.
The method described above results in the persistence of EBS volumes after the termination of AWS Spot Instances.
How does the creation of AMI and snapshots have an effect on the overall billed amount on AWS?
AMI's are effectively a collection of snapshots that are stored in S3. You'll pay the regional S3 prices for the snapshot storage depending on which region you are working in. Please note that Amazon EBS snapshots are incremental backups, meaning that only the blocks on the device that have changed since your last snapshot will be saved. For example: If you have a device with 100 GBs of data, but only 5 GBs of data has changed since your last snapshot, only the 5 additional GBs of snapshot data will be stored back to Amazon S3. You can find further information under "EBS Snapshots" here: http://aws.amazon.com/ebs/.
Hope this helps,
Snapshots are charged by AWS @ $0.095 per GB-month of data stored. Snapshots are stored compressed. Here is what AWS says:
"Snapshot storage is based on the amount of space your data consumes in Amazon S3. Because data is compressed before being saved to Amazon S3, and Amazon EBS does not save empty blocks, it is likely that the size of a snapshot will be considerably less than the size of your volume."
For more info, please read (bottom of the page):
I have had anywhere from 7-28 Snapshots last month. The size of the EBS volumes that these Snapshots were created from is in the range 130-520 GB. I was charged $0.80 for 8.439 GB-mo of Snapshot data stored last month.
HI Mani and ALL!
CAN THE AWS EBS VOLUME USED FOR BACKUPS BE REMOVED/REPLACED WHEN FULL?
We have 95% capacity on the volume, cannot delete the backups (too many, large, and/or not enough memory) - and are trying to avoid catastrophic disaster...
Studio - can't delete backups and was not sure if we could just stop HANA, remove the EBS and replace with a new volume mount? We have snapshots, to created replica of the volume via snapshot, AMIs and made a complete backup of everything - so we are prepared.
ANY ADVICE or guidance for the above would be greatly appreciated!
The 7 volumes currently take about 130 GB which is way over the free allowance and Amazon charges almost $20 for storage only every month.
Can I delete 100 GB worth volumes and still be able to use the Instance when I next login? I am not concerned about the data in the instance and happy to lose data after my work is done.
Do these volumes contain only data and not the HANA install?
Appreciate your response
1) Once you have created the 7 snapshots of the normal (non-spot) instance EBS volumes (7 "in-use" volumes -- 130 gig), you may delete the 7 EBS volumes (not the snapshots). If you terminated the normal instance, these 7 EBS volumes will be deleted. Make sure you issue the command ec2-describe-instance-attribute and save the output before terminating the instance.
2) Create an AMI and spot instance from snapshots and do your work.
3) When the spot instance terminates -- whether initiated by the user or by AWS -- your work will be saved -- whether you want it or not -- in EBS volumes ("available" state).
4) If you want to use the saved data for your next session or login, you can create 7 snapshots from the EBS volumes in step 3 above. Otherwise, keep using the original AMI created (Step 2) from the 7 snapshots from Step 1 above. In either case, you may delete the 7 EBS volumes ("available" state) in Step 3.
5) If you created new snapshots in step 4, loop back to Step 2 during your next session or login.
Using the above approach, you will have EBS volumes only when you are using the normal instance or spot instance. Snapshots do not cost much since they are in compressed format (refer to my response above on July 2, 2013).
The 7 volumes contain the OS (SUSE), HANA DB, and system/user data.
Couldn't one write an script to automate the whole procedure after spot instance termination?
IE make snapshots, create new AMI with the snapshots, delete old AMI and old snapshots, delete EBS volumes
I was able to create AWI and re-launch it into another new running instance. I associated the same elastic with the new running instance.
However, I am unable to use HANA Studio to connect to the new running instance. The error message is "cannot access system on host: ec2-.......com".
Do you have this problem, and how do you solve this problem?
No, I don't have this problem. When you create an AMI, AWS creates a security group by default which has only SSH port 22 open. HANA Studio uses tcp port 30015.
Locate the security group under the description tab of the running instance. Make sure this group has the inbound rules below:
On the left pane: Menu: Network & Security >> Security Groups. Choose the security group associated with your instance, add port range, choose Add rule, and apply rule changes under the inbound tab to add ports.