24 June 2013

Applications running on Amazon EC2 frequently use EBS for persistent storage. It survives reboots, and it facilitates convenient backup models via snapshots. The snapshots are stored incrementally such that you only incur storage expenses when data differs between snapshots. Thus, if a volume has strictly-growing data (e.g., log files), then taking a snapshot at time A and B would take about the same amount of storage as taking an extra 10 snapshots between those two times. While they can be created manually online, automated EBS snapshots are a thousand times more helpful. It takes a few minutes to setup, but hopefully these instructions will prove helpful.

System Requirements

Although possible on any system, the instructions assume the following Linux setup.

  1. Ubuntu Linux with SSH or terminal access
  2. Amazon EC2 AMI Tools : Developer Tools : Amazon Web Services
  3. One solution is to start with the Ubuntu AMI: Ubuntu Amazon EC2 AMI Finder
  4. PHP

Instructions

  1. Go to the IAM Console
  2. Create a user "backup-ebs-user" (or use any alternate name)
    1. AWS's instructions for adding an IAM User
    2. Record the user name and security credentials for future use (not necessary for this task)
  3. Create a certificate for CLI
    1. Creating and Uploading a User Signing Certificate
    2. In terminal, run (the third command includes a number of user prompts that you can address as you see fit):
      cd /path/to/secure/location/
      openssl genrsa 1024 > backup-ebs-user.private-key.pem
      openssl req -new -x509 -nodes -sha1 -days 365 -key backup-ebs-user.private-key.pem -outform PEM > backup-ebs-user.certificate.pem
    3. Upload the .pem file to Amazon: Click "Manage Signing Certificates" for the user and paste the backup-ebs-user.certificate.pem file
  4. Update the permissions for the backup user
    1. Overview of AWS IAM Policies
    2. AWS Policy Generator
    3. The key steps for adding a policy are to go to the user, click Permissions, click "Attach User Policy", and click "Custom Policy" to initiate the process of adding the secure policy for this user.
    4. The user needs the ability to view and create EBS snapshots, which involves this minimal policy:
      {
        "Statement": [
          {
            "Sid": "1",
            "Action": [
              "ec2:CreateSnapshot",
              "ec2:DeleteSnapshot",
              "ec2:DescribeVolumes",
              "ec2:DescribeSnapshots"
            ],
            "Effect": "Allow",
            "Resource": "*"
          }
        ]
      }
  5. Install the script to create snapshots nightly
    1. Copy backup-ebs.php to your server
    2. Edit the file and customize the configuration variables at the top of the file (e.g., paths to the .pem files)
    3. Change the mode to allow execution: chmod 755 /path/to/backup-ebs.php
    4. Update crontab to run as frequently as you like:
      1. In terminal: crontab -e
      2. Add line: 30 23 * * * /path/to/backup-ebs.php

WARNINGS/DISCLAIMERS

Use this script at your own risk. Make sure you understand what the script is doing!!!!

  1. Snapshots result in charges to your AWS account.
  2. Snapshots are automatically deleted after the configured time limit. That data is lost forever.
  3. AMI snapshots are never automatically deleted by this script, so costs can continue to escalate for those snapshots.
  4. If you are using a micro instance (or if you run other resource-intensive applications), then be warned that snapshot creation can be killed by the system if resources are insufficient. Thus, whenever possible, run this on a larger instance -- or on a non-EC2 server since snapshots do not need to be triggered from the cloud.

Script: backup-ebs.php

#!/usr/bin/php
<?php
// Configure the script.
$AWSDIR = "/path/to/secure/location";
$AWSCERT = "--cert=$AWSDIR/backup-ebs-user.certificate.pem --private-key=$AWSDIR/backup-ebs-user.private-key.pem";
$MAXDAYS = 60;
// -----------------------------------------------------------------------
// NO NEED TO EDIT BELOW THIS LINE.
// Delete any old snapshots.
$EBS = "ec2-create-snapshot $AWSCERT ";
$snapshots = array();
$cmd = "ec2-describe-snapshots $AWSCERT";
exec($cmd, $snapshots);
foreach ($snapshots as $snapshot) {
  // Parse the snapshot data.
  $snapshot = preg_split("@\[email protected]", trim($snapshot), 9);
  $snapshotId = $snapshot[1];
  if ($snapshot[3] !== 'completed') {
    continue;
  }
  // Never delete AMI snapshots.
  if (isset($snapshot[8]) && strpos($snapshot[8], 'ami-') !== FALSE) {
    continue;
  }
  // Delete any snapshots over MAXAGE.
  $time = strtotime($snapshot[4]);
  if ($time < time() - 86400 * $MAXDAYS) {
    $cmd = "ec2-delete-snapshot $AWSCERT $snapshotId";
    echo "Deleting $snapshotId from $snapshot[4]\n";
    system($cmd);
  }
}
// Create snapshots of any volumes that currently exist on the account.
$volumes = array();
$cmd = "ec2-describe-volumes $AWSCERT | egrep 'TAG.*Name' | cut -f 3,5";
exec($cmd, $volumes);
foreach ($volumes as $vol) {
  list($volId, $volName) = preg_split("@\[email protected]", trim($vol), 2);
  $volName = str_replace('ami-', 'ami_', $volName);
  $volName = escapeshellarg($volName);
  $cmd = "$EBS -d $volName $volId";
  echo "Creating snapshot $volName for $volId\n";
  system($cmd);
}
Technologies:


blog comments powered by Disqus