Some users may need to use custom images to fulfill business or security requirements. CycleCloud supports building clusters from images created and owned entirely by the user.

Custom User Images

CycleCloud supports custom images. Specify the image id directly with ImageId, or add the image to the image registry. When the image is in the registry, reference it with either Image or ImageName on your node. It will appear in the image dropdown on the cluster creation page.

Images in the image registry consist of a Package record that identifies the contents of the logical image and one or more corresponding Artifact records that specify the actual image id in the appropriate cloud provider. For example, a custom image with R installed on it might consist of this Package record:

AdType = "Package"
Name = "r_execute"
Version = "2.1.1"
PackageType = "image"
Label = "R"

Once you add that record, you can specify that image by including either Image = R or ImageName = r_execute in the cluster template.

If this image existed as a single EC2 AMI in us-east-1 with an id of ami-123456, it would need to have the following artifact stored:

AdType = "Artifact"
Package = "r_execute"
Version = "2.1.1"
Name = "aws/us-east-1"
Provider = "aws"
Region = "us-east-1"
ImageId = "ami-123456"

You must specify Provider on the artifact (and in the case of AWS, Region).

You can add as many artifacts as you want for a given image package, but you must include all the artifacts required to use that image in all the “locations” you want (one per cloud provider account, regions, projects, etc). The name of the artifact is not important, except that it must be unique to all artifacts for a given package and version. Using a combination of the provider and provider-specific details (eg region) is usually recommended. CycleCloud automatically picks the right artifact to match the provider and any provider-specific details, but it uses the Provider attribute (and Region, etc) rather than parsing the Name.

If you add more than one image package with the same name, they must have different version numbers. When starting an instance, CycleCloud will automatically pick the image with the highest version number, by treating the version number as a dotted string and comparing each part as a number. For instance, 2.0 is newer than 1.2, 1.10 is newer than 1.2, and 1.2.3 is newer than 1.2. To override this, specify ImageVersion on the node, as either a literal (eg 1.2) or a wildcard (1.x).

You can also automatically import a set of image ids to be used in CycleCloud.

Azure Custom Images

Private custom Azure images can be specified in the template file with the ImageID attribute. This ID can be found in the Azure portal as the Resource ID for the image. In addition, the ImageOS attribute must be set to either windows or linux:

[[node demo]]

  ImageId = /subscriptions/xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/MyResourceGroup/providers/Microsoft.Compute/images/MyCustomImage
  ImageOS = linux

  Azure.Publisher = OpenLogic
  Azure.Offer = CentOS-HPC
  Azure.Sku = 7.4
  Azure.ImageVersion = 7.4.20180301
  Azure.OS = linux

  InstallJetpack = true
  JetpackVersion = 7.1.0
  JetpackPlatform = centos-7
  AwaitInstallation = true

jetpackplatform = windows works for all versions of Windows, or with centos-6, centos-7, ubuntu-14, or ubuntu-16 only. If jetpack will not be installed, set the installjetpack attribute to false and leave out the remaining information.

Azure Marketplace Images

Azure Marketplace images can be directly specified as well. The following attributes must be set to identify the image:

[[node demo]]

Azure.Publisher = OpenLogic Azure.Offer = CentOS-HPC Azure.Sku = 7.4 Azure.ImageVersion = 7.4.20180301 Azure.OS = linux

InstallJetpack = true JetpackVersion = 7.1.0 JetpackPlatform = centos-7 AwaitInstallation = true

As an alternative to specific version most publishers support referencing the latest image with a label: Azure.ImageVersion = latest.

AWS Machine Images

When creating cluster template files an Image, ImageName, or ImageId must be specified for each node created. Bear in mind that in EC2 each image is specific to both the OS as well as the Amazon Region the machine is in.

Default AWS Images

Users can specify default AWS images, and Jetpack will be installed at boot:

[[node demo]]

  ImageId = $Image
  ImageName = cycle.image.centos7
  InstallJetpack = true
  JetpackVersion = 6.6.0
  JetpackOs = linux
  AwaitInstallation = true

Jetpack

CycleCloud can launch instances using any Machine Image. However, to build clusters using CycleCloud and to benefit from CycleCloud’s orchestration layer, Cycle Computing’s Jetpack package must be installed on the Image. If the image does not have Jetpack installed, then many of the features of CycleCloud will be lost.

Users have two options for installing Jetpack:

  1. Use images provided by Cycle Computing that already have Jetpack installed
  2. Install Jetpack on an existing image to create a custom Jetpack image

For new images that are not managed by CycleCloud, you must specify which version and platform of Jetpack to install:

[[node demo]]

  InstallJetpack = true
  JetpackVersion = 7.1.0
  JetpackPlatform = centos-7
  AwaitInstallation = true

Jetpack packages are currently built for the following operating systems: – Enterprise Linux 6, 7 (CentOS, RHEL) – Ubuntu 12.04, 14.04, 16.04 – Microsoft Windows

CycleCloud Image Requirements

  1. Jetpack must be installed on the Image.
  2. Jetpack uses Chef to configure instances launched using the Image. In order for Chef to function, other infrastructure configuration management tools (such as CloudInit in AWS) should be disabled.

Note

If a configuration management tool, such as Chef or Puppet, is already in use, it is generally possible to configure the systems to work together. Please contact Cycle Computing support for more help in this case.

  1. Management ports, such as TCP port 22, SSH (Secure Shell), and port 3389 RDP (Remote Desktop Protocol), should be open in the security group and on an instance’s firewall during the image baking process.

Building a Custom Image using the AWS Console

For users accustomed to building AMIs via the AWS Console, the easiest way to begin building custom images for CycleCloud is to continue using your current image building method or follow the method, using the AWS Console, described here.

If you intend to build many AMIs or update them regularly, then you may eventually want to switch to using CycleCloud to build images directly.

Select a Base AMI

The first step in building a custom image in AWS is selecting the base AMI and making note of its AMI ID. Your organization may have an approved list of base AMIs. You may also select any AMI of a supported platform available from within the AWS Console.

This guide refers to AMI ID ami-f0b23b98, but this should be replaced with the correct base AMI ID.

Launch the Instance

  1. Go to the Services -> EC2 dashboard in the AWS Console and select the AMIs page.
  2. Select the AMI you wish to use as your base AMI, and click Launch to start an instance with the base AMI.
  1. For details on configuring and launching an instance via the AWS Console, see the AWS User Guide.
  2. For most users, using an EBS-backed instance with a root volume of at least 8GB in size is sufficient.
  3. If the base image is configured to use Cloud-Init, then do not attach an ephemeral drive to the instance at launch, otherwise Cloud-Init will attempt to mount the drive and add it to the fstab. If you do need the ephemeral drive to build the image, then be sure to clear out the fstab as described in the Clean Up section below.
  4. Be sure to select a security group with the management ports opened (e.g. SSH, Remote Desktop), and select a security key-pair for which you have access to the private key (this guide will assume that you have used a keypair named cyclecloud as described in the CycleCloud Quickstart Guide).
  1. After launching the instance, collect the instance ID and hostname of the image builder instance.

Custom Configuration

Once the new instance has started, use the private key to log in to the instance. For the example in this guide, the keypair allows direct access to the instance as root. If your base image uses a different default user, be sure that the user has sudo access.

Next, install any custom software and configurations that your cluster requires.

It is recommended to create a shell script which automates the software download and installation. Such a script can later be used with CycleCloud to orchestrate image rebuilding.

Chef/Cluster-Init vs. Image Baking

All user-level software is installed and configured using Chef and/or Cluster-Init at cluster startup. You may be able to reduce launch times by pre-installing some software or pre-downloading the packages for software that may only be installed after launch. When making the decision what to burn into the image, here are some questions to think about:

  1. Do all users of this image need this software? Or is it required by policy?
  • A great candidate for baking into the image would be software such as anti-virus programs that are required by policy and may be updated dynamically on each launch.
  1. Can this software be updated when instances are launched?
  • If not, the you may end up baking new images every time a new version is released.
  1. Does this software require customization at install time that can only be done after instance launch?
  • Grid-enabled software often needs to be configured with hostnames and ips of actual, running instances based on cluster search results. (You may still be able to install the software prior to baking the image, and use Chef to re-configure it at instance start-up.)
  1. Does this software package belong on the root volume of the instance?
  • If the software should be installed on EBS or the ephemeral drive, then it cannot be installed directly on the image.

If the software cannot easily be installed prior to image baking, consider including a copy of the installer for use by CycleCloud’s Thunderball cookbook to avoid the download at instance startup until the installer is updated.

Installing Jetpack

Jetpack has no external dependencies, and it includes an easy-to-use installer for all supported platforms.

On Linux, Jetpack will be installed at: /opt/cycle/jetpack

On Windows, Jetpack will be installed at: C:cyclejetpack

First, contact Cycle Computing support to request a copy of Jetpack for your platform, then follow the instructions below to install it in your instance.

Windows

To install Jetpack on Windows run the following commands from a Powershell session as Administrator:

PS> unzip jetpack.zip
PS> cd jetpack
PS> install.cmd

If you do not have 7zip or another unzip command available on the command line, you can extract jetpack.zip using Windows Explorer.

Should the installation fail for any reason, error output will be saved in install.log in the same directory as install.cmd.

Linux

To install Jetpack on a supported Linux distribution:

$ tar xvzf jetpack.tar.gz
$ cd jetpack
$ chmod +x install.sh
$ sudo ./install.sh        # if not running as root

Should the installation fail for any reason, error output will be saved in install.log in the same directory as install.sh.

Note

The Linux installer requires root privileges.

Clean-Up

An important part of building a new image is to ensure only the files that you want on every instance, launched from a saved AMI, exist on the instance at the time the AMI is baked.

Prior to baking the image it is a good idea to remove the installer and all other temporary files from the instance. Otherwise, those files will be baked into the image and permanently clutter it.

Here are the a few common clean-up steps (in order):

  1. Disable password based login for all users:

    <$ passwd -l root
  2. (Optional) Configure sshd_config according to your policies.

  3. (Optional) Configure the instance level firewall according to your policies.

  4. Remove any temporary files and installers for you custom installations.

  5. Remove the Jetpack installer and install dir:

    $ cd /tmp
    $ rm -rf jetpack*
  6. Remove any system logs that may contain sensitive data.

  7. If you mounted and formatted the ephemeral drive for the instance (or let Cloud-Init do it for you), then be sure to remove the mount configuration from the /etc/fstab file.

  8. Clear the bash history for all users, but in particular for root:

    $ sudo su -
    $ history -w
    $ history -c

9. Remove the authorized key for the key-pair you used to log in: (Do this last: once you perform this step, you won’t be able to log back into the instance):

$ rm ~/.ssh/authorized_keys

Bake the New Image

The new image is ready for baking.

Return to the AWS Console and locate the running instance in the EC2 instances list. Select the instance and select Action -> Create Image. Give the image an appropriate name and description, and then click the Create Image button.

The image creation process will take several minutes. Once the process has completed, the AMI will be ready for use in CycleCloudStore, and this new AMI ID can be used in your cluster templates.

Import the Image

The image ids generated above can be automatically added to the image registry in CycleCloud with the cyclecloud image add command:

cyclecloud image add --name custom.image --label "My Image" --os linux ami-123 ubuntu14_rstudio

This would attempt to find images with an id or name of ami-123 and ubuntu14_rstudio in all the cloud accounts you have configured, and save the resulting image package and artifacts. If that command succeeds, you can then use either Image = My Image or ImageName = custom.image in your cluster templates, and My Image will appear in the image dropdown for a cluster-creation form. The “–os” option should be specified so that the resulting package is correctly labeled as a linux or windows image.

The above command will version the package with the latest version of custom.image that is stored, or 1.0 if there is not currently an image package named custom.image. To automatically increment to the next version, include --bump-version:

cyclecloud image add --name custom.image --label "My Image" --bump-version minor ami-123 ubuntu14_rstudio

The option to --bump-version can be one of major, minor, or patch, which will increment the first, second, or third part of the version number, respectively. You can also set the version directly with --package-version (for instance, --package-version 2.0).

The command prints out a summary like the following:

Image name/id             Description
-----------------------   --------------------------------------------------------------------
ami-123                   AWS image in account prod-aws, region us-east-1 (standard HVM), 8 GB
ubuntu14_rstudio          GCP image in account prod-google, 10 GB

Added image custom.image, v1.0 with 2 artifacts from 2 accounts (ubuntu14_rstudio, prod-google)

If it cannot match all the images, the command will fail. To test out what it would find, include the --dry-run option, which prints out the same summary but does not store anything.