You can use spot instances in your cluster to reduce overall cost. In general, you can specify a bid price for each instance, which is the amount you are willing to pay per hour. If there is an instance available in your price range you will be granted that instance until the market price exceeds your bid price, at which time your instance will be terminated. Please see the official Amazon Documentation for details on on how the spot market works along with the current prices of various instance types.

You can easily modify the template for a cluster to include bid prices by enabling and modifying the following option on a node-by-node basis in the cluster template:

[[nodearray execute]]
BidPrice = 0.10
# Other parameters removed for brevity

The above directives state that execute nodes will be a spot instance with a bid price of $.10/instance hour.

When using spot instances, the output from the CLI tools detailing the state of your cluster will differ from on-demand instances based on the state of your spot request, for example:

$ cyclecloud start_cluster spot_cluster
Starting cluster spot_cluster....
------------
spot_cluster
------------
Zone:
Keypair: cyclecloud
Cluster nodes:
    execute-1 Waiting for spot instance request to be fulfilled
    master  Waiting for spot instance request to be fulfilled
Total nodes: 2

...

$ cyclecloud showcluster spot_cluster
------------
spot_cluster
------------
Zone:
Keypair: cyclecloud
Cluster nodes:
    execute-1 Waiting for instance to start running i-85dd96f9
    master  Waiting for instance to start running i-8bdd96f7
Total nodes: 2

Warning

Spot instances will take longer to start than on-demand instances since they have to go through the spot market fulfillment process. If your bid price is lower than the market price, your spot request may not be fulfilled for a long time, if at all.

Working With EBS Volumes

CycleCloud supports configuring volumes for specific Elastic Block Store (EBS) features including snapshot and ephemeral volumes. Snapshots can be used to attach common data to multiple instances at once, for example giving every execute node in a cluster the same reference data stored on a snapshot. Ephemeral volumes will automatically be attached to your instances. They exist for the life of the instance and provide a large amount of storage while the instance is running. For more information about EBS volumes, please see the official documentation.

Customizing EBS Volumes

The [[[volume]]] section supports several EBS-only settings:

Type
Specifies the type of the volume. Current options are standard (Magnetic), gp2 (General Purpose SSD, the default), and io1 (Provisioned IOPS)
IOPS
Specifies the provisioned IOPS rating.
DeleteOnTermination
Attaches this volume with the “delete on termination” setting. This is set by default for new volumes that are not marked Persistent=true,
but may be specified manually if needed.

Existing Volumes

To attach an existing EBS volume, you must know the ID of the volume (which looks like ‘vol-xxxxxxxx’). You can specify that it should be attached to an instance by setting VolumeId in the [[[volume]]] block within your cluster template:

[[node master]]

  [[[volume example-vol]]]       # 'example-vol' is the name of the volume within CycleCloud
  VolumeId = vol-abcd1234        # This is your volume ID as defined by AWS

Note

The instance will automatically be placed in the same availability zone as your volume, because AWS will not let you attach volumes to an instance in a different zone. If you specify more than one volume by ID, they must all be in the same zone. If you specify a subnet manually, it must be in the same zone as all volume ID’s.

EBS Snapshots

You can also create a volume from an existing EBS snapshots. This will create a volume on the instance from the snapshot. As with EBS volumes, you will have to create the snapshot from an existing volume in the AWS console. Once you have created a snapshot, you specify SnapshotId in a [[[volume]]] section of your template:

[[nodearray execute]]

  [[[volume reference-data]]]    # 'reference-data' is the name of the volume within CycleCloud
  SnapshotId = snap-1234abcd

EBS snapshots are available across an entire region, so you can attach a snapshot to an instance in any zone. As with EBS volumes, you can specify a device to attach to or CycleCloud will automatically pick one for you.

Note

If you also specify a size, it must be at least as large as the snapshot. If you do not specify a size, the size of the snapshot is used.

Warning

You can ensure automatic volumes are kept after instance termination if you specify Persistent=True. However, you may end up with a large number of volumes, increasing your cloud-provider bill. This option is for advanced use only.

Ephemeral/Instance Storage

Ephemeral disks are handled automatically for you by CycleCloud.

For AWS specific details about ephemeral storage, please see the official EC2 documentation.

Custom Security Groups

In the Quick Start Guide, users created and updated the default security group for use on all of your cluster nodes. You can customize your security groups from AWS and assign them to the nodes in the cluster template files. Typical reasons for customizing security groups include the default security group being used for another purpose, wanting to open or close additional ports, etc.

You can create and modify security groups from the AWS console as you did in the Quick Start Guide, and then inside a cluster template you can modify the security groups settings to reference your newly created security group:

SECURITY_GROUPS = sg-1234abcd, sg-abcd1234

This setting will instruct CycleCloud to start nodes using the two security groups listed. To use the default security group, comment out the line.

Note

You can specify different security groups for each class of node. For example your master/head nodes could use one security group that opens ports 22 and 8080 (for CycleServer access), while the worker/execute nodes use a different security group that only opens port 22 for ssh access.

Remember: You need to allow all nodes within your cluster to talk to each other. Make sure at least one of your security groups has TCP/UDP ports open to other members of the security group and is used by every class of node.

Custom Key Pair

In the quickstart you created a keypair named cyclecloud. To use a different AWS keypair, specify the name and location of the keypair in the cluster template file:

[cluster your_cluster]

  [[node defaults]]
  # This will set the keypair for all nodes to 'custom'
  KeyPair = custom
  KeyPairLocation=~/.ssh/custom.pem

Network Interfaces

You can configure the standard network device or attach additional devices with the network-interface element:

[cluster your_cluster]

  [[node simple]]
  MachineType = m1.small

  [[[network-interface]]]
  SecurityGroups = sg-12345678
  PublicIp = 198.51.100.1

This configures the first network device to apply rules from security group sg-12345678 to its traffic, and use 198.51.100.1 as its public IP. (This assumes that you have already allocated 198.51.100.1 in your cloud provider, and thus can use that IP.)

If you simply want to use an existing network interface, you can include its ID in the section:

[[[network-interface]]]
InterfaceId = eni-23456789

This will use the eni-23456789 network interface, assuming it exists and is not already attached to another instance. If you do not specify an ID, then one will be created and set to be automatically deleted when the instance is terminated.

Note

You cannot specify a SubnetId with an existing interface (when InterfaceId is supplied), because the interface already has a subnet.

As an alternative to specifying a public IP manually, you can ask for one to be created for you with the AssociatePublicIpAddress parameter:

[[[network-interface]]]
AssociatePublicIpAddress = true

This is only necessary for VPC, which does not assign public IP addresses by default.

VPC instances may also have a fixed Private IP address assigned to their network interfaces using the PrivateIp parameter:

[[[network-interface]]]
PrivateIp = 10.0.1.10

This attribute is only valid in VPC. To use it, simply select a valid IP within your VPC’s subnet to dedicate to the associated node.

Amazon Virtual Private Cloud

CycleCloud supports launching clusters using Amazon’s Virtual Private Cloud. You will have to configure your network through the AWS Console’s VPC section. Once you have configured your network, you can alter cluster templates to use your network by specifying the subnet ID to use when launching instances:

# VPC Settings
SubnetId = subnet-abcd1234

This will instruct CycleCloud to launch instances in this VPC subnet instead of public EC2.

Amazon Placement Groups

CycleCloud supports launching cluster compute class nodes in placement groups for connection-intensive computing. A placement group allows the machines to be physically located very close together to take advantage of high speed interconnect between the machines, making clusters like this good for MPI-type workloads. You will have to create a placement group in the AWS/EC2 console, for example one named ‘demo-pg’. Once you have created a placement group you can tell your execute machines to launch in this placement group:

[[nodearray execute]]
MachineType = cc2.8xlarge   # Cluster Compute 2 instance
PlacementGroup = demo-pg
Zone = us-east-1a

In the above example we are creating execute nodes using the Cluster Compute class instances, telling Amazon to place them all in the demo-pg group in the availability zone us-east-1a.

Note

When using placement groups you should always specify the zone you want the machines to be placed in as Amazon will refuse any requests to put machines from different zones in the same placement group.

Note

Cluster compute class machines require HVM (Hardware Virtual Machine) images. CycleCloud provides and supports these, but you should make sure to properly specify the image when attempting to launch Cluster Compute instances.

Using Limited Identity Access Management (IAM) Credentials

While we suggest that you use your root AWS Keys which have full access to your account for ease of setup, this level of access is not required for CycleCloud to function. CycleCloud requires full EC2 access (to manage compute on your behalf) and read/write access to a single bucket in S3 (for data storage). Access to other AWS services is not needed at this time. If your AWS account is used by other people at your organization or you use other AWS services, you may want to use limited IAM credentials with the CycleCloud software.

Drop the following records into a text file at “${CS_HOME}/config/data/initial_setup.txt”. Replace the variables with your account settings ([ YOUR_LOCKER_URL_HERE, COMMON_CHEF_VERSION, INITIAL_ADMIN_PASSWORD, CYCLE_PORTAL_SITE_NAME, CYCLE_PORTAL_ACCOUNT_NAME, CYCLE_PORTAL_ACCOUNT_PASS ]). You may also need to customize the Region attributes of various records if not using us-east-1.

AdType = “Application.Setting” Name = “cycleserver.sendAnonymizedData” Value = false

AdType = “Application.Setting” Name = “cycleserver.installation.complete” Value = true

AdType = “AuthenticatedUser” Name = “admin” RawPassword = “${INITIAL_ADMIN_PASSWORD}” Roles = {“Administrator”}

AdType = “Application.Setting” Name = “site_name” Value = “${CYCLE_PORTAL_SITE_NAME}”

Category = “Support” AdType = “Application.Setting” Description = “The account login for this installation” Value = “${CYCLE_PORTAL_ACCOUNT_NAME}” Name = “support.account.login”

Category = “Support” AdType = “Application.Setting” Description = “The account password for this installation” Value = “${CYCLE_PORTAL_ACCOUNT_PASS}” Name = “support.account.password”

AWSUseInstanceProfile = true AWSAccessKey = “” AWSSecretKey = “” AdType = “Credential” Name = “cloud-locker” DefaultRegion = “us-east-1” CredentialType = “AWS” CommonName = “cloud-locker” Owner = undefined AccountName = “cloud” Default = false

AdType = “Credential” Name = “cloud” DefaultRegion = “us-east-1” CredentialType = “AWS” Login = ifThenElse(CredentialType===”PrivateKey”,owner_name(Name),undefined) CommonName = “cloud” Owner = undefined AccountName = “cloud” Default = true AWSUseInstanceProfile = true Provider = “aws”

AdType = “Cloud.ProviderAccount” Name = “cloud” MasterCredentials = “cloud” DefaultRegion = “us-east-1” Provider = “aws” Default = true

AdType = “Cloud.Locker” Credentials = “cloud-locker” Name = “cloud-storage” AccountName = “cloud” AccountId = “default” Location = “s3://${YOUR_LOCKER_URL_HERE}/” Shared = false Region = “us-east-1”

Creating a Bucket

If root credentials are not being used, you will have to create a bucket for use with CycleCloud since it cannot create the bucket for you. The naming scheme used by the ‘initialize’ command when generating buckets is ‘com.cyclecloud.name.region.locker’ where name is a name of your choice (for instance, it might be your organization name), and region is the AWS region, e.g. us-east-1. You will have to create a bucket following this scheme, tag it appropriately, and then grant the IAM user read/write (full) access to this bucket. As an example, if we were to simply use’limited’ as name and ‘us-east-1’ as region you would create a bucket named ‘com.cyclecloud.limited.us-east-1.locker’. Once created, you need to add a single tag to this bucket so that CycleCloud can detect it. The key for the tag should be “CycleCloudAccountId” and the value of the tag should be “limited”, or whatever you have chosen as name.

IAM Policy

As mentioned above, the IAM policy can be anything you want as long as it has full EC2 access as well as read/write access to the bucket created and tagged in the step above. An example IAM policy for the bucket above may look like:

{
  "Statement": [
    {
      "Sid": "Stmt1362611954648",
      "Action": [
        "ec2:*"
      ],
      "Effect": "Allow",
      "Resource": [
        "*"
      ]
    },
    {
      "Sid": "Stmt1362611973688",
      "Action": [
        "s3:*"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::com.cyclecloud.limited.us-east-1.locker"
      ]
    },
    {
      "Sid": "Stmt1362611991617",
      "Action": [
        "s3:*"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::com.cyclecloud.limited.us-east-1.locker/*"
      ]
    }
  ]
}

Initializing CycleCloud

When you initialize CycleCloud with your IAM credentials you will use the access key and secret key for the IAM account instead of the root credentials. When prompted for a region, you must specify the region identical to that used when naming the bucket. When prompted for a name, you have to specify the exact name used when creating and tagging your bucket (in the above example you would enter ‘limited’). CycleCloud will then be configured to use this bucket for all data storage purposes.

Enabling Usage Tracking

AWS provides billing reports that can be programmatically accessed. Cycle uses these reports for billing, auditing and troubleshooting. Many of our customers use these reports to analyze usage data across individual clusters. We recommend that you follow these instructions to enable billing reports so that reports are collected from the beginning. They will provide you with an audit trail for all usage.

Enabling Billing Reports

Create a S3 bucket to hold the usage reports. Update the bucket’s usage policy so that AWS has access to write the reports.

Follow the AWS instructions for enabling the usage reports. In short you will be following these steps:

  • Select ‘Receive Billing Reports’.
  • Enter your usage reports bucket name (and make sure that AWS has access by attaching the AWS sample policy to it).
  • Select the ‘Cost allocation report’ and the ‘Detailed billing report with tags’.
  • Select the appropriate tags.
  • Save settings.

https://docs.cyclecomputing.com/wp-content/uploads/2018/04/usage_report_preferences-1.png

Adding Cycle Tags to Reports

You will want to add a number of tags to the billing report so that the report can be filtered by cluster and node usage. For more information on tagging, see the AWS documentation .

Select the tags listed on the Tags page and enable these tags:
  • Name
  • ClusterName
  • CycleOwner

Granting Cycle Access

Cycle uses these billing reports to audit their internal usage tracking. For customers that are billed based on usage we also use these reports to determine your Cycle bill.

Create a Cycle user on the users page.
  • Click ‘Create New User’.
  • Enter “cyclecomputing” as user name.
  • Select the ‘Generate an access key’ checkbox.
  • Click ‘Create’.
  • Save the access key and the secret key.
Grant access to the usage bucket for the Cycle user.
  • Select the ‘cyclecomputing’ user and “Attach User Policy”.
  • Add the following policy and replace BUCKET_NAME with the name of your usage bucket.
{
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::BUCKET_NAME",
                "arn:aws:s3:::BUCKET_NAME/*"
            ]
        }
    ]
}

Note

Please provide the usage bucket name, access key and secret key to your Cycle Computing support contact.

Using the Reports

Reports are updated periodically throughout the day to provide an accurate view into your usage. By importing the csv report into Excel and using pivot tables on the ClusterName tags you can get detailed usage information for your individual CycleCloud clusters.