This document describes a VPC setup in which you create a public zone that can be accessed from the Internet (with appropriate credentials), and a private zone that cannot be accessed directly from the Internet (but can be accessed by servers in the public zone).

Overview

The high-level steps to configuring a VPC are:

  • Create the VPC with private and public subnets using the VPC Wizard
  • Modify the public subnet to auto-assign public IP addresses
  • Create a ‘public’ security group
  • Restrict outbound traffic from the default security group
  • (Optional) Create a dedicated bastion server

Note

There are a lot of lower-level steps to setting up the VPC, such as creation of an Internet Gateway, a NAT Gateway, Route Tables, etc. This document uses the “VPC Wizard” to handle the creation and configuration of these resources. You can learn more about the lower-level details in the official AWS VPC User Guide.

Accessing Your Cluster

There are three ways to connect to instances running in a VPC:

  • Your machine is part of your organization’s VPN
  • Create a bastion host with a elastic IP
  • For each cluster you launch with CycleCloud, configure one node as a proxy through which the rest of the cluster is accessed

Corporate VPN Connection

This is the simplest method and recommended when running in a production scenario in your own network. Instances inside the VPC are directly reachable by your machine.

AWS offers several options to connect your internal corporate network to a VPC via a VPN. Please see the official Amazon documentation for more information on those options.

Bastion Server with an Elastic IP

Outside of connecting your corporate network directly to your VPC, the next best option is to create an external-facing server (also known as a “bastion server”) with a publicly accessible IP address or an elastic IP.

The key distinction between elastic IP addresses (static IP addresses designed for dynamic cloud computing and associated with your AWS account) and auto-assigned IP addresses is that elastic IP addresses are guaranteed to not change for the life of the instance, and can quickly be remapped to a different instance. The same is not true for auto-assigned public IP addresses, which can change any time without warning.

Note

It is highly recommended that you lock down inbound access to the server to only a few protocols (typically only SSH), and only allow access to known IP addresses.

Proxy Server

Instead of using a dedicated bastion server, you can configure one of the nodes in your cluster to act as a proxy for communicating back to CycleCloud. The benefit of this is that you will not have to pay for a residual EC2 instance after you have torn down your cluster. For this to work, you will need to configure the public subnet to automatically assign public IP addresses:

[cluster htcondor]
  [[node proxy]]
  # this attribute configures the instance to act as a proxy
  IsReturnProxy = true
  credentials = cloud
  MachineType = t2.micro
  # this is the public subnet
  subnetid = subnet-1234557
  ImageName = cycle.image.centos7

  [[node private]]
  # this is the private subnet
  subnetid = subnet-1234557

Please note that proxy node in this cluster template only proxies communication from instances to CycleCloud. It does not proxy communication to the larger Internet.

You can use SSH tunneling to access the private node in this example via the proxy node.

Outbound Access to the Internet

After providing connectivity to your cluster, your cluster must still access the internet. For this to work, you must provide access to S3 and NTP in order to use CycleCloud.

For access to S3, a VPC Endpoint is the recommended option. We suggest creating a NAT Gateway for NTP access, and also recommend using the NAT Gateway to grant instances in the private subnet access to arbitrary services on the Internet.

A NAT instance can also be used to route traffic to the Internet but it is not as performant and takes more effort to configure and manage. A comparison of NAT gateways and NAT instances is available on AWS.

This document describes how to setup a “bastion server” which can also be used as a NAT
Instance
with additional configuration steps not detailed here.

Note

The bastion server should belong to both the ‘public’ and ‘private’ security groups.

Creating a VPC

Use the AWS VPC Wizard to set up your VPC. We recommend using a full /16 block such as 172.16.0.0/16.

  1. Browse to the VPC dashboard in the AWS Console
  2. Click “Start VPC Wizard”
  3. Select the “VPC with Public and Private Subnets” VPC configuration
  4. Click Select
  5. Enter 172.16.0.0/16 as the IPv4 CIDR block
  6. Name the VPC something meaningful like ‘cyclecloud’
  7. For “Public subnet’s IPv4 CIDR” enter 172.16.0.0/24
  8. Set the Public Subnet name to ‘public’
  9. For “Private subnet’s IPv4 CIDR” enter 172.16.16.0/20
  10. Set the Private Subnet name to ‘private’
  11. Place your cursor in the text box for Elastic IP Allocation ID
    and choose the first available option from the drop-down
  12. Click “Add Endpoint” to create a VPC endpoint that grants access to S3
  13. Accept all other defaults and click “Create VPC”

Once you click “Create VPC”, the process can take a few minutes to complete.

The VPC Wizard does a lot of work for you. It has created:

  • A NAT Gateway providing outbound Internet access for private subnet
  • An Internet Gateway that allows instances in the public subnet to
    be accessed from the Internet
  • A VPC endpoint for S3 access that allows instances in the public and
    private subnets to access S3 without traversing the Internet
  • Route Tables for the private an public subnets to
    configure routing to appropriately use the NAT Gateway, Internet Gateway,
    and VPC endpoint

To complete your VPC setup, several other items must be configured.

Turn on Auto-Assign Public IP Addresses

Once the Public security group has been created, modify it to auto-associate public IP addresses:

  1. Click “Subnets” in the lefthand menu
  2. Click “Subnet Actions” drop-down with the public subnet selected
  3. Select “Modify Auto-assign IP settings”
  4. Select “Enable auto-assign public IPv4 address”
  5. Click Save

Restrict Access to the Public Subnet

Create a ‘public’ security group that only allows SSH access. The “public” security group should be locked down as much as possible, since it faces the public internet. We recommend opening port 22 inbound for source 0.0.0.0/0.

  • SSH, source: 0.0.0.0/0

Next, remove all outbound rules and add a single one for SSH. This rule is needed for SSH port forwarding.

  • SSH, destination: default security group

Note

Bastion servers must belong to both the ‘default’ and ‘public’ security groups.

Private Security Group

The VPC wizard creates the ‘default’ security group with very permissive outbound rules. You may wish to restrict outbound access to NTP and S3.

To do so, remove the default outbound rule “ALL Traffic” and add the following outbound rules:

  • Custom UDP, 123, destination: 0.0.0.0/0 (NTP)
  • HTTP, destination: pl-63a5400a
  • HTTPS, destination: pl-63a5400a
  • All Traffic, destination: the public security group

pl-63a5400a is a shorthand reference for the set IP ranges used by S3 in the us-east-1 region. You will need a different reference for S3 in other regions.

To view the current pl-xxxx ID for S3 in various regions, you can use the describe-prefix-lists command.

Launching a Bastion Server

Once your VPC is set up, create a bastion server to verify the configuration.

To create a bastion, follow these steps:

  1. Click on “VPC Dashboard” in the left menu
  2. Click “Launch EC2 Instances”
  3. Select the default Amazon Linux AMI
  4. Choose “t2.micro” (or other instance)
  5. Select “EC2 VPC” and pick the public subnet (172.16.0.0/24)
  6. Click “Next”
  7. Create a tag with the key ‘Name’ and the value “bastion”
  8. Important choose both the “public” and “default” (private) security groups and click “Continue”
  9. Click Launch
  10. Choose the keypair you want to use and click “Continue”

Finally, if you did not auto-assign a public IP address you will need to attach an elastic IP address to the bastion server.

  1. Click on “Elastic IPs” in the left menu
  2. Click on “Allocate New Address”, select “VPC” from drop-down and click “Allocate”
  3. Click on “Associate Address”, select the bastion instance and then click “Associate”

Connecting to these instances is covered in the following section of the Administration Guide.