Endpoints are logical connections to storage providers and can be any of the following:

Endpoint Types

Type Description
Azure Block Storage Microsoft Azure’s storage service
Amazon S3 Amazon’s Simple Storage Service
Google Cloud Storage Google Cloud’s object storage service
Host Data residing on servers accessible to the CycleCloud Data Manager. These can be local directories or other mounted file systems.

To view and manage endpoints, select Data then Transfer Manager from the CycleCloud menu bar.

Configuring a Local Endpoint

On the left side of the “Transfer Manager” screen, you can see that an initial local endpoint has been created by the installer, and is configured to use the /tmp directory. Click “Save” to create this endpoint.

Adding Another Local Directory

To add another directory to the local endpoint, click the endpoint name next to the gear in the top left, and select Edit Endpoint.

https://docs.cyclecomputing.com/wp-content/uploads/2018/04/edit_endpoint.png

Click the + to generate a new line, then enter the path. Click Save to continue.

https://docs.cyclecomputing.com/wp-content/uploads/2018/04/local_path.png

Note

Any directories in the “Allowed Paths” must be accessible by the user running CycleCloud.

Host Endpoint Attributes

Attribute Description
Name Name of the endpoint
Type The type of this endpoint (see table above)
Allowed Paths A list of system paths accessible by this endpoint
Encryption Key The encryption key used to encrypt all files transferred to this endpoint
Local When checked, indicates that folders are local to the host
Hostname Hostname for host endpoints
SSH User Remote login name
Credential The SSH keypair to use when logging into a remote host
SSH Port Port for connections via SSH
Pogo Path The path to the Pogo executable on the remote host

Adding Cloud Provider Credentials

You will need to add credentials for each of the cloud provider accounts you wish to use with the CycleCloud Data Manager.

  • Click on the host name in the upper left and select Add Endpoint
  • Click on the green plus sign next to Credential, which will open a new window called Create Credential
  • Enter a descriptive name for the credential
  • Select the appropriate Type:
    • Azure: Azure Access Key
    • AWS: Amazon
    • Google: Google Cloud Private Key
  • Enter your Access and Secret Keys
  • Select your default region
  • Confirm your entries by clicking Test
  • Click “Save” to save your credentials. You can now use the credentials to create an Amazon S3, Azure Block Storage, or Google Cloud Storage endpoint.

Restricted S3 Credentials

If you wish to restrict privileges to a minimum necessary set, we recommend using the following IAM policy:

{
"Action": [
    "s3:ListAllMyBuckets"
],
"Effect": "Allow",
"Resource": "*"
}

{
"Action": [
    "s3:AbortMultipartUpload",
    "s3:DeleteObject",
    "s3:GetObject",
    "s3:GetObjectVersion",
    "s3:PutObject",
    "s3:ListBucket",
    "s3:CreateBucket",
    "s3:DeleteBucket",
    "s3:GetBucketLocation",
    "s3:ListBucketMultipartUploads",
    "s3:ListMultipartUploadParts"
],
"Effect": "Allow",
"Resource": [
    "arn:aws:s3:::com.example.bucket",
    "arn:aws:s3:::com.example.bucket/*"
]
}

Adding Endpoints

Adding a Microsoft Azure Endpoint

Once you’ve added Azure credentials, you can set up an Azure Storage endpoint.

  • From the New Endpoint window, enter a descriptive Name
  • Select Azure Storage as the Type
  • Enter the Allowed Path. Use the + to add additional paths if needed.
  • Select the Encryption Key
  • Choose the appropriate Azure Credential
  • Select the Storage Account and Storage Container via the dropdown menus
  • Click Save to add your Azure endpoint

https://docs.cyclecomputing.com/wp-content/uploads/2018/04/add_endpoint-azure.png

Adding an AWS S3 Endpoint

After you’ve created AWS credentials, you can set up an AWS S3 endpoint. From the Add Endpoint window, enter a descriptive name in the “Name” field. Select “Amazon S3” from the “Type” drop-down, then select the credentials you created above. Next, select the desired bucket from the drop-down or click the green plus sign to create a new bucket. Click “Save” to save the endpoint.

https://docs.cyclecomputing.com/wp-content/uploads/2018/04/add_endpoint-s3.png

Note

If you do not have an S3 bucket created, you can click the green plus button next to the bucket dropdown to create one.

Adding a GC Endpoint

Once you’ve added Google Cloud credentials, you can set up a Google Cloud Storage endpoint. From the Add Endpoint window, enter a descriptive name in the “Name” field. Select “Google Cloud Storage” from the “Type” drop-down. Select the GC credentials you created earlier. Select a bucket from the drop-down or click the green plus sign to create a new bucket. Click “Save” to save the endpoint.

https://docs.cyclecomputing.com/wp-content/uploads/2018/04/add_endpoint-gcp.png

Adding a Remote Host Endpoint

For server-to-server transfers, you can create another “Host” type endpoint. To add a remote host endpoint, select “Add Endpoint” from the “Type” drop-down. Select “Host” as the endpoint type. Name the endpoint, and make sure the “Local Host” checkbox is unchecked. Fill in your remote host login credentials, then select SSH Private Key Credential that will be used to connect to the host.

https://docs.cyclecomputing.com/wp-content/uploads/2018/04/add_endpoint-remotehost.png

Endpoint Monitoring

CycleCloud monitors endpoints for directory structure, file size (used to calculate transfer times), and last modified times. For large amounts of data, the monitoring process can take up a significant amount of processing time. CycleCloud has implemented a daily monitor limit, the default of which is set to 10 minutes.

The default limit will suffice for the majority of users, but can be changed. You can also choose to turn off monitoring for a particular endpoint. To edit the settings, click the gear icon in the Transfer Manager and select the appropriate endpoint. Click the gear again to select Edit Endpoint. Under Advanced, deactivate the checkbox to disable monitoring. To increase the monitor limit from the default, enter the number of minutes you wish to allocate to monitoring.

If you wish to retain the directory data for the endpoint after disabling monitoring, click the appropriate checkbox.

After making your changes, click Save to apply.

Note

The 10-minute daily limit will not prevent the monitor from fully traversing endpoints when doing so would exceed that limit.