Endpoints are logical connections to storage providers and can be any of the following:
|Amazon S3||Amazon’s Simple Storage Service|
|Azure Block Storage||Microsoft Azure’s storage service|
|Google Cloud Storage||Google Cloud’s object storage service|
|Host||Data residing on servers accessible to the CycleCloud Data Manager. These can be local directories or other mounted file systems.|
To view and manage endpoints, select Data then Transfer Manager from the CycleCloud menu bar.
Configuring a Local Endpoint
On the left side of the “Transfer Manager” screen, you can see that an initial local endpoint has been created by the installer, and is configured to use the /tmp directory. Click “Save” to create this endpoint.
Adding Another Local Directory
To add another directory to the local endpoint, click the endpoint name next to the gear in the top left, and select Edit Endpoint.
Click the + to generate a new line, then enter the path. Click Save to continue.
Any directories in the “Allowed Paths” must be accessible by the user running CycleCloud.
Host Endpoint Attributes
|Name||Name of the endpoint|
|Type||The type of this endpoint (see table above)|
|Allowed Paths||A list of system paths accessible by this endpoint|
|Encryption Key||The encryption key used to encrypt all files transferred to this endpoint|
|Local||When checked, indicates that folders are local to the host|
|Hostname||Hostname for host endpoints|
|SSH User||Remote login name|
|Credential||The SSH keypair to use when logging into a remote host|
|SSH Port||Port for connections via SSH|
|Pogo Path||The path to the Pogo executable on the remote host|
Adding Cloud Provider Credentials
You will need to add credentials for each of the cloud provider accounts you wish to use with the CycleCloud Data Manager.
- Click on the host name in the upper left and select Add Endpoint
- Click on the green plus sign next to Credential, which will open a new window called Create Credential
- Enter a descriptive name for the credential
- Select the appropriate Type:
- AWS: Amazon
- Azure: Azure Access Key
- Google: Google Cloud Private Key
- Enter your Access and Secret Keys
- Select your default region
- Confirm your entries by clicking Test
- Click “Save” to save your credentials. You can now use the credentials to create an Amazon S3, Azure Block Storage, or Google Cloud Storage endpoint.
Adding an AWS S3 Endpoint
After you’ve created AWS credentials, you can set up an AWS S3 endpoint. From the Add Endpoint window, enter a descriptive name in the “Name” field. Select “Amazon S3” from the “Type” drop-down, then select the credentials you created above. Next, select the desired bucket from the drop-down or click the green plus sign to create a new bucket. Click “Save” to save the endpoint.
If you do not have an S3 bucket created, you can click the green plus button next to the bucket dropdown to create one.
Adding a Microsoft Azure Endpoint
Once you’ve added Azure credentials, you can set up an Azure Storage endpoint.
- From the New Endpoint window, enter a descriptive Name
- Select Azure Storage as the Type
- Enter the Allowed Path. Use the + to add additional paths if needed.
- Select the Encryption Key
- Choose the appropriate Azure Credential
- Select the Storage Account and Storage Container via the dropdown menus
- Click Save to add your Azure endpoint
Adding a GC Endpoint
Once you’ve added Google Cloud credentials, you can set up a Google Cloud Storage endpoint. From the Add Endpoint window, enter a descriptive name in the “Name” field. Select “Google Cloud Storage” from the “Type” drop-down. Select the GC credentials you created earlier. Select a bucket from the drop-down or click the green plus sign to create a new bucket. Click “Save” to save the endpoint.
Adding a Remote Host Endpoint
For server-to-server transfers, you can create another “Host” type endpoint. To add a remote host endpoint, select “Add Endpoint” from the “Type” drop-down. Select “Host” as the endpoint type. Name the endpoint, and make sure the “Local Host” checkbox is unchecked. Fill in your remote host login credentials, then select SSH Private Key Credential that will be used to connect to the host.
CycleCloud monitors endpoints for directory structure, file size (used to calculate transfer times), and last modified times. For large amounts of data, the monitoring process can take up a significant amount of processing time. CycleCloud has implemented a daily monitor limit, the default of which is set to 10 minutes.
The default limit will suffice for the majority of users, but can be changed. You can also choose to turn off monitoring for a particular endpoint. To edit the settings, click the gear icon in the Transfer Manager and select the appropriate endpoint. Click the gear again to select Edit Endpoint. Under Advanced, deactivate the checkbox to disable monitoring. To increase the monitor limit from the default, enter the number of minutes you wish to allocate to monitoring.
If you wish to retain the directory data for the endpoint after disabling monitoring, click the appropriate checkbox.
After making your changes, click Save to apply.
The 10-minute daily limit will not prevent the monitor from fully traversing endpoints when doing so would exceed that limit.