CycleCloud supports both on-demand and scheduled data transfers between endpoints. Both of these modes are managed via the Transfer Manager page, which is accessed from the Data dropdown menu.

https://docs.cyclecomputing.com/wp-content/uploads/2018/04/data_transfer.png

On-Demand Transfers

To begin an on-demand transfer, select the appropriate endpoint on the left side of the page. On the right, select a cloud provider endpoint. Use the file browsers for each endpoint to locate the data to transfer and the desired destination. Click the right arrow to upload to cloud storage, or the left arrow to download to your endpoint. Please note that the transfer request is submitted for execution, to be run as soon as a worker is available to perform the transfer. The number of workers can be configured by editing the Queue records and changing the worker attribute to the desired number. The default setting is 1, which is recommended for most purposes.

Note

If the source is a directory, Data Manager will recursively transfer all files and subdirectories.

Scheduled or Recurring Transfers

Scheduled transfers are done the same way as on-demand transfers, but use the clock arrows instead. This will allow you to transfer files:

  • Immediate: transfer right away
  • Scheduled: transfer once, at a set time
  • Recurring: transfer on a regular schedule

If you wish to encrypt your data, you can import or create a new encryption key in the Advanced section of the Scheduler.

Once your scheduled or recurring transfer has been created, click Go to schedule it. To view, edit, or delete your scheduled transfer(s), click the Scheduled Transfers tab at the bottom of the page.

Monitoring Transfer Details

When an immediate transfer is initiated or a scheduled or recurring transfer begins, transfer details are displayed in the bottom pane of the Transfer Manager page. You may need to click the refresh button for an accurate transfer progress reading.

https://docs.cyclecomputing.com/wp-content/uploads/2018/04/transfer_pane.png

Double-clicking on a transfer displays further transfer details.

https://docs.cyclecomputing.com/wp-content/uploads/2018/04/transfer_details_window.png

Clicking on the Scheduled Transfers tab displays a queue of future transfers that have already been scheduled.

https://docs.cyclecomputing.com/wp-content/uploads/2018/04/scheduled_transfer_pane.png

Double-clicking a queued transfer displays further details about the scheduled transfer.

https://docs.cyclecomputing.com/wp-content/uploads/2018/04/transfer_request_details.png

Transfer Listeners

CycleCloud supports writing plugins that are executed based on transfer events, such as individual file transfer completion or failure, or entire transfer completion or failure. Documentation for creating these plugins can be found in the CycleCloud Admin Guide.

Advanced Settings

In order to tune DataMan transfers, a number of settings can be adjusted. Navigate to the settings page by clicking on the user drop-down menu on the top right of the screen and selecting Settings.

https://docs.cyclecomputing.com/wp-content/uploads/2018/04/settings_dropdown.png

On the settings page, double clicking on the DataMan entry displays the DataMan settings form as shown below.

https://docs.cyclecomputing.com/wp-content/uploads/2018/04/dataman_settings.png

Transfer Threads This setting modifies the number of threads used to transfer data. Recommended value of this setting depends on your network bandwidth and the number of CPU cores available on the DataMan host. To prevent http connection errors during uploads, we recommend that this value not be set greater than 32. Default: 16.

Transfer Part Size This is the size of a single chunk of a multi-part transfer. Default: 16,777,216.

Proxy Server Configuration To enable using a proxy server for network operations, check the Use Proxy checkbox and put the details of the proxy server in the following fields. Default: Unchecked.