CycleCloud’s orchestration process is designed to be extended with two forms of customization: user-defined phases that are part of the orchestration, and must finish for the resource to move on; and user-defined listeners that are notified asynchronously of resources that enter a given state.
The canonical example is a node resource that creates a matching virtual machine, and does any pre-configuration or post-configuration required for that virtual machine. This takes many steps, executed in sequence, and some of those steps will require waiting on an external operation to complete or an external resource to reach the right state (eg, for the instance to be ready to interact with).
The orchestration process is designed to group resources in the same phase and process them in bulk. This lets phase plugins take advantage of the primary methods for ensuring code is efficient:
- batching calls to query or modify resources
- caching information to be used for multiple resources in turn
For this reason, CycleCloud tracks resources as a group by the phase(s) they are in. Whenever there is at least one resource that just entered a phase, CycleCloud collects all resources waiting to run that phase and executes the plugin for that phase with those resources. Some phases will need to be called repeatedly if they are waiting for an external operation, and when they are called they are given all the resources ready to run or rerun that phase. This means that any internal progress the plugin needs to track must be saved on each resource, because the group of resources given to the plugin can vary for each execution.
A phase plugin is responsible for invoking the behavior for a single step in the orchestration process. Each time the plugin is called, it is given a list of resources that are currently in that phase. It must execute the code for those resources (either one at a time, or as a bulk operation where possible), and determine for each resource whether that operation succeeded, failed, or is still pending. Note that although the phase is called with a set of resources for efficiency, each resource has its own separate status with respect to that phase, so the documentation describes each resource in isolation for simplicity.
To make a phase, you first create a plugin that indicates which states it handles. For instance, a very simple plugin that creates instances for the acme cloud service provider (CSP) would be:
HandlesStates = Allocation ResourceType = Cloud.Node PhaseName = Acme.CreateInstance Phase.Description = Starting Acme instance ForProvider = acme
In this case, CycleCloud will create a phase named Acme.CreateInstance once any Cloud.Node resource that is using the acme provider reaches the Allocation state. The plugin itself simply needs to call the create-instance method for each node:
import acme def handle(handler): for node in handler.getResources(): # convert node settings to the format that acme uses instance_params = _convert_node_to_instance(node) # look up the keys for this node creds = _lookup_credentials(node.getAsString("Credentials")) # create the instance instance_id = acme.create_instance(creds, instance_params) # update the node with the instance id node.setString("InstanceId", instance_id) handler.setCompleted(node)
This plugin makes a call to a fictional acme API that can create an instance in the cloud provider. It converts the node settings and the credential to the format that acme.create_instance wants, and then calls it. It stores the instance id on the node. Note that the resources are saved automatically when the call exits.
This plugin assumes the instance is created and ready to go immediately, assuming the call does not fail. A more realistic example would need to check back on the instance periodically to see when it becomes ready. However, implementing that as a loop that sleeps and then queries the instance is definitely not the right approach. See the section on asynchronous plugins for more information.
Phase plugins should never block on asynchronous external operations! This means that they can call external APIs (eg, HTTP requests), which may take seconds to execute, but should not do any waiting or sleeping inside the plugin.
If the operation succeeds for a resource, the phase will succeed for that resource, and the resource will exit that phase. If the resource is in any other phases in the current state, it waits for those phases to complete; if not, the resource moves on to the next state in line. If the next state is the terminal state (eg Started for Cloud.Node resources), then the orchestration process is complete for that resource.
If the operation fails for a resource, the status for that resource and that phase, will be Failed for that resource. In addition, the resource as a whole will be marked as failed (specifically, PhaseFailed will be set to true on the resource). If the resource is in any other phases, those phases will continue normally, but when they complete, the resource will not exit the current state, since the failed phase is not complete. Note that if a plugin throws an exception, all resources it was given (except for those it had marked completed before it threw the exception) are considered to have failed. Resources that have failed a phase can be retried by the user, possibly after they correct the issue that was failing.
If the operation is still pending, the phase will be sleeping for that resource. The plugin will be called again after a delay (which defaults to 15 seconds, but can be overridden). Note that when the plugin is called again, it gets all waiting resources, including any new ones that are now in that phase.
Each resource has a separate status for each phase that it has ever been in:
- Running: the plugin for this phase is currently running with this resource
- Waiting: the plugin will be called with this resource the next time the phase runs
- Sleeping: the plugin will be called with this resource after a given time in the future
- Completed: the plugin already ran for this resource and succeeded
- Failed: an error occurred while running the plugin for this resource
- Canceling: the plugin is running, but the phase is being canceled so the resource can move to some other target state
The status is stored along with other phase-specific information in the PhaseMap attribute, which is a nested record on the resource that holds information for all the phases for this resource. Each attribute in the record is the name of a phase, and the entry for the phase holds the resource phase data. CycleCloud uses certain attributes on the phase data, but plugins can also store phase-specific data there. For instance, this node has three phases, two active:
Name = "Node1" PhaseMap = [ Phase1 = [ Status = "Completed"; ]; Phase2 = [ Status = "Sleeping"; Message = "Performing operation 2"; OperationId = "abc123" ]; Phase3 = [ Status = "Running"; Message = "Performing operation 3"
The plugin for Phase2 has set the OperationId attribute to the id of an operation it is tracking for that resource. The existence of that attribute and its meaning are completely determined by that plugin. A plugin can get the record for its phase and update it with the getResourcePhaseData() method:
phase_data = handler.getResourcePhaseData(resource) phase_data.setString("OperationId", current_operation_id)
Updates to the phase are automatically persisted to the resource after the plugin runs.
Most real-world plugins will end up calling a remote service, typically over HTTP, but a simpler model that illustrates the same aspects is an SSH call. Suppose you need to create a “host file” in a directory on a remote server for each node you are starting, once it reaches the Configuration state. First, you need to define the plugin’s configuration to indicate it is a phase plugin, and which state it should be in:
HandlesStates = Configuration ResourceType = Cloud.Node PhaseName = Example.CreateFile Phase.Description = Creating host files
In this case, CycleCloud will create a phase named Example.CreateFile once any Cloud.Node resource reaches the Configuration state. The plugin itself simply executes touch over ssh:
import subprocess def handle(handler): for resource in handler.getResources(): try: subprocess.check_call(["ssh", "user@remotehost", "touch", resource.getAsString("Name")]) handler.setCompleted(resource) except subprocess.CalledProcessError, e: handler.setFailed(resource, "Failed to create file: %s" % e)
It executes one-by-one, which is easy but not very efficient for a high-latency operation (eg an SSH or HTTPS call). This could also be written as a bulk operation that builds up a single command line and runs that:
import subprocess def handle(handler): args = ["ssh", "user@remotehost", "touch"] args.extend([resource.getAsString("Name") for resource in handler.getResources()]) subprocess.check_call(args) handler.setCompleted(handler.getResources())
In this case, we rely on the fact that if an exception is thrown, all resources in that phase will be marked as failed, using the text from the exception. The downside is that it doesn’t indicate which specific resources could not get their files created, so a more realistic example might parse the output from the command, or perhaps subdivide the resources in half on a failure and try each half independently (recursing to subdivide each failing half), or some other option. That complexity is the cost you often have to pay in order to get fast operations in the common case when things tend to succeed.
Note that in this example, we are not waiting on any asynchronous operation to succeed, so we mark the resources completed as soon as the call fails without error. If we did not call setCompleted for a resource, it would be considered waiting, and the plugin would automatically be called again for that resource after a delay.
Suppose the operation that needs to be done will take a while, like running a lengthy install script on each machine when it boots. In that case, it would be bad to run a blocking SSH call inside the plugin, for several reasons:
- The thread would be mostly idle, waiting on the remote machine. This occupies a thread in CycleCloud, and threads are limited so we don’t oversubscribe the heap or the CPU.
(In general, we do not know what a phase plugin will do, so we have to limit conservatively.) This can therefore starve other phases.
- If you start multiple machines, the call is executed on each in sequence, greatly increasing the time it takes.
- If CycleCloud is restarted, the SSH connection would die, and when the phase is re-run, the command would be re-run on the machine.
This will probably interfere with the first command, which may still be executing, and it certainly takes more total time.
A much better solution is to run the command asynchronously on the machine (eg, using nohup) in a script file that creates a status file when it is done that indicates success or failure. The plugin would then run the command for each resource, and just “check in” periodically until the command completes, and set the resource to be completed or failed accordingly:
def handle(handler): for node in handler.getResources(): phase_data = handler.getResourcePhaseData(node) if not phase_data.isDefined("Directory"): # run the command and get the temporary directory with its output dirname = _call_remote_command(node) phase_data.setString("Directory", dirname) else: dirname = phase_data.getAsString("Directory") # ssh to the machine and determine if done completed, error = _check_command(dirname, node) if completed: if error is not None: handler.setFailed(node, error) else: handler.setCompleted(node) else: # still not done, so CycleCloud will call again later for this resource pass
(Note that realistic error handling is left out of this example.)
In this case, we track the temp directory on the server that contains the command’s status in the phase-specific data for this resource. This lets us distinguish between the first time this phase is run and subsequent times, and also stores the name so we know what file to check on the remote machine.
The SSH calls to start the command or check on the file are still done serially, but the plugin does not itself block on the command completing. This counters the disadvantages listed above:
- The thread is not idle (except for a small amount of time in SSH negotiation), so other phase plugins can run in between.
- Each command runs (nearly) simultaneously on the remote machines.
- If CycleCloud is restarted, this plugin will resume where it left off, either running the command for those resources that had not run it yet, or checking for those that had.
(Note there is still a race condition if the CycleCloud process is killed while starting a command.)
Similarly, if this command fails, this phase will be marked as Failed, and can be retried. When it is retried, CycleCloud will clear the phase data for this resource. The plugin would not find Directory set on it, so it would start the command again and track it anew. Since the state is stored per resource, other resources that are ongoing would continue to check the status.
Not every phase should run for every resource. For instance, a node-based phase that attaches a public IP only needs to run on nodes that require a public IP. If the public IP address is stored on nodes in the FixedPublicIp attribute, then to just run the the phase for nodes that have it defined, you would include this in your plugin definition:
ResourceConstraint := FixedPublicIp isnt undefined
Note that this is declared with := because it is an expression.
Some phases cannot run until an earlier phase has completed. For instance, if you must allocate a storage volume as a boot disk before you create an instance using that disk, you must first allocate the volume, then wait for it to complete, then create the instance. This should be broken down into two phases:
- a phase that makes the request to allocate a volume, then waits for the volume to be ready
- a phase that makes the request to create an instance, and (optionally) waits for the instance to reach some state
As of CycleCloud 6.1, you can specify dependencies on phases to avoid having to manually handle the dependency. These dependencies are specified in the Dependencies attribute on the phase, as a list of other phases that must be completed before this phase can run. These phases must all be for the same state, since phases in a later state are implicitly dependent on phases in earlier states.
For example, using the example above, first you would create a phase plugin to create the volume, defining a phase called Example.CreateVolume:
HandlesStates = Allocation ResourceType = Cloud.Node PhaseName = Example.CreateVolume Phase.Description = Creating volumes
Then you would define the phase plugin that creates the instance, which cannot run until the volumes are created:
HandlesStates = Allocation ResourceType = Cloud.Node PhaseName = Example.CreateInstance Phase.Description = Creating instances Phase.Dependencies = Example.CreateVolume
Resources that cannot run since their dependencies are running are in the Blocked status for that phase. Once all the phases they depend on have completed, the phase moves to Waiting and is then executed normally.
Phase Plugin Attributes
Phase plugins are defined with the following attributes:
- The name of the state (or a list of states) during which this plugin is called. Identifies this as a phase plugin.
- The type or types of resources this plugin handles (typically Cloud.Node)
- The name of the phase for this plugin. Names must be unique.
- An optional description of this phase. Any attributes of the form “Phase.ATTRIBUTE” will be stored as ATTRIBUTE on the phase.
(Note that prior to 6.1, this had to be written as Phase_ATTRIBUTE.)
- The cloud provider (eg “aws”, “azure”, “gcp”) that this plugin is intended for, or none if this plugin is not provider-specific (only for Cloud.Node resources)
- A filter that limits the resources for which this plugin is called
When a resource of a given type reaches each state, CycleCloud calls any registered state listeners for that type and state. Listeners run asynchronously and cannot block the progression of the resource. They are intended for triggering behavior that should happen when the resource reaches a given state (for example, logging an event or connecting to a node). Note that the state of the listener is not tracked anywhere, nor is the the success or failure of past listeners.
A listener plugin must have a single function:
This method is called with a StateEvent object that has the following methods:
- Returns the type of resource the event is for
- Returns the state the resources were in. Note that in general, unless the state is the target state,
the resources will not be in that state any more by the time the listener is called.
- The cluster that the resources are in
- A list (specifically a Series object) containing all the resources that transitioned to the state in this event
Listener plugins support the following attributes:
- Must be set to true to register this plugin as a state listener
- The list of states being listened to (or a simple string if there is just one)
- The list of types being listened to (or a simple string if there is just one)
The status of the phase for each resource is tracked in the resource itself (on the PhaseMap attribute). The state of the phase as a whole (ie, what the server is doing), is tracked in a Cloud.Phase record.
These are the activities for an overall phase, stored in the Cloud.Phase record:
- Running: this phase is actively running with some resources
- Sleeping: there are no running resources in this phase, but there are resources sleeping until a point in the future
- Inactive: there are no resources associated with this phase
- Canceling: this phase is active, but it is being canceled
When a phase completes, there are three possibilities: it goes to Running again, because while it was running there were resources that entered the phase; it goes to Sleeping, because there are resources in the phase that are sleeping (either from this phase execution or an earlier one); or it goes to Inactive.