A project is a collection of resources necessary to define and run a cluster within CycleCloud. A project consists of three parts:
- Cluster Template
- Custom Chef
Create a New Project
To create a new project, use the CLI command cyclecloud project init myproject, where myproject is the name of the project you wish to create. A directory tree will be created with skeleton files you will amend to include your own information.
The following directories will be created by the project command:
The templates directory will hold your cluster templates, while specs will contain the specifications defining your project. spec has two subdirectories: cluster-init and custom chef. cluster-init contains directories which have special meaning, such as the scripts directory (contains scripts that are executed in lexicographical order on the node), files (raw data files to will be put on the node), and tests (contains tests to be run when a cluster is started in testing mode).
The custom chef subdirectory has three directories: site-cookbooks (for cookbook definitions), data_bags (databag definitions), and roles (chef role definition files).
When creating a new project, a single default spec is defined. You can add additional specs to your project via the cyclecloud project add_spec command.
You can upload the contents of your project to any locker defined in your CycleCloud install via the command cyclecloud project upload <locker>, where <locker> is the name of a cloud storage locker in your CycleCloud install. This locker will be set as the default target. Alternatively, you can see what lockers are available to you with the command cyclecloud locker list. Details about a specific locker can be viewed with cyclecloud locker show <locker>.
If you add more than one locker, you can set your default with cyclecloud project default_target <locker>, then simply run cyclecloud project upload. You can also set a global default locker that can be shared by projects with the command cyclecloud project default locker <locker> -global.
Note: Default lockers will be stored in the cyclecloud config file (usually located in ~/.cycle/config.ini), not in the project.ini. This is done to allow project.ini to be version controlled.
Uploading your project contents will zip the chef directories and sync both chef and cluster init to your target locker. These will be stored at:
By default, all projects have a version of 1.0.0. You can set a custom version as you develop and deploy projects by setting version=x.y.z in the project.ini file.
For example, if “locker_url” was s3://com.cyclecomputing.O66.projects, project was named “Order66”, version was “1.6.9”, and the spec is “default”, your url would be:
- s3://com.cyclecomputing.O66.projects/projects/Order66/1.6.9/default/cluster-init - s3://com.cyclecomputing.O66.projects/projects/Order66/1.6.9/default/chef
Specify Project within a Cluster Template
Project syntax allows you to specify multiple specs on your nodes. To define a project, use the following:
[[[cluster-init myspec]]] Project = myproject # inferred from name Version = x.y.z Spec = myspec # (optional, inferred from section definition) Locker = default # (optional, will use default locker for node)
Note: The name specified after ‘spec’ can be anything, but can and should be used as a shortcut to define some common properties.
You can also apply multiple specs to a given node as follows:
[[node master]] [[[cluster-init myspec]]] Project = myproject Version = x.y.z Spec = myspec # (optional, inferred from section definition) Locker = default # (optional, will use default locker for node) [[cluster-init otherspec]] Project = otherproject Version = a.b.c Spec = otherspec # (optional, inferred from section definition)
By separating the project name, spec name, and version with colons, CycleCloud can parse those values into the appropriate Project/Version/Spec settings automatically:
[[node master]] [[[cluster-init myproject:myspec:x.y.z]]] [[[cluster-init otherproject:otherspec:a.b.c]]]
Specs can also be inherited between nodes. For example, you can share a common spec between all nodes, then run a custom spec on the master node:
[[node defaults]] [[[cluster-init my-project:common:1.0.0]]] Order = 2 # optional [[node master]] [[[cluster-init my-project:master:1.0.0]]] Order = 1 # optional [[nodearray execute]] [[[cluster-init my-project:execute:1.0.0]]] Order = 1 # optional
This would apply both the common and master specs to the master node, while only applying the common and execute specs to the execute nodearray.
By default, the specs will be run in the order they are shown in the template, running inherited specs first. Order is an optional integer set to a default of 1000, and can be used to define the order of the specs.
If only one name is specified in the [[[cluster-init]]] definition, it will be assumed to be the spec name. For example:
[[[cluster-init myspec]]] Project = myproject Version = 1.0.0
is a valid spec setup in which Spec=myspec is implied by the name.
Warning: If you are using Projects, you cannot use pre-v6.5.4 ClusterInit (pre 6.5.4). They are mutually exclusive.
The zipped chef files will be downloaded during the bootstrapping phase of node startup. They are downloaded to $JETPACK_HOME/system/chef/tarballs and unzipped to $JETPACK_HOME/system/chef/chef-repo/, and used when converging the node.
Note: To run custom cookbooks, you MUST specify them in the run_list for the node.
The cluster-init files will be downloaded to /mnt/cluster-init/<project>/<spec>/. For ‘my-project’ and ‘my-spec’, you will see your scripts, files, and tests located in /mnt/cluster-init/my-project/my-spec.
Log files generated when running cluster-init are located in $JETPACK_HOME/logs/cluster-init/<project>/<spec>.
When a cluster-init script is run successfully, a file is placed in /mnt/cluster-init/.run/<project>/<spec> to ensure it isn’t run again on a subsequent converge. If you want to run the script again, delete the appropriate file in this directory.
When CycleCloud executes scripts in the scripts directory, it will add environment variables to the path and name of the spec and project directories:
CYCLECLOUD_PROJECT_NAME CYCLECLOUD_PROJECT_PATH CYCLECLOUD_SPEC_NAME CYCLECLOUD_SPEC_PATH
On linux, a project named “test-project” with a spec of “default” would have paths as follows:
CYCLECLOUD_PROJECT_NAME = test-project CYCLECLOUD_PROJECT_PATH = /mnt/cluster-init/test-project CYCLECLOUD_SPEC_NAME = default CYCLECLOUD_SPEC_PATH = /mnt/cluster-init/test-project/default
Run Scripts Only
To run ONLY the cluster-init scripts:
jetpack converge --cluster-init
Output from the command will both go to STDOUT as well as jetpack.log. Each script will also have its output logged to:
Custom chef and Composable Specs
Each spec has a chef directory in it. Before a converge, each spec will be untarred and extracted into the local chef-repo, replacing any existing cookbooks, roles, and data bags with the same name(s). This is done in the order the specs are defined, so in the case of a naming collision, the last defined spec will always win.