Design Goals

The plugin architecture was designed to make it possible to:

  • Write and test simple plugins easily
  • Write and test cohesive sets of plugins (components) to support complex features
  • Write plugins in multiple languages, using a common interface
  • Add, update and remove plugins on a live system without causing any service interruption

A plugin is uniquely identified by its name. There are two major types of plugins: server-side plugins and client-side plugins.

Server-side plugins are written in Java or Python. A Java plugin is a single instance of a class, and the methods on that class are the plugin’s functions. A Python plugin is a set of functions defined in a .py file. Server-side plugins provide customization such as HTTP handlers, timers, background tasks, or triggers and event handlers.

Note

IMPORTANT CycleServer supports the Jython implementation of Python, which is distinct from the default C-based implementation of Python. Jython offers many performance advantages over CPython and the important benefit that it can seamlessly access services provided by the CycleServer platform. As of 6.1.0, Jython currently supports Python 2.7.

Since Jython runs on the JVM, it cannot access lower-level libraries written in C, as many public Python projects do. When porting python code from CPython to Jython, carefully check for any dependencies on C modules as you will need to replace them with Java constructs.

A plugin may be called directly by the system or by other plugins, or as a result to a RESTful call to the URI associated with that plugin. Because plugins are executed in a multithreaded environment, these functions must be thread-safe. The simplest way to make plugins thread-safe is to make them stateless.

Client-side plugin code can be static HTML or Javascript files that are processed by the browser. They do not contain any functions available to the server.

Packaging

Plugins may exist either directly on disk or in a .zip file for a component, structured in the following way. As an example, this is a component that contains one plugin, foo.bar:

src/main/component/component.cfg
                   plugins/foo/bar.py
                   plugins/foo/bar.cfg

The component.cfg file is in the standard plugin config format. It must exist and it must contain a Name attribute which names the component (the name of the zip or jar file is not used in determining the name). It is always named component.cfg. The plugins that this component contains are in the plugins directory, which is treated the same as CycleServer’s top-level plugins directory.

Here is a very basic component.cfg:

Name: ChefCleanup
Label: Chef Server clean up
Description: Remove references to terminated nodes from a Chef Server
Version: 1.0
  • Name – name of the component, this is a unique identifier that should not be changed
  • Label – human-readable name
  • Description – summary or explanation of what the component does
  • Version – Version of component

To deploy a component, copy the zip file into the top-level components directory in CycleServer. This will add the component to CycleServer. CycleServer will delete the zip file from the components directory once it has fully processed the component. You can also create an expanded component by creating a ComponentSource record with the Directory attribute pointing to the location of your component.cfg on disk. In this case, CycleServer will monitor the expanded directory for changes and reload the plugins when they change.

The code for each plugin is packaged in its own file under the plugins/ directory. For example, the code for Python plugin a.b.c.p is stored in the file plugins/a/b/c/p.py. The properties of each plugin are stored in a file of the same name as the source file, but with the .cfg extension. For example, the properties of plugin a.b.c.p are stored in the configuration file plugins/a/b/c/p.cfg (click for additional information on plugin attributes). The existence of either the source file or the configuration file (the “primary files”) indicates a plugin.

As a convenience for writing very simple plugins, a CycleServer installation includes a top-level plugins directory to which admins can directly add plugin files. This functions as an implicit “site local” component.

When CycleServer initially starts or a new plugin is added to the plugins directory, that plugin is automatically instantiated. When the primary files for a given plugin are updated, that plugin is reloaded. When the primary source files are deleted, the plugin is unloaded.

Configuration

A plugin gets its default attributes from its associated .cfg file. However, these can be overridden in two ways for a given CycleServer installation, e.g. for a plugin named acme.foo:

  • Create a file named $CS_HOME/config/plugin_configs/acme/foo.cfg which is parsed like a normal plugin config file

or

  • Update the record of type PluginConfig which has a Name attribute of “acme.foo”

In either case, the attributes included will override any specified on the plugin itself. Changing the configuration via either of these methods will reload the plugin immediately.

A plugin can find the current effective value on itself with the application.manifest plugin:

from application import manifest

def someFunction():
    print manifest.ad.getAsString("MySetting")

You can find the effective value on any plugin, not just your own, by querying the Plugin record:

from application import datastore

def someFunction():
    print datastore.getRequiredAd("Plugin", "foo.bar").getAsString("MySetting")

Dependencies

Components define source code that is available to other plugins within that component, and optionally to plugins in other components. When the source code for one plugin in a component changes, the entire component must be reloaded. Additionally, a plugin may reference plugins in other components, meaning that it may need to be reloaded when those plugins change.

Components may declare their dependencies on other components. This is the only way to get source code (eg, classes) from libraries directly available to plugins in the component. All plugins in a component implicitly depend on their component and the component(s) that their component depends on.

Plugins can declare their dependencies on other plugins (either in their component, or in another component) in their configuration. In addition, while loading Python or Java plugins, the system will detect imports and use that to amend the dependencies list. Since the list of plugins in a component is not known until the component is loaded, it is an error if a plugin references a plugin in a component not yet loaded. (The solution is to update the component’s dependencies to include the component that contains the plugin, or use a dynamic dependency as described below.)

When loading a component, the dependency tree will be built from the dependencies list on each plugin. Each plugin will then be loaded in order according to its dependency tree. All plugins in the same component are loaded at the same time, with the LoadOrder used to break ties for that component (ie, it does not impose a global ordering).

When a plugin is changed or unloaded, all dependent plugins are reloaded or unloaded.

There are three types of dependencies that a plugin P can have on another plugin D in component C:

Static
P is compiled with classes provided by C. Changing the source code in C means P must be
recompiled. All plugins are statically bound to their component and to their component’s
dependencies. Component-component dependencies are static.
Referential
P holds a reference to D, so changes to D mean P must be cloned, and the reference to D updated
to the new version of D. Importing plugins is a referential dependency (but using classes in C
implies a static dependency). Explicitly listed plugin-plugin dependencies are referential.
Dynamic
P looks up D every time it runs. Changes to D do not need to modify or reload P, so this is not
tracked as a dependency by the system.

Plugins which are designed to call extension points that other plugins may implement should use a dynamic dependency so that new plugins do not trigger a reload. This can be done in Python by embedding the import call inside your function, so it is not at global scope. This can be done in Java by looking up the plugin from the registry every time the function is called.

Both components and plugins can declare their dependencies with the Dependencies attribute. For example, a plugin with this declaration depends on both the foo.graphics and bar.math plugins:

Dependencies:= { "foo.graphics", "bar.math" }

Components can only depend on components, and plugins can only depend on plugins (although they implicitly depend on the plugins in their component’s dependencies).

Current Limitations

When a component is reloaded or unloaded, the components that depend on it are not reloaded or unloaded automatically. They must be reloaded or removed manually. The system logs a warning with the components that are affected in this case.

Importing Static Configuration

Plugins can include initial data that is needed for their operation, or sample data for a new install. That data can be defined as a resource on the plugin. Any resource named ads/*.xml or ads/*.txt is automatically parsed as classads (in XML or text format, respectively) and stored when the plugin is loaded. Note: we also support the older PLUGIN-INF/ads/*.xml for backwards compatibility.

Records that are imported are only stored if they are changed. In addition, if the user edits the record after the plugin is stored, those changes are kept if the plugin is later upgraded with a changed record. Only the attributes that the user changed are kept; unmodified attributes are updated to match the version in the plugin. Note that this is not always the correct merging behavior. For this reason, the following message is logged when a merge is done:

Possible conflict while importing new record "Type","NewType" in Foo.Bar;
user has modified attributes [ A, B, C ]

Imported records are tagged with the plugin and component they come from. When the plugin is removed, its corresponding imported records are removed.

Note

Exception: currently types are preserved since removing them would require cascading to remove all records of that type. Attributes are preserved for a similar reason.

If a plugin is updated to not contain a record anymore, that record is removed. (Currently it is removed even if modified by the user, though that behavior may change in the future.)

API

There is an API for storing data as well in application.importer. It has a single method, importAds, that imports the list of records given to it.

For example, this would create a new type called NewType on load:

from application import importer, datastore

def onLoad():
   type = datastore.newAd("Type", "NewType")
   type.setString("KeyAttributes", "Name")
   importer.importAds([ type ])

Calling this more than once in a single onLoad is not recommended since it will remove the ones imported earlier. However, resource-based imported records will be unaffected.

Life Cycle

The system keeps track of all known plugins as records (see Plugins Configuration for the description of all default attributes).

Components are defined by a ComponentSource record. This record can either point to an directory that contains an expanded component or can point to an archive file that has been copied internally to CycleServer.

Install

Components are installed by creating a ComponentSource record. As a shortcut, a component can also be installed by copying its archive file into the components directory. (Archives put in components are automatically loaded as components, and the file is deleted.)

With the exception of files in the plugins directory, all plugins are contained inside components. The plugins are installed with the component. If the component is expanded on disk in development mode, plugins can be installed or uninstalled without installing/uninstalling the entire component.

A component is installed with the following steps:

  1. Determine what dependencies the component has and make sure they are loaded.
  2. Load the plugins for the component as described below.

A plugin is installed with the following steps:

  1. Determine what dependencies the plugin has and make sure they are loaded.
  2. Load and compile the source.
  3. Store record defined in the plugin, tagging them with the plugin they came from.
  4. If onInstall is defined on the plugin, call it.
  5. If onLoad is defined on the plugin, call it.
  6. Make the plugin available to the system.

During plugin development, it is possible to configure CycleServer so that it hot-reloads your plugin every time you save a file in your project. To do this, simply create a ComponentSource record that points to the directory which contains the component.cfg for your component.

The easiest way to create the ComponentSource record is through CycleServer’s data browser. Go to the url http://localhost:8080/types. Ensure that the “Show: ___ Types” label in the upper left-hand corner displays the text “Show: All Types” scroll down the left-hand column and select ComponentSource.

https://docs.cyclecomputing.com/wp-content/uploads/2018/04/find_component_source.png

Next, click the Create button, type the name of your component, and the path to the component.cfg. Finally, click Save

https://docs.cyclecomputing.com/wp-content/uploads/2018/04/create_component_source.png

Upgrade

A plugin can be upgraded to a later version by modifying the source file or configuration file. The upgrade process is similar to the uninstall/install pair, except that onUninstall and onInstall are not called. Similarly, onUnload and onLoad are not called (although there does need to be a means of transferring active state from the old instance to the new, if the plugin needs it). Finally, records installed with the plugin need to be updated to match the new version.