One essential aspect of CycleServer is the ability to run code periodically. This serves many features:
- Collecting ads from remote systems
- Sampling ads to create time-series data like usage or load
- Data purging per retention policies
- Machine monitoring to detect machines or daemons that are not responding
- “Health” monitoring to detect jobs that have run too long, that have waited too long to run, etc.
Plugins can be run periodically by creating an Application.Timer ad.
Type Definition for Application.Timer
|Name||String||Required||Uniquely identifies this timer|
|Plugin||String||Required||The name of the plugin that runs each time this timer ticks.|
|Function||String||Optional||The name of the plugin function to call. If not present,
the default is “run”.
|Arguments||List||Optional||If present, the arguments that will be passed to the
plugin. If not present, the function will be called with
the timer ad.
|Interval||RelTime||Optional||Number of seconds to wait between running the
plugin. Ignored if Schedule is present in 3.9.6.
|Disabled||Boolean||Optional||If true, this timer will not run.|
|Schedule||String||Optional||The cron expression used to
control when the plugin runs. Schedule and Interval are
mutually exclusive. Warning! The cron expression
syntax used by Quartz is different from the standard
syntax. See this tutorial
for the actual syntax.
|InitialDelay||RelTime||Optional||How long to wait before firing for the first time (since
restart). Either StartTime or InitialDelay can be
specified, but not both. Note: this attribute
only affects the start of a timer span.
|StartTime||AbsTime||Optional||When to fire this timer for the first time. Either
StartTime or InitialDelay can be specified, but not
both. Note: this attribute only affects the start of a
|StrictTiming||Boolean||Optional||If true, timers will be run with strict timing. Defaults
|FireTime||AbsTime||Automatic||The time this timer actually fired, if it is still
running, or undefined if not. See ScheduledFireTime.
|ScheduledFireTime||AbsTime||Automatic||The time this timer was scheduled to fire, if it is
still running, or undefined if not. See FireTime.
|LastFireTime||AbsTime||Automatic||The last time this timer ran, or undefined if it has
not run yet
|LastScheduledFireTime||AbsTime||Automatic||The last time this timer was scheduled to run, or
undefined if it has not run yet
|NextFireTime||AbsTime||Automatic||The upcoming time this timer will run, or undefined if
it is disabled
|FireCount||Integer||Automatic||The number of times this timer has fired|
|MisfireCount||Integer||Automatic||The number of times this timer has misfired|
|Misfired||Boolean||Automatic||True if the timer could not run the last time it was
scheduled, either due to the server not running or
insufficient resources. This is not cleared until the
next run completes.
|LastRuntime||RelTime||Automatic||The amount of time last spent running this timer, or
undefined if it has not yet run
|LastStatus||Boolean||Automatic||True if successful the last time it ran; false if
error; undefined if it has not run yet
|LastStatusMessage||String||Automatic||The message from the task the last time it ran;
undefined if it has not run yet
|CurrentState||String||Automatic||Idle if this timer has not fired yet; Waiting if it has
been scheduled to run; Running if it has run; Disabled
|EnteredCurrentState||Timestamp||Automatic||The time this timer transitioned to the state in
|TimerType||String||Required||ONE_SHOT if the plugin only needs to run once after the
initial delay; FIXED_RATE if each execution is
scheduled relative to the scheduled execution time of
the initial execution; FIXED_DELAY if each execution is
scheduled relative to the completion time of the
|Logging||Integer||Optional||Whether to output extra logging (1) or not (0) for this
There are two primary forms of timing: interval-based, in which execution is scheduled after a fixed number of seconds; or schedule-based, in which execution is scheduled on human time units, which can vary in length (for instance, some days have 23 or 25 hours, due to daylight savings time), and may not even be uniform at all (for instance, a timer that fires at noon and 5PM).
StartTime and InitialDelay are mutually exclusive, and both only affect the start of a timer span. A span starts when any of the following occur:
- A timer ad is created
- A disabled timer ad is re-enabled
- The Interval or Schedule attributes are changed
- CycleServer is started
When a new timer span begins, the “first-fire” time is calculated. This depends on StartTime or InitialDelay, as well as the type of timing:
|Attribute||Interval-based timing||Schedule-based timing|
|StartTime||The timer will first fire at StartTime, or
at an integral number of Intervals after
|The timer will fire on the first scheduled
time that is on or after StartTime
|InitialDelay||The timer will be delayed by InitialDelay,
and will then fire
|The timer will be delayed by InitialDelay,
and then be eligible to fire
In general, if you want uniform intervals, even across reboots, use StartTime. If you want delayed initial execution, use InitialDelay (including an InitialDelay of 0 to force the execution to be immediately after boot).
Timers can optionally be flagged as “strict” (by setting StrictTiming = true) which changes their behavior in several ways. If a timer misses an entire execution, strict timers will wait until the next scheduled time to fire (non-strict timers will fire as soon as possible). Second, the priority by default is higher for strict timers. (This will also affect proposed features: manually firing a strict timer will have no effect.)
To simplify creating plugins that run periodically, a plugin whose config includes Timer = true will have a timer created for it automatically. All attributes starting with Timer_ will be used to define the timer. For instance, setting Timer_Interval = 60 on the plugin would make the timer’s Interval attribute be 60 (ie, 60 seconds). The timer will be updated when the plugin’s configuration is changed, and removed when the plugin is deleted.
- Forces the given timer to run now rather than waiting until its next cycle.
As of 5.4, timers that are forced have Rescheduled=true set on the timer supplied to the plugin (but not saved).
In some instances, you need to create timers automatically, one for each item which needs to be run periodically. For instance, if you have a plugin that monitors multiple systems of some kind, it is simpler to make a timer for each system than to make one timer that loops through each system. (Not only will the monitoring run simultaneous, but you can poll different systems at different rates.) But the code to maintain the parallel set of timers can be tricky. To make this easier, we added support for scheduled types: types whose instances correspond directly to timers.
To use this, add the following attributes to your type:
|Scheduled||Boolean||Optional||If true, instances of this type will get timers
associated with them.
|SchedulerPlugin||String||Required||The name of the plugin that creates the timers for each
The SchedulerPlugin must identify a plugin with the following method:
This function will be called once for each instance that is created or updated. (It is not called for instances that are deleted. Their timers are deleted automatically.) It returns a list of timer ads. The name and type are ignored, so they do not need to be set. This plugin should return all the timers for the given instance. (Some systems may require both high- and low-frequency monitoring, for instance.) If you return more than one timer, each timer must have a “Suffix” attribute that uniquely identifies it in the list you return. The suffix is added to the timer name. If a particular instance does not need any timers, return an empty list.
Note that all timers that were created from a previous call to createTimers are replaced with the timers you return, so you must make sure you return all the timers needed for the given instance, rather than just the new timers that need to be added.
Suppose you have a plugin that monitors a remote SMTP mail server by querying it every minute. If there is a type called SMTP.Server, it might be configured this way:
AdType = "Type" ForType = "SMTP.Server" KeyAttributes = "Name" Scheduled = true SchedulerPlugin = "server.collector"
The server.collector plugin is as follows (imports removed). Note that it is both the timer and the scheduler plugin for convenience, but it does not need to be.
def createTimers(server): timers =  if not server.getAsDefaultBoolean("Disabled", False): timer = Ad("Application.Timer") timer.setString("ServerName", server.getAsString("Name")) timer.setString("Plugin", "server.collector") timer.setRelativeTime("Interval", server.getAsRelativeTime("Interval")) timers.append(timer) return timers def run(timer): server = adstore.getAd("SMTP.Server", timer.getAsString("ServerName")) # connect to the server and get data....
This example takes a server ad and copies the name into the timer (so that the full server ad can be looked up when the timer runs), as well as the interval. If the server ad is disabled, then no timers are returned, so the existing ones are deleted.
Tasks are used to specify and track code that runs on a thread. Each task is associated with a plugin, which it calls when it runs.
Tasks can be tagged with an “exclusive resource” which will help control load on a given resource. If two tasks have the same value for the ExclusiveLock attribute, then only one of those tasks will execute at a time. (Other tasks with a different resource value, or none at all, may overlap these tasks.) This enables you to make sure that tasks do not interfere with each other.
Note: Completed tasks older than a certain age are periodically deleted from the ad store.
Type Definition: Application.Task
|Id||Identifier||Automatic||A unique identifier for this task|
|Description||String||Optional||Describes the task|
|Plugin||String||Required||The name of the plugin that this task runs.|
|Function||String||Optional||The name of the plugin function to call. If not
present, the default is “run”.
|Arguments||List||Optional||If present, the arguments that will be passed to the
plugin. If not present, the function will be called
with the task ad.
|Timer||String||Automatic||Name of the timer that generated this task. Undefined
for tasks that were not generated by a timer.
|ExclusiveLock||String||Optional||Name of a resource that should only have one associated
task running at any time.
|StartTime||AbsTime||Automatic||Time at which this task was added|
|EndTime||AbsTime||Automatic||Time at which this task finished, either successfully
|CurrentState||String||Automatic||Current state of this task. See below for possible values|
|LastMessage||String||Automatic||Most recent message about this task’s current
state. Typically set in case of error.
|StateChangetime||AbsTime||Automatic||Time at which this task’s state last changed.|
|OperationId||Identifier||Suggested||If present, groups tasks that were created at the same
time for the same purpose
|Priority||Integer||Optional||If present, tasks with higher priority values run
first. If not, the default is 0.
|Queue||String||Optional||If present, tasks will run on the given named queue. If
not present, they run on a shared built-in queue.
Possible values for CurrentState:
- Waiting if this task is queued to run
- Running if this task is executing
- Completed if this task completed
- Failed if an error occurred while scheduling or executing this task
- Executes the given task.
- Executes the given collection of tasks.
The task argument must be an ad of type Application.Task. It specifies the plugin to call, the arguments to use, and other attributes for tagging what type of task it is. The state of the task as it runs is tracked in the same Task ad. Tasks can be grouped into an operation, which is a set of tasks that are performing a logical unit of work (eg, restarting all the daemons on a grid, each of which is a separate task). If the OperationId attribute is not specified when tasks are added, all tasks in one call to execute() are set to a new unique identifier. You can set the operation yourself in case the the grouping by execute() does not match the logical grouping.
- Cancels the task with the given identifier. If the task is running, it is interrupted. If not, it is removed from the queue so it will not run.
- cancelTask(taskId, force)
- Cancels the task forcibly, if force is true. (If force is false this is the same as the previous
cancelTask method). If the task is running and does not respond to interruption in a short
interval, the thread itself is stopped. Note this should be used as a last resort as CycleServer
cannot be guaranteed to be in a proper state after this operation.
- Returns true if this task should abort. This is checked automatically by most blocking calls, but
“normal” code should check this periodically if it could loop for an extended period of time.
These methods are available when running unit tests. In this mode, tasks are not executed automatically. This ensures that you can check the state before and after a task that is created by code being tested, without causing race conditions. The cancel methods above work, though they will never need to interrupt the code since tasks are executed synchronously.
- Runs the next task to completion. An exception is thrown if there is no task waiting.
- Runs any remaining tasks to completion.
- Returns the number of tasks waiting to run. This can be used to ensure the code being tested created the correct number of tasks.
Tasks are run on a queue, which is an ordered list of tasks serviced by a thread pool. You can create new queues in addition to the built-in queue. Each queue has a unique name and is represented by Application.TaskQueue records. The built-in queue is named “Global”.
Type Definition Application.TaskQueue
|Name||String||Required||Uniquely identifies this task|
|Description||String||Optional||Describes the queue|
|Workers||Integer||Optional||The number of threads to allocate to this
queue. Defaults to the number of available processors.
|WorkersPerProcess||Integer||Optional||If present, and Workers is not defined, this computes an
effective number of as WorkersPerProcess * processor_count
|MinWorkers||Integer||Optional||If present, limits the worker count to no less than this|
|MaxWorkers||Integer||Optional||If present, limits the worker count to no more than this|
|EffectiveWorkers||Integer||ReadOnly||Automatically set to the number of workers currently
in effect for this queue