Timers

One essential aspect of CycleServer is the ability to run code periodically. This serves many features:

  • Collecting ads from remote systems
  • Sampling ads to create time-series data like usage or load
  • Data purging per retention policies
  • Machine monitoring to detect machines or daemons that are not responding
  • “Health” monitoring to detect jobs that have run too long, that have waited too long to run, etc.

Plugins can be run periodically by creating an Application.Timer ad.

Type Definition for Application.Timer

Attribute Value Type Status Description
Name String Required Uniquely identifies this timer
Plugin String Required The name of the plugin that runs each time this timer ticks.
Function String Optional The name of the plugin function to call. If not present,
the default is “run”.
Arguments List Optional If present, the arguments that will be passed to the
plugin. If not present, the function will be called with
the timer ad.
Interval RelTime Optional Number of seconds to wait between running the
plugin. Ignored if Schedule is present in 3.9.6.
Disabled Boolean Optional If true, this timer will not run.
Schedule String Optional The cron expression used to
control when the plugin runs. Schedule and Interval are
mutually exclusive. Warning! The cron expression
syntax used by Quartz is different from the standard
syntax. See this tutorial
for the actual syntax.
InitialDelay RelTime Optional How long to wait before firing for the first time (since
restart). Either StartTime or InitialDelay can be
specified, but not both. Note: this attribute
only affects the start of a timer span.
StartTime AbsTime Optional When to fire this timer for the first time. Either
StartTime or InitialDelay can be specified, but not
both. Note: this attribute only affects the start of a
timer span.
StrictTiming Boolean Optional If true, timers will be run with strict timing. Defaults
to false.
FireTime AbsTime Automatic The time this timer actually fired, if it is still
running, or undefined if not. See ScheduledFireTime.
ScheduledFireTime AbsTime Automatic The time this timer was scheduled to fire, if it is
still running, or undefined if not. See FireTime.
LastFireTime AbsTime Automatic The last time this timer ran, or undefined if it has
not run yet
LastScheduledFireTime AbsTime Automatic The last time this timer was scheduled to run, or
undefined if it has not run yet
NextFireTime AbsTime Automatic The upcoming time this timer will run, or undefined if
it is disabled
FireCount Integer Automatic The number of times this timer has fired
MisfireCount Integer Automatic The number of times this timer has misfired
Misfired Boolean Automatic True if the timer could not run the last time it was
scheduled, either due to the server not running or
insufficient resources. This is not cleared until the
next run completes.
LastRuntime RelTime Automatic The amount of time last spent running this timer, or
undefined if it has not yet run
LastStatus Boolean Automatic True if successful the last time it ran; false if
error; undefined if it has not run yet
LastStatusMessage String Automatic The message from the task the last time it ran;
undefined if it has not run yet
CurrentState String Automatic Idle if this timer has not fired yet; Waiting if it has
been scheduled to run; Running if it has run; Disabled
if disabled
EnteredCurrentState Timestamp Automatic The time this timer transitioned to the state in
CurrentState
TimerType String Required ONE_SHOT if the plugin only needs to run once after the
initial delay; FIXED_RATE if each execution is
scheduled relative to the scheduled execution time of
the initial execution; FIXED_DELAY if each execution is
scheduled relative to the completion time of the
previous execution.
Logging Integer Optional Whether to output extra logging (1) or not (0) for this
timer

Timing

There are two primary forms of timing: interval-based, in which execution is scheduled after a fixed number of seconds; or schedule-based, in which execution is scheduled on human time units, which can vary in length (for instance, some days have 23 or 25 hours, due to daylight savings time), and may not even be uniform at all (for instance, a timer that fires at noon and 5PM).

StartTime and InitialDelay are mutually exclusive, and both only affect the start of a timer span. A span starts when any of the following occur:

  • A timer ad is created
  • A disabled timer ad is re-enabled
  • The Interval or Schedule attributes are changed
  • CycleServer is started

When a new timer span begins, the “first-fire” time is calculated. This depends on StartTime or InitialDelay, as well as the type of timing:

Attribute Interval-based timing Schedule-based timing
StartTime The timer will first fire at StartTime, or
at an integral number of Intervals after
StartTime
The timer will fire on the first scheduled
time that is on or after StartTime
InitialDelay The timer will be delayed by InitialDelay,
and will then fire
The timer will be delayed by InitialDelay,
and then be eligible to fire

In general, if you want uniform intervals, even across reboots, use StartTime. If you want delayed initial execution, use InitialDelay (including an InitialDelay of 0 to force the execution to be immediately after boot).

Timers can optionally be flagged as “strict” (by setting StrictTiming = true) which changes their behavior in several ways. If a timer misses an entire execution, strict timers will wait until the next scheduled time to fire (non-strict timers will fire as soon as possible). Second, the priority by default is higher for strict timers. (This will also affect proposed features: manually firing a strict timer will have no effect.)

Automatic Timers

To simplify creating plugins that run periodically, a plugin whose config includes Timer = true will have a timer created for it automatically. All attributes starting with Timer_ will be used to define the timer. For instance, setting Timer_Interval = 60 on the plugin would make the timer’s Interval attribute be 60 (ie, 60 seconds). The timer will be updated when the plugin’s configuration is changed, and removed when the plugin is deleted.

API

application.timer plugin

runNow(timerName)
Forces the given timer to run now rather than waiting until its next cycle.

Note

As of 5.4, timers that are forced have Rescheduled=true set on the timer supplied to the plugin (but not saved).

Scheduled Types

In some instances, you need to create timers automatically, one for each item which needs to be run periodically. For instance, if you have a plugin that monitors multiple systems of some kind, it is simpler to make a timer for each system than to make one timer that loops through each system. (Not only will the monitoring run simultaneous, but you can poll different systems at different rates.) But the code to maintain the parallel set of timers can be tricky. To make this easier, we added support for scheduled types: types whose instances correspond directly to timers.

To use this, add the following attributes to your type:

Attribute Value Type Status Description
Scheduled Boolean Optional If true, instances of this type will get timers
associated with them.
SchedulerPlugin String Required The name of the plugin that creates the timers for each
instance.

The SchedulerPlugin must identify a plugin with the following method:

list createTimers(ad)

This function will be called once for each instance that is created or updated. (It is not called for instances that are deleted. Their timers are deleted automatically.) It returns a list of timer ads. The name and type are ignored, so they do not need to be set. This plugin should return all the timers for the given instance. (Some systems may require both high- and low-frequency monitoring, for instance.) If you return more than one timer, each timer must have a “Suffix” attribute that uniquely identifies it in the list you return. The suffix is added to the timer name. If a particular instance does not need any timers, return an empty list.

Note that all timers that were created from a previous call to createTimers are replaced with the timers you return, so you must make sure you return all the timers needed for the given instance, rather than just the new timers that need to be added.

Example

Suppose you have a plugin that monitors a remote SMTP mail server by querying it every minute. If there is a type called SMTP.Server, it might be configured this way:

AdType = "Type"
ForType = "SMTP.Server"
KeyAttributes = "Name"
Scheduled = true
SchedulerPlugin = "server.collector"

The server.collector plugin is as follows (imports removed). Note that it is both the timer and the scheduler plugin for convenience, but it does not need to be.

def createTimers(server):
    timers = []
    if not server.getAsDefaultBoolean("Disabled", False):
        timer = Ad("Application.Timer")
        timer.setString("ServerName", server.getAsString("Name"))
        timer.setString("Plugin", "server.collector")
        timer.setRelativeTime("Interval", server.getAsRelativeTime("Interval"))
        timers.append(timer)

   return timers

 def run(timer):
    server = adstore.getAd("SMTP.Server", timer.getAsString("ServerName"))
    # connect to the server and get data....

This example takes a server ad and copies the name into the timer (so that the full server ad can be looked up when the timer runs), as well as the interval. If the server ad is disabled, then no timers are returned, so the existing ones are deleted.

Tasks

Tasks are used to specify and track code that runs on a thread. Each task is associated with a plugin, which it calls when it runs.

Tasks can be tagged with an “exclusive resource” which will help control load on a given resource. If two tasks have the same value for the ExclusiveLock attribute, then only one of those tasks will execute at a time. (Other tasks with a different resource value, or none at all, may overlap these tasks.) This enables you to make sure that tasks do not interfere with each other.

Note: Completed tasks older than a certain age are periodically deleted from the ad store.

Type Definition: Application.Task

Attribute Value Type Status Description
Id Identifier Automatic A unique identifier for this task
Description String Optional Describes the task
Plugin String Required The name of the plugin that this task runs.
Function String Optional The name of the plugin function to call. If not
present, the default is “run”.
Arguments List Optional If present, the arguments that will be passed to the
plugin. If not present, the function will be called
with the task ad.
Timer String Automatic Name of the timer that generated this task. Undefined
for tasks that were not generated by a timer.
ExclusiveLock String Optional Name of a resource that should only have one associated
task running at any time.
StartTime AbsTime Automatic Time at which this task was added
EndTime AbsTime Automatic Time at which this task finished, either successfully
or not.
CurrentState String Automatic Current state of this task. See below for possible values
LastMessage String Automatic Most recent message about this task’s current
state. Typically set in case of error.
StateChangetime AbsTime Automatic Time at which this task’s state last changed.
OperationId Identifier Suggested If present, groups tasks that were created at the same
time for the same purpose
Priority Integer Optional If present, tasks with higher priority values run
first. If not, the default is 0.
Queue String Optional If present, tasks will run on the given named queue. If
not present, they run on a shared built-in queue.

Possible values for CurrentState:

  • Waiting if this task is queued to run
  • Running if this task is executing
  • Completed if this task completed
  • Failed if an error occurred while scheduling or executing this task

API

application.executor plugin

execute(task)
Executes the given task.
execute(tasks)
Executes the given collection of tasks.

The task argument must be an ad of type Application.Task. It specifies the plugin to call, the arguments to use, and other attributes for tagging what type of task it is. The state of the task as it runs is tracked in the same Task ad. Tasks can be grouped into an operation, which is a set of tasks that are performing a logical unit of work (eg, restarting all the daemons on a grid, each of which is a separate task). If the OperationId attribute is not specified when tasks are added, all tasks in one call to execute() are set to a new unique identifier. You can set the operation yourself in case the the grouping by execute() does not match the logical grouping.

cancelTask(taskId)
Cancels the task with the given identifier. If the task is running, it is interrupted. If not, it is removed from the queue so it will not run.
cancelTask(taskId, force)
Cancels the task forcibly, if force is true. (If force is false this is the same as the previous
cancelTask method). If the task is running and does not respond to interruption in a short
interval, the thread itself is stopped. Note this should be used as a last resort as CycleServer
cannot be guaranteed to be in a proper state after this operation.
isCanceled()
Returns true if this task should abort. This is checked automatically by most blocking calls, but
“normal” code should check this periodically if it could loop for an extended period of time.

Unit testing

These methods are available when running unit tests. In this mode, tasks are not executed automatically. This ensures that you can check the state before and after a task that is created by code being tested, without causing race conditions. The cancel methods above work, though they will never need to interrupt the code since tasks are executed synchronously.

runNext()
Runs the next task to completion. An exception is thrown if there is no task waiting.
runAll()
Runs any remaining tasks to completion.
getRemainingTaskCount()
Returns the number of tasks waiting to run. This can be used to ensure the code being tested created the correct number of tasks.

Queues

Tasks are run on a queue, which is an ordered list of tasks serviced by a thread pool. You can create new queues in addition to the built-in queue. Each queue has a unique name and is represented by Application.TaskQueue records. The built-in queue is named “Global”.

Type Definition Application.TaskQueue

Attribute Value Type Status Description
Name String Required Uniquely identifies this task
Description String Optional Describes the queue
Workers Integer Optional The number of threads to allocate to this
queue. Defaults to the number of available processors.
WorkersPerProcess Integer Optional If present, and Workers is not defined, this computes an
effective number of as WorkersPerProcess * processor_count
MinWorkers Integer Optional If present, limits the worker count to no less than this
MaxWorkers Integer Optional If present, limits the worker count to no more than this
EffectiveWorkers Integer ReadOnly Automatically set to the number of workers currently
in effect for this queue