RULE Command

This command is one of the most recent extensions I have developed for the system. It extends the system’s functionality with an alerting mechanism.

NOTE: The functionality described here is covered by the tests: issue42_rule, described in the appendix Integration Tests.

The syntax of the RULE command is as follows:

RULE rule_name
ON data_stream_name
WHEN logical_condition
DO DUMP steps_back TO steps_forward [RETENTION segments]

Or like this:

RULE rule_name
ON data_stream_name
WHEN logical_condition
DO SYSTEM system_command

Fig. 8. RULE command syntax diagram

The railroad diagram in Fig. 8 is generated from the ANTLR4 rule_statement rule (RQL.g4). After DO it has three branches: DUMP writes a record range, SYSTEM executes a shell command, and SET ?NAME = expression requests a plan variable update. Rounded boxes denote literal symbols, rectangles denote user-supplied values, and bypass tracks denote optional elements.

Events defined this way attach to defined data streams. A rule name must be unique within the stream selected by ON; different streams may have rules with the same name. The data stream must be defined before the rule-creation command appears in the rql file.

In every form of the RULE command, a rule name, a logical condition, and the name of the stream to which the process launched by the DO command is attached are created. The condition can use fields of the ON stream, LET constants, and current VAR values.

In the first version of the command, which includes the DO DUMP clause, we define a process that allows collecting data that will arrive in the future. If we omit the RETENTION clause, the dump goes directly to a file named after the rule, prefixed with the stream name. If we add the RETENTION clause, the files will be subject to retention within the range defined by the ‘segments’ parameter. Sequential numbers will be appended to the end of each dump. Dumps are binary and preserve the schema of all fields of the source data stream. It is worth noting here that the command creates a process in the system which, once the logical condition becomes true, pulls data from the past and also expects its arrival and registration in the future. Nothing prevents us, however, from collecting data only from the past or only from the future. If the step_* values are negative, they refer to the past (i.e. to data that is historical relative to the moment the event described by the logical condition occurred).

The DO SYSTEM clause allows a system event to be triggered once the logical condition based on recorded data is met. This way, an arbitrary system command can be invoked.

Examples of rule declarations in RQL:

RULE testrule1 ON str1 WHEN str1[0] > 11 DO DUMP -5 TO 5 RETENTION 100

RULE testrule2 \
ON str1 \
WHEN str1[0] = 13 OR str1[0] = 11 \
DO SYSTEM 'echo "systemcall"'

Assume that a stream str1 has previously been defined, whose data - integer values - arrives once per second. In this case, the first rule, attached to this stream, waits for data whose value exceeds 11. Should such an event occur, a dump of the data is made, covering the range from 5 seconds before to 5 seconds after the event described by the logical condition.

The second rule, with a slightly different logical condition, prints the text “systemcall” to the screen from which the RetractorDB process was started.

RULE Command Syntax

The full syntax of the RULE command is:

RULE <name>
ON <stream>
WHEN <condition>
DO <action>

Where <action> is exactly one of three forms:

SYSTEM '<system_command>'
DUMP [-]<step_back> TO [-]<step_forward> [RETENTION <n>]
SET ?NAME = <scalar_expression>

Restriction

A rule can only be attached to a stream declared with a SELECT command (an artifact or substrate). Attaching it to a DECLARE input stream is a compilation error:

# INVALID - core0 is a declaration, a rule cannot be attached to it
RULE r1 ON core0 WHEN core0[0] > 10 DO SYSTEM 'echo alarm'

The WHEN condition

The condition is a logical expression evaluated to true/false after every new sample of the stream.

Comparison operators: =, !=, <, >, <=, >=. Logical operators: OR, AND, NOT. Examples:

WHEN str1[0] > 100
WHEN str1[0] = 0 OR str1[0] = 255
WHEN str1[0] >= 10 AND str1[0] <= 90
WHEN NOT str1[0] = 0

The DO SYSTEM action

The DO SYSTEM action executes the given shell command (via a system(3) call) the moment the condition is satisfied. RetractorDB logs the command’s exit code - a non-zero code is reported as an error in the log.

RULE alert1 \
ON results \
WHEN results[0] > 1000 \
DO SYSTEM 'curl -s http://monitoring/alert'

Any program available on PATH can be used in the command: shell scripts, Python programs, REST calls, sending notifications, etc.

A DO SYSTEM rule may be asked for only in the plan file the instance starts from - the author of that file is whoever runs the service. Both IPC channels refuse it: xqry --adhoc accepts DO DUMP and DO SET rules, and xqry --reset rejects the whole plan carrying DO SYSTEM, because that channel carries no authorship (→ xqry). An operator who deliberately hands the reset channel over sets service.unrestricted = true in the TOML configuration (→ xretractor).

The DO SET action

DO SET ?NAME = expression writes a variable declared by VAR. Its right-hand side is a scalar expression over the current ON record, LET constants, and plan variables; its type must match the variable’s type. A LET constant cannot be a target. NULL replaces the previous value.

VAR COUNT = 0
DECLARE x INTEGER STREAM src, 1 TEXTFILE 'signal.txt'
SELECT src[0], ?COUNT STREAM counted FROM src
RULE tick ON counted WHEN 1 = 1 DO SET ?COUNT = ?COUNT + 1

A counted record shows the counter’s pre-slot state. Updates are buffered and visible from the next slot. Among multiple writes to one variable, the last firing rule in text order wins. Each rule executes one action; combining SET and DUMP requires two rules with the same condition.

A SET rule on an entire generator family is refused. See VAR and DO SET for contexts, types, ad hoc and reset behavior, hysteresis, and uc10 examples.

The DO DUMP action

The DO DUMP action writes a window of stream samples to a binary file the moment the condition is satisfied. It lets you preserve the context of an event: data before it occurred and data after it.

RULE event ON results WHEN results[0] > 500 DO DUMP -10 TO 5

Range parameters:

Total number of dumped records: abs(step_forward - step_back). Example: DUMP -5 TO 5 → 10 records (5 historical + 5 subsequent). DUMP 0 TO 1 → 1 record (the current sample).

The step_back bound must be strictly less than step_forward. Equal or reversed bounds are rejected by the parser, including when attaching a rule ad hoc. The step_back value can be negative (history) or non-negative (delay). Both bounds may be negative: DUMP -5 TO -2 writes three records at positions t-5, t-4 and t-3 relative to the triggering record t. The right bound is excluded; this range contains no current or future records.

Dump files

Files are created in the directory configured by the STORAGE directive. Naming convention:

<stream>_<rule_name>_dump.tmp          # without RETENTION
<stream>_<rule_name>_dump_<n>.tmp      # with RETENTION (n = 0..N-1)

The file format is raw binary data matching the stream descriptor (no header). The xtrdb tool can be used to read the file.

A dump carries values only. Neither a .desc nor a .meta file accompanies it, so the schema has to be supplied from outside, and the NULL map and the transmission gaps have no representation in it whatsoever: a NULL field is written as the substitute value of its type, and a record the engine did not have is written as zeros. Absence and gaps are notions of the engine’s interior and do not leave it - the full contract, together with the routes that do preserve fidelity, is described in Alerting implementation.

The RETENTION option

The RETENTION <n> parameter limits the number of stored dumps - the oldest file is overwritten by the new one (a circular buffer). Without RETENTION, every trigger overwrites a single _dump.tmp file.

RULE event ON results WHEN results[0] > 500 DO DUMP -10 TO 5 RETENTION 20

The example above keeps the last 20 dumps in files results_event_dump_0.tmp … results_event_dump_19.tmp.

Multiple rules for a single stream

Any number of rules of different types can be attached to a single stream:

RULE high_alert \
ON measurements \
WHEN measurements[0] > 900 \
DO SYSTEM 'notify-send "Threshold exceeded"'

RULE low_alert \
ON measurements \
WHEN measurements[0] < 10 \
DO SYSTEM 'notify-send "Value too low"'

RULE anomaly_log \
ON measurements \
WHEN measurements[0] > 900 \
DO DUMP -20 TO 10 RETENTION 5

All rules for a given stream are evaluated on every new sample.