Ad Hoc Queries

By Ad Hoc queries we mean queries directed at a running system. In the typical scenario originally envisioned during system development, the initial working assumption was that a system user would know all the queries and data sources needed to obtain the processed time series.

During development, however, additional scenarios emerged, assuming that the system’s operation should not be interrupted, and that additional queries should be attached to the query execution plan. We call this kind of functionality Ad Hoc queries - attached to the system while it is running, without interrupting its operation.

Fig. 48 shows the control flow described above. A file with queries and directives is first directed to the xretractor process. Then, through shared memory, the xqry process pulls data from xretractor. Using that same process, we can send a command to the xretractor process. In this command we include the text of the additional query that xretractor should attach to the tree being processed.

What can be attached at run time

The ad hoc channel accepts exactly one SELECT, DECLARE, or RULE statement. Compiler directives and programs containing multiple statements are rejected without changing the active plan.

A syntax error or invalid value is returned to the client with its reason, and the server continues to accept commands. The parser rejects, among other cases, out-of-range numeric literals, a zero denominator, a zero interval (in DECLARE and with &, %, or -), and an empty FILE name. If attachment fails after import, the server restores the previous plan and releases newly claimed stream names and storage files on the bus. The same command can be retried after the cause is fixed.

Import rollback covers both a failure returned as a status and an exception during import, compilation, or registration of new streams. In the latter case, xqry -a prints the reason from the server response, for example command processor failure: ..., and exits with a nonzero code. Existing streams remain available, and the rejected names disappear from the plan and bus. adhoc_register_rollback, including its -throw variant, checks both paths and retrying the same command.

If the bus mutex becomes unusable during import rollback, the plan is restored but the new claims cannot be released. They remain claimed until the instance ends; an ERROR log lists the stream names and storage paths with the cause. A refused import does not guarantee restoration of the bus registry in this case.

Before attaching new streams, the server checks the STORAGE directory, retained descriptors, source existence, kinds and read permissions, and whether output files and applicable .shadow files can be opened. A new SELECT descriptor must match the plan’s schema and must not introduce an unplanned REF; its TYPE must be supported. The check includes descriptors without ROTATION and for MEMORY, because import does not delete them. Only new streams are checked. The open check creates no new files; for a missing file, it checks the parent directory. A refusal returns the reason to the client, such as cannot open output file, without changing the plan. If the directory becomes unavailable after this check, a later failure to open a POSIX or POSIXSHD accessor, or a DEFAULT/DIRECT segment, is also returned as a refusal: the server rolls back the import and new claims on the bus, stays running, and allows the command to be retried. This handling of late open failures does not cover the STORAGE GENERIC profile.

A generated substrate name colliding with a different program causes a compilation refusal (Fail local chain compiler:Substrate name '...' denotes two different programs) without terminating the server. This also applies to ordinary names with underscores, such as A_B + C and A + B_C. A corrected command can be submitted after refusal; coverage: it_plan_error_e4-run and it_storage_refusal-adhoc.

A new source can be declared without stopping a running engine:

$ xqry -a "DECLARE a BYTE STREAM C, 1 TEXTFILE 'data3.txt'"

Exit code 0 with no message means that the declaration was accepted. A declaration attached after the zero step receives its logical-index base in its first due slot. If a query attached later needs a window or a time shift, emission waits until the source has accumulated the complete required history. HOLD is not required; it remains an optional directive that delays the physical read. Repeating DECLARE for an existing name is rejected rather than treated as a configuration update.

With multiple live instances, DECLARE alone cannot identify an owner because it has no FROM clause. The target must then be selected explicitly:

$ xqry --server measurements -a "DECLARE a BYTE STREAM C, 1 TEXTFILE 'data3.txt'"

Attaching the first declaration to a server started with an empty plan is not yet supported; the ad hoc channel requires an active data model.

A rule attached at run time may execute DO DUMP or DO SET. DO SYSTEM remains available only in the plan file the instance starts from, because exposing it through IPC would let a client run arbitrary shell commands as the server account. The same boundary holds on the xqry --reset channel, which also carries a complete plan but carries no authorship either: a plan with a DO SYSTEM rule is refused there as a whole, unless the operator deliberately sets service.unrestricted = true (→ xqry). The ad-hoc channel refuses unconditionally and does not read that key. The ON target must be an existing stream created by SELECT.

For a historical DUMP -H TO M range, the WHEN condition is first evaluated no earlier than record H+1 after attachment, when the preceding H records were also produced after attachment. For a range without history (H=0), evaluation starts with the first new record. A MEMORY store must have at least H+1 slots; for H > 0 and capacity N <= H, the request is rejected without attaching the rule. Attachment does not enlarge the existing store. See Alerting implementation for details.

xqry --server measurements -a \
  "RULE alarm ON temperature WHEN temperature[0] > 80 DO DUMP -10 TO 5"

On a MEMORY stream, the example above requires at least 11 slots; with capacity 10, the request is rejected.

With multiple instances, the client routes a SELECT according to the owners of streams in FROM, and a RULE according to the stream in ON. A query combining sources from several servers is rejected. New stream names and storage files are claimed on the bus before the active plan is changed, so ad hoc commands cannot overwrite another instance’s resource.

Ad hoc commands extend the current plan. Use xqry --reset file.rql to replace it fully and atomically, including on an idle instance.

Plan variables and DO SET

Ad hoc can see the active plan’s VAR variables: it can read them in SELECT and WHEN and write them through an attached DO SET rule. VAR and LET declarations are refused; the plan file’s LET constants are invisible. For the hysteresis example, submit:

xqry --server measurements -a "SELECT status[0], ?ALARM STREAM monitor FROM status"
xqry --server measurements -a "RULE force_alarm ON status WHEN status[0] >= 300 DO SET ?ALARM = 1"

The rule starts evaluating from the next new target record. Its update is committed at the end of the slot and becomes visible from the following slot. In a conflict with a plan rule or an earlier ad hoc rule, the later attached firing rule wins. There is no separate configuration key for SET; the DO SYSTEM refusal is unchanged.

The right-hand side must have the variable’s type. A refusal preserves active values and rules. Accepted ad hoc additions retain existing variable state; an accepted --reset creates new state from declarations and removes the old plan’s ad hoc rules. A SET rule can target an existing individual generator instance such as cell$0; the entire family is not a live-plan target. Coverage: it_var_adhoc-run.

Where an ad hoc stream begins

A plan built from the start of system operation numbers records from the logical origin computed by the compiler. A query attached after the zero step has no such history - its base corresponds to the first due slot in which the runtime processed it, rather than slot zero of the plan. Import is atomic: the compiled tree and its stream instances are published under a common lock, and the execution loop rebuilds the timeline without rewinding, even when the new query introduces a new rate.

The model can accept commands before the zero step, for example with the --xqrywait (-x) gate. A file source attached in that window belongs to the epoch’s initial plan: the zero step assigns its compiler-computed base and prepares its first record, just as for sources from the plan file. DEVICE sources have no zero-step read; their first record is read in a due slot. During the zero step itself, the common lock blocks import until initialization finishes. it_hash_string-adhoc-zero-step checks attaching a source before that step.

NOTE: This behavior is covered by the issue227_join_alignment test (the adhoc-origin case).

Example

We’ll start the example by preparing a simple query:

DECLARE a BYTE STREAM A, 1 TEXTFILE 'data1.txt'
DECLARE a BYTE STREAM B, 2 TEXTFILE 'data2.txt'
SELECT * STREAM str1 FROM A+B

We’ll save the query file under the name qplan1.rql. For the query to run correctly, we also need to prepare the files data1.txt and data2.txt. I suggest filling data1.txt with consecutive numbers from 1 to 6, each on a new line, and filling data2.txt with numbers from 10 to 15. In a directory prepared this way, we run the command:

$ xretractor qplan1.rql

Without ROTATION, startup removes earlier SELECT output files, so an old str1 schema does not block this example. When ROTATION preserves the files, an incompatible descriptor causes a refusal before startup with descriptor schema mismatch and the file path. The operator must restore a compatible plan or deliberately remove the stream’s file family and start it afresh. Ad hoc import preserves existing files and also refuses an incompatible descriptor.

The xretractor process will begin processing data. At this point, open another terminal and issue the command:

$ xqry -d
name | duration | size | count | location  | cap
-----+----------+------+-------+-----------+----
str1 | 1        | 48   | 24    |           | 0
A    | 1        | -1   | 3     | data1.txt | 1
B    | 2        | -1   | 2     | data2.txt | 1

This will display, in tabular form, what’s currently being processed in the system - how many bytes have already arrived, which files the data is being read from, and how much data has already been processed. If a more descriptive format is desired, we can issue the following command:

$ xqry -d -y
---
apiVersion: xqry/v1
streams:
  - name: str1
    delta: 1
    size: 214
    count: 107
  - name: A
    delta: 1
    count: 86
    location: data1.txt
  - name: B
    delta: 2
    count: 43
    location: data2.txt

The response is given in YAML form.

To add another query to the system, we need to issue the command:

$ xqry -a "SELECT * STREAM str2 FROM A#B"

A command in this form sends a new query to the xretractor process. No message and exit code 0 mean that it was accepted. The system compiles it and merges it into the query plan tree; on rejection, xqry returns a non-zero code and writes the diagnostic reason.

If we check the system’s state again, we’ll see the following picture:

$ xqry -d
name | duration | size | count | location  | cap
-----+----------+------+-------+-----------+----
str2 | 2/3      | 10   | 10    |           | 0
A    | 1        | -1   | 23    | data1.txt | 1
str1 | 1        | 312  | 156   |           | 0
B    | 2        | -1   | 12    | data2.txt | 1

Or like this:

$ xqry -d -y
---
apiVersion: xqry/v1
streams:
  - name: str2
    delta: 2/3
    size: 7
    count: 7
  - name: A
    delta: 1
    count: 16
    location: data1.txt
  - name: str1
    delta: 1
    size: 298
    count: 149
  - name: B
    delta: 2
    count: 8
    location: data2.txt

Taking a closer look at the queries via the xqry command, we’ll see the following system response for the str1 query:

$ xqry -t str1 -y
---
apiVersion: xqry/v1
stream:
  name: str1
  delta: 1
query: SELECT * STREAM str1 FROM A+B
fields:
  str1.A_0:
    type: BYTE
  str1.B_1:
    type: BYTE

and for the str2 query:

$ xqry -t str2 -y
---
apiVersion: xqry/v1
stream:
  name: str2
  delta: 2/3
query: SELECT * STREAM str2 FROM A#B
fields:
  str2.a:
    type: BYTE

As you can see, the additional query str2 was correctly merged into the existing query execution plan. You can also see that far less data has accumulated compared to str1.

NOTE: The functionality described here is covered by the test: issue6_adhoc, described in the appendix Integration Tests.