xretractor

The xretractor program is RetractorDB’s core process. It compiles files containing RQL queries and executes the data-processing plan. It’s built to run autonomously as a systemd daemon process.

Modes of operation

xretractor starts in one of two modes:

Calling -h shows a different option list depending on the mode - option shorthands overlap, so pay attention to which mode a given option applies in.

Processing mode (default)

$ xretractor -h
xretractor - compiler & data processing tool.

Usage: xretractor queryfile [option]

Available options:
  -h [ --help ]               Show program options
  -b [ --build-info ]         show optimizer build configuration
  -c [ --onlycompile ]        compile only mode
  -q [ --queryfile ] arg      query set file
  -r [ --quiet ]              no output on screen, skip presenter
  -s [ --status ]             check service status
  -o [ --cleanup ]            remove leftovers of dead instances and exit
  -v [ --verbose ]            verbose mode (show stream params)
  -x [ --xqrywait ]           wait with processing for first query
  -n [ --name ] arg           instance name; own IPC area and lock
  -a [ --autoname ]           generate a docker-style instance name
  -k [ --noanykey ]           do not wait for any key to terminate
  -j [ --service ]            service mode: log to stderr (journald)
  -t [ --realtime ]           enable real-time scheduling
  -f [ --no-clock ]           offline mode: compute slots without waiting
  -u [ --until-eof ]          forces one-shot all sources
  -g [ --config ] arg         config file (TOML); overrides search
  -m [ --llimitqry ] arg (=0) loop iteration limit, 0 - no limit

Processing-mode options

Multiple instances

Several named instances can run concurrently:

xretractor measurements.rql --name measurements --noanykey &
xretractor diagnostics.rql --name diagnostics --noanykey &
xqry --bus

Each receives its own lock and IPC objects. The shared bus nevertheless rejects a plan that collides with a live instance by stream name, written storage file, or :ROTATION counter file. The check happens before artifacts are removed. Omitting --name preserves the historical unnamed instance. After the service-mode exclusivity check, an instance with the same name as a live bus entry, including a second unnamed one, is refused with exit code ENOLCK (37 on Linux) even when its lock file was deleted from the temporary directory.

Service mode provides a separate guarantee: exactly one service instance may run in each RDB_NAMESPACE; in the default namespace it is named service. See Multiple Instances and the Bus.

The instance lock file must be a regular file with a single name, owned by the account that starts the instance. A symbolic link (dangling ones included), a second hard link, a directory or a FIFO at that path, as well as a file owned by another account - also for an instance started as root - stop startup without changing the contents of the target file. The stderr message, shown in Release builds as well, names the path and the cause, and exit code 5 (EIO) distinguishes this refusal from an identity that is already taken (ENOLCK, 37 on Linux, message is already running). A leftover of another account after a crash is neither taken over by the instance nor removed by --cleanup; an operator with rights to the lock directory removes it by hand. Reading presence and IPC identity locks of other accounts remains allowed.

Cleaning up leftovers

xretractor --cleanup

The command does not start a plan. It attempts to acquire locks on recognized resources and removes only those without a live owner holding the lock. The same mechanism runs when an instance exits.

The scope is not restricted to an instance selected with --name or a single RDB_NAMESPACE. Filesystem permissions still control access to resources. The command does not enumerate or remove client response queues; it also leaves v5 and older segments untouched because they do not participate in the presence-lock protocol.

The output counts removed instance locks, IPC identity sets, and segments, for example:

Removed leftovers of dead instances: 1 instance lock(s), 1 IPC identity set(s), 1 bus segment(s).

The IPC-set count is not a count of individual queues. Completion does not establish that resources outside the command’s scope were removed. For a custom lock directory, select the appropriate TOML using --config.

Clock-free batch processing

The simplest run over a complete input file without manually selecting an iteration count is:

xretractor query.rql --no-clock --until-eof --noanykey --quiet

--no-clock removes sleeps only. It does not change slot order or artifact content, making it suitable for fast verification after completion. It can outrun an xqry client, however, so it is not intended for live observation.

--until-eof prevents a sequential source from returning to the start of its file. EOF is checked after a slot has been processed, exactly before the first record that would otherwise have to use synthetic NULL beyond the input. A DEVICE source is read at the start of the slot, so its exhaustion is checked before the slot - with the same effect. With several sources, the first exhausted one stops the run so the plan does not continue with a missing input. It may be combined with -m N; whichever condition occurs first wins.

A missing declared source, denied read access or a failed open ends execution with an environment error, K4, and code EIO (5 on Linux). This applies to TEXTFILE, BINFILE and DEVICE, including without -u. The message identifies the stream, path and cause. The startup check leaves previous results unchanged; compile-only mode (-c) does not require an available input. See source checks.

⚠️ Warning Short options depend on the mode. During execution, -f means --no-clock and -u means --until-eof. With -c, the same letters mean --fields and --rules, respectively, and do not start processing.

NOTE: noclock_offline verifies equivalence between paced and offline execution; untileof_stop verifies first-EOF termination and the non-wrapping control.

Compile-only mode (-c)

$ xretractor -h -c
xretractor - compiler & data processing tool.

Usage: xretractor -c queryfile [option]

Available options:
  -h [ --help ]          show help options
  -b [ --build-info ]    show optimizer build configuration
  -c [ --onlycompile ]   compile only mode
  -q [ --queryfile ] arg query set file
  -r [ --quiet ]         no output on screen, skip presenter
  -v [ --verbose ]       verbose mode (show stream params)
  -d [ --dot ]           create dot output
  -m [ --csv ]           create csv output
  -f [ --fields ]        show fields in dot file
  -t [ --tags ]          show tags in dot file
  -s [ --streamprogs ]   show stream programs in dot file
  -u [ --rules ]         show rules in dot file
  -i [ --hideruleprog ]  hide rule program in rules (-u) output
  -p [ --transparent ]   make dot background transparent
  -w [ --diagram ] arg   create diagram output
  -z [ --shmbudget ]     show shared memory budget of the compiled plan
  -g [ --config ] arg    config file (TOML); overrides search

In this mode, options for creating diagrams and diagnostic dumps, described in more detail elsewhere in this work, are available. -v is also available when compiling without running the plan.

Visualization and diagnostic options

Configuration file (TOML)

The --config option (short form -g) points at a configuration file, including in -c mode; without it the program searches two locations in layered fashion, in the order given, each later layer overriding keys from the previous one:

1. /etc/retractor/retractor.toml - system layer,
2. $XDG_CONFIG_HOME/retractor/retractor.toml (or ~/.config/retractor/retractor.toml) - user layer.

The absence of discovered files is a valid state - the program starts with default values. An existing file with invalid TOML stops startup: the program returns a nonzero exit code and reports an ERROR diagnostic naming the file and cause, including in Release builds. Every loaded layer must be valid, even if a later layer would override its keys. With an explicit path (--config), a missing file remains an error as well. The same configuration loader is used by xqry (--config, short option -e) and xtrdb; these programs also refuse to run with a malformed file. The --help option works in all three programs regardless of the configuration; xretractor then reports the error in the Config: line of its help. The [ipc] and [timing] sections apply to the engine and the xqry client.

IPC objects created by the server - the response segment, command queue, subscription queues, and bus segment - receive an explicit 0600 mode. Queue and response-segment creation uses create_only; opening an existing object requires its owner to match the process’s effective UID and its mode to be 0600. xqry must therefore run with the same effective UID as the server, including when invoked by an administrator, for example through sudo -u retractor xqry --server service --dir. Foreign objects are rejected and are not removed during cleanup. See the IPC access boundary for collision, restart, and bus behavior.

Out-of-range values do not stop the service: the program logs a warning and uses the default. The exceptions are storage.dir, whose invalidity would mean results landing somewhere unintended, or nowhere, and sources.timeout_s, whose silent replacement with the default would change how sources are waited for - both are hard errors of xretractor. xqry does not check sources.timeout_s. An invalid storage.ref_dirs is also a hard error, but for xtrdb: the list of permitted write locations must not be guessed.

Example file:

[storage]
dir = "/var/lib/retractor"
default_retention = [1000, 4]

[sources]
timeout_s = 0.01

[limits]
history_memory_mib = 1024

[ipc]
queue_buffer_seconds = 30

[scheduling]
rt_priority = 60

[paths]
lock_dir = "/var/run/retractor"

[server]
autoname = false

NOTE: Layer loading and validation are covered by the ut_appConfig unit test; hard rejection of an invalid storage.dir, a malformed TOML file and a relative paths.lock_dir by the config_storage_validation integration test, and the precedence and rejection of sources.timeout_s by the device_timeout integration test.

Service and plan replacement

Starting without an .rql file, or with an empty one, creates an idle instance with working IPC. The first or a later complete plan can be loaded without restarting:

xqry --server service --reset plan.rql

The server parses and compiles the complete contents, checks retained .desc files, storage compatibility under :ROTATION, and resource collisions, reserves the new set, and then switches plans at a slot boundary. A malformed, empty, or incompatible descriptor causes a refusal with a reason; the old plan and the service startup file remain unchanged. An empty reset file returns the server to idle. On a service instance, the server also attempts to write accepted contents to the service’s startup file. A failed write is logged and does not stop the new plan, but a restart may restore the previous contents. The reset’s OK response does not guarantee plan durability.

The alternative xretractor new-plan.rql path detects a running systemd unit and validates the set, stores, descriptors, and rotation counter before atomically overwriting its startup file and requesting a restart. A plan refusal returns EPROTO (71 on Linux), while an environment refusal returns EIO (5); the message includes nothing was changed. The running service keeps its plan and is not restarted. Explicitly selecting another identity with --name or --autoname starts a separate instance instead.

Startup, plan delivery, reset, and ad hoc import share storage checks before changing the plan. They check the STORAGE directory (which must exist even for a plan using only MEMORY), retained profiles and descriptors, source existence, kinds and read permissions, and whether output data files and applicable .shadow files can be opened. The open check creates no new files. A bad .desc produces a diagnostic naming its path; a syntax error also identifies the line and column. DECLARE source descriptors are always checked; on startup and reset, output and substrate descriptors are checked when :ROTATION preserves their files. Ad hoc import checks new streams’ descriptors even without rotation and for MEMORY, because it does not delete them first.

Plan refusal and environment failure

The core propagates execution errors through Result. The xretractor boundary distinguishes Errc::IO from an internal error and runs the corresponding termination procedure; rdb::fatal() uses the handler installed by the launcher. The systemd unit since #475 also uses ExecStopPost: after SIGABRT or OOM kill it clears the plan file, because these terminations do not run atexit. An ordinary I/O failure preserves the plan. This policy requires the current unit and xretractor-stop-post helper, rather than only replacing the binary. it_service_abort-run guards it.

EIO identifies the process error category; causes such as EACCES or ENOSPC appear in diagnostics. Outside a unit, the operator’s .rql file remains unchanged. Compile-only mode (-c) neither checks declared sources, storage files or the counter nor clears the query file, even inside a unit. A reset or ad hoc request refused before acceptance returns the reason to the client, while the running server keeps its plan.

A successful check does not guarantee a later open: files can change on disk in between. A model-construction refusal retains its cause class: a plan error yields EPROTO, an environment error yields EIO, and no partial model is published. On reset, a change after the OK response but before model construction can terminate the process with the same code; the previous model has already been dismantled. Reset does not roll back this late failure. The checks are covered by it_storage_refusal-*, it_plan_error_e4-run, it_env_error_exit-run, it_untileof_stop-missing-sources, and ut_fatalError.

The supplied systemd unit template sets Restart=on-failure, RestartSec=5, RestartSteps=5, and RestartMaxDelaySec=5min. On versions supporting these keys, retry delays grow from 5 seconds to 5 minutes in five steps. The template sets no StartLimit* options, so administrator limits still apply; unlimited retries are not guaranteed. Check older systemd versions on the target host. Keeping the plan after an environment failure is not a checkpoint: a restart does not guarantee state restoration or freedom from lost or repeated data.

Version Information

At the end of every help message, a line with build information is displayed:

Branch: issue_31-doc:2707ce0,
Code compiler: GNU Ver. 13.3.0,
Build time: 2512211449,
Type: Debug

The next line indicates the log file location:

Log: /tmp/xretractor.log

The file /tmp/xretractor.log records the history of invocations and the system’s internal events. In a production environment, this file should be cleaned up or rotated regularly.

The Config: Defaults line means that no TOML files were loaded. If configuration was loaded, Config: lists the file paths in load order. For a syntax error in a plan loaded from a file, the parser’s log entry also includes that file’s path.

The last line contains MIT license information, which allows safe use of the code in corporate applications.