Field Expressions and Scalar Functions

A field expression computes one value of an output record. It occurs in the SELECT list, in a RULE condition, and as the argument of a record-history aggregate. Do not confuse it with the stream expression in FROM, which constructs and schedules an entire stream.

Building an expression

Operands are numeric and text literals, $ inside a stream generator, field references, and function results. A field may be addressed by name, stream-qualified name, or flat index, for example temperature, src.temperature, and src[0]. For a numeric declaration a T[N], bare a denotes the complete array entry and is not a scalar operand; use a[0]…a[N-1]. STRING[N] is one text field.

The basic arithmetic operators are +, -, *, /, and ^. Parentheses change grouping. * and / bind more strongly than + and -; exponentiation ^ binds most strongly and is right-associative:

DECLARE v INTEGER, w INTEGER STREAM source, 1 TEXTFILE 'source.txt'
SELECT v-w+1, v/w*2 STREAM precedence FROM source

For input row 12 3, the result is 10, 8. Equal precedence within each pair applies since fix #554. Previously, + bound more strongly than -, and * more strongly than /: these fields yielded 8, 2. To retain the earlier grouping deliberately, write it explicitly as v-(w+1) and v/(w*2).

Exponentiation examples:

SELECT v*w^2, v^w^2, (v*w)^2 STREAM powers FROM source

These fields mean v*(w^2), v^(w^2), and (v*w)^2. A negative literal is one grammar atom: -2^2 means (-2)^2, while -v^2 means -(v^2).

For integer and rational types, a non-negative integral power has the semantics of repeated multiplication, including type promotion and overflow. Other cases use floating-point computation; an infinite or NaN result becomes NULL. Text operands are not allowed.

The product of two BYTE values has type INTEGER; the same promotion applies to exponentiation, including a zero exponent. For BYTE fields b and e with e = 0, b^e yields integer 1, including when b = 0, and -(b^e) yields -1. Negation acts on an INTEGER result here, so it is arithmetic. The cases exact_power_zero_exponent_is_one_of_the_product_type and negated_byte_power_with_zero_exponent_is_minus_one in ut_expeval guard this behavior.

NULL propagates through ordinary arithmetic. Division by zero yields NULL for every numeric type and does not stop later stream processing. Comparison and three-valued logic in a RULE condition are documented in Logical Condition.

Addition, subtraction, and multiplication on UINT fields are checked: a sum or product outside the unsigned 32-bit range, or a negative difference, yields NULL instead of a wrapped value. An operation on an INTEGER and a UINT operand is computed on the exact values, and only the result is narrowed to UINT: it yields NULL only when the result does not fit. With u = 10 and i = -2, u + i yields 8 while u * i yields NULL; comparing such a pair, for example i < u, is exact as well. INTEGER and RATIONAL arithmetic also yields NULL on overflow.

⚠️ Warning After an interleave A#B, do not refer to its components as A[0], A.field, A[_], or A.*. An interleave has one shared schema; use the output stream name or recover a component with & or %. See Aliasing.

Unary operators

The behavior in this section requires an engine version containing the fix for #328.

The operators +, -, and ~ accept a field, a function result, or a parenthesized expression. Each item in the SELECT list still produces one output field, and an operand in a RULE condition remains part of that condition.

For a BYTE value of 0, both -a and ~a yield 255; for 1 they yield 254, and for 255 they yield 0. For UINT, the corresponding results for 0 and 1 are 4294967295 and 4294967294. This is neither subtraction from zero nor logical NOT. To obtain an arithmetic negative value from an unsigned field, explicitly convert its type first, for example -to_double(a).

The type is that of the computed operand, rather than just the source field. The literal -1 is a negative INTEGER; it does not use the bitwise-complement rule for BYTE or UINT. An allowed operator passes NULL through unchanged. Overflow in arithmetic negation of an INTEGER or RATIONAL yields NULL; for example, -a over an INTEGER value of -2147483648 yields NULL.

The operand of a unary operator includes exponentiation, multiplication, and division, but stops before binary + or -. Thus -a+1 means (-a)+1, while -(a+1) negates the entire sum. -a*b means -(a*b); write (-a)*b to negate only a. This distinction can change the operand type, the bitwise-complement result, or where overflow occurs. Likewise, ~a*b means ~(a*b). The exponentiation distinction remains: -2^2 yields 4, whereas -a^2 means -(a^2); write (-a)^2 to square a negated field.

DECLARE b BYTE, u UINT, i INTEGER STREAM src, 1 TEXTFILE 'source.txt'
SELECT -src[0], ~src[0], -src[1], ~src[1], +src[2], -(src[2]+1), -src[2]+1 STREAM unary FROM src
RULE negative ON unary WHEN -unary[4] < 0 DO DUMP -1 TO 0

For input 0 1 5, the result has seven fields: 255, 255, 4294967294, 4294967294, 5, -6, -4. The rule checks the negation of the fifth field, that is -5 < 0.

-a over STRING and ~a over INTEGER, RATIONAL, FLOAT, DOUBLE, or STRING are rejected during compilation, both in SELECT and in a RULE condition. Diagnostics state the reason: unary '-' is not defined for STRING or unary '~' is defined only for BYTE and UINT, not for .... When checking a file with xretractor -c, the reason appears in Check result:. Rejecting such an ad-hoc query (xqry -a) leaves the running plan and service active.

Available scalar functions

The sole name-and-arity list shared by the compiler and evaluator is the rqlFunctions.hpp table. Matching is case-insensitive and the canonical spelling is stored in the plan. An unknown function or invalid arity stops compilation rather than being deferred to runtime.

Every function takes one expression argument. The only exception is the optional output field width in to_string(expression : width).

Predicates and missing values

• isnull(x) returns 1 for NULL and 0 for a present value;
• null2zero(x) maps NULL to integer zero but passes a present value without changing its type;
• IsZero(x) and IsNonZero(x) return integer 1 or 0 for a numeric argument.

null2zero is lossy: afterwards, an original missing value cannot be distinguished from an actual zero. It does not replace the NULL bitmap stored in .meta.

String length

Length(x) accepts a string only. It counts the actual value up to the first zero byte, not the declared STRING[N] width. A STRING[8] field containing alpha therefore yields 5. A numeric argument is a runtime error.

Conversions

to_integer, to_float, and to_double convert a numeric or textual value to the named type. NULL passes through unchanged. to_integer truncates toward zero rather than flooring: to_integer(-8/3) yields -2. A floating-point value that INTEGER cannot hold - NaN and infinity included - yields NULL, exactly like arithmetic overflow. The same rule covers mathematical functions over an integer field whose result returns to the argument type: Sqrt(-4) and log(0) over an INTEGER field yield NULL. The range is checked after truncating the fractional part: for a DOUBLE argument, the value 2147483647.5 yields 2147483647, whereas 2147483648.0 yields NULL. The rule does not stop at integer types: NaN and infinity have no rational approximation, so a floating-point value written into a RATIONAL field yields NULL just as it does for INTEGER, not zero.

Converting an integer or rational value to a narrower integer type also yields NULL if the result does not fit: this includes a negative value directed to UINT, a UINT above INT_MAX directed to INTEGER or RATIONAL, and a value outside 0..255 directed to BYTE. Conversion from RATIONAL to an integer type first truncates toward zero, then checks the range. Invalid numeric text yields NULL. Approximation of a finite floating-point number as RATIONAL stops at the last fraction whose numerator and denominator both fit in int. For a finite FLOAT or DOUBLE argument, the integer part of its absolute value is checked first: 2147483647.5 and -2147483647.5 can be approximated, while 2147483648.0 and -2147483648.0 yield NULL.

Converting a string to INTEGER, FLOAT, or DOUBLE must consume the entire text. to_integer('12') yields 12, but to_integer('12tail') and to_double('1.5tail') yield NULL; a valid numeric prefix is insufficient. These conversions do not trim surrounding whitespace, so a trailing space also yields NULL.

String-to-RATIONAL conversions in the C++ library accept the complete forms 3/4 and 3 (meaning 3/1), including whitespace around the numerator, separator, and denominator. The numerator and denominator must fit in int; the denominator must be neither zero nor INT_MIN. A negative denominator is normalized: 3/-4 yields -3/4, unless a numerator equal to INT_MIN prevents sign reversal. This is separate from reading a binary RATIONAL field, which requires a positive denominator. Incomplete 3/, a suffix such as 3/4tail, nonnumeric bad, and overflow yield NULL. This rule introduces neither an RQL to_rational function nor RATIONAL support in a TEXTFILE source. ut_convertTypes, case cast_contract.string_prefixes_and_pair_parsing_match_both_interfaces, checks the shared contract of both C++ conversion interfaces.

to_string creates a text field. Without a second part its width is 32 bytes; to_string(x : N) declares N bytes. The separator is a colon because a comma separates fields in the SELECT list:

N must be in 1..65536. to_string(x : 0) is a parser error (to_string width 0 must be greater than zero), and the resulting string concatenation must also fit the field limit. See Plan Size Limits.

SELECT to_string(value : 10), Length(label), null2zero(optional) \
STREAM converted FROM source

The output text width is inferred after field references have been resolved, so a pure copy of STRING[N] and text concatenation retain the correct descriptor. The type of the whole expression, including numeric expressions, is determined by compiler::inferFieldShapes() after field references are resolved; see Type Promotion.

The declared width is a property of the field, not a side effect of one particular compilation pass. It also holds when the whole argument is constant: to_string(42 : 16) yields STRING[16], not STRING[2]. Expression simplification folds the argument underneath the call but never removes to_string itself - otherwise the declaration would disappear together with the program on a second compilation of the plan, that is after an ad-hoc query (xqry -a), which compiles the live plan a second time.

sin, cos, and exp

All three functions take a numeric argument and return DOUBLE, regardless of the argument’s type. sin and cos interpret angles in radians. For an INTEGER field k, the expressions sin(k), cos(k), and exp(k) therefore produce DOUBLE fields without truncating fractional results. A NULL argument yields NULL; a non-finite result, such as exp(1000), also yields NULL without stopping the stream.

A RATIONAL argument is the exception: the compiler rejects it and requires an explicit to_double, exactly as it does for Sqrt - see the section below.

Changing the result type of sin and cos can change the .desc descriptor and record layout relative to an older engine: INTEGER and FLOAT occupy 4 bytes, while DOUBLE occupies 8. An existing artifact with the old schema must be recreated or written to a separate output stream.

Irrational functions over a RATIONAL value

Sqrt, sin, cos, exp, tan, log, and log2 do not accept a RATIONAL argument - the compiler rejects such a query through the Check result: channel and names the workaround. In practice this concerns stream reducers over BYTE, INTEGER, UINT, and RATIONAL fields, whose result type is RATIONAL. Reducers over FLOAT and DOUBLE preserve the input type:

SELECT * STREAM m FROM AVG(src)
SELECT Sqrt(m[0]) STREAM o FROM m              // rejected at compilation
SELECT Sqrt(to_double(m[0])) STREAM o FROM m   // correct

One rule applies: a function with an irrational range over an exact rational value requires an explicit to_double. The same rule covers a rule condition (RULE ... WHEN), which the compiler checks in a separate pass.

For Sqrt, tan, log, and log2, the reason for the gate is the return from a double calculation to RATIONAL: the result used to be approximated by a fraction with a large denominator (for the rational argument 2/1, the square root yielded 19601/13860 and the logarithm 2731/3940). In an older version, two further multiplications of such an approximation could silently overflow its 32-bit numerator or denominator; Sqrt(x)*Sqrt(x)*Sqrt(x) returned -4.247 instead of +2.828. Arithmetic on INTEGER and RATIONAL field values now detects overflow and writes NULL, but these functions still require an explicit to_double.

For sin, cos, and exp the reason is different: these three end at DOUBLE and never return to RATIONAL, so they would compute correctly. Their rejection is a language contract decision, made so that no list of exceptions has to be remembered - one rule instead of seven separate behaviours. The price is a to_double in every query computing, say, an RMS over a reducer.

The restriction extends neither to the remaining functions nor to other argument types. The rounding functions Floor, Ceil, round, and trunc over RATIONAL are safe, because their result is integral and therefore has denominator 1, and Abs operates on the value directly and never touches the denominator.

The gate covers only the pairing with RATIONAL and changes no function’s result type. tan, log, and log2 over INTEGER still yield INTEGER, that is, they truncate the fractional part - an explicit and intended loss, not an overflow. Bringing them to DOUBLE, as sin, cos, and exp are, would change the field type in .desc, so it is a separate task.

NOTE: Functions and type propagation are covered by the integration tests fncall_runtime_case, string_field_passthrough, issue121_isnull, issue128_numeric_to_string, and issue128_string_to_numeric, and by the ut_compiler, ut_expeval, and ut_facctxtsrc unit tests.