Files

This chapter describes the five files that make up the complete file set of an artifact or substrate: the schema descriptor (.desc), the main binary data file, the metadata index (.meta), the data shadow file (.shadow), and the index shadow file (.meta.shadow). For each file, the binary format, field semantics, and read/write rules are presented. The chapter also covers the metaData class - the RLE compression mechanism, transmission-gap handling, the update interface, and persistence across restarts. The final section shows the relationships between all the files at the level of append, update, and read operations.

The scope of this chapter does not include the file-rotation mechanism between sessions (→ Rotation) or the xtrdb -s inspection tool (→ Inspection Tool).

File opens and symbolic links

Storage files - data, .shadow, .meta, .meta.shadow, and .desc - are opened with O_NOFOLLOW: a symbolic link at the final filename causes the open to be refused instead of writing to or truncating its target. For GENERIC, refusal may occur only at the first write. An unavailable metadata index may disable its persistence without stopping the stream; refusal to open a link therefore does not always mean rejection of the entire plan.

The exception is the main data file named by an explicit REF supplied by the caller (a plan or a schema in xtrdb). That operator choice can authorize a final data-file link. A REF read only from an existing .desc does not grant this permission, nor does allowing a directory through storage.ref_dirs. The exception covers neither auxiliary files nor retention-segment names generated by the engine.

O_NOFOLLOW checks only the last path component. The storage directory and its ancestors may still be links; replacement of a parent directory during path resolution remains outside this guarantee. ut_rdb checks refusals and permitted controls. DUMP files have a separate policy that removes the final entry and exclusively creates a new file - see Alerting Implementation.

The descriptor file (.desc)

The .desc file describes the record structure. It is parsed by an ANTLR4 grammar (DESC.g4) and can contain data fields, storage-type meta-information, and a retention policy.

Syntax

{ <statement>* }

Each statement is one of the following:

BYTE     name [N]          # array of N bytes (default N=1)
INTEGER  name [N]          # 32-bit signed integers
UINT     name [N]          # 32-bit unsigned
FLOAT    name [N]          # 32-bit floating point (IEEE 754)
DOUBLE   name [N]          # 64-bit floating point
RATIONAL name [N]          # pair of int32: numerator and denominator
STRING   name [size]       # fixed-length string
REF      "path/file"       # data file (relative paths use the process's working directory)
TYPE     identifier        # storage type (DEFAULT, MEMORY, POSIXSHD, …)
RETENTION capacity segment # cyclic on-disk retention
RETMEMORY capacity         # cyclic in-memory retention

Example .desc files

A default artifact - two numeric fields, DEFAULT storage (data file + shadow file):

{
  INTEGER  ts
  FLOAT    value
  TYPE     DEFAULT
}

A SELECT result or substrate in RAM - MEMORY storage with a one-record ring (ephemeris sources declared with DECLARE have type BINFILE, TEXTSOURCE, or DEVICE):

{
  DOUBLE   x
  DOUBLE   y
  TYPE     MEMORY
  RETMEMORY 1
}

A substrate with retention - up to 1000 records on disk (10 segments of 100):

{
  INTEGER  ts
  FLOAT    a
  FLOAT    b
  TYPE     DEFAULT
  RETENTION 100 10
}

A binary source declaration (DECLARE in RQL generates this schema):

{
  INTEGER  a
  FLOAT    b
  TYPE     BINFILE
  REF      "sensor/data.bin"
}

Field type sizes

For array fields name[N], the total size = type_size × N. The TYPE, REF, RETENTION, and RETMEMORY fields take no space in the record - they are descriptor metadata.

Record size R = the sum of the sizes of all data fields.

The RATIONAL field layout

A RATIONAL field stores a rational number as a pair of signed integers, written into the record directly, with no header and no type tag:

offset +0   int32   numerator
offset +4   int32   denominator

The byte order is the machine’s native one - little-endian on x86-64 and ARM64, the same as for INTEGER and UINT fields. The field occupies 8 bytes; in an array field RATIONAL name[N] the pairs follow one another, 8 × N bytes in total.

The value is always stored in lowest terms, and the denominator is always positive - the sign is carried by the numerator alone. This follows from boost::rational arithmetic, which normalizes the result on every assignment; it is not a writing convention. In particular:

• zero is stored as 0/1, never as 0/0 or 0/5;
• a whole number is stored as n/1 - a RATIONAL field with denominator 1 is exactly an integer, with no rounding involved (the slot-divisibility test in the query tree traversal algorithm relies on the same invariant);
• the denominator is never zero - no write path in the system produces such a pair.

The invariant describes writing, however, not every file that can be handed to the engine. Since 23 September 2026 the read path checks it at the bytes-to-value boundary: a pair whose denominator is less than or equal to zero yields NULL, not a rational number. This covers a zeroed record (0/0), a denominator equal to INT_MIN, and the pair INT_MIN/-1 - exactly the bytes on which the previous version terminated the process with SIGFPE. A pair with a positive denominator also passes through normalization, so a reducible representation comes back reduced (2/4 as 1/2). The pair 1/-2 yields NULL because its denominator is negative. For a file written by the system this changes nothing - it contains no such pairs.

RATIONAL fields are produced by the MIN, MAX, AVG, and SUMC reducers when the input value has type BYTE, INTEGER, UINT, or RATIONAL. This covers current-record reducers in FROM, the deprecated .min/.max/.avg/.sumc notation, and record-history AGG(expression : W) in the SELECT list. FLOAT and DOUBLE inputs preserve their types (→ Aggregate Operators). A reducer over integer or rational input is, in practice, the main source of RATIONAL in an artifact.

A measured example

A plan computing the mean over a window of three samples:

DECLARE v INTEGER STREAM src, 1 TEXTFILE 'data.txt'

SELECT * STREAM ravg FROM AVG(src@(1,3))

fed with -3, -3, -2, 7, 7, 7, … yields the descriptor { RATIONAL avg } and a data file whose first record is eight bytes long:

f8 ff ff ff   03 00 00 00
└ numerator ┘ └ denominator ┘
   -8              3            →  -8/3

The fourth record is 07 00 00 00 01 00 00 00, that is 7/1 - the mean of three sevens, stored as a rational number with denominator 1 rather than as an INTEGER.

Reading the value without decoding bytes

The pair layout only matters when reading the binary file directly. That a field is of type RATIONAL and occupies 8 bytes is reported by xtrdb -s name from the descriptor (→ Inspection Tool) - the tool shows structure, not values. The value itself is obtained by passing the field through a conversion in the query; three functions offer three different trade-offs:

For export to text-based systems to_string is the right choice, because it preserves the value exactly; to_integer is convenient but drops the fractional part, and does so differently from Python’s flooring // - the rounding rule is documented alongside the expression functions.

The TYPE field and storage strategy

The TYPE field in the descriptor directly determines which accessor (FileInterface) is used by storage::initializeAccessor(). The absence of a TYPE field is equivalent to DEFAULT. The value is case-insensitive (MEMORY = memory).

The binary data file

The data file is a sequence of fixed-length records, written one after another with no header at all. The size of a single record R is determined by the descriptor as the sum of all field sizes in bytes.

Every record contains the packed field values in the order defined by the descriptor:

The append operation (adding a new record) writes data to the end of the file. The update operation (modifying an existing record) - if a shadow file exists - goes into the shadow file, not the main file.

Example

DECLARE a INTEGER, b FLOAT STREAM src, 0.1 BINFILE 'data.dat'
SELECT * STREAM str1 FROM src

Record size: INTEGER (4 B) + FLOAT (4 B) = 8 bytes. After 50 records have been written, the output file str1 occupies 50 x 8 = 400 bytes. With a 0.1 s interval, this represents 5 seconds of data. data.dat is an existing read-only source; DECLARE does not grow it.

Reading a record that does not exist

A request for an index past the last record - or a read from an empty store - is not a successful read. Since 23 September 2026 storage::read() and storage::revRead() return a separate NoSuchRecord status, zero the destination buffer, and set the entire null pattern to true: a record that does not exist is an undetermined value, not a zero record. This is the same convention dataModel::fetchBack() and fetchForward() apply to a read beyond the accumulated history.

This matters for the result, not only for diagnostics. Previously that branch returned success and marked the zeroed record explicitly as not null, so NULL absorption did not kick in and the MIN, MAX, SUMC, and AVG reducers folded a fabricated zero into the result instead of skipping the missing record (→ Aggregate Operators). The xtrdb tool distinguishes an initial out-of-range refusal (record out of range, with no payload change) from a failed read of an allowed index (error and, for a list, fetch error). See xtrdb for details.

The null pattern lives in the payload and in the .meta index, that is, inside the engine. A DO DUMP dump does not carry it - a non-existent record is written there as zeros indistinguishable from data (→ Alerting implementation).

The metadata file (.meta)

The .meta file is an index of null values and transmission gaps. It stores information about which record fields have null values, and where gaps occurred - without duplicating the data itself.

File format

RLE entry format

Each entry describes a run of consecutive records with an identical null pattern:

The null pattern belongs to a particular record: a bit set for one field does not change the other fields or other records described by the same descriptor. A numeric array field T[N] is one descriptor entry and has one shared null bit for all N elements; a set bit means the entire field is null. When N scalar fields are converted into one T[N] field, a null in any of them sets that shared bit.

RLE compression

Consecutive records with the same null pattern are merged into a single entry by incrementing recordCount. A new entry is only created once the pattern changes.

10 records, 2 fields, no nulls:

Field 1 becomes null starting at record 5:

Transmission gap after record 3:

The transmission-gap marker

A transmission gap (e.g. a system shutdown, a lost signal) is recorded as an entry with isGap=true and all null bits set to true. The count parameter stores the length of the gap in units of the stream’s interval. The binary data file itself contains no additional records for the gap - that information lives solely in the .meta file.

NOTE: The functionality described here is covered by the tests: issue113_meta_internal, issue113_meta_autocreate, described in the appendix Integration Tests.

The metaData class

The .meta file is managed by the rdb::metaData class. It acts as a coordinator: it keeps the RLE policy itself (lazy overwrite of the last entry, record numbering) and delegates the specialized parts to separate units:

metaData itself encapsulates three areas of responsibility:

1. In-memory RLE aggregation - it buffers the current segment (the most recent run of records with an identical null pattern) in the currentEntry_ field, without writing it to the file on every record.
2. Data persistence - only completed segments (when the pattern changes, or on an explicit call to flushCurrentEntry()) are written to the file as committed entries.
3. A query index - it exposes an interface for querying the null pattern of any record and for detecting transmission gaps.

The class holds two states:

Object lifecycle

The state diagram (Fig. 16) shows the transitions between phases of a metaData object:

%% pdf-width: 30%
stateDiagram-v2
    [*] --> Construction : constructor
    Construction --> Active : loadIndex()
    Active --> Active : onRecordAppended()
    Active --> Active : onRecordModified()
    Active --> Active : onTransmissionGap()
    Active --> Active : flushCurrentEntry()
    Active --> [*] : destructor (auto flush)

Fig. 16. Lifecycle of a metaData object

Constructor (metaData(descriptor, path)):

• Initializes an empty currentEntry_ based on the number of fields in the descriptor.
• Calls loadIndex() - if the file exists, it loads all committed segments, determines committedRecordCount_, and moves the last non-gap segment back into currentEntry_ (allowing the RLE run to continue after a restart).
• If the file does not exist, it creates it and writes the header (8 reserved bytes, zeros).

The destructor automatically calls flushCurrentEntry(), guaranteeing that the current buffer reaches disk even when the program exits normally.

The update interface

The class distinguishes three scenarios for changing metadata state:

onRecordAppended(nullBitset)

Called by storage after every new record is appended to the data file.

pattern identical to currentEntry_?
├─ YES → increment currentEntry_.recordCount (RLE accumulation, no I/O)
└─ NO  → flushCurrentEntry() (previous segment to disk)
          set currentEntry_ = {nullBitset, count=1}

I/O only happens when the pattern changes - for a run of identical records, the cost is a single in-memory counter increment.

onRecordModified(index, nullBitset)

Called by storage when updating an existing record. Behavior depends on the operating mode:

Normal mode (no data shadow file): it locates the record within the RLE segments and splits the segment into up to three parts: before the modified record, the record itself, and after it.

is the record in currentEntry_ (memory)?
├─ YES → splitSegment() in memory, new fragments appended to the file
└─ NO  → read the file, splitSegment(), rewrite the file (rewriteFile)

Example of splitting a segment [allNull × 5] when modifying record 2:

Before: [allNull × 5]
After:  [allNull × 2] [allPresent × 1] [allNull × 2]

Shadow variant (storageShadow, injected in place of the base metaData for stores that maintain a .shadow file): instead of modifying the main index, it appends a single null-pattern override to the .meta.shadow file. The main .meta index remains untouched and consistent with the main data file.

index object is a storageShadow?
├─ YES → metaShadow::appendOverride(index, nullBitset) → entry in .meta.shadow
└─ NO  → modify the main index → splitSegment()

onTransmissionGap(duration)

Records a transmission gap of the given length (in units of the stream’s interval). It first commits the current segment (flushCurrentEntry()), then appends an entry with isGap=true to the file (Fig. 17).

sequenceDiagram
    participant S as storage
    participant M as metaData
    participant F as .meta file

    S->>M: onTransmissionGap(5)
    M->>F: flushCurrentEntry() - write [normal, count=N]
    M->>F: appendEntry(isGap=true, count=5)
    Note over F: the file now contains a gap marker

Fig. 17. Gap-recording sequence - onTransmissionGap

Safety mechanism: flushCurrentEntry() and overwriting (tail_.dirty)

The storage class calls flushCurrentEntry() after a successful physical record append, following the metadata update by onRecordAppended(). A record absorbed by the transmission-gap mechanism and a refused append to a read-only source return from write() earlier. Overwriting an existing record calls onRecordModified(), which updates either the main index or its shadow, depending on the store type.

flushCurrentEntry() writes the current index entry to the file, reducing the amount of metadata held only in process memory. This write does not include fsync after every record or a transactional commit of the data together with the index, so it does not by itself guarantee consistency after a process crash or power loss. The ut_metaData_usage test checks an append followed by an explicit flush; it does not simulate a process crash.

Rewriting the index through MetaIndexStore::rewrite() returns Result with Errc::IO and errno if opening or writing the temporary file, or replacing .meta, fails. onRecordModified() propagates that error before changing the in-memory index state; this does not provide a transaction with an earlier data write. A rewrite failure during loadIndex() is an initialization error and detaches persistence. The separate appendEntry, overwriteLast and saveHeader paths still lack this error propagation; port #511 does not guarantee detection of every metadata-write failure. ut_metaData guards failed replacement.

A naive implementation would append a new entry to the file on every flush - causing file growth proportional to the number of records, even without any change in the null pattern.

The solution: a lazy overwrite mechanism flagged by tail_.dirty.

flushCurrentEntry() → write [pattern, count=2] to disk
onRecordAppended(the same pattern):
    currentEntry_.count = 2 (restored from disk)
    tail_.markDirty()        ← the next flush will overwrite, not append
    currentEntry_.count++    → count = 3
flushCurrentEntry() → seek to the last entry, overwrite [pattern, count=3]
    (file size unchanged)

The sequence diagram for storage’s typical pattern (append + flush after every record) is shown in Fig. 18:

%% pdf-height: 55%
sequenceDiagram
    participant S as storage
    participant M as metaData
    participant F as .meta file

    S->>M: onRecordAppended([F,F])
    S->>M: flushCurrentEntry()
    M->>F: appendEntry([F,F], count=1)

    S->>M: onRecordAppended([F,F])
    S->>M: flushCurrentEntry()
    Note over M: tail_.dirty=true, overwrite last entry
    M->>F: overwrite last entry: [F,F] count=2

    S->>M: onRecordAppended([F,F])
    S->>M: flushCurrentEntry()
    M->>F: overwrite last entry: [F,F] count=3

    S->>M: onRecordAppended([T,F])
    Note over M: different pattern → new entry
    S->>M: flushCurrentEntry()
    M->>F: appendEntry([T,F], count=1)

Fig. 18. The lazy-overwrite mechanism - overwriting the last .meta entry

Thanks to this, the .meta file grows only when the null pattern changes - not on every record. With continuous, uniform data arrival, the file has a constant size regardless of the number of records.

Persistence and state recovery

After the process restarts, a new metaData object loads the file via loadIndex() (sequence shown in Fig. 19):

1. It skips the header - 8 reserved bytes; nothing in them is interpreted.
2. It loads all committed entries from the file.
3. If the last entry is not a gap, it moves it back into currentEntry_ and removes it from the file (allowing the RLE run to continue after a restart without duplication).
4. It computes committedRecordCount_ as the sum of recordCount over all non-gap entries remaining in the file.

sequenceDiagram
    participant Proc1 as First session
    participant F as .meta file
    participant Proc2 as Second session

    Proc1->>F: writes segments [A×500][B×200]
    Note over Proc1: destructor → flushCurrentEntry()
    Proc1->>F: last segment committed

    Proc2->>F: loadIndex()
    F-->>Proc2: reads all segments
    Note over Proc2: last segment moved into currentEntry_
    Note over Proc2: ready to continue the RLE run
    Proc2->>Proc2: totalRecords() = 700

Fig. 19. Persistence and state recovery after a restart

Query interface

The index-shadow interface

Methods of the storageShadow class - the index variant injected by makeMetaIndex() for stores that maintain a data shadow file. The base metaData does not have them, and there is no mode switch: the presence of a shadow is decided by the choice of class at store initialization.

Usage example - a typical production scenario

storage.write(rec0)           → onRecordAppended([F,F,F]) + flushCurrentEntry()
storage.write(rec1)           → onRecordAppended([F,F,F]) + flushCurrentEntry()
storage.write(rec2_val_null)  → onRecordAppended([T,F,F]) + flushCurrentEntry()
storage.write(rec3)           → onRecordAppended([F,F,F]) + flushCurrentEntry()

The .meta file after the above operations (4 flushes, 2 segments):
  [isGap=F, count=2, bitset=[F,F,F]]   ← entry 0
  [isGap=F, count=1, bitset=[T,F,F]]   ← entry 1  (rec2)
  [isGap=F, count=1, bitset=[F,F,F]]   ← entry 2  (rec3, currently in memory)

getNullBitset(2) → [T,F,F]   (field 0 of record 2 is null)
isGapBefore(2)  → false
totalRecords()  → 4

The shadow file (.shadow)

The shadow file allows modification of recorded records without destroying the original data. Deleting the .shadow file restores the original state of the data.

Entry format

Every modification appends a new entry to the end of the shadow file. With multiple modifications of the same record, the file may contain multiple entries for the same position - the most recent one is the current one.

Read priority

Read priority is the rule for deciding which source the system should return a record’s value from, when the same index could appear in both the main file and the shadow file at once. In RetractorDB, priority is defined deterministically: .shadow is checked first (from the end, to pick the most recent modification), and only if there’s no entry is a read performed from the main file. This concept concerns the consistency and read versioning of data after modifications, not the physical record-storage format of the binary file itself.

%% pdf-width: 100%
flowchart LR
    Q["Read record at position P"]
    Q --> SH{"Look up P in .shadow\n(from the end)"}
    SH -->|found| RET1["Return data from .shadow\n(the most recent modification)"]
    SH -->|not found| MAIN["Read from the main file\npread(fd, pos=P×R)"]
    MAIN --> RET2["Return the original data"]

Fig. 20. Record read priority relative to the shadow file

Fig. 20 shows the record-read logic: the system first checks for an entry in .shadow, and only reads the record from the main file if there is none.

Merging (merge)

The merge() operation merges changes from the shadow file into the main file and clears the shadow file. After merging, the original data is irrecoverably overwritten.

sequenceDiagram
    participant App
    participant Shadow as .shadow
    participant Main as main file

    App->>Shadow: read all entries (i, data_i)
    loop for every entry
        Shadow-->>App: (position=i, data=data_i)
        App->>Main: pwrite(data_i, offset=i×R)
    end
    App->>Shadow: ftruncate(0) - clear the shadow file

Fig. 21. Merging the shadow file into the main file

Fig. 21 shows the flow of merge(): successive (position, data) entries from .shadow are written to the main file, and once finished, the shadow file is cleared.

Example: modifying a record

# Stream str1: 2 INTEGER fields (4B each), recordSize = 8B
# Record 2 (original): [100, 200]
# Modification: field 0 → 999

# The .shadow file after the modification:
# offset 0: [position=2 (8B)][999, 200 (8B)]

Reading record 2 will return [999, 200]. Reading records 0 and 1 will return data from the main file (they have no entries in the shadow file).

The index shadow file (.meta.shadow)

The .meta.shadow file is the counterpart of .shadow at the null-index level. It records overrides of null patterns for individual records without modifying the main .meta file, keeping the pairing consistent: main file ↔ .meta and shadow file ↔ .meta.shadow.

When it’s created

The .meta.shadow file is created automatically when two conditions are met:

1. The store is of type DEFAULT or POSIXSHD - i.e. one that keeps record modifications in a .shadow file (not in the main file).
2. At least one modification of an existing record (storage::write() at an index other than the maximum) is made during the given session.

Condition 1 is not a mode switch but a choice of class. At store initialization the makeMetaIndex() factory (accessorFactory.hpp) asks the accessor for hasShadow() and returns:

storageShadow derives from metaData and overrides the virtual onRecordModified(), getNullBitset(), and reset(); the .meta.shadow file itself is managed by its metaShadow member. As a result, storage contains no branching on shadow mode.

File format

The .meta.shadow file has no header. It is a sequence of entries in the same binary format as the entries in the .meta file, with the difference that the recordCount field stores the absolute record index (not the number of records in an RLE run):

Every call to onRecordModified() in shadow mode appends one entry to the end of the file. Multiple entries for the same position are allowed - the last entry governs (last-write-wins semantics, matching the .shadow file).

Read priority

storageShadow::getNullBitset(i) scans the list of overrides from the end. If it finds an entry for index i, it returns that entry’s null pattern without consulting the main index (Fig. 22):

flowchart TD
    Q["getNullBitset(i)"]
    Q --> SM{"object is a storageShadow?"}
    SM -->|yes| SCAN{"metaShadow::lookup(i)\n(from the end): entry for i?"}
    SCAN -->|found| RET1["Return the nullBitset from the override\n(the most recent wins)"]
    SCAN -->|not found| MAIN["Look up in the main index\n(RLE segments on disk)"]
    SM -->|no| MAIN
    MAIN --> RET2["Return the pattern from .meta"]

Fig. 22. Null-pattern read priority - main index vs. index shadow

Lifecycle

The .meta.shadow file is managed in parallel with the data shadow file:

Persistence across restarts

After the process restarts, a new storageShadow object restores the shadow state already in its constructor, via metaShadow::load() (Fig. 23):

1. It reads all entries from .meta.shadow (no header - a direct format).
2. It loads them into the override list in write order.
3. getNullBitset() and subsequent calls to onRecordModified() behave exactly as they did before the restart.

%% pdf-width: 100%
sequenceDiagram
    participant Proc1 as First session
    participant MS as .meta.shadow
    participant Meta as .meta

    Proc1->>Meta: onRecordAppended([F,F,F]) × 5
    Proc1->>MS: onRecordModified(2, [T,T,T]) → append entry (index=2)
    Note over Meta: .meta unchanged [allNull×5]
    Note over MS: .meta.shadow: [(index=2, [T,T,T])]

    Note over Proc1: restart

    participant Proc2 as Second session
    Proc2->>MS: storageShadow constructor → metaShadow::load()
    MS-->>Proc2: [(index=2, [T,T,T])]
    Note over Proc2: getNullBitset(2) → [T,T,T]
    Proc2->>Meta: mergeShadow() → applyModificationToMainIndex(2, [T,T,T])
    Proc2->>MS: delete the .meta.shadow file

Fig. 23. Index shadow - restoring null patterns after a restart

Usage example - correcting a record while preserving consistency

# 5 records in stream str1, 3 FLOAT fields
# Record 2 has a null value in field 0: nullBitset=[T,F,F]
# The operator corrects field 0 of record 2 → pattern changes to [F,F,F]

# Operations:
storage.write(rec2_corrected, pos=2)
  → .shadow: append (position=2, data_corrected)
  → storageShadow.onRecordModified(2, [F,F,F])
    → .meta.shadow: append (index=2, [F,F,F])

# File state:
# .meta        - unchanged: [isGap=F, count=2, [F,F,F]],
# >> [isGap=F, count=1, [T,F,F]], [isGap=F, count=2, [F,F,F]]
# .meta.shadow - new entry: [gapFlag=0, recordCount=2, bitset=[F,F,F]]

# Read:
getNullBitset(2) → [F,F,F]  (from .meta.shadow)
getNullBitset(1) → [F,F,F]  (from .meta)

# After merging:
storage.merge() → .shadow absorbed into the main file
storageShadow.mergeShadow() → .meta rebuilt, .meta.shadow deleted
# .meta after merge: [isGap=F, count=5, [F,F,F]]  (all records complete)

NOTE: The .meta.shadow mechanism is covered by the unit tests ut_metaShadow_usage (shadow-file format and lifecycle) and ut_storageShadow_usage (integration with metaData and the accessor).

The relationship between the files

In this section, the relationships between the files are shown at two levels. The structural level describes how the data file carries the records, the .desc descriptor defines their format, the .meta file stores information about null values and transmission gaps, .shadow collects data modifications without destroying the original, and .meta.shadow similarly collects overrides of null patterns. The operational level (Fig. 24) shows the read and write flow: reads check .shadow and .meta.shadow first and fall back to the main file and .meta only when there is no entry. In this way the append, update, and read operations keep the data and metadata consistent throughout the artifact’s lifecycle. Merging with merge() / mergeShadow() is not part of this flow: it is a separate API operation that the engine never invokes on its own (Fig. 21).

%% pdf-width: 100%
graph LR
    UP["update<br/>write(N), N < count"] -->|append| SHD
    AP["append<br/>write(N), N ≥ count"] -->|append at the end| MAIN

    subgraph SHD["shadow layer"]
        direction TB
        S[".shadow<br/>(N·size, data)"]
        MS[".meta.shadow<br/>(N, nullBitset)"]
    end

    subgraph MAIN["main layer"]
        direction TB
        D["main file<br/>data"]
        M[".meta<br/>nullBitset"]
    end

    SHD ==>|"1. entry N exists"| RD["read(pos=N)<br/>data + nullBitset"]
    MAIN -->|"2. no entry N"| RD

Fig. 24. The relationship between an artifact’s write, modify, and read operations (DEFAULT and POSIXSHD types)

Fig. 24 shows the flow of append, update, and read operations through the storage layer, and their direct effect on the data file, .meta, .shadow, and .meta.shadow. The record index decides the kind of write: N equal to or greater than the record count is an append, a smaller one is an update. An entry in .shadow is keyed by a byte offset (N·size, relative to the segment when retention is used), an entry in .meta.shadow by the record index N. A record made up solely of null values outside the nullfill phase does not reach the main file - it leaves only a gap entry in .meta. The shadow layer exists only for the DEFAULT and POSIXSHD types; in the remaining types (POSIX, DIRECT, GENERIC, MEMORY) an update overwrites the record directly in the main file and in .meta.

Starting point - a binary file without metadata

The simplest possible way to record a time series is a sequence of raw values in a binary file: fixed record size, no header, no structure description. This approach has one advantage - minimal overhead - and a number of significant limitations:

• Interpreting the data requires knowledge external to the file (field names, types, order).
• No information about transmission gaps - continuity of the data is only apparent.
• Every modification of a historical record irreversibly destroys the original data.
• A change to the record structure invalidates the entire file.

RetractorDB records data from sensors operating in real time, where power interruptions, signal loss, and the need for retrospective data correction are normal operational occurrences, not exceptions. The five-file structure directly addresses each of these limitations.

What each file contributes

The descriptor (.desc) - self-description and independence from code

A binary data file is useless without knowledge of the record structure. The descriptor stores that knowledge alongside the data, which means:

• Data can be read and interpreted without access to the source code or configuration - the .desc file is enough.
• The xtrdb tool can analyze any artifact without additional parameters.
• Changes to a stream’s structure (adding a field, changing a type) are explicit and versionable.
• The TYPE field in the descriptor determines the storage strategy, allowing the same engine to handle durable artifacts, ephemeral ephemerides, and external data sources without changing the query logic.

The metadata file (.meta) - trustworthiness of the time series

A time series with gaps, treated as continuous, leads to incorrect time-window computations, incorrect aggregations, and false correlations. The .meta file provides:

• The ability to distinguish a record with a zero value from a record that is absent (null) - semantically completely different states.
• Recording of transmission gaps without inserting fake records into the data file - the binary file stays dense and positionally addressable.
• RLE compression - typical time series have long stretches without nulls, so the metadata cost is close to zero for good-quality data.
• The ability to reconstruct the exact recording schedule, including gap lengths, which is necessary when computing intervals in the stream algebra.

The shadow file (.shadow) - non-destructive data correction

In measurement systems, correcting faulty samples after the fact is a standard procedure. Overwriting the binary file is irreversible and destroys the evidence of the original measurement. The shadow file:

• Lets you correct any historical record without modifying the main file.
• Preserves the original measurement as the default - deleting the .shadow file fully restores the initial state.
• Allows merging (merge) corrections into the main file only when that is a deliberate decision by the operator, not a side effect of writing.
• Separates certified data (the main file) from working data (the shadow file), which matters in applications requiring auditability.

The index shadow file (.meta.shadow) - metadata consistency during correction

A correction to a record in the data shadow file must be reflected in the null index - otherwise getNullBitset() would return a stale pattern from the main .meta. The .meta.shadow file:

• Maintains consistency between the pairs: main file ↔ .meta and .shadow ↔ .meta.shadow.
• Lets getNullBitset() return the current null pattern for a corrected record without modifying the main index.
• Tracks the lifecycle of the data shadow file - merged and deleted exactly alongside .shadow.
• Enables full state recovery after a restart: overrides loaded from .meta.shadow are immediately available without re-scanning the data shadow file.