1.2. Async pipelines#

Async pipelines buffer writes in server memory. The server merges compatible writes and writes them to disk in larger operations. This process reduces write amplification for frequent, small inserts.

Use async pipelines for continuous ingestion, such as market data or sensor data. For large and infrequent batches, use the batch inserter with fast insertion mode.

For more information about storage and compaction, see Compaction and Trimming.

1.2.1. Behavior as of 3.14.3#

As of QuasarDB 3.14.3, async pipelines give immediate feedback and immediate data visibility. They also give the client control of flow and retry operations.

When the client sends an async push, the server does these steps:

  1. The server validates the request.

  2. The server checks storage and pipeline capacity.

  3. The server accepts the request into a pipeline or returns an error.

The server returns validation errors immediately. For example, it rejects an invalid schema, column, option, or insertion mode in the client request. The client does not have to find this error later in the server log. Do not retry an invalid request without a correction.

The server also returns an error immediately if it cannot accept the request. This can occur when a pipeline is full or when the persistence layer is full. The server does not wait and retry the push. Thus, the client keeps control. The client can reduce its write rate and retry a valid request when the server can accept it.

Important

Keep the data until the server accepts the request. For a retryable busy or pipeline-full error, use a bounded backoff before you retry. Do not retry in a tight loop.

1.2.2. Data visibility and persistence#

After the server accepts an async push, queries can read the data immediately. The data does not have to wait for a disk write.

A successful async push means that the server accepted the data into memory. It does not mean that the server wrote the data to disk. A disk write can still fail after the server returned success to the client. This late failure cannot be returned in the response to the original push.

If a disk write gets a retryable error, the server retries the write one time. If this retry fails, the server does not retry it again. A non-retryable write error fails without a retry. Monitor write failures and the server log to find late failures.

1.2.3. Flush behavior#

Each pipeline runs in a separate thread. A pipeline does these operations:

  1. It buffers accepted requests in memory.

  2. It merges compatible requests for the same table shard.

  3. It starts a flush when the buffered data reaches half of the configured buffer size or when the flush deadline expires.

  4. It writes the merged requests to disk.

1.2.4. Configuration#

Use these settings in the QuasarDB configuration file:

Setting

Default

Valid values

Effect

local.depot.async_ts.pipelines

1

0 to 1024

Sets the number of pipelines. A value of 0 disables async inserts.

local.depot.async_ts.pipeline_buffer_size

1 GiB on 64-bit systems; 100 MiB on 32-bit systems

1 MiB to 128 GiB

Sets the maximum buffer size for each pipeline.

local.depot.async_ts.flush_deadline

4 s

1 ms to 24 h

Sets the maximum time between pipeline flushes.

The possible pipeline memory use is approximately the number of pipelines multiplied by the buffer size. Make sure that the server has sufficient memory before you increase these settings.

For all command-line arguments and environment variables, see Asynchronous time series inserter.

1.2.5. Client error handling#

Use the error type to select the client action:

Result

Client action

Success

The server accepted the data. Keep monitoring for late disk-write failures.

Invalid request

Correct the request. Do not retry the same input.

Busy or pipeline full

Keep the data, reduce the write rate, and retry with a bounded backoff.

Persistence full

Stop or pause the writes. Retry only after storage becomes available.

An API can report a busy or full pipeline as qdb_e_async_pipe_full or as a retry-later error. The exact exception or error name depends on the client API.

1.2.6. Monitoring#

Monitor these metrics:

  • async_pipelines.busy_denied_count counts requests that the server did not accept because a pipeline was full.

  • async_pipelines.[pipe_nb].buffer_bytes shows the current buffer size for one pipeline.

  • async_pipelines.write.failures.total_count counts late disk-write failures across all pipelines.

  • async_pipelines.write.successes.total_count counts successful disk writes across all pipelines.

  • async_pipelines.flush.total_count counts flushes across all pipelines.

For all pipeline metrics, see Storage - Async pipelines. For dashboard guidance, see Async Pipelines.