quasardb daemon#

Introduction#

The quasardb daemon is a highly scalable data repository that handles requests from multiple clients. The data is cached in memory and persisted on disk. It can be distributed on several servers to form a cluster.

The quasardb daemon does not require privileges (unless listening on a port under 1024) and can be launched from the command line. From this command line it can safely be stopped with CTRL-C. On UNIX, CTRL-Z will also result in the daemon being suspended.

Important

Without a valid license (see License), quasardb will run in “community edition” mode. The community edition is limited to 16 GiB of storage and 4 GiB of RAM per node, with a maximum of two nodes per cluster.

Quick Reference#

Configuration#

Global and local options#

When a node connects to a ring, it will first download the configuration of this ring and overwrite its parameters with the ring’s parameters.

This way, you can be sure that parameters are consistent over all the nodes. This is especially important for parameters such as replication where you need all nodes to agree on a single replication factor.

This is also important for persistence as having a mix of transient and non-transient nodes will result in undefined behaviour and unwanted data loss. Transient nodes store no data on disk while RocksDB nodes persist data, creating inconsistent cluster behavior during recovery and replication.

However, not all options are taken from the ring. It makes sense to have a heterogenous logging threshold for example, as you may want to analyze the behaviour of a specific part of your cluster.

In addition, some parameters are node specific, such as the listening address or the node ID.

An option that applies cluster-wide is said to be global whereas other options are said to be local. The value of a global option is set by the first node that creates the ring, and all other nodes copy these parameters. Local options come from each node’s server configuration, whether supplied through the configuration file, command-line arguments, or environment variables. See the server configuration reference for the current option names and forms.

Network distribution#

qdbd distribution is peer-to-peer. This means:

The unavailability of one server does not compromise the whole cluster

The memory load is automatically distributed amongst all instances within a cluster

Each server within one cluster needs:

An unique address on which to listen (you cannot use the any address) (-a)

At least one node within the cluster to contact (--peer)

Note

It’s counter-productive to run several instances on the same node. qdbd is hyper-scalar and will be able to use all the memory and processors of your server. The same remark applies for virtual machines: running quasardb multiple times in multiple virtual machines on a single physical server will not increase the performances.

The daemon will automatically launch an appropriate number of threads to handle connection accepts and requests, depending on the actual hardware configuration of your server.

Logging#

By default, a non-daemonized qdbd will log to the console. If daemonized, logging is disabled unless configured to log to files (--log-directory) or to the syslog (--log-syslog) on Unix.

There are six different log levels: detailed, debug, info, warning, error and panic. You can change the log level (--log-level), it defaults to info.

You can also change the log flush interval (--log-flush-interval), which defaults to 3,000 ms.

Cache#

QuasarDB caches data in RAM in addition to the cache provided by the persistence layers. QuasarDB avoids loading data if it can answer a query based on information contained in the indexes, and will use a LRU strategy for timeseries buckets. Inserting data in parallel of querying it is well supported and will not pollute the cache.

The daemon will start to evict entries when the process memory usage reaches the soft limit. It will stop all processing and evict has much as it can when the process memory usage hits the hard limit. When QuasarDB evicts entries, it will throttle down queries to prevent a situation where users would load data faster than QuasarDB could evict it.

Thus, the memory usage is kept between the soft and the hard limit, possibly below the soft limit.

The memory usage measurement is based on the process size, which means that file system caches, and persistence layer caches are included in this measurement. It is thus important to ensure that the soft limit is well above the sum of all caches of the persistence layer. Failure to do so may result in continuous eviction and poor performance.

By default, the hard limit is 50% of the physical RAM available to the process, including any container memory limit. The soft limit is half of the hard limit.

On a machine with 64 GiB of available RAM, the hard limit is therefore 32 GiB and the soft limit is 16 GiB.

Each parameter can be configured independently, the hard limit must always be greater than the soft limit.

Ideally, you want your working set to fit in memory. The working set for a single node is close to the total working set divided by the number of nodes.

Note

The cache size has a huge impact on performance. Your QuasarDB solutions architect will be happy to assist you in finding the optimal setting for your usecase.

Data Storage#

Note

Data storage options are global for any given ring.

QuasarDB has two storage engines:

RocksDB (default) - Persistent storage for production use
Transient - In-memory storage for automated testing only (see Transient Storage Engine)

Entries are often kept resident in a write cache so the daemon can rapidly serve a large amount of simultaenous requests. Data may not be synced to the disk at all times.

The RocksDB persistence engine allows you to sync every write to disk, if needed, thanks to the “sync” option setting.

For more information, see Data Storage and Data Transfer.

RocksDB#

RocksDB is an open-source, persistent, key-value store for fast storage environments. It is based on LevelDB and uses LSM trees.

To enable RocksDB, one sets the storage engine configuration parameter to “rocksdb” and set a “root” directory in the rocksdb configuration section. QuasarDB will then write the data under this directory using our tuned RocksDB implementation. The data stored is 100% compatible with RocksDB and can be used via the RocksDB API, if needed.

The directory can be absolute or relative, for production we recommend using absolute directory.

It is possible to limit the amount of space a node will occupy with the “max_size” option. The writes to the node will fail when the disk usage reaches that limit, warnings being emitted before that point. The write-ahead log is not accounted in the space usage meaning that the actual disk usage may be greater than the limit. Compression may also reduce the actual disk usage.

Note

RocksDB is a safe default, however it can limit the performance of your QuasarDB cluster.

Persistent read cache#

Note

The persistent read cache is only available for the RocksDB persistence layer.

The persistent read cache optimizes I/O in buffering data from a (potentially remote) storage into a faster, local storage (See Persistent read cache). The persistent read cache does not buffer writes.

There are three configuration settings:

The path of the persistent read cache. It should be used on local, fast storage only (SSD, NVMe, or Optane). Using remote or slow storage will have a detrimental effect on performance.

The maximum size of the persistent read cache

Should the persistent read cache be optimized for NVMe.

The persistent read cache is disabled by default.

Partitions#

A partition can be seen as a worker thread. The more partitions, the more work can be done in parallel. However if the number of partitions is too high relative to your server capabilities to actually do parallel work, performance will decrease.

quasardb is highly scalable and partitions do not interfere with each other. The daemon’s scheduler will assign incoming requests to the partition with the least workload.

The ideal number of partitions is close to the number of physical cores your server has. By default the daemon chooses the best compromise it can. If this value is not satisfactory, you can use the partitions_count config file option to set the value manually.

Note

Unless a performance issue is identified, it is best to let the daemon compute the partition count.

Asynchronous time series inserter#

The server has an asynchronous time series inserter. It buffers compatible batch inserts in memory and writes merged operations to disk.

As of QuasarDB 3.14.3, the server validates an async push and returns the result immediately. If a pipeline is full, the server does not wait and retry the push. It returns control to the client. The client can reduce its write rate and retry with a bounded backoff.

Accepted data is visible to queries before the disk write. A successful push means that the server accepted the data into memory. A disk write can still fail later.

For validation, retry, persistence, configuration, and monitoring details, see Async pipelines.

Statistics#

Runtime statistics are enabled by default and can be disabled with global.cluster.enable_statistics. They cannot be enabled with the transient storage engine. Statistics are stored in blobs or integer keys for convenient consumption and represent the value at the latest refresh. Configure the normal snapshot interval with global.cluster.statistics_refresh_interval.

Host, resident-cache, and persistence memory gauges use the normal statistics snapshot. Periodic TBB allocator gauges have an additional control and are not collected by default. Set local.limiter.memory_statistics_refresh to at least 1000 milliseconds to enable that allocator instrumentation. A value of 0 leaves it disabled.

Statistic keys have the form $qdb.statistics.{node id}.{stat name}. For example, for node ID 1-0-0-0, the accumulated number of bytes written to disk is stored in $qdb.statistics.1-0-0-0.persistence.bytes_written.

For the complete, maintained list of metrics, types, and descriptions, see the Metrics reference. For all related settings, see Configuration.

Performance data#

When performance profiling is enabled, each request will store an accumulator of the time spent, in nanoseconds, in each step of the process.

The performance metrics are stored in the “$qdb.statistics.{node id}.perf.” subfield.

Operating limits#

Theoretical limits#

Entry size: An entry cannot be larger than the amount of virtual memory available on a single node. This ranges from several megabytes to several gigabytes depending on the amount of physical memory available on the system. It is recommended to keep entries size well below the amount of available physical memory.
Key size: As it is the case for entries, a key cannot be larger than the amount of virtual memory available on a single node.
Number of nodes in a grid: The maximum number of nodes is $2^{63}$ (9,223,372,036,854,775,808)
Number of entries on a single grid: The maximum number of entries is $2^{63}$ (9,223,372,036,854,775,808)
Node maximum capacity: The node capacity depends on the available disk space on a given node. The community edition is limited to 16 GiB on disk and 4 GiB in RAM.
Total amount of data: The total amount of data a single grid may handle is 16 EiB (that’s 18,446,744,073,709,551,616 bytes).

Practical limits#

Entry size: Very small entries (below a hundred bytes) do not offer a very good throughput because the network overhead is larger than the payload. This is a limitation of TCP. Very large entries (larger than 10% of the node RAM) impact performance negatively and are probably not optimal to store on a quasardb cluster “as is”. It is generally recommended to slice very large entries in smaller entries and handle reassembly in the client program. If you have a lot of RAM (several gigabytes per node) do not be afraid to add large entries to a quasardb cluster. For optimal performance, it’s better if the “hot data” - the data that is frequently acceded - can fit in RAM.
Simultaneous clients: A single instance can serve thousands of clients simultaneously. The actual limit is the network bandwidth, not the server. You can set the -s to a higher number to handle more simultaneous clients per node. Also you should make sure the clients connects to the nodes of the cluster in a load-balanced fashion.

Parameters Reference#

Parameters can be supplied in any order and are prefixed with --. The arguments format is parameter dependent.

Instance specific parameters only apply to the instance, while global parameters are for the whole ring. Global parameters are applied when the first instance of a ring is launched.

Instance specific#

-h, --help#

Displays basic usage information.

Example

To display the online help, type:

qdbd --help

-v, --version#: Displays qdbd version information.

--gen-config#

Generates a JSON configuration file that omits settings whose values match their defaults, then prints it to standard output.

Example

To create a new config file with the name “qdbd_default_config.json”, type:

qdbd --gen-config > qdbd_default_config.json

--gen-complete-config#

Generates a complete JSON configuration file including default values and prints it to standard output. Use this output to inspect the settings and defaults supported by the installed server version.

Example

To create a complete configuration named qdbd.conf:

qdbd --gen-complete-config > qdbd.conf

-c, --config#

Specifies a configuration file to use. See Configuration file.

Argument

The path to a valid configuration file.

Example

To use a configuration file named “qdbd_default_config.json”, type:

qdbd --config=qdbd_default_config.json

-d, --daemonize#

Runs the server as a daemon (UNIX only). In this mode, the process will fork and prevent console interactions. This is the recommended running mode for UNIX environments.

Example

To run as a daemon:

qdbd -d

Note

Logging to the console is not allowed when running as a daemon.

--license-file#

Specifies the location of the license file. A valid license is required to run the daemon (see License).

Argument

The path to a valid license file.

Default value

Empty. Without a license file or inline license key, the daemon runs in free mode.

Example

Load the license from license.txt:

qdbd --license-file=license.txt

-a <address>:<port>, --address=<address>:<port>#

Specifies the address and port on which the server will listen.

Argument

A string representing one address the server listens on and a port. The address string can be a host name, an IP address or an interface (BSD and Linux only).

Default value

127.0.0.1:2836, the IPv4 localhost and the port 2836

Example

Listen on localhost and the port 5910:

qdbd --address=localhost:5910

Listen on eth0 port 2836:

qdbd --address=eth0:2836

Note

The unspecified address (0.0.0.0 for IPv4, :: for IPv6) is not allowed.

-s <count>, --sessions=<count>#

Deprecated. Specifies simultaneous sessions per partition. Use --total-sessions instead and do not set both options.

Argument

A nonnegative number of simultaneous sessions per partition. 0 selects automatic session sizing.

Default value

0 (automatic)

Example

Allow 10,000 simultaneous session:

qdbd --sessions=10000

Note

The sessions count determines the number of simultaneous clients the server may handle at any given time. Increasing the value increases the memory load. This value may be limited by your license.

--total-sessions=<count>#

Specifies the maximum total number of concurrent server connections.

Argument: A nonnegative integer below 1,000,000. 0 selects an automatic value based on the resolved partition and thread counts.
Default value: 0 (automatic)

--idle-timeout=<duration>#

Sets the timeout after which inactive sessions will be considered for termination.

Argument

An integer representing the number of milliseconds after which an idle session will be considered for termination.

Default value

900,000 (900 seconds, 15 minutes)

Example

Set the timeout to one minute:

qdbd --idle-timeout=60000

--request-timeout=<timeout>#

Sets the timeout after which a request from the server to another server must be considered to have timed out.

Argument

An integer representing the number of milliseconds after which a request must be considered to have timed out.

Default value

300,000 (300 seconds, 5 minutes)

Example

Set the timeout to two minutes:

qdbd --request-timeout=120000

--peer=<address>:<port>#

The address and port of a peer to which to connect within the cluster. It can be any server belonging to the cluster.

Argument

The address and port of a machines where a quasardb daemon is running. The address string can be a host name or an IP address.

Default value

None

Example

Join a cluster where the machine 192.168.1.1 listening on the port 2836 is already connected:

qdbd --peer=192.168.1.1:2836

--id=<id string>#

Sets the node ID.

Argument

A string representing the ID to of the node. This can either be a 256-bit number in hexadecimal form, the value “random” and use the indexed syntax. This value may not be zero (0-0-0-0).You are strongly encouraged to use the indexed syntax. See Clustering.

Default value

Unique random value.

Example

Set the node ID to 1-a-2-b:

qdbd --id=1-a-2-b

Set the node ID to a random value:

qdbd --id=random

Set the node to the ideal value for the third node of a cluster totalling 8 nodes:

qdbd --id=3/8

Warning

Having two nodes with the same ID on the ring leads to undefined behaviour. By default the daemon generates an ID that is guaranteed to be unique on any given ring. Only modify the node ID if the topology of the ring is unsatisfactory and you are certain no two node IDs are the same.

-l <path>, --log-directory=<path>#

Logs in the designated directory.

Argument

A string representing a path to a directory where log files will be created.

Example

qdbd --log-directory=/var/log/qdb

--log-syslog#: UNIX only, activates logging to syslog.

--log-level=<value>#

Specifies the log verbosity.

Argument

A string representing the amount of logging required. Must be one of:

detailed (most output)

debug

info

warning

error

panic (least output)

Default value

info

Example

Request a debug level logging:

qdbd --log-level=debug

--log-flush-interval=<delay>#

How frequently log messages are flushed to output, in milliseconds.

Argument

An integer representing the number of milliseconds between each flush.

Default value

3,000

Example

Flush the log every minute:

qdbd --log-flush-interval=60000

--rocksdb-max-open-files=<count>#

Sets the maximum number of open files for the RocksDB persistence layer. An explicit value must fit within the operating system’s file-descriptor limit.

Argument

An integer representing the maximum number of open files at any point in time.

Default value

0 (automatic detection)

Example

Increase the number to 10,240:

qdbd --rocksdb-max-open-files=10240

--rocksdb-max-bytes=<size-in-bytes>#

Sets the maximum RocksDB size for each node. Once the configured size is reached, storage operations fail with a no-space error. The write-ahead log is not included in this limit.

Argument

A nonnegative size in bytes. 0 means no configured limit.

Default value

0 (unlimited)

Example

Limit RocksDB data on each node to 12 TiB:

qdbd --rocksdb-max-bytes=13194139533312

--limiter-max-bytes-hard=<value>#

The hard limit after which the system will take drastic measures to lower memory usage. When the hard limit is reached, processing is temporarly stopped, every entry is evicted from memory, file cache is flushed, and every buffer is purged to free memory.

The threshold is evaluated against the process’s resident-memory size.

Argument

An integer representing the hard limit, in bytes.

Default value

0 (automatic, 50% of the physical memory available to the process, including a container memory limit).

Example

To allow only 100 KiB of entries:
qdbd --limiter-max-bytes-hard=102400
To allow up to 8 GiB:
qdbd --limiter-max-bytes-hard=8589934592

Note

This value has a huge impact on performance.

--limiter-max-bytes-soft=<value>#

The limit after which the system will start to evict entries. When the soft limit, queries will be throttled down to let the system clear memory. The soft limit must always be lower than the hard limit.

The threshold is evaluated against the process’s resident-memory size.

Argument

An integer representing the soft limit, in bytes.

Default value

0 (automatic, half of the resolved hard limit).

Example

To allow only 100 KiB of entries:
qdbd --limiter-max-bytes-soft=102400
To allow up to 8 GiB:
qdbd --limiter-max-bytes-soft=8589934592

Note

This value has a huge impact on performance.

-r <path>, --rocksdb-root=<path>#

Specifies the directory where data will be persisted when using the RocksDB storage engine for the node where the process has been launched.

Argument

A string representing a full path to the directory where data will be persisted.

Default value

The “db” subdirectory relative to the current working directory.

Example

Persist data in /var/quasardb/db

qdbd --rocksdb-root=/var/quasardb/db

Note

Although this parameter is global, the directory refers to the local node of each instance.

--security=<boolean>#

Enables or disables cluster security.

Argument

A boolean specifying whether security should be enabled.

Default value

False

Example

To disable security completely:

qdbd --security=false

Note

Security requires a cluster private key and a user list.

--cluster-private-file=<path>#

A path to the cluster private key file.
Argument
A string representing a full path to the cluster private key file.

Example
Use the file /etc/qdbd/cluster_private.key:
qdbd --cluster-private-file=/etc/qdbd/cluster_private.key
Note

A cluster private key file is required for security to work (see quasardb cluster key generator).

--user-list=<path>#

A path to the user lists containing the user names and their respective public key in JSON format.

Example

Use the file /etc/qdbd/users.cfg:

qdbd --user-list=/etc/qdbd/users.cfg

Note

A users list is required for security to work (see quasardb user adder).

Global#

--replication=<factor>#

Specifies the replication factor (global parameter). For more information, see Data replication.

Argument

A positive integer between 1 and 5 (inclusive) specifying the replication factor. If the integer is higher than the number of nodes in the cluster, it will be automatically reduced to the cluster size.

Default value

1 (replication disabled)

Example

Have one copy of every entry in the cluster:

qdbd --replication=2

--storage_engine=<engine>#

Specifies the storage engine.

Argument

A string representing the storage engine to use. Can be ‘transient’ (for automated testing only) or ‘rocksdb’ (production use).

Default value

“rocksdb”

Example

Use the transient storage engine for automated tests:

qdbd --storage_engine=transient

Warning

The transient storage engine is designed exclusively for automated testing. It provides no data persistence and is not suitable for production use. See Transient Storage Engine for details.

Configuration file#

The daemon accepts a JSON configuration file through -c. Generate a version-matched template instead of copying a static default from the documentation:

qdbd --gen-complete-config > qdbd.conf

The generated file contains the settings and defaults supported by the installed server. Settings can also be supplied through command-line arguments and environment variables. For the maintained mapping between all three forms, see the server configuration reference.

Performance considerations#

Persistence#

RocksDB defaults are suitable for getting started, but production settings should reflect the storage hardware and durability requirements.

Important settings to look at:

local.depot.rocksdb.disable_wal trades durability for write performance.
local.depot.rocksdb.max_open_files must be compatible with the operating system’s file-descriptor limit. A low value makes RocksDB reopen files more often.
local.depot.rocksdb.sync_every_write adds a synchronous disk operation to every write and can significantly reduce write throughput.

See Storage and the RocksDB tuning guide before changing lower-level cache or threading settings.

Network#

Size the network settings for the expected number of concurrent connections and the available CPU resources.

Important settings to look at:

local.network.total_server_sessions controls the maximum total concurrent connections. A value of 0 selects automatic sizing. The older local.network.server_sessions setting is deprecated.
local.network.parallelism is the preferred control for allocating query processing cores. Use local.network.partitions_count and local.network.threads_per_partition only when manual control is required.
local.network.idle_timeout controls when inactive server sockets can be recycled.

See Parallelism and Performance tuning for the current guidance.

Memory management#

QuasarDB loads data from disk when needed and evicts cached entries when memory usage exceeds the configured thresholds.

If the eviction thresholds are too low, QuasarDB will not properly use all the memory available and may spend a lot of time paging in and out entries.

If the evictions thresholds are too high, system memory may be exhausted, which at best results in dramatic performance loss and at worst in operating system failure.

Use local.limiter.max_bytes_soft and local.limiter.max_bytes_hard to control eviction and the emergency memory limit. The percentage settings in the performance configuration determine when the corresponding actions begin. The soft limit must remain below the hard limit.

local.limiter.memory_statistics_refresh is separate: it controls periodic TBB allocator instrumentation and does not define an eviction threshold. That instrumentation is disabled when its value is 0; the minimum enabled interval is one second.

Security#

Authentication has little to no impact on server performance, however, encrypt the traffic will cap the bandwidth to the encryption speed of your server. QuasarDB uses AEGIS-256 for traffic encryption. On 10 GBit networks this can result in an observable drop in maximum transfer bandwidth.