QuasarDB stores its configuration options in a qdbd.conf file. The precise location on where this file resides depends on on your method of installation, but is typically /etc/qdb/qdbd.conf.
This section is organised per component, and describes the configuration options relevant to each of those.
You can set a license either by file or by putting the license key as a string directly into your configuration.
Variable |
Description |
|---|---|
|
Absolute location to the license file. Be careful to ensure the license file is readable by the |
|
License key as a string. |
Warning
Improper configuration of the paralellism will negatively impact performances.
QuasarDB has been designed for multicore architectures. Queries coming from clients are processed in parallel into disctinct shards called “partitions”. Each partition has a configurable number of threads.
In most instances, you want to use the unified setting that gives a “budget” to let QuasarDB pick for you the right number of partitions and threads per partitions.
Variable |
Description |
|---|---|
|
How many cores to allocate to process queries. The default of 0 will let QuasarDB pick a number depending on the computing resources available on the computer. |
It’s possible to have total control over the configuration in using the partitions_count and threads_per_partition settings. If these values are not 0, the parallelism setting will be ignored and QuasarDB will use the exact configured number of partitions and threads per partition. It is strongly advised to set partitions_count and threads_per_partition to 0 and use the parallelism setting instead.
Note
The storage layer has independent multithreading settings that are not affected by the settings above. In other words, the cores attributed to storage come in addition to the ones used for queries processing.
Variable |
Description |
|---|---|
|
Specifies which storage engine to use. Accepted values: |
|
Location to the root folder of where your database files are stored. Defaults to |
|
Location to the Write-Ahead-Log (WAL) folder. Defaults to empty, which will make it a subdirectory of the root directory called “wal”. |
|
An array in the form [{“path”: “path1”, “size”: size1}, {“path”: “path2”, “size”: size2}] which specified, which directory, in order should be used for storage. When a storage exceeds capacity, older data is moved to the next storage in the list. Actual usage may exceed the size specified. Size is specified in bytes. |
|
The compaction strategy to use. By default it is “leveled”, but “universal” is also accepted. Universal improves write amplification at the cost of space amplification. |
|
When this option is set to true, writes will not go to the Write-Ahead-Log (WAL), increasing performance at the cost of durability. Writes go to the WAL by default. Use this option with caution. |
|
Should be increased if the daemon is under constant insertion pressure, and there is still available CPU: it increases the amount of threads the storage engine has available for background operations such as compaction. |
|
The number of threads allocated to high-priority operations such as flushes to disk. The default setting of 2 is sufficient for most cases. Increase if the server cannot keep up with the persistence layer. |
|
Size, in bytes, of the Write-Ahead-Log (WAL). The WAL is where every update gets written. When the WAL is full, memtables are flushed to disk. If the WAL is too small, memtables may thus be flushed before they are full, impacting write speed. However, large WALs means increased memory usage and potentially higher compaction. Your WAL should be large enough to absorbs spikes, but cannot be used to compensate for a persistence layer too slow to absorb the load or memtables being too small. See |
|
The number of high-priority threads, in addition to |
|
The size of the block cache, in bytes. Increasing this may increase the performance at the cost of additional memory usage. |
|
The mem budget for each column family. Defaults to 1 GiB. For write heavy usages it’s recommended to significantly increase this value. |
|
How many threads will be dedicated to writing data to disk. Write heavy scenarios may benefit from a higher count. Defaults to |
|
The maximum amount of data to store, in bytes. Inserting data will fail with “quota exceeded” when the limit is reached. If set to 0, no check is done. |
|
A path to a local disk to be used as a persistent cache. May increase performance when data is stored in a remote disk. |
|
The maximum size of the persistent cache, in bytes. Cannot be zero if the persistent cache path is specified. |
|
If your persistent cache is on an NVME, enabling this option may increase performance. |
|
A string either “none” or “aws” specifying your cloud provider to use to store the cloud data. |
|
The size of the local SST cache, in bytes. If 0, no SST file will be kept locally. The default value is 18446744073709551615, which means every SST file will be kept locally and if the files are missing at startup, they will be downloaded in the background. |
|
The timeout used for cloud queries, in milliseconds. The default of 0 means that the cloud provider’s default will be used. |
|
Will create the bucket if it does not exist. |
|
The first half of the bucket name where to store the data. |
|
The second half of the bucket name where to store the data. On AWS, the bucket name must be unique. If both are empty, the database will not upload its content to the cloud. |
|
The first half of the bucket name where to read the data from. |
|
The second half of the bucket name where to read the data from. On AWS, the bucket name must be unique. If both are empty, the database will not read its content from the cloud. |
|
Quasardb will use |
|
The region where the bucket is located. |
|
The AWS access key id to use for authentication. |
|
The secret key to use for authentication. You need both the access key and secret key set for authentication to work. |
|
The configuration file to use for authentication. When using a configuration file, |
|
Use the AWS transfer manager to upload and download files which can greatly increase the performance. |
|
The number of threads to be used by the AWS transfer manager. |
|
The buffer size for the AWS transfer manager. Files larger will be broken into smaller pieces, transfered in parallel. |
The QuasarDB daemon uses TCP for network communications and listens to two ports. The primary port is configurable and is the port used by client applications. Its default value is 2836. The secondary port, is used for cluster discovery mechanisms and queries run on separate, higher priority partitions. This design ensures that when the server is busy processing queries, it can still service cluster management requests and preserve stability.
The value of the secondary port is set to the value of the primary port plus one (1). Thus, by default, the value of the secondary port is 2837.
Variable |
Description |
|---|---|
|
Local address and port to bind to for accepting new connections. Can also refer to network adapter, e.g. |
|
Specify if the local address we bind to is not the same as other nodes or clients should use to connect to the daemon (e.g. when behind a NAT). Defaults to |
|
Time (in milliseconds) until a server socket is automatically recycled. |
|
Time (in milliseconds) until a client connection without activity is automatically disconnected. |
|
Size of the buffer the server allocates for receiving data from clients. Increase this size if you experience unexpected closed connections from the client, or the server logs mention |
|
Size of the buffer the server allocates for sending data to clients. Increase this size if you experience unexpected closed connections from the client, or the server logs mention |
|
Enable changes publications on the firehose. |
|
Endpoint the firehose should listen to subscribers. Only active if the firehose publisher is enabled. |
Variable |
Description |
|---|---|
|
Unique identifier for this node. See also: Node ID configuration. |
|
One or more peers that should be connected to to discover and bootstrap the cluster. Peers should be specified as a JSON array of tuples of address:port, e.g. |
|
Replication factor of data stored inside the cluster. Must be equal to or lower than the amount of nodes in your cluster, and not higher than |
Variable |
Description |
|---|---|
|
Boolean that determines whether security should be enabled. Valid values are |
|
Absolute location to the file that contains the QuasarDB users. Typically should be |
|
Absolute location to the cluster private key file. Typically should be |
|
Boolean that determines whether node-to-node communication should be encrypted. Valid values are |
Variable |
Description |
|---|---|
|
How many sessions (connections) each server thread is able to handle concurrently. Too few will cause significant performance degradations. Increase this value if you see “out of free server sessions” messages in the logs. |
|
The number of threads in the main worker pool. Handles blocking I/O operations, and should be increased if your clients are waiting on the server but the server is not at 100% CPU. These are the queries handled by the main port. |
|
The number of threads dedicated for high-priority, service, messages. The default value of 2 is adequate for most setups and should be only increased for cluster with more than 20 nodes. These are the queries handled by the secondary port. |
|
Maximum number of open files that the storage engine is allowed to use. This should typically be close to the number of files the |
|
The maximum amount of memory (in bytes) QuasarDB will use. Any usage above this will cause QuasarDB to evict entries from its cache. |
|
Same like |
Variable |
Description |
|---|---|
|
An integer representing verbosity. Use |
|
Absolute path where log files will be written into. Defaults to |
|
Enables performance profiling from the server side, so that clients can start running performance traces. It has a small performance impact, and is disabled by default. |
|
Enables runtime statistics collected server side. It has a small performance impact, and is disabled by default. |
|
Time (in milliseconds) between refreshes of the statistics. Should typically be set to the interval in which you query the node statistics; e.g. if you only poll your statistics once a minute, this value should be |
The asynchronous timeseries inserter is a mechanism in QuasarDB to buffer inserts server-side, which you can tune using the configuration options below.
Variable |
Description |
|---|---|
|
The number of asynchronous time series pipelines. Increase this number if you are experiencing slow inserts, but CPU is not yet at 100%. |
|
Maximum size of the buffer in bytes for each of the pipelines. Increase this value to if you are experiencing slow inserts and CPU is at 100%. |
|
Maximum number of entries to buffer for each of the pipelines. Increase this value to if you are experiencing slow inserts and CPU is at 100%. |
|
Maximum time duration between consequent pipeline flushes. |
The QuasarDB Rest Server qdb_rest can be configured using either command line arguments, a json config file or environmental variables. See below for examples of each configuration as well as the various configuration options:
# Environment variable based configuration
QDB_CLUSTER_URI=qdb://192.168.0.1:2837 qdb_rest
# Command line based configuration
qdb_rest --cluster=qdb://192.168.0.1:2837
# File based configuration
qdb_rest --gen-config > rest.conf # Generate a new config file and edit
qdb_rest --config-file rest.conf
Command Line Variable |
Environment Variable |
Description |
Default |
|---|---|---|---|
|
Allowed origins for cross origin resource sharing |
[] |
|
|
QDB_CLUSTER_URI |
URI of the cluster to connect to |
qdb://127.0.0.1:2836 |
|
QDB_CLUSTER_PUBLIC_KEY_FILE |
Key file used for cluster security |
|
|
TLS_CERTIFICATE |
The certificate to use for secure connections |
|
|
TLS_PRIVATE_KEY |
The private key to use for secure conections |
|
|
TLS_PORT |
The port to listen on for secure connections |
40493 |
|
HOST |
The IP to listen on |
localhost |
|
PORT |
The port to listen on for insecure connections |
40080 |
|
QDB_REST_LOG_FILE |
The path to the log file |
/var/log/qdb/qdb_rest.log |
|
QDB_REST_ASSETS_DIR |
The path to the directory served by the rest api |
/var/lib/qdb/assets |
As it can a bit tedious to edit a configuration file inside a Docker container, the bureau14/qdb docker container provides several environment variables you can use to configure the most common configuration options.
Below you find an overview of the different environment variables the image supports:
Variable |
Description |
Example usage |
|---|---|---|
|
QuasarDB license key (as string) |
docker run -d \
-e QDB_LICENSE="$your_license_key" \
bureau14/qdb
|
|
QuasarDB license key (as file) |
docker run -d \
-v /path/to/my/qdb.key:/qdb.key \
-e QDB_LICENSE_FILE=/qdb.key \
bureau14/qdb
|
|
Controls replication factor, defaults to 1 (no replication) |
docker run -d \
-e QDB_REPLICATION=3 \
bureau14/qdb
|
|
Controls the soft memory limit, defaults to 50% of host memory. |
docker run -d \
-e QDB_MEMORY_LIMIT_SOFT=17179869184 \
-e QDB_MEMORY_LIMIT_HARD=25769803776 \
bureau14/qdb
|
|
Controls the hard memory limit, defaults to 75% of host memory. |
docker run -d \
-e QDB_MEMORY_LIMIT_SOFT=17179869184 \
-e QDB_MEMORY_LIMIT_HARD=25769803776 \
bureau14/qdb
|
|
If “true”, enables security; requires private key and user list to be specified |
docker run -d \
-e QDB_ENABLE_SECURITY=true \
-e QDB_CLUSTER_PRIVATE_KEY=... \
-e QDB_USER_LIST=... \
bureau14/qdb
|
|
QuasarDB cluster private key (as string) |
docker run -d \
-e QDB_ENABLE_SECURITY=true \
-e QDB_CLUSTER_PRIVATE_KEY=... \
-e QDB_USER_LIST=... \
bureau14/qdb
|
|
QuasarDB cluster private key (as file) |
docker run -d \
-e QDB_ENABLE_SECURITY=true \
\
-v /path/to/my/qdb/private.key:/private.key \
-e QDB_CLUSTER_PRIVATE_KEY_FILE=/private.key \
\
-e QDB_USER_LIST=... \
bureau14/qdb
|
|
QuasarDB user list (as string) |
docker run -d \
-e QDB_ENABLE_SECURITY=true \
-e QDB_CLUSTER_PRIVATE_KEY=... \
-e QDB_USER_LIST=... \
bureau14/qdb
|
|
QuasarDB user list (as file) |
docker run -d \
-e QDB_ENABLE_SECURITY=true \
-e QDB_CLUSTER_PRIVATE_KEY=... \
\
-v /path/to/my/qdb/user.list:/user.list \
-e QDB_USER_LIST_FILE=/user.list \
\
bureau14/qdb
|