2. Configuration

2. Configuration

2.1. QuasarDB Daemon

QuasarDB stores its configuration options in a qdbd.conf file. The precise location on where this file resides depends on on your method of installation, but is typically /etc/qdb/qdbd.conf.

This section is organised per component, and describes the configuration options relevant to each of those.

2.1.1. License

You can set a license either by file or by putting the license key as a string directly into your configuration.

Variable

Description

local.user.license_file

Absolute location to the license file. Be careful to ensure the license file is readable by the qdb user, otherwise the license will not activate.

local.user.license_key

License key as a string.

2.1.2. Parallelism

Warning

Improper configuration of the paralellism will negatively impact performances.

QuasarDB has been designed for multicore architectures. Queries coming from clients are processed in parallel into disctinct shards called “partitions”. Each partition has a configurable number of threads.

In most instances, you want to use the unified setting that gives a “budget” to let QuasarDB pick for you the right number of partitions and threads per partitions.

Variable

Description

local.network.parallelism

How many cores to allocate to process queries. The default of 0 will let QuasarDB pick a number depending on the computing resources available on the computer.

It’s possible to have total control over the configuration in using the partitions_count and threads_per_partition settings. If these values are not 0, the parallelism setting will be ignored and QuasarDB will use the exact configured number of partitions and threads per partition. It is strongly advised to set partitions_count and threads_per_partition to 0 and use the parallelism setting instead.

Note

The storage layer has independent multithreading settings that are not affected by the settings above. In other words, the cores attributed to storage come in addition to the ones used for queries processing.

2.1.3. Storage

Variable

Description

global.cluster.storage.engine

Specifies which storage engine to use. Accepted values: rocksdb or transient. Defaults to rocksdb.

local.depot.rocksdb.root

Location to the root folder of where your database files are stored. Defaults to /var/lib/qdb.

local.depot.rocksdb.wal

Location to the Write-Ahead-Log (WAL) folder. Defaults to empty, which will make it a subdirectory of the root directory called “wal”.

local.depot.tiered_storage

An array in the form [{“path”: “path1”, “size”: size1}, {“path”: “path2”, “size”: size2}] which specified, which directory, in order should be used for storage. When a storage exceeds capacity, older data is moved to the next storage in the list. Actual usage may exceed the size specified. Size is specified in bytes.

local.depot.rocksdb.compaction

The compaction strategy to use. By default it is “leveled”, but “universal” is also accepted. Universal improves write amplification at the cost of space amplification.

local.depot.rocksdb.disable_wal

When this option is set to true, writes will not go to the Write-Ahead-Log (WAL), increasing performance at the cost of durability. Writes go to the WAL by default. Use this option with caution.

local.depot.rocksdb.threads

Should be increased if the daemon is under constant insertion pressure, and there is still available CPU: it increases the amount of threads the storage engine has available for background operations such as compaction.

local.depot.rocksdb.hi_threads

The number of threads allocated to high-priority operations such as flushes to disk. The default setting of 2 is sufficient for most cases. Increase if the server cannot keep up with the persistence layer.

local.depot.max_total_wal_size

Size, in bytes, of the Write-Ahead-Log (WAL). The WAL is where every update gets written. When the WAL is full, memtables are flushed to disk. If the WAL is too small, memtables may thus be flushed before they are full, impacting write speed. However, large WALs means increased memory usage and potentially higher compaction. Your WAL should be large enough to absorbs spikes, but cannot be used to compensate for a persistence layer too slow to absorb the load or memtables being too small. See local.depot.rocksdb.table_mem_budget.

local.depot.rocksdb.hi_threads

The number of high-priority threads, in addition to local.depot.rocksdb.threads. Should typically be left to 2.

local.depot.rocksdb.data_cache

The size of the block cache, in bytes. Increasing this may increase the performance at the cost of additional memory usage.

local.depot.rocksdb.table_mem_budget

The mem budget for each column family. Defaults to 1 GiB. For write heavy usages it’s recommended to significantly increase this value.

local.depot.rocksdb.max_background_flushes

How many threads will be dedicated to writing data to disk. Write heavy scenarios may benefit from a higher count. Defaults to 1`.

local.depot.rocksdb.max_bytes

The maximum amount of data to store, in bytes. Inserting data will fail with “quota exceeded” when the limit is reached. If set to 0, no check is done.

local.depot.rocksdb.persistent_cache_path

A path to a local disk to be used as a persistent cache. May increase performance when data is stored in a remote disk.

local.depot.rocksdb.persistent_cache_size

The maximum size of the persistent cache, in bytes. Cannot be zero if the persistent cache path is specified.

local.depot.rocksdb.persistent_cache_nvme_optimization

If your persistent cache is on an NVME, enabling this option may increase performance.

local.depot.rocksdb.cloud.provider

A string either “none” or “aws” specifying your cloud provider to use to store the cloud data.

local.depot.rocksdb.cloud.local_sst_cache_size

The size of the local SST cache, in bytes. If 0, no SST file will be kept locally. The default value is 18446744073709551615, which means every SST file will be kept locally and if the files are missing at startup, they will be downloaded in the background.

local.depot.rocksdb.cloud.request_timeout

The timeout used for cloud queries, in milliseconds. The default of 0 means that the cloud provider’s default will be used.

local.depot.rocksdb.cloud.create_bucket_if_missing

Will create the bucket if it does not exist.

local.depot.rocksdb.cloud.bucket.destination_bucket

The bucket name where to store the data. On AWS, the bucket name must be unique. If name is empty, the database will not upload its content to the cloud.

local.depot.rocksdb.cloud.bucket.source_bucket

The bucket name where to read the data from. On AWS, the bucket name must be unique. If name is empty, the database will not read its content from the cloud.

local.depot.rocksdb.cloud.bucket.path_prefix

Quasardb will use local.depot.rocksdb.root as the path inside the bucket to store the data in the cloud. If path_prefix is not empty, a prefix will be added to this path. That way, different nodes can use the same local path and write to different paths within a same bucket.

local.depot.rocksdb.cloud.bucket.region

The region where the bucket is located.

local.depot.rocksdb.cloud.aws.access_key_id

The AWS access key id to use for authentication.

local.depot.rocksdb.cloud.aws.secret_key

The secret key to use for authentication. You need both the access key and secret key set for authentication to work.

local.depot.rocksdb.cloud.aws.config_file

The configuration file to use for authentication. When using a configuration file, access_key_id and secret_key must be left blank.

local.depot.rocksdb.cloud.aws.use_transfer_manager

Use the AWS transfer manager to upload and download files which can greatly increase the performance.

local.depot.rocksdb.cloud.aws.transfer_manager_threads

The number of threads to be used by the AWS transfer manager.

local.depot.rocksdb.cloud.aws.transfer_manager_buffer_size

The buffer size for the AWS transfer manager. Files larger will be broken into smaller pieces, transfered in parallel.

2.1.4. Networking

The QuasarDB daemon uses TCP for network communications and listens to two ports. The primary port is configurable and is the port used by client applications. Its default value is 2836. The secondary port, is used for cluster discovery mechanisms and queries run on separate, higher priority partitions. This design ensures that when the server is busy processing queries, it can still service cluster management requests and preserve stability.

The value of the secondary port is set to the value of the primary port plus one (1). Thus, by default, the value of the secondary port is 2837.

Variable

Description

local.network.listen_on

Local address and port to bind to for accepting new connections. Can also refer to network adapter, e.g. eth0:2836. Defaults to 127.0.0.1:2836. The daemon will listen to the specified port and the port + 1 for service messages. For example, if the server is configured to listen to 127.0.0.1:2836 it will also listen to 127.0.0.1:2837.

local.network.advertise_as

Specify if the local address we bind to is not the same as other nodes or clients should use to connect to the daemon (e.g. when behind a NAT). Defaults to 127.0.0.1:2836.

local.network.idle_timeout

Time (in milliseconds) until a server socket is automatically recycled.

local.network.client_timeout

Time (in milliseconds) until a client connection without activity is automatically disconnected.

local.network.max_in_buffer_size

Size of the buffer the server allocates for receiving data from clients. Increase this size if you experience unexpected closed connections from the client, or the server logs mention cannot send data: Broken pipe.

local.network.max_out_buffer_size

Size of the buffer the server allocates for sending data to clients. Increase this size if you experience unexpected closed connections from the client, or the server logs mention cannot send data: Broken pipe.

global.cluster.publish_firehose

Enable changes publications on the firehose.

local.network.firehose_endpoint

Endpoint the firehose should listen to subscribers. Only active if the firehose publisher is enabled.

2.1.5. Clustering

Variable

Description

local.chord.node_id

Unique identifier for this node. See also: Node ID configuration.

local.chord.bootstrapping_peers

One or more peers that should be connected to to discover and bootstrap the cluster. Peers should be specified as a JSON array of tuples of address:port, e.g. ["192.168.1.1:2836", "192.168.1.2:2836"].

local.chord.replication_factor

Replication factor of data stored inside the cluster. Must be equal to or lower than the amount of nodes in your cluster, and not higher than 4. A replication factor of 1 means no replication.

2.1.6. Security

Variable

Description

global.security.enabled

Boolean that determines whether security should be enabled. Valid values are true and false. Defaults to false. Enabling this to true requires you to set valid values for user_list and cluster_private_file.

global.security.user_list

Absolute location to the file that contains the QuasarDB users. Typically should be /etc/qdb/users.txt. The file can be created and managed using the QuasarDB user management tool, and needs to be identically replicated accross all QuasarDB daemon nodes.

global.security.cluster_private_file

Absolute location to the cluster private key file. Typically should be /etc/qdb/cluster_private.key. The file can be created and managed using the QuasarDB cluster keygen, and needs to be identically replicated accross all QuasarDB daemon nodes.

global.security.encrypt_traffic

Boolean that determines whether node-to-node communication should be encrypted. Valid values are true and false. Defaults to false.

2.1.7. Performance tuning

Variable

Description

local.network.server_sessions

How many sessions (connections) each server thread is able to handle concurrently. Too few will cause significant performance degradations. Increase this value if you see “out of free server sessions” messages in the logs.

local.network.partitions_count

The number of threads in the main worker pool. Handles blocking I/O operations, and should be increased if your clients are waiting on the server but the server is not at 100% CPU. These are the queries handled by the main port.

local.network.hi_partitions_count

The number of threads dedicated for high-priority, service, messages. The default value of 2 is adequate for most setups and should be only increased for cluster with more than 20 nodes. These are the queries handled by the secondary port.

local.depot.rocksdb.max_open_files

Maximum number of open files that the storage engine is allowed to use. This should typically be close to the number of files the qdb user is allowed to open (i.e. the output of ulimit -n). Too low will cause significant performance degradations, and you are recommended to use at least 60000 in production.

local.limiter.max_bytes_soft

The maximum amount of memory (in bytes) QuasarDB will use. Any usage above this will cause QuasarDB to evict entries from its cache.

local.limiter.max_bytes_hard

Same like local.limiter.max_bytes_soft, except that processing is temporarily halted when this limit is reached.

2.1.8. Observability

Variable

Description

local.logger.log_level

An integer representing verbosity. Use 0 for most verbose logs, 5 for only fatal logs. Defaults to 2. Level 1 (debug) will also enable separate storage engine logs.

local.logger.log_directory

Absolute path where log files will be written into. Defaults to /var/log/qdb.

local.network.profile_performance

Enables performance profiling from the server side, so that clients can start running performance traces. It has a small performance impact, and is disabled by default.

global.cluster.enable_statistics

Enables runtime statistics collected server side. It has a small performance impact, and is disabled by default.

global.cluster.statistics_refresh_interval

Time (in milliseconds) between refreshes of the statistics. Should typically be set to the interval in which you query the node statistics; e.g. if you only poll your statistics once a minute, this value should be 60000.

2.1.9. Asynchronous Timeseries Inserter

The asynchronous timeseries inserter is a mechanism in QuasarDB to buffer inserts server-side, which you can tune using the configuration options below.

Variable

Description

local.depot.async_ts.pipelines

The number of asynchronous time series pipelines. Increase this number if you are experiencing slow inserts, but CPU is not yet at 100%.

local.depot.async_ts.pipeline_buffer_size

Maximum size of the buffer in bytes for each of the pipelines. Increase this value to if you are experiencing slow inserts and CPU is at 100%.

local.depot.async_ts.pipeline_queue_length

Maximum number of entries to buffer for each of the pipelines. Increase this value to if you are experiencing slow inserts and CPU is at 100%.

local.depot.async_ts.flush_deadline

Maximum time duration between consequent pipeline flushes.

2.2. QuasarDB Rest Server

The QuasarDB Rest Server qdb_rest can be configured using either command line arguments, a json config file or environmental variables. See below for examples of each configuration as well as the various configuration options:

# Environment variable based configuration
QDB_CLUSTER_URI=qdb://192.168.0.1:2837 qdb_rest

# Command line based configuration
qdb_rest --cluster=qdb://192.168.0.1:2837

# File based configuration
qdb_rest --gen-config > rest.conf # Generate a new config file and edit
qdb_rest --config-file rest.conf

Command Line Variable

Environment Variable

Description

Default

--allowed-origins

Allowed origins for cross origin resource sharing

[]

--cluster

QDB_CLUSTER_URI

URI of the cluster to connect to

qdb://127.0.0.1:2836

--cluster-public-key-file

QDB_CLUSTER_PUBLIC_KEY_FILE

Key file used for cluster security

--tls-certificate

TLS_CERTIFICATE

The certificate to use for secure connections

--tls-key

TLS_PRIVATE_KEY

The private key to use for secure conections

--tls-port

TLS_PORT

The port to listen on for secure connections

40493

--host

HOST

The IP to listen on

localhost

--port

PORT

The port to listen on for insecure connections

40080

--log-file

QDB_REST_LOG_FILE

The path to the log file

/var/log/qdb/qdb_rest.log

--assets-dir

QDB_REST_ASSETS_DIR

The path to the directory served by the rest api

/var/lib/qdb/assets

2.3. Docker

As it can a bit tedious to edit a configuration file inside a Docker container, the bureau14/qdb docker container provides several environment variables you can use to configure the most common configuration options.

Below you find an overview of the different environment variables the image supports:

Variable

Description

Example usage

QDB_LICENSE

QuasarDB license key (as string)

docker run -d \
       -e QDB_LICENSE="$your_license_key" \
       bureau14/qdb

QDB_LICENSE_FILE

QuasarDB license key (as file)

docker run -d \
       -v /path/to/my/qdb.key:/qdb.key \
       -e QDB_LICENSE_FILE=/qdb.key \
       bureau14/qdb

QDB_PORT

QuasarDB port to bind to, defaults to 2836.

docker run -d \
       -e QDB_PORT=102836 \
       bureau14/qdb

QDB_ADVERTISED_ADDRESS

When behind a NAT, QuasarDB will advertise this address as its publicly reachable IP.

docker run -d \
       -e QDB_ADVERTISED_ADDRESS="172.16.64.8" \
       bureau14/qdb

QDB_LOG_LEVEL

Numeric value of QuasarDB log level, 1 meaning debug and 4 meaning error. Defaults to 2, info-level logging.

docker run -d \
       -e QDB_LOG_LEVEL=1 \
       bureau14/qdb

QDB_LOG_PATH

Path where new logs are written to. Defaults to /opt/qdb/log.

docker run -d \
       -v /my/qdb/log/output:/logs \
       -e QDB_LOG_PATH="/logs" \
       bureau14/qdb

QDB_DEPOT_PATH

Where QuasarDB stores its data files. Override this value to tell QuasarDB to use a custom path.

docker run -d \
       -v /mnt/fast_disks/qdb:/db \
       -e QDB_DEPOT_PATH="/db" \
       bureau14/qdb

QDB_ENABLE_SECURITY

If “true”, enables security; requires private key and user list to be specified

docker run -d \
       -e QDB_ENABLE_SECURITY=true \
       -e QDB_CLUSTER_PRIVATE_KEY=... \
       -e QDB_USER_LIST=... \
       bureau14/qdb

QDB_CLUSTER_PRIVATE_KEY

QuasarDB cluster private key (as string)

docker run -d  \
       -e QDB_ENABLE_SECURITY=true \
       -e QDB_CLUSTER_PRIVATE_KEY=... \
       -e QDB_USER_LIST=... \
       bureau14/qdb

QDB_CLUSTER_PRIVATE_KEY_FILE

QuasarDB cluster private key (as file)

docker run -d \
       -e QDB_ENABLE_SECURITY=true \
       \
       -v /path/to/my/qdb/private.key:/private.key \
       -e QDB_CLUSTER_PRIVATE_KEY_FILE=/private.key \
       \
       -e QDB_USER_LIST=... \
       bureau14/qdb

QDB_USER_LIST

QuasarDB user list (as string)

docker run -d \
       -e QDB_ENABLE_SECURITY=true \
       -e QDB_CLUSTER_PRIVATE_KEY=... \
       -e QDB_USER_LIST=... \
       bureau14/qdb

QDB_USER_LIST_FILE

QuasarDB user list (as file)

docker run -d \
       -e QDB_ENABLE_SECURITY=true \
       -e QDB_CLUSTER_PRIVATE_KEY=... \
       \
       -v /path/to/my/qdb/user.list:/user.list \
       -e QDB_USER_LIST_FILE=/user.list \
       \
       bureau14/qdb

QDB_REPLICATION

Sets QuasarDB replication factor

docker run -d \
       -e QDB_REPLICATION=3 \
       \
       bureau14/qdb

QDB_TOTAL_SERVER_SESSIONS

Sets the amount of server-side sessions the server will allocate.

docker run -d \
       -e QDB_TOTAL_SERVER_SESSIONS=8192 \
       \
       bureau14/qdb

QDB_PARALLELISM

QuasarDB will dedicate this amount of threads to serve requests. Use in conjunction with ROCKSDB_THREADS to properly tune for your available processing power.

docker run -d \
       --cpu-number=32 \
       -e QDB_PARALLELISM=24 \
       -e QDB_ROCKSDB_THREADS=8 \
       -e QDB_ROCKSDB_HI_THREADS=1 \
       \
       bureau14/qdb

QDB_PUBLISH_FIREHOSE

When true, QuasarDB will publish a realtime activity stream, which is required for continuous queries. Defaults to false.

docker run -d
       -e QDB_PUBLISH_FIREHOSE="true" \
        \
        bureau14/qdb

QDB_FIREHOSE_PORT

When provided, QuasarDB will bind the firehose publisher to this port. Defaults toport 3836.

docker run -d
       -e QDB_PUBLISH_FIREHOSE="true" \
       -e QDB_FIREHOSE_PORT=103836 \
       \
       bureau14/qdb

QDB_FIREHOSE_PUBLISHING_THREADS

When provided, QuasarDB will dedicate this amount of threads to the firehose publisher.

docker run -d
       -e QDB_PUBLISH_FIREHOSE="true" \
       -e QDB_FIREHOSE_PUBLISHING_THREADS=4 \
       \
       bureau14/qdb

QDB_MEMORY_LIMIT_SOFT

Controls the soft memory limit, defaults to 50% of host memory.

docker run -d \
           -e QDB_MEMORY_LIMIT_SOFT=17179869184 \
           -e QDB_MEMORY_LIMIT_HARD=25769803776 \
           bureau14/qdb

QDB_MEMORY_LIMIT_HARD

Controls the hard memory limit, defaults to 75% of host memory.

docker run -d \
       -e QDB_MEMORY_LIMIT_SOFT=17179869184 \
       -e QDB_MEMORY_LIMIT_HARD=25769803776 \
       bureau14/qdb

QDB_ROCKSDB_THREADS

Sets the number of threads dedicated to RocksDB compaction.

docker run -d \
       -e QDB_ROCKSDB_THREADS=8 \
       bureau14/qdb

QDB_ROCKSDB_HI_THREADS

Sets the number of threads dedicated to RocksDB flushes. You will likely want to keep this at the default value.

docker run -d \
       -e QDB_ROCKSDB_THREADS=16 \
       -e QDB_ROCKSDB_HI_THREADS=2 \
       bureau14/qdb

QDB_CLOUD_PROVIDER

Enables a specific cloud provided configuration. Currently only single value aws is supported. Required for S3 cloud storage backend.

docker run -d \
       -e QDB_CLOUD_PROVIDER="aws" \
       bureau14/qdb

QDB_CLOUD_BUCKET_REGION

Set an S3 bucket region, required for S3 cloud storage backend.

docker run -d \
       -e QDB_CLOUD_PROVIDER="aws" \
       -e QDB_CLOUD_BUCKET_REGION="us-east-1" \
       -e QDB_CLOUD_BUCKET="qdb-data" \
       -e QDB_CLOUD_BUCKET_PATH_PREFIX="instance1" \
       bureau14/qdb

QDB_CLOUD_BUCKET

S3 bucket name, required for S3 storage backend.

docker run -d \
       -e QDB_CLOUD_PROVIDER="aws" \
       -e QDB_CLOUD_BUCKET_REGION="us-east-1" \
       -e QDB_CLOUD_BUCKET="qdb-data" \
       -e QDB_CLOUD_BUCKET_PATH_PREFIX="instance1" \
       bureau14/qdb

QDB_CLOUD_BUCKET_PATH_PREFIX

Prefix to use for path within S3 bucket. Useful if you intend on storing data of multiple QuasarDB instances in the same bucket.

docker run -d \
       -e QDB_CLOUD_PROVIDER="aws" \
       -e QDB_CLOUD_BUCKET_REGION="us-east-1" \
       -e QDB_CLOUD_BUCKET_PREFIX="qdb-" \
       -e QDB_CLOUD_BUCKET_SUFFIX="data" \
       -e QDB_CLOUD_BUCKET_PATH_PREFIX="instance1" \
       bureau14/qdb

QDB_CLOUD_AWS_ENABLE_INSTANCE_AUTH

Enable IAM credential discovery through AWS instance authentication. This is the preferred credential discovery method in production. Defaults to true.

docker run -d \
       -e QDB_CLOUD_PROVIDER="aws" \
       -e QDB_CLOUD_AWS_ENABLE_INSTANCE_AUTH="false" \
       -e QDB_CLOUD_AWS_ACCESS_KEY_ID="AKIA...." \
       -e QDB_CLOUD_AWS_SECRET_ACCESS_KEY="STS...." \
       -e QDB_CLOUD_BUCKET_REGION="us-east-1" \
       -e QDB_CLOUD_BUCKET_PREFIX="qdb-" \
       -e QDB_CLOUD_BUCKET_SUFFIX="data" \
       -e QDB_CLOUD_BUCKET_PATH_PREFIX="instance1" \
       bureau14/qdb

QDB_CLOUD_AWS_ACCESS_KEY_ID

Reflects AWS_ACCESS_KEY_ID environment variable. Discouraged in production use.

docker run -d \
       -e QDB_CLOUD_PROVIDER="aws" \
       -e QDB_CLOUD_AWS_ENABLE_INSTANCE_AUTH="false" \
       -e QDB_CLOUD_AWS_ACCESS_KEY_ID="AKIA...." \
       -e QDB_CLOUD_AWS_SECRET_ACCESS_KEY="STS...." \
       -e QDB_CLOUD_BUCKET_REGION="us-east-1" \
       -e QDB_CLOUD_BUCKET_PREFIX="qdb-" \
       -e QDB_CLOUD_BUCKET_SUFFIX="data" \
       -e QDB_CLOUD_BUCKET_PATH_PREFIX="instance1" \
       bureau14/qdb

QDB_CLOUD_AWS_SECRET_ACCESS_KEY

Reflects AWS_SECRET_ACCESS_KEY environment variable. Discouraged in production use.

docker run -d \
       -e QDB_CLOUD_PROVIDER="aws" \
       -e QDB_CLOUD_AWS_ENABLE_INSTANCE_AUTH="false" \
       -e QDB_CLOUD_AWS_ACCESS_KEY_ID="AKIA...." \
       -e QDB_CLOUD_AWS_SECRET_ACCESS_KEY="STS...." \
       -e QDB_CLOUD_BUCKET_REGION="us-east-1" \
       -e QDB_CLOUD_BUCKET_PREFIX="qdb-" \
       -e QDB_CLOUD_BUCKET_SUFFIX="data" \
       -e QDB_CLOUD_BUCKET_PATH_PREFIX="instance1" \
       bureau14/qdb