QuasarDB requires certain tasks be performed regularly to maintain optimal performace. The tasks discussed in this chapter are required, but they are repetetive in nature and can easily be automated using standard tools.
QuasarDB ensures data consistency by using a Write-Ahead Log (WAL) <https://en.wikipedia.org/wiki/Write-ahead_logging> on top of its data store. This ensures that, in the event of a fatal system crash, we can restore a node to a known good state and as such achieve single-node durability.
This recovery procedure can be expensive, may cause a full database repair upon next boot, and is typically something you want to avoid in production whenever possible.
Additionally, forcefully disconnecting a node from a cluster will lead to timeouts on clients, and a delayed cluster rebalancing.
To work around these limitations, QuasarDB supports a graceful shutdown. On UNIX-like systems, this can be achieved by sending the QuasarDB daemon the
SIGQUIT signal. This will ensure QuasarDB gracefully leaves the cluster, and does the necessary housekeeping to ensure the data store is in a consistent state.
If you have installed QuasarDB through one of our packages, our packaging scripts typically take care of this. However, if you are using custom scripts to start/stop QuasarDB, please ensure a
SIGTERM is sent before termination, and not killed. For example, with systemd, you will want to ensure you use an infinite termination timeout as follows:
[Service] Type=simple User=qdb Group=qdb ExecStart=/usr/bin/qdbd -c /etc/qdb/qdbd.conf Restart=on-failure LimitNOFILE=65536 TimeoutStopSec=infinity
Here we explicitly tell systemd to wait as long as it takes to have QuasarDB terminate gracefully.
QuasarDB requires periodic trimming of the dataset. When trimming, it performs several operations:
it compacts the Write-Ahead-Log and merges this into the database;
it removes MVCC-transaction metadata;
it reclaims any unused space.
While these tasks are also done automatically by the database, we strongly recommend periodic explicit trimming, for example using a daily or weekly cron job. As trimming can be an expensive operation, we recommend to avoid doing this during peak hours.
You can issue a cluster trim using the QuasarDB shell:
$ qdbsh qdbsh > cluster_trim
If you observe the logs of the database daemon, you should see something like this:
2019.11.05-188.8.131.523826008 3339 3438 info trimming all 412,160 entries (325,396 versions in memory) 2019.11.05-184.108.40.2063002389 3339 3438 debug successfully cleaned allocator buffers 2019.11.05-220.127.116.113022210 3339 3438 info 434,001/325,594 entries were trimmed (for a size of 14.91 GiB). we now have 428,327 entries in total and 309,597 entries in memory for a total size of 60.238 GiB (16.0 EiB in memory) 2019.11.05-18.104.22.1683021310 3339 3438 info requesting complete database compaction 2019.11.05-22.214.171.1240022210 3339 3438 info complete database compaction took 47 ms
QuasarDB does not support log rotation out of the box, and its log files will increase in size over time. We recommend:
weekly rotation of all log files stored in
all logs older than 7 days should be removed.
Additionally, you might want to setup a centralised log aggregator to ease the management of your logs.
Although it’s not required, it’s recommended you take periodic backups of your database and store them in a safe, offsite location.
We give the following process recommendations for backups:
Schedule your backups during off-hours, for example at night;
Before making a backup, issue a cluster trim in order to remove any unnecessary metadata and compact the Write-Ahead-Log (WAL);
Shut the QuasarDB daemon you are about to take a backup of, for example using
systemctl stop qdbd. Make sure to send the normal termination signal
SIGTERM so that the process gracefully shutds down.
Copy the data directory, including the WAL, to your backup location. The precise location of your data directory depends upon your storage configuration, but is typically
After the copy operation has completed, you can restart the QuasarDB daemon.
Repeat this process for each of the nodes you want to back up.