1.6. Memtables#

A memtable is an in-memory buffer that stores the most recent writes before they are flushed to disk as immutable SSTables. Memtables are a core component of QuasarDB’s LSM-tree storage architecture.

1.6.1. How Memtables Work#

When you write data to QuasarDB:

  1. Write-Ahead Log (WAL): The write is first recorded to the Write-Ahead Log on disk for crash recovery.

  2. Memtable Buffer: The data is then stored in the active memtable in memory.

  3. Flush to Disk: When the memtable reaches its size limit, it becomes immutable and is flushed to disk as an SSTable.

This design enables high write throughput by batching multiple writes in memory before performing sequential disk I/O.

1.6.2. Crash Protection#

Memtables are protected against crashes through the Write-Ahead Log (WAL):

  • Every write to a memtable is first written to the WAL on disk.

  • If the database crashes, uncommitted memtable data can be recovered from the WAL.

  • Once a memtable is successfully flushed to an SSTable, the corresponding WAL entries can be discarded.

This ensures durability without sacrificing write performance.

1.6.3. Memory and Performance#

Memtables directly impact both memory usage and write performance:

Memory Usage

  • Active and immutable memtables consume heap memory.

  • The total memory used by memtables is controlled by write_buffer_size multiplied by max_write_buffer_number.

  • Monitor memtable memory usage via the memory.persistence.memtable_bytes metric.

Write Performance

  • Larger memtables reduce flush frequency, improving write throughput.

  • Too many memtables can cause RocksDB to stall writes until flushes complete.

  • Proper sizing of memtables and WAL prevents write stalls during ingestion spikes.

1.6.4. Configuration#

Key configuration parameters:

  • write_buffer_size: Size of a single memtable (default: 1/4 of table memory budget).

  • max_write_buffer_number: Maximum number of memtables (active + immutable) before writes stall.

  • min_write_buffer_to_merge: Minimum memtables to merge before flushing to L0, enabling deduplication.

  • max_total_wal_size: Size of the WAL; when full, triggers memtable flush.

1.6.5. See Also#