6. Column Partitioning Implementation History#

This page documents the column partitioning changes on sc-11549/v5 in the QuasarDB repository, from master (506e33475ad6) to HEAD (68e10a0d4e96), as inspected on 14 October 2026. Sections follow commit ancestry order, rather than author dates. They describe the changes at each step; intermediate restrictions do not necessarily apply at HEAD.

The range contains 15 commits, presented as 13 changes. The plan-only commit eedffab8b6 is grouped with efe807e2ec, and the serialization include fix 4446295a48 is grouped with aa1634192b, in preparation for folding each into the larger commit with the same title. Source references use the original commit identities and will need updating after that history rewrite.

The feature is intended for QuasarDB 4.0.0. The branch currently reports version 3.15 and uses that version in acceptance fixtures. For user-facing syntax, behavior and limits, see Column partitioning.

Coverage below describes tests added or changed by the commits. It does not claim that those tests were rerun for this documentation change.

6.1. Partitioning policies and SQL grammar#

Commit: 285e7f554b — Add partitioning policies and SQL grammar.

Introduces qdb/partitioning and a policy consisting of a stable column index, an encoding version, and one of four strategies:

  • Timestamp partitioning on a TIMESTAMP column.

  • Distinct-value partitioning on an INT64 or STRING column.

  • Hash partitioning on an INT64 or STRING column, with 1 to 65,536 slots.

  • Range partitioning on an INT64 column, with a positive range width.

SQL parsing accepts PARTITION BY key, PARTITION BY HASH(key) PARTITIONS n, PARTITION BY RANGE(key, width), and PARTITION BY TIMESTAMP(key). Omitting the clause retains the existing $timestamp layout. Validation rejects unsupported column types, reserved columns and policy versions. Non-default creation is still rejected by the query client at this stage.

Hash version 1 uses FNV-1a over a type tag and canonical value bytes; INT64 encoding is independent of host byte order. Range indexes use floor division, so negative keys route consistently: with width 1,000, key -1 belongs to range -1, while key 0 belongs to range 0. NULL keys cannot be routed.

Adds policy unit tests and the initial SINGLE_COLUMN_PARTITIONING_PLAN.md. The plan describes intended work as well as implementation status; it is not itself evidence of completed storage support.

6.2. Persisted metadata and initial protocol compatibility#

Commit: 678884b8c2 — Persist partitioning metadata with protocol compatibility.

Adds the partitioning policy to table roots and CREATE/get-columns protocol messages. Root serialization gains a new version and conversions for earlier roots. Legacy roots acquire the default timestamp policy, preserving existing table kinds, shard sizes, TTL and column metadata.

Versioned request and reply adapters retain older CREATE/get-columns layouts. The server validates non-default metadata, but creation remains gated in the query client; persisting a policy does not yet route rows into partitions. Serialization and protocol unit tests cover round trips and legacy conversion, and metadata expectations are updated for the additional root field.

6.3. Policy-owned shard sizes and legacy request adapters#

Commit: ee506a7074 — Refactor partitioning metadata and preserve protocol compatibility.

Moves timestamp shard size into timestamp_partitioning and removes the redundant field from the current table root. ts_root_shard_size derives the size from the policy, returning the infinite size for non-timestamp strategies. Timestamp policy validation now checks the shard size.

Propagates policies through query table metadata and metadata consumers in the client. Legacy roots and requests carrying a separate shard size are converted into the timestamp policy. Versioned adapters cover batch writes, reads, aggregations, mutations and maintenance messages, retaining their earlier wire layouts. Protocol and serialization tests are updated accordingly.

6.4. Typed partition identities and index lifecycle#

Commits: efe807e2ec and eedffab8b6 — Add partition identities and index lifecycle support.

Adds distinct physical key types for timestamp bases, INT64 values, STRING values, hash slots and range indexes. route_partition converts a selected column value into a typed key, while partition_payload describes the stored row-time layout. A selected column’s partition boundaries cannot be treated as bounds on $timestamp.

Canonical partition bucket and column aliases use the reserved ..pt.1. prefix. Parsing and validation check policy/key compatibility and alias limits; default timestamp partitioning retains its legacy alias format. Partition columns use stable physical identities, independent of their display names.

A separate partition index records populated keys without enlarging legacy shard entries. Integrates registration, removal, entry merging/migration, recovery and index repair with the table directory. Recovered identities are validated against authoritative table metadata before becoming visible. Adds routing, alias, index and repair unit tests, plus dbtool recognition and client error mappings.

The grouped eedffab8b6 commit changes only the implementation plan’s status paragraph. Its description mentions the broader implementation, but its diff adds no routing or index code. That paragraph describes a creation gate; the actual preceding SQL enablement remains in force at this point in ancestry.

6.5. Temporary creation and metadata gates#

Commit: 7839f1e90e — Gate non-default partitioning until storage support is complete.

Explicitly rejects non-default CREATE requests in the server and non-default get-columns replies with qdb_e_not_implemented. Retains the query client’s creation guard to avoid exposing policies before ingestion and reads support them.

Adds SQL expectations for rejected creation, defers SHOW TABLE coverage and adds a disabled bulk-reader rejection test. These are historical restrictions: the next SQL integration commit lifts the creation and metadata gates.

6.6. SQL policy resolution, creation and metadata presentation#

Commit: 3fcdb31444 — Integrate partitioning policies with SQL and enable table creation.

Resolves parsed column names against creation-time column indexes, excluding $timestamp from ordinary column numbering. Carries SHARD_SIZE in the timestamp strategy and validates column existence, type, policy parameters and TTL. Non-default layouts require a timeseries schema, without TTL or aggregated table metadata.

Enables ordinary CREATE TABLE with supported policies and restores get-columns metadata access. Partitioned CREATE TABLE AS SELECT remains unsupported. Creation alone still does not provide partition-aware ingestion or reads at this step.

Adds partition_by to SHOW TABLE and $sys.tables, rendered as the selected column, HASH(key) PARTITIONS n, RANGE(key, width), or TIMESTAMP(key). Default layouts display TIMESTAMP($timestamp). Presentation resolves the stable column index to its current name, including after rename.

Updates parser, policy, system-table and query-client tests, enables creation and SHOW TABLE SQL fixtures, and adds invalid-policy cases.

6.7. Derived partition and payload widths#

Commit: cbc1bbde14 — Refactor partitioning metadata and legacy compatibility.

Removes the redundant shard size from query table_info. The partition_shard_size accessor derives the partition width from the policy; payload_shard_size retains that width for the default layout and returns infinite width for column partitions.

This distinction keeps stored row timestamps independent of the selected key, including when a TIMESTAMP key differs substantially from $timestamp. Updates writers, readers, processors, query preparation and fixtures to use the appropriate derived width. Simplifies legacy message adapters and adds tests for policy-derived widths and compatibility conversions.

6.8. Partitioned INSERT and basic SELECT#

Commit: 8996110170 — Implement partitioned INSERT and SELECT MVP.

Preflights batch tables before sending writes, groups rows by physical partition, and applies the same row selection to all columns to preserve alignment. Splits messages according to their estimated byte size and the network input budget. Synchronous transactional and fast ingestion are supported; asynchronous, truncate and ingestion deduplication modes remain guarded.

Every row must provide a valid non-NULL partition key. Empty STRING keys are rejected because existing column storage represents them as NULL. Alias and timestamp-domain checks occur before sending writes. The server validates column identities, types, row counts, canonical bucket aliases and the routed key of each row in a partition batch.

SELECT traverses populated partition keys and merges aligned results into logical $timestamp order before final query processing. Stable physical column indexes preserve access after rename. Removal and index repair gain partition-aware handling.

At this stage, advanced SELECT features, most mutations and maintenance, bulk export, and raw table export remain rejected. Adds SQL fixtures for all four strategies, collisions, domain limits, preflight, rename, repair/isolation and legacy timestamp behavior, plus result-ordering and raw-export tests.

6.9. Partition-aware reads, transfers and maintenance#

Commits: aa1634192b and 4446295a48 — Support partition-aware reads and maintenance.

Bulk-reader protocol version 2 carries query table metadata, including the partition policy, with adapters for older requests. The client reads populated partitions and orders merged bulk results by logical row timestamp. Multi-table bulk reads validate layout compatibility as well as schema compatibility.

Introduces shared traversal over a snapshot of populated partition keys so maintenance can safely remove buckets during iteration. Extends timestamp reads, COPY/export, IMPORT routing, ANALYZE, SHOW DISK_USAGE, TRIM and rewrite repair, reusing table ACL checks. Rewrite repair recognizes partition aliases rather than assuming that aliases encode row-time intervals.

Removes the corresponding query-client guards, fixes pending-request handling and adds bulk-reader, transfer and maintenance coverage. Mutation and advanced SELECT guards still exist at this stage. Logical bulk export support does not remove the separate raw-table export restrictions.

The grouped 4446295a48 commit adds only the qdb/serialization/ts_bulk_reader.hpp include to compatibility.cpp, providing the bulk-reader serialization declarations used by the compatibility implementation. It introduces no additional read or maintenance behavior.

6.10. Logical SELECT evaluation across partitions#

Commit: 41833891c4 — Enable full SQL SELECT semantics for column partitions.

Removes the basic SELECT restrictions and evaluates windows, transformations, ordered aggregations, logical GROUP BY/HAVING, ASOF joins, lookback and interpolation over the appropriate merged logical streams. SQL window PARTITION BY and grouping remain independent of physical storage partitions.

Partitioned native aggregation scans all populated partitions before reduction: overlapping row times make per-partition early limits unsafe for FIRST/LAST. Ordered operations and time grouping use the existing client evaluation facilities where required. Final ordering, limits and offsets apply to the logical results.

For RESTRICT TO, PREWHERE remains on the server; restriction and subsequent WHERE evaluation run after merging rows with equal timestamps across physical partitions. This preserves the required evaluation order.

Adds query-family fixtures for windows, transformations, grouping, ordered aggregations, restrictions, joins, lookback/interpolation and transformed INSERT FROM SELECT. The commit title describes SELECT support, not completion of all mutations or cluster lifecycle validation.

6.11. Partition-local mutations, schema changes and row IDs#

Commit: b6baacff4e — Support partition-local mutations, schema changes and row IDs.

Extends UPDATE, DELETE and schema changes to populated partitions. Non-key updates are supported. A key update succeeds only when every affected value continues to route to its existing physical partition, such as another value within the same range or hash slot. Cross-partition relocation returns qdb_e_not_implemented.

DELETE WHERE DUPLICATE is supported when comparisons include the partition key, or when a one-slot hash layout guarantees one physical partition. Other comparisons remain rejected because they require global coordination. Adding columns, renaming the key and dropping non-key columns are supported; dropping the partition key returns qdb_e_invalid_argument. TTL changes remain unsupported for column-partitioned tables.

Creates opaque row IDs from the physical bucket alias and local ordinal before logical row merging. Equal ordinals in different partitions remain distinct, and non-key UPDATE preserves identity. Insertion or deletion may change later ordinals, so these IDs are not persistent primary keys.

Fixes persistent ranged deletion by reading all columns before shortening their shared timestamp vector, and fixes processor lifetime handling. Adds mutation, rollback/guard, ranged deletion, schema and row-ID fixtures; updates earlier rejection expectations to reflect the newly supported operations.

6.12. Partition pruning for SELECT and aggregation#

Commit: fb4c469259 — Prune column partitions for SELECT and aggregation.

Derives bounds from PREWHERE and WHERE before opening partition buckets:

  • Distinct-value and hash INT64/STRING layouts use direct key equality.

  • INT64 RANGE and selected TIMESTAMP layouts use equality and ordered comparisons.

  • AND intersects constraints. Ordered OR uses a conservative enclosing range; equality OR retains a constraint only when its branches require the same value or one branch is impossible.

  • Contradictions can produce an empty partition range. Unsupported expressions retain a conservative range or fall back to scanning all populated partitions.

Handles qualified and reversed comparisons, negative ranges, integer extremes, finite/infinite timestamp layouts and nanosecond endpoints. Row filtering remains in place: hash collisions and range boundaries still require evaluating the original predicate. $timestamp restrictions remain independent of selected key bounds.

Adds ten SQL fixtures with explicit expected results and paired predicates on the partition key and a non-key mirror column. Each pair checks SELECT and COUNT/SUM. Index unit tests check bounded snapshots; an API integration test uses existing performance profiles to assert actual bucket visits, including one-bucket equality, empty contradictions, interval scans and full-scan controls. Adds query performance benchmarks across partitioning strategies.

6.13. RocksDB partition prefix benchmark#

Commit: 68e10a0d4e — Add prefix extractor benchmark.

Adds the microbench_partition_prefix target and its usage documentation. The benchmark compares partition aliases with ordinary aliases, for both metadata keys and data keys containing UUIDs. It uses the real partition alias generators and labels results with the active extractor name.

Each iteration performs 8,192 point reads, equally divided between hits and misses, across 128 tables and 32 partitions per table. Cases use 8 or 32 SSTs with overlapping key ranges, disabled compaction, prefix-only Bloom filters and hash indexes. Setup, warmup and cleanup are outside the measured loop.

Reports throughput, prefix checks/rejections per read and data-cache misses per read. Filters and indexes are warmed; data blocks are not admitted to the cache, but the OS page cache may be warm. Cache misses therefore do not measure physical disk reads. This commit adds measurement infrastructure only: it does not change the prefix extractor or establish a performance improvement.

6.14. Behavior and remaining limits at HEAD#

At the inspected HEAD, supported column layouts can be created, populated and queried with logical table semantics. Reads and aggregations can prune supported key predicates. Bulk transfers, maintenance, partition-local mutations and schema changes have partition-aware paths. Existing default timestamp layouts retain legacy identities and compatibility conversions.

Remaining restrictions include cross-partition key relocation, globally coordinated duplicate deletion, TTL, partitioned CREATE TABLE AS SELECT, asynchronous/truncate/deduplicating ingestion, raw-table export and dropping the partition key. Empty STRING keys cannot be ingested.

The branch’s acceptance-test notes distinguish fresh-cluster checks from persistent restart, damaged-storage recovery, replica movement and restricted user ACL coverage. Those lifecycle scenarios and broader query/type/layout combinations still require validation before declaring the feature complete.