1.6. quasar cryptocurrency feed handler¶
Important
The Quasar cryptocurrency feed handler is still under development and is not considered ready for production. It is included in the Quasar distribution package starting with version 3.13.2.
1.6.1. Introduction¶
The Quasar cryptocurrency feed handler is a small footprint, high-throughput, low-latency feed handler that automatically connects to cryptocurrency exchanges, captures the data, and stores it in a QuasarDB instance.
The feed handler understands the exchange protocol it connects to, decodes the message, and organizes the data into tables.
The feed handler is designed to capture the full order book and store that at the lowest latency possible. It can also capture trades and ticker information.
It is written in C++ 20 and is delivered as a single binary without any dependency.
1.6.2. Description¶
1.6.2.1. Supported exchanged, channels, and products¶
The feed handler currently only supports Coinbase. Exchanges are added as development progress. All products supported by Coinbase are supported.
The feed handler can capture ticker information, trade information, and full order book (a.k.a. Level 3).
1.6.2.2. Full order book data capture¶
The feed handler captures data to guarantee correct order book reconstruction if needed.
When capturing data from the exchange, the feed handler will do a proper handshake:
It will first register to the full channel of the exchange and start capturing messages with their sequence number
Then, it will request through the REST API the full state of the order book
The state of the order book is stored as a snapshot into QuasarDB in a format the order book engine understands
When building the order book from the data stored in QuasarDB:
The execution engine will look for the most recent snapshot available
Using the snapshot, the engine will create an internal state of the order book
It will proceed to execute all orders whose sequence number are higher than the sequence number of the snapshot until the requested time point
1.6.2.3. Data structure¶
Messages coming from the exchange are decoded in-place using zero-copy networking programming. Data is pushed to Quasar using the batch API. Currently, only asynchronous writes using micro-batches are supported. All insertions modes supported by Quasar will eventually be supported.
The tables’ names are currently hardcoded as exchange + product + data type. For example, the BTC/USD ticker coming from Coinbase is stored in the table “coinbase_btc_usd_ticker”.
Data is organized as such:
Full order book: two tables per product, one for the buy-side, one for the sell-side, containing all orders (one row per order)
Trades: one table per product, containing all trade information (one row per trade)
Ticker: one table per product, containing the price, trade size, best bid, and best ask (one row per ticker)
1.6.2.4. System architecture¶
The feed handler is single-threaded and uses an asynchronous web socket connection to handle incoming messages from the exchange.
Currently, one product per instance is supported.
The proposed system architecture is to have one instance of the handler running per product you wish to capture. Since the footprint of the feed handler is a couple of megabytes of RAM, running multiple instances on a single machine isn’t an issue. It’s also possible to run each instance in a separate container.
The multi-instance approach gives you greater resiliency as if for any reason one instance stops, it does not compromise other ongoing captures.
1.6.3. Usage¶
The feed handler will default to capture everything: ticker, trades, and orders. You can disable each from the command line or using a configuration file. At least one data stream needs to be captured or the feed handler will exit with an error.
Each feed handler instance captures one product from one exchange. There is no default value for the product to capture. If no product is configured, the feed handler will exit with an error.
By default, the feed handler attempts to connect to a Quasar instance running on the same machine and listening to the IPv4 loopback address.
In 99% of the cases, there are three settings you want to change:
The exchange to use
The product to capture
The address of the QuasarDB instance to write data to
1.6.4. Command line options¶
To capture the BTC/USD from Coinbase, you would thus type:
qdb_crypto_feed --product=btc-usd
1.6.5. Configuration file¶
Here is the default configuration file:
{
"qdb_url": ""qdb://127.0.0.1:2836",
"exchange": "coinbase",
"product": "",
"websocket_host": "ws-feed.pro.coinbase.com",
"websocket_port": "443",
"rest_host": "api.pro.coinbase.com",
"rest_port": "443"
"load_snapshot": true,
"capture_orders": true,
"capture_ticker": true,
"capture_trades": true
}
The hosts for the websocket and REST API are configurable if need be. Note that the port number is stored as a string, not an integer.
The settings of interest are the “qdb_url” and “product”. It allows you to select which product to capture and the address of the QuasarDB instance to write to.
You may opt to disable the capture or orders, ticker, and trades at your convenience. At least one channel must be captured. Otherwise, the feed handler will exit and error.