2.6. quasar cryptocurrency feed handler#
Important
The Quasar cryptocurrency feed handler is still under development and is not considered ready for production. It is included in the Quasar distribution package starting with version 3.13.2.
2.6.1. Introduction#
The Quasar cryptocurrency feed handler is a small footprint, high-throughput, low-latency feed handler that automatically connects to cryptocurrency exchanges, captures the data, and stores it in a QuasarDB instance.
The feed handler understands the exchange protocol it connects to, decodes the message, and organizes the data into tables.
The feed handler is designed to capture the full order book and store it at the lowest latency possible. It can also capture trades and ticker information.
It is written in C++ 20 and is delivered as a single binary without any dependency.
2.6.2. Description#
2.6.2.1. Supported exchanged, channels, and products#
The feed handler currently only supports Coinbase. Exchanges are added as development progresses. All products supported by Coinbase are supported.
The feed handler can capture ticker information, trade information, and full order book (a.k.a. Level 3).
2.6.2.2. Full order book data capture#
The feed handler captures data to guarantee correct order book reconstruction if needed.
When capturing data from the exchange, the feed handler will do a proper handshake:
It will first register to the full channel of the exchange and start capturing messages with their sequence number
Then, it will request through the REST API the full state of the order book
The state of the order book is stored as a snapshot into QuasarDB in a format the order book engine understands
When building the order book from the data stored in QuasarDB:
The execution engine will look for the most recent snapshot available
Using the snapshot, the engine will create an internal state of the order book
It will proceed to execute all orders whose sequence number are higher than the sequence number of the snapshot until the requested time point
2.6.2.3. Data structure#
Messages coming from the exchange are decoded in-place using zero-copy networking programming. Data is pushed to Quasar using the batch API. By default data is written asynchronously to Quasar, you can change to synchronous writes in the configuration file.
The feed handler will retain the data for a configurable amount of time before pushing it to Quasar, regardless of the push mode.
The tables’ names are currently hardcoded as exchange + product + data type. For example, the BTC/USD ticker coming from Coinbase is stored in the table “coinbase_btc_usd_ticker”.
Data is organized as such:
Full order book: two tables per product, one for the buy-side, one for the sell-side, containing all orders (one row per order)
Trades: one table per product, containing all trade information (one row per trade)
Ticker: one table per product, containing the price, trade size, best bid, and best ask (one row per ticker)
2.6.2.4. System architecture#
The feed handler is single-threaded and uses an asynchronous web socket connection to handle incoming messages from the exchange.
Currently, one product per instance is supported.
The proposed system architecture is to have one instance of the handler running per product you wish to capture. Since the footprint of the feed handler is a couple of megabytes of RAM, running multiple instances on a single machine isn’t an issue. It’s also possible to run each instance in a separate container.
The multi-instance approach gives you greater resiliency: if for any reason one instance stops, it does not compromise other ongoing captures.
2.6.3. Error management#
Caution
Because the feed handler requests a full snapshot of the order book at startup, frequent restarts of the feed handler may get your IP banned by the exchange when snapshotting is enabled.
If the feed handler loses connectivity with the exchange, it will log the error, flush all in-flight packets to the database, and exit with a failure error code. This behavior lets you see when packets may have been dropped and take the appropriate action.
In other words, the feed handler will not attempt to reconnect on error: it will exit. Certain exchanges, like Binance, will unconditionally disconnect you after 24 hours. Disconnection from exchanges is just a fact of life you need to manage and prepare for.
The feed handler will currently not backfill missing ticks and trades on restart. However, when snapshotting is enabled, consistent and coherent order book reconstruction is guaranteed except during the brief period during which the connection was lost.
2.6.4. Usage#
The feed handler will default to capture everything: ticker, trades, and orders. You can disable each from the command line or using the configuration file. At least one data stream needs to be captured or the feed handler will exit with an error.
Each feed handler instance captures one product from one exchange. There is no default value for the product to capture. If no product is configured, the feed handler will exit with an error.
By default, the feed handler attempts to connect to a Quasar instance running on the same machine and listening to the IPv4 loopback address.
In 99% of the cases, there are three settings you want to change:
The exchange to use
The product to capture
The address of the QuasarDB instance to write data to
2.6.5. Order book reconstruction#
The feed handler will capture all possible order data to enable full order book reconstruction at any point in time. How precise the order capture is depends on the crypto exchange as not all of them full level 3 market data access.
To rebuild the order book, you can either use Quasar’s accelerated order book reconstruction routines or you own programs.
When rebuilding the order book, there are a number of caveats to keep in mind:
Buy and sell orders are stored in two different tables per product. For example, if you capture the orders for the BTC/USD pair, you will have one table with the buy orders, one table with the sell orders.
Crypto markets function 24/7, you need to start your order book reconstruction from an order book snapshot. Quasar does this automatically.
Quasar starts capturing the order data _before_ snapshoting the order book, ensuring you will not be missing orders. If you are using your own program to rebuild the order book it is your responsibility to ignore every order whose sequence number is anterior to the order book snapshot.
If you reconstruct the order book “in the present”, ensure you have all orders in the database. If you use the crypto feed handler with asynchronous writes (default) there can be a lag of a couple of seconds which can result in an incorrect or inconsistent order book while orders are being flushed to disk. Consider using the “fast push” write mode if you want the lowest lag possible, but keep in mind, that, no matter what, there will always be a non-zero lag.
2.6.6. Command line options#
To capture the BTC/USD from Coinbase, you would thus type:
qdb_crypto_feed --product=btc-usd
2.6.7. Configuration file#
Here is the default configuration file:
{
"qdb_url": ""qdb://127.0.0.1:2836",
"push_mode": "async",
"push_delay": 3000,
"handshake_timeout": 30000,
"idle_timeout": 0,
"exchange": "coinbase",
"product": "",
"websocket_host": "ws-feed.pro.coinbase.com",
"websocket_port": "443",
"rest_host": "api.pro.coinbase.com",
"rest_port": "443"
"load_snapshot": true,
"capture_orders": true,
"capture_ticker": true,
"capture_trades": true
}
The hosts for the websocket and REST API are configurable if need be. Note that the port number is stored as a string, not an integer.
The settings of interest are the “qdb_url” and “product”. It allows you to select which product to capture and the address of the QuasarDB instance to write to.
You may opt to disable the capture or orders, ticker, and trades at your convenience. At least one channel must be captured. Otherwise, the feed handler will exit and error.
You can configure how the crypto feed handler pushes data to Quasar by changing the “push_mode” field. The supported values are:
“async” (default): data is pushed to Quasar which will commit it to the database asynchronously, as per cluster configuration
“fast”: data is pushed to Quasar which will write it to the database synchronously, but without any transactional (ordering) guarantee
“transactional”: data is pushed to Quasar which will write it to the database using read-comitted transactions
The “push_delay” parameters configures for how long the feed handler will retain the data before pushing to Quasar, in milliseconds. Its default value is 3,000. Lower values reduce the visibility lag at the cost of higher CPU and I/O usage. Values higher than 10,000 have no real advantage in asynchronous mode.
The delay period does not have a hard guarantee and may be effectively milliseconds higher than the specified amount.
If you combine the “fast” push mode with a low delay, you can achieve low data lag. For example, a delay of 1,000 in fast push mode means the data is only 1s + network delay behind the exchange.
You can configure the timeouts of the websockets which defaults to 30s for the handshake and disabled (0) for idle.