ArrowAsyncProducer Struct Reference

#include <arrow_abi.h>

Data Fields
ArrowDeviceType	device_type
	The device type that this stream produces data on.
void(*	request )(struct ArrowAsyncProducer *self, int64_t n)
void(*	cancel )(struct ArrowAsyncProducer *self)
const char *	additional_metadata
void *	private_data
	producer-specific opaque data.

Detailed Description

EXPERIMENTAL: ArrowAsyncProducer represents a 1-to-1 relationship between an async producer and consumer. This object allows the consumer to perform backpressure and flow control on the asynchronous stream processing. This object must be owned by the producer who creates it, and thus is responsible for cleaning it up.

Field Documentation

const char* ArrowAsyncProducer::additional_metadata

Any additional metadata tied to a specific stream of data. This must either be NULL or a valid pointer to metadata which is encoded in the same way schema metadata would be. Non-null metadata must be valid for the lifetime of this object. As an example a producer could use this to provide the total number of rows and/or batches in the stream if known.

void(* ArrowAsyncProducer::cancel)(struct ArrowAsyncProducer *self)

This cancel callback signals a producer that it must eventually stop making calls to on_next_task. It must be idempotent and thread-safe. After calling cancel once, subsequent calls must be NOPs. This must not call any consumer-side handlers other than on_error.

It is not required that calling cancel affect the producer immediately, only that it must eventually stop calling on_next_task and subsequently call release on the async handler. As such, a consumer must be prepared to receive one or more calls to on_next_task even after calling cancel if there are still requested arrays pending.

Successful cancellation should not result in the producer calling on_error, it should finish out any remaining tasks and eventually call release.

Any error encountered during handling a call to cancel must be reported via the on_error callback on the async stream handler.

void(* ArrowAsyncProducer::request)(struct ArrowAsyncProducer *self, int64_t n)

A consumer must call this function to start receiving on_next_task calls.

It must be valid to call this synchronously from within on_next_task or on_schema, but this function must not immediately call on_next_task so as to avoid recursion and reentrant callbacks.

After cancel has been called, additional calls to this function must be NOPs, but allowed. While not cancelled, calling this function must register the given number of additional arrays/batches to be produced with the producer. The producer should only call on_next_task at most the registered number of arrays before propagating backpressure.

Any error encountered by calling request must be propagated by calling the on_error callback of the ArrowAsyncDeviceStreamHandler.

While not cancelled, any subsequent calls to on_next_task, on_error or release should be scheduled by the producer to be called later.

It is invalid for a consumer to call this with a value of n <= 0, producers should error if given such a value.

---

The documentation for this struct was generated from the following file:

• qdb/arrow_abi.h