Skip to main content

Create a new connector.

Create a new connector.

Request Body REQUIRED
config object REQUIRED

A data connector's configuration

enable_output_buffer boolean

Enable output buffering.

The output buffering mechanism allows decoupling the rate at which the pipeline pushes changes to the output transport from the rate of input changes.

By default, output updates produced by the pipeline are pushed directly to the output transport. Some destinations may prefer to receive updates in fewer bigger batches. For instance, when writing Parquet files, producing one bigger file every few minutes is usually better than creating small files every few milliseconds.

To achieve such input/output decoupling, users can enable output buffering by setting the enable_output_buffer flag to true. When buffering is enabled, output updates produced by the pipeline are consolidated in an internal buffer and are pushed to the output transport when one of several conditions is satisfied:

  • data has been accumulated in the buffer for more than max_output_buffer_time_millis milliseconds.
  • buffer size exceeds max_output_buffer_size_records records.

This flag is false by default.

max_output_buffer_size_records integer

Maximum number of updates to be kept in the output buffer.

This parameter bounds the maximal size of the buffer. Note that the size of the buffer is not always equal to the total number of updates output by the pipeline. Updates to the same record can overwrite or cancel previous updates.

By default, the buffer can grow indefinitely until one of the other output conditions is satisfied.

NOTE: this configuration option requires the enable_output_buffer flag to be set.

max_output_buffer_time_millis integer

Maximum time in milliseconds data is kept in the output buffer.

By default, data is kept in the buffer indefinitely until one of the other output conditions is satisfied. When this option is set the buffer will be flushed at most every max_output_buffer_time_millis milliseconds.

NOTE: this configuration option requires the enable_output_buffer flag to be set.

format object

Data format specification used to parse raw data received from the endpoint or to encode data sent to the endpoint.

config object

Format-specific parser or encoder configuration.

name string REQUIRED

Format name, e.g., "csv", "json", "bincode", etc.

max_queued_records int64

Backpressure threshold.

Maximal number of records queued by the endpoint before the endpoint is paused by the backpressure mechanism.

For input endpoints, this setting bounds the number of records that have been received from the input transport but haven't yet been consumed by the circuit since the circuit, since the circuit is still busy processing previous inputs.

For output endpoints, this setting bounds the number of records that have been produced by the circuit but not yet sent via the output transport endpoint nor stored in the output buffer (see enable_output_buffer).

Note that this is not a hard bound: there can be a small delay between the backpressure mechanism is triggered and the endpoint is paused, during which more data may be queued.

The default is 1 million.

transport REQUIRED

Transport-specific endpoint configuration passed to crate::OutputTransport::new_endpoint and crate::InputTransport::new_endpoint.

description string REQUIRED

Connector description.

name string REQUIRED

Connector name.

Responses
201

Connector successfully created

Schema OPTIONAL
connector_id uuid

Unique connector id.

409

A connector with this name already exists in the database

Schema OPTIONAL
details object

Detailed error metadata. The contents of this field is determined by error_code.

error_code string

Error code is a string that specifies this error type.

message string

Human-readable error message.