Feldera API
With Feldera, users create data pipelines out of SQL programs and data connectors. A SQL program comprises tables and views. Connectors feed data to input tables in a program or receive outputs computed by views.
This API allows users to create and manage data pipelines, and the programs and connectors that comprise these pipelines.
API concepts
Program. A SQL program with a unique name and a unique ID attached to it. A program contains tables and views. A program needs to be compiled before it can be executed in a pipeline.
Connector. A data connector that can be used to feed input data to SQL tables or consume outputs from SQL views. Every connector has a unique name and identifier. We currently support Kafka and Redpanda. We also support directly ingesting and consuming data via HTTP; see the
pipelines/{pipeline_id}/ingressandpipelines/{pipeline_id}/egressendpoints.Service. A service with a unique name and ID. It represents a service (such as Kafka, etc.) that a connector can refer to in its config. Services are declared separately to reduce duplication and to make it easier to create connectors. A service has its own configuration, which generally includes hostname, port, authentication, and any service parameters.
Pipeline. A pipeline is a running instance of a program and some attached connectors. A client can create multiple pipelines that make use of the same program and connectors. Every pipeline has a unique name and identifier. Deploying a pipeline instantiates the pipeline with the then latest version of the referenced program and connectors. This allows the API to accumulate edits to programs and connectors before use in a pipeline.
Concurrency
All programs have an associated version. This is done to prevent race conditions due to multiple users accessing the same program concurrently. An example is user 1 modifying the program, while user 2 is starting a pipeline for the same program. It would be confusing if the pipeline could end up running the old or the new version.
A version is a monotonically increasing number, associated with each program and pipeline. Every request to compile the program or start a pipeline must include the program id and version number. If the version number isn't equal to the current version in the database, this means that the last version of the program observed by the client is outdated, so the request is rejected.