Every data pipeline has to deal with historical data, before it can tell you anything interesting. How that happens depends on the system. In Spark, a batch job rereads your historical data on every run, recomputing its results from scratch, even when only a small fraction of the data has changed. The cost of each run and time grow with the size of your data, not the size of the incoming changes.
Feldera takes a different approach. As an incremental view maintenance engine, it reads your history once, up front, then computes only what changes from then on. That one-time step is called a backfill.
What is a backfill
In Feldera, a backfill is when a Feldera pipeline ingests a snapshot of your existing data and builds the internal state it needs to perform incremental updates. Feldera processes inserts, updates, and deletes as changes, and a backfill is one very large batch of changes. Once it completes, Feldera maintains every view by looking only at new changes, without ever recomputing old data again.
How often will I need to backfill?
- Once, when a pipeline starts fresh. A backfill happens when you run a new pipeline, or restart one with its storage cleared.
- Sometimes, when your queries change. Adding views that build on existing tables and views can reuse state Feldera already has. Adding a new table or column from a source requires recomputing that table and anything downstream of it.
- Got late-arriving data? No problem. A late event is just another change. It propagates through the pipeline and updates the affected views, with no reprocessing window.
Feldera can ingest millions of rows per second on a single host. In most cases, this is more than enough. That said, when we see 10s of TBs and/or highly complex SQL, sometimes additional horsepower is required to meet backfill SLAs, and that’s precisely what Feldera’s scale-out feature is for.
Introducing Feldera Scale-Out
To improve backfill performance on the largest use cases, we’re introducing scale-out as a new Feldera enterprise feature. This adds the ability for a Feldera pipeline to run in a cluster that spans multiple machines. In recent POCs, we’ve seen backfill times cut in half. Scale-out builds on Feldera’s Transactions feature, which groups historical data into a single efficient operation, and then scales that work out to as many hosts as you need.
What scale-out means for your team
Every system has to backfill. Scale-out makes that step faster
- Get incremental pipelines live faster. Cutting backfill time means new pipelines compute real-time results sooner.
- Less downtime due to a backfill. Evolve your pipelines freely, add a column or reload a source with ease, so you can meet tighter SLAs on availability.
- Handle enterprise scale with confidence. Terabytes of history and billions of rows are normal for enterprise data, and scale-out grows with the workload as you add hosts.
- Turn it on with one setting. Set the number of hosts, and Feldera does the rest, with no changes to your existing SQL.
How It Works
A Feldera pipeline divides input data into shards, one per worker thread. Each worker independently processes the data in its shard. At some points in the data analysis process, the data needs to be split up differently among workers, or “resharded”. In a single-host pipeline, resharding just moves data around in memory among the workers, but in a multiple-host pipeline, much of the data needs to be sent to other hosts. Feldera does that over the network: each host maintains a connection to each of the other hosts and sends data to be resharded across it.
A single-host instance of Feldera handles all of the input and output connectors on that host. For multi-host pipelines, Feldera instead evenly distributes the input and output connectors across the hosts. For input connectors, this spreads the work of communicating with the sources of the input across the hosts, making better use of CPU cycles and network bandwidth for fetching large amounts of data. For output connectors, it does the same thing for sending the output data to the output sinks.
While data processing is distributed across the cluster, some tasks need to be centralized. A scaled-out Feldera pipeline has an additional process called a “coordinator” that runs in the same pod as the first pipeline. The coordinator initializes the cluster by helping each of the pods find the other ones and divvying up the input and output connectors among the pods. After initialization, its biggest job is to ensure that the pods process data together: when data arrives for an input connector for one pod, the pod starts processing it and notifies the coordinator, which tells the other pods to start processing, too, even if they have not yet received data themselves. This is important because, due to resharding, all the pods must run data processing in lockstep.
The coordinator supports the Feldera pipeline API, the same one that each pipeline pod implements itself. It implements most of the API by calling into each of the pipelines. For example, when the pipeline manager requests pipeline statistics with the /stats API, the coordinator calls /stats on each of the pipeline hosts and combines their results, e.g. by summing the amount of memory used by each pipeline pod to report the total amount of memory used by the pipeline.
Some of the API calls require a little more sophistication to implement across the cluster. Usually this amounts to making sure that the pipeline pods are operating in synchronization. Checkpoints, for example, require the coordinator to tell each of the pipeline pods to prepare for a checkpoint, then in some cases to process further data in lockstep until all of the input connectors across the pods are in a safe spot for a checkpoint, then to complete the checkpoint. Only if all of the pods successfully checkpoint does the overall checkpoint succeed. Transactions are another example: the coordinator ensures that all of the pipeline pods start and complete transactions at the same time rather than independently.
Get started
Scale-out is available in Feldera Enterprise.
For users, scaling out a pipeline only requires setting the “hosts” configuration parameter to a value greater than 1, and perhaps increasing the number of “workers” to match, since Feldera spreads the workers across the hosts. Then, when Feldera launches the pipeline, it creates the specified requested number of Kubernetes pods, which work together to execute the pipeline.
A scaled-out pipeline typically has more CPU cores available to it than a single-host pipeline, since it can use the cores on multiple systems; at the same time, those systems can have fewer CPUs than a single scaled-up pod would require. Similarly, a scaled-out pipeline typically has access to more RAM, more local storage, and more I/O bandwidth than a single machine. The latter can be particularly helpful to fetch historical data more quickly.
