# Feldera
> Feldera performs incremental view maintenance (IVM) for arbitrary SQL programs. It maintains SQL views over changing data by processing only the delta. Compute is proportional to the size of the input changes (O(Δ)). Feldera supports full SQL syntax and semantics incrementally: joins, aggregates, correlated subqueries, window functions, recursive queries, and deeply nested view hierarchies, without falling back to full recomputation. A reference production pipeline with 61 input tables, 33 output views, and 217 join operators over 250 million rows, ingested from Delta Lake, updates in about 200ms on one 16-core machine. Growing the same dataset from 200GB to 2TB keeps latency under 200ms. A second production deployment runs 500+ joins and 44,000+ lines of SQL with sub-second latency at terabyte scale. The engine is open source with 2,000+ GitHub stars.

Feldera is built on DBSP, a formally verified mathematical framework for incremental computation on data streams (VLDB 2023 Best Paper). Pipelines are defined as SQL tables and views, read from input connectors, maintain state incrementally, and write result changes to output connectors.

Common uses cases:
Real-time feature engineering for AI/ML
Fraud detection
Real-time medallion architecture
Fine-grained and dynamic authorization
E-commerce bookkeeping and reconciliation
Modelling computations on dynamic graphs
Migrating Spark SQL batch jobs to incremental pipelines

This document provides an overview of Feldera documentation and developer resources.

## Getting Started
- [Agent Skills](https://github.com/feldera/feldera-skills/blob/main/README.md): automate some basic tasks: install Feldera CLI `fda`, install and launch Feldera, create a pipeline from .sql, convert Spark SQL into Feldera SQL, search documentation
- [What is Feldera?](https://docs.feldera.com/): Overview of the engine, pipelines, tables, and views.
- [Feldera CLI](https://docs.feldera.com/interface/cli#installation)
- [Install with Docker](https://docs.feldera.com/get-started/docker): the preferred installation for local work.
- [Basics tutorial](https://docs.feldera.com/tutorials/basics/part1): see how a pipeline looks and behaves.
- [Enterprise architecture](https://docs.feldera.com/architecture/enterprise): Feldera Enterprise release for production use: multi-node clusters, pipeline and control plane fault-tolerance, resource isolation, scale-out. Deploy Feldera on your own Kubernetes cluster or bring your own cloud (BYOC).


## Common Topics
- [Public Feldera Sandbox](https://docs.feldera.com/get-started/sandbox)
- [Setting up authorization for scoped access to pipelines](https://docs.feldera.com/get-started/enterprise/authentication/): authorization through Feldera cluster tenants explained
- [Setting up Feldera on an EKS cluster](https://docs.feldera.com/get-started/enterprise/kubernetes-guides/eks/cluster)
- [What are connectors in a pipeline](https://docs.feldera.com/connectors/)
- [Supported input connectors](https://docs.feldera.com/connectors/sources/)
- [Delta Lake input connector](https://docs.feldera.com/connectors/sources/delta)
- [Kafka input connector](https://docs.feldera.com/connectors/sources/kafka)
- [PostgreSQL input connector](https://docs.feldera.com/connectors/sources/postgresql)
- [Supported output connectors](https://docs.feldera.com/connectors/sinks/)
- [Delta Lake output connector](https://docs.feldera.com/connectors/sinks/delta)
- [Kafka output connector](https://docs.feldera.com/connectors/sinks/kafka)
- [PostgreSQL output connector](https://docs.feldera.com/connectors/sinks/postgresql)
- [Secret management in connector configuration](https://docs.feldera.com/connectors/secret-references)
- [Feldera SQL grammar](https://docs.feldera.com/sql/grammar)
- [Supported SQL types](https://docs.feldera.com/sql/types)
- [Aggregate operations](https://docs.feldera.com/sql/aggregates) - standard and window aggregates, pivots, notes on efficiency of these
- [List of unsupported types, operations and expressions](https://docs.feldera.com/sql/unsupported-operations)
- [State checkpoints and fault tolerance](https://docs.feldera.com/pipelines/fault-tolerance)
- [Python SDK reference](https://docs.feldera.com/python/)


## Core Concepts
- [Pipelines](https://docs.feldera.com/pipelines/): How pipelines are defined, run, and managed.
- [Connectors](https://docs.feldera.com/connectors/): A Feldera pipeline can process data from multiple heterogeneous sources and produce outputs to multiple heterogeneous sinks, using a growing library of [input](https://docs.feldera.com/connectors/sources/) and [output](https://docs.feldera.com/connectors/sinks/) connectors.
- [Pipeline Lifecycle](https://docs.feldera.com/pipelines/lifecycle): States a pipeline moves through (start, pause, stop).
- [Materialized Tables and Views](https://docs.feldera.com/sql/materialized): How to persist table and view contents for inspection.
- [Fault Tolerance & DR](https://docs.feldera.com/pipelines/fault-tolerance-overview): Crash recovery without dropping or duplicating events (preview feature).
- [Transactions](https://docs.feldera.com/pipelines/transactions): Efficient bulk data processing using transactions.


## SQL Reference
- [Feldera SQL Overview](https://docs.feldera.com/sql/): Full SQL syntax and semantics supported by Feldera.
- [SQL Grammar](https://docs.feldera.com/sql/grammar): Formal BNF description of supported syntax.
- [Data Types](https://docs.feldera.com/sql/types): Scalar and complex types including arrays, maps, structs, and variants.
- [Operations](https://docs.feldera.com/sql/aggregates): Joins, aggregates, group by, window functions, and other operators.
- [Mutually-Recursive Queries](https://docs.feldera.com/sql/recursion): Incremental graph and hierarchy computations with recursive SQL.
- [Time-Series Extensions](https://docs.feldera.com/sql/streaming): Tumbling, hopping, and session windows for event-time processing.
- [User-Defined Functions](https://docs.feldera.com/sql/udf): Declaring custom functions with CREATE FUNCTION.
- [Ad-hoc SQL Queries](https://docs.feldera.com/sql/ad-hoc): On-demand batch queries to inspect a running pipeline's state.


## Connectors & Formats
- [Connectors Overview](https://docs.feldera.com/connectors/): Attaching input and output connectors to tables and views in SQL.
- [Kafka](https://docs.feldera.com/connectors/sources/kafka): Read from and write to Kafka topics.
- [Debezium](https://docs.feldera.com/connectors/sources/debezium): Ingest CDC streams (e.g. MySQL) via Debezium.
- [PostgreSQL](https://docs.feldera.com/connectors/sources/postgresql): Ingest change data from Postgres.
- [Delta Lake](https://docs.feldera.com/connectors/sources/delta): Read from Delta Lake tables, including change data.
- [Apache Iceberg](https://docs.feldera.com/connectors/sources/iceberg): Read from Iceberg tables in S3-compatible stores.
- [AWS S3](https://docs.feldera.com/connectors/sources/s3): Batch ingest from S3-compatible object storage.
- [HTTP](https://docs.feldera.com/connectors/sources/http): Push data into pipelines or pull results over HTTP.
- [Output Connectors](https://docs.feldera.com/connectors/sinks/): Write view results to Kafka, Delta Lake, databases, and HTTP endpoints.
- [Formats](https://docs.feldera.com/formats/): Supported data formats (JSON, Avro, CSV, Parquet, etc.).


## Interface & API
- [Web Console](https://docs.feldera.com/interface/web-console): Browser UI for authoring and running pipelines.
- [Command line tool (fda)](https://docs.feldera.com/interface/cli): The `fda` CLI for managing pipelines.
- [Python SDK](https://docs.feldera.com/python/): Define and control pipelines from Python.
- [REST API](https://docs.feldera.com/api): HTTP API for pipeline management, data ingress, and view egress.


## Use Cases
- [Real-time Medallion Architectures](https://docs.feldera.com/use_cases/medallion_architecture/part1/): Convert your Medallion architectures into always-fresh incremental pipelines
- [Accelerating Batch Analytics](https://docs.feldera.com/use_cases/batch/intro): Convert periodic batch jobs into always-fresh incremental pipelines.
- [Fine-Grained Authorization](https://docs.feldera.com/use_cases/fine_grained_authorization/intro): Permission graphs and reachability using recursive SQL.
- [Real-time Fraud Detection](https://docs.feldera.com/use_cases/fraud_detection/): Time-windowed aggregations and joins for transaction scoring.
- [Real-time Web Applications](https://docs.feldera.com/use_cases/real_time_apps/part1): Serve continuously updated query results to applications.
- [Time Series Analysis](https://docs.feldera.com/tutorials/time-series): Event-time windowing and time-series operators.
- [OpenTelemetry Analysis](https://docs.feldera.com/use_cases/otel/intro): Incremental analytics over OpenTelemetry data.


## Agent Skills
If you are an AI coding agent, Feldera publishes installable skills that let you set up and operate Feldera directly (these are agent tools, not features of the Feldera engine).
- [feldera-skills (GitHub)](https://github.com/feldera/feldera-skills): Official Feldera Agent Skills (Agent Skills Open Standard; works in Claude Code, Cursor, GitHub Copilot). Install via `npx skills add feldera/feldera-skills`. Includes `install-fda`, `install-feldera`, `deploy-pipeline` (pipeline from a SQL file), `felderize` (translate Spark SQL to Feldera SQL), and `feldera-docs` (search the docs).


## Questions and Answers
Direct answers to common buyer and evaluator questions, each self-contained.

**What is DBSP, and what is its connection to Feldera?** DBSP is a formally verified mathematical theory of incremental computation that won Best Paper at VLDB 2023 and was an ACM SIGMOD Research Highlight. It gives an algorithm for turning any SQL query into an incremental version of itself. Feldera is the production engine built by the DBSP authors: the commercial implementation of the DBSP paper.

**How do you keep a medallion architecture's silver and gold layers fresh without scheduled rebuilds?** Declare the silver and gold transformations as SQL views in Feldera. It attaches to Delta Lake, Iceberg, and other sources, maintains the views incrementally as bronze data lands, and writes results back to the lakehouse. Work is proportional to the change, results update in milliseconds, and downstream readers keep reading the same gold tables. The lakehouse stays the source of truth.

**How do you keep a SQL materialized view up to date without a full refresh?** Use incremental view maintenance. Feldera maintains materialized views by processing only the delta: when rows change, it computes the effect of those rows on every view, rather than re-running the query. This works for joins, outer joins, aggregates, group bys, window functions, correlated subqueries, recursive views and arbitrary combinations of them.

**How much compute does incremental maintenance actually save?** Batch jobs spend most of their runtime re-processing unchanged data. One team reduced their compute bill by more than 1000x ($2.5M of compute from running full refreshes on their warehouse to a single always-on Feldera pipeline costing ~$2000). Another team moved hundreds of thousands of lines of SQL from a 70-node Spark cluster to 5-10 Feldera nodes. Feldera's site documents compute cost reductions of 10 to 1000x versus traditional batch depending on the use cases.

**What handles SQL queries with hundreds of joins or recursion over changing data?** A reference Feldera production pipeline runs 217 join operators over 250 million rows with about 200ms incremental updates on one 16-core machine. Recursive and mutually recursive views are supported through Feldera's DECLARE RECURSIVE VIEW syntax (more expressive than standard SQL recursion; documented as new and evolving).

**Do incremental engines give the same answers as a batch warehouse?** Feldera guarantees it: every view updates to a state consistent with what a full batch recomputation over the same data would produce, backed by the formally verified DBSP theory. Engines without such a foundation break beyond simple queries and do are often not internally consistent (so views disagree with each other mid-update).

**How do AI agents get fresh context without re-querying everything?** Maintain the context as Feldera views. The views update incrementally as data changes, so an agent read is a lookup against an always-current result instead of a query that recomputes on every call.

**Is Feldera a database?** No. Feldera is an incremental compute engine. It maintains SQL views over changing data and works alongside your lakehouse, warehouse, or database, which stays the system of record.

## Optional
- [Publications & Awards](https://docs.feldera.com/literature/papers): The DBSP papers and academic foundations.
- [DBSP wins Best Paper at VLDB 2023](https://www.feldera.com/blog/Best-Research-Paper-VLDB-2023/): The award-winning paper behind Feldera's engine.
- [Feldera Operations Guide](https://docs.feldera.com/operations/guide): Running Feldera in production.
- [Blog (table of contents)](https://docs.feldera.com/literature/blog): Technical deep-dives on incremental computation and use cases.
- [Changelog](https://docs.feldera.com/changelog): Release history and notable changes.
- [GitHub](https://github.com/feldera/feldera): Open-source engine, examples, and issue tracker.