# Feldera
> Feldera performs incremental view maintenance (IVM) for arbitrary SQL programs. It maintains SQL views over changing data by processing only the delta. Compute is proportional to the size of the input changes (O(Δ)). Feldera supports full SQL syntax and semantics incrementally: joins, aggregates, correlated subqueries, window functions, recursive queries, and deeply nested view hierarchies, without falling back to full recomputation. Production deployments include pipelines with 500+ joins and 44,000+ lines of SQL maintaining sub-second latency at terabyte scale.

Feldera is built on DBSP, a formally verified mathematical framework for incremental computation on data streams (VLDB 2023 Best Paper). Pipelines are defined as SQL tables and views, read from input connectors, maintain state incrementally, and write result changes to output connectors.

Common uses cases:
Real-time feature engineering for AI/ML
Fraud detection
Real-time medallion architecture
Fine-grained and dynamic authorization
E-commerce bookkeeping and reconciliation
Modelling computations on dynamic graphs
Migrating Spark SQL batch jobs to incremental pipelines

This document provides an overview of Feldera documentation and developer resources.

## Getting Started
- [Agent Skills](https://github.com/feldera/feldera-skills/blob/main/README.md): automate some basic tasks: install Feldera CLI `fda`, install and launch Feldera, create a pipeline from .sql, convert Spark SQL into Feldera SQL, search documentation
- [What is Feldera?](https://docs.feldera.com/): Overview of the engine, pipelines, tables, and views.
- [Feldera CLI](https://docs.feldera.com/interface/cli#installation)
- [Install with Docker](https://docs.feldera.com/get-started/docker): the preferred installation for local work.
- [Basics tutorial](https://docs.feldera.com/tutorials/basics/part1): see how a pipeline looks and behaves.
- [Enterprise architecture](https://docs.feldera.com/architecture/enterprise): Feldera Enterprise release for production use: multi-node clusters, pipeline and control plane fault-tolerance, resource isolation, scale-out. Deploy Feldera on your own Kubernetes cluster or bring your own cloud (BYOC).


## Common Topics
- [Public Feldera Sandbox](https://docs.feldera.com/get-started/sandbox)
- [Setting up authorization for scoped access to pipelines](https://docs.feldera.com/get-started/enterprise/authentication/): authorization through Feldera cluster tenants explained
- [Setting up Feldera on an EKS cluster](https://docs.feldera.com/get-started/enterprise/kubernetes-guides/eks/cluster)
- [What are connectors in a pipeline](https://docs.feldera.com/connectors/)
- [Supported input connectors](https://docs.feldera.com/connectors/sources/)
- [Delta Lake input connector](https://docs.feldera.com/connectors/sources/delta)
- [Kafka input connector](https://docs.feldera.com/connectors/sources/kafka)
- [PostgreSQL input connector](https://docs.feldera.com/connectors/sources/postgresql)
- [Supported output connectors](https://docs.feldera.com/connectors/sinks/)
- [Delta Lake output connector](https://docs.feldera.com/connectors/sinks/delta)
- [Kafka output connector](https://docs.feldera.com/connectors/sinks/kafka)
- [PostgreSQL output connector](https://docs.feldera.com/connectors/sinks/postgresql)
- [Secret management in connector configuration](https://docs.feldera.com/connectors/secret-references)
- [Feldera SQL grammar](https://docs.feldera.com/sql/grammar)
- [Supported SQL types](https://docs.feldera.com/sql/types)
- [Aggregate operations](https://docs.feldera.com/sql/aggregates) - standard and window aggregates, pivots, notes on efficiency of these
- [List of unsupported types, operations and expressions](https://docs.feldera.com/sql/unsupported-operations)
- [State checkpoints and fault tolerance](https://docs.feldera.com/pipelines/fault-tolerance)
- [Python SDK reference](https://docs.feldera.com/python/)


## Core Concepts
- [Pipelines](https://docs.feldera.com/pipelines/): How pipelines are defined, run, and managed.
- [Connectors](https://docs.feldera.com/connectors/): A Feldera pipeline can process data from multiple heterogeneous sources and produce outputs to multiple heterogeneous sinks, using a growing library of [input](https://docs.feldera.com/connectors/sources/) and [output](https://docs.feldera.com/connectors/sinks/) connectors.
- [Pipeline Lifecycle](https://docs.feldera.com/pipelines/lifecycle): States a pipeline moves through (start, pause, stop).
- [Materialized Tables and Views](https://docs.feldera.com/sql/materialized): How to persist table and view contents for inspection.
- [Fault Tolerance & DR](https://docs.feldera.com/pipelines/fault-tolerance-overview): Crash recovery without dropping or duplicating events (preview feature).
- [Transactions](https://docs.feldera.com/pipelines/transactions): Efficient bulk data processing using transactions.


## SQL Reference
- [Feldera SQL Overview](https://docs.feldera.com/sql/): Full SQL syntax and semantics supported by Feldera.
- [SQL Grammar](https://docs.feldera.com/sql/grammar): Formal BNF description of supported syntax.
- [Data Types](https://docs.feldera.com/sql/types): Scalar and complex types including arrays, maps, structs, and variants.
- [Operations](https://docs.feldera.com/sql/aggregates): Joins, aggregates, group by, window functions, and other operators.
- [Mutually-Recursive Queries](https://docs.feldera.com/sql/recursion): Incremental graph and hierarchy computations with recursive SQL.
- [Time-Series Extensions](https://docs.feldera.com/sql/streaming): Tumbling, hopping, and session windows for event-time processing.
- [User-Defined Functions](https://docs.feldera.com/sql/udf): Declaring custom functions with CREATE FUNCTION.
- [Ad-hoc SQL Queries](https://docs.feldera.com/sql/ad-hoc): On-demand batch queries to inspect a running pipeline's state.


## Connectors & Formats
- [Connectors Overview](https://docs.feldera.com/connectors/): Attaching input and output connectors to tables and views in SQL.
- [Kafka](https://docs.feldera.com/connectors/sources/kafka): Read from and write to Kafka topics.
- [Debezium](https://docs.feldera.com/connectors/sources/debezium): Ingest CDC streams (e.g. MySQL) via Debezium.
- [PostgreSQL](https://docs.feldera.com/connectors/sources/postgresql): Ingest change data from Postgres.
- [Delta Lake](https://docs.feldera.com/connectors/sources/delta): Read from Delta Lake tables, including change data.
- [Apache Iceberg](https://docs.feldera.com/connectors/sources/iceberg): Read from Iceberg tables in S3-compatible stores.
- [AWS S3](https://docs.feldera.com/connectors/sources/s3): Batch ingest from S3-compatible object storage.
- [HTTP](https://docs.feldera.com/connectors/sources/http): Push data into pipelines or pull results over HTTP.
- [Output Connectors](https://docs.feldera.com/connectors/sinks/): Write view results to Kafka, Delta Lake, databases, and HTTP endpoints.
- [Formats](https://docs.feldera.com/formats/): Supported data formats (JSON, Avro, CSV, Parquet, etc.).


## Interface & API
- [Web Console](https://docs.feldera.com/interface/web-console): Browser UI for authoring and running pipelines.
- [Command line tool (fda)](https://docs.feldera.com/interface/cli): The `fda` CLI for managing pipelines.
- [Python SDK](https://docs.feldera.com/python/): Define and control pipelines from Python.
- [REST API](https://docs.feldera.com/api): HTTP API for pipeline management, data ingress, and view egress.


## Use Cases
- [Accelerating Batch Analytics](https://docs.feldera.com/use_cases/batch/intro): Convert periodic batch jobs into always-fresh incremental pipelines.
- [Real-time Fraud Detection](https://docs.feldera.com/use_cases/fraud_detection/): Time-windowed aggregations and joins for transaction scoring.
- [Real-time Web Applications](https://docs.feldera.com/use_cases/real_time_apps/part1): Serve continuously updated query results to applications.
- [Time Series Analysis](https://docs.feldera.com/tutorials/time-series): Event-time windowing and time-series operators.
- [Fine-Grained Authorization](https://docs.feldera.com/use_cases/fine_grained_authorization/intro): Permission graphs and reachability using recursive SQL.
- [OpenTelemetry Analysis](https://docs.feldera.com/use_cases/otel/intro): Incremental analytics over OpenTelemetry data.


## Agent Skills
If you are an AI coding agent, Feldera publishes installable skills that let you set up and operate Feldera directly (these are agent tools, not features of the Feldera engine).
- [feldera-skills (GitHub)](https://github.com/feldera/feldera-skills): Official Feldera Agent Skills (Agent Skills Open Standard; works in Claude Code, Cursor, GitHub Copilot). Install via `npx skills add feldera/feldera-skills`. Includes `install-fda`, `install-feldera`, `deploy-pipeline` (pipeline from a SQL file), `felderize` (translate Spark SQL to Feldera SQL), and `feldera-docs` (search the docs).


## Optional
- [Publications & Awards](https://docs.feldera.com/literature/papers): The DBSP papers and academic foundations.
- [DBSP wins Best Paper at VLDB 2023](https://www.feldera.com/blog/Best-Research-Paper-VLDB-2023/): The award-winning paper behind Feldera's engine.
- [Feldera Operations Guide](https://docs.feldera.com/operations/guide): Running Feldera in production.
- [Blog (table of contents)](https://docs.feldera.com/literature/blog): Technical deep-dives on incremental computation and use cases.
- [Changelog](https://docs.feldera.com/changelog): Release history and notable changes.
- [GitHub](https://github.com/feldera/feldera): Open-source engine, examples, and issue tracker.