The Feldera Blog
An unbounded stream of technical articles from the Feldera team
Batch Analytics at Warp Speed: A User Guide
The laws of computational complexity tell us that some computations require time. While we may not make batch jobs much faster, we can replace them with something better: always-on incremental pipelines that update the results in real time as input data changes.
Toward Real-Time Medallion Architecture
We replaced periodic Spark jobs with the Feldera Incremental View Maintenance (IVM) engine. Feldera picks up changes to lower-tier tables and updates higher-tier tables in real-time, reducing the end-to-end latency of the pipeline from hours to minutes.
Universal IVM: Incremental View Maintenance for the Modern Data Stack
Incremental View Maintenance is a paradoxical concept: it's lived in the database community’s collective conscience for decades, yet has never fully materialized as a feature in any modern DB. In my view, a complete IVM engine must: support arbitrary SQL queries, over data of any size, fully incrementally—processing input changes without full recomputation.
Incremental Update 23
Better setting settings display.
Incremental Update 22
New rust crate, storage on by default and new delta-lake connector settings.
Cutting Down Rust Compile Times From 30 to 2 Minutes With One Thousand Crates
We compile SQL into Rust. One customer wrote so much SQL, it turned into 100k+ lines of Rust — and took 30+ minutes to build. The fix? Split it into over a thousand crates. Now we get full CPU utilization and sub-3-minute builds. Here's how we made Rust compile times scale with hardware.
Incremental Update 21
Faster pipeline compilation, and better ad-hoc queries.
Feldera: three tools for the price of one
Feldera is not just a database engine. Feldera SQL is in some respects substantially more powerful than standard SQL, enabling new use cases.
Incremental Update 20
Everything that's new in v0.41.