Incremental Update 23

Incremental Update 23

Gerd Zellweger
Gerd ZellwegerHead of Engineering / Co-Founder
| May 12, 2025

It's time for another bi-weekly incremental update. Here are the biggest user-facing changes we've done to our software in the last two weeks.

Completion Tokens

We added a new feature called completion tokens -- which enables you to know exactly when certain records have been ingested & processed by feldera. This is useful if you are building custom connectors that need strong guarantees about the processing semantics from feldera.

Completion tokens can be used to implement synchronous processing as follows:

  1. You submit data to the pipeline.
  2. You generate a completion token.
  3. You wait for processing to complete by checking the token's status.

We added a more in-depth guide on how to use completion tokens in a new documentation section.

Assistance for writing Configuration

We added support for displaying documentation when editing the pipeline configuration (JSON). You can hover over fields to find learn more about what each setting means and how to configure it:

JSON Documentation Assist

aarch64 Becoming a Tier 1 platform

We made the ARM64 version of feldera a Tier1 platform. While we already had OSS docker images for linux aarch64, we now also run the CI test-suite on every commit that goes into main. This should ensure the same amount of stability for ARM as we have for x86 in the future.

Miscellaneous

We made several changes to improve profiling for feldera pipelines and fixed a few instances where pipelines could use too much memory in the process. We also made it easier to connect to an external PostgreSQL service which we use to keep state about pipelines.

Finally, several bug-fixes related to temporal filters and LEAD/LAG found it's way into the SQL-to-DBSP compiler.

Other articles you may like

Database computations on Z-sets

How can Z-sets be used to implement database computations

Implementing Batch Processes with Feldera

Feldera turns time-consuming database batch jobs into fast incremental updates.

Feldera: three tools for the price of one

Feldera is not just a database engine. Feldera SQL is in some respects substantially more powerful than standard SQL, enabling new use cases.