It's time for another bi-weekly incremental update. Here are the biggest user-facing changes we've done to our software in the last two weeks.
Completion Tokens
We added a new feature called completion tokens -- which enables you to know exactly when certain records have been ingested & processed by feldera. This is useful if you are building custom connectors that need strong guarantees about the processing semantics from feldera.
Completion tokens can be used to implement synchronous processing as follows:
- You submit data to the pipeline.
- You generate a completion token.
- You wait for processing to complete by checking the token's status.
We added a more in-depth guide on how to use completion tokens in a new documentation section.
Assistance for writing Configuration
We added support for displaying documentation when editing the pipeline configuration (JSON). You can hover over fields to find learn more about what each setting means and how to configure it:
aarch64 Becoming a Tier 1 platform
We made the ARM64 version of feldera a Tier1 platform. While we already had OSS docker images for linux aarch64, we now also run the CI test-suite on every commit that goes into main. This should ensure the same amount of stability for ARM as we have for x86 in the future.
Miscellaneous
We made several changes to improve profiling for feldera pipelines and fixed a few instances where pipelines could use too much memory in the process. We also made it easier to connect to an external PostgreSQL service which we use to keep state about pipelines.
Finally, several bug-fixes related to temporal filters and LEAD/LAG found it's way into the SQL-to-DBSP compiler.
