Incremental Update 10

Today, we’re happy to announce the release of feldera v0.30! This release includes various bug-fixes for storage, the web-console and our compiler and a new SQL type, row.

ROW

We added ROW which can be helpful in situations where you need to assemble some fields into a single one. ROW works similar to struct (and it's represented as a struct under the hood). The difference is that rows which are constructed as part of views can have unnamed fields. Here is an example that uses the ROW type twice, once in a table and once in a view:

CREATE TABLE employees (
    employee_id INTEGER NOT NULL PRIMARY KEY,
    employee_name ROW(first_name varchar, last_name varchar),
    department VARCHAR
);

CREATE MATERIALIZED VIEW q0 AS SELECT ROW(employees.employee_name.first_name, department) FROM employees;

You can interact with row values similar to how you would interact with a struct, for example here we insert into the employees table and then query the q0 view from the SQL snippet above:

Storage

We've also focused on optimizing storage for a better experience. Here’s two regressions that we addressed:

1. Automatic Cleanup of Old Files: Previously, pipelines would sometimes hold onto old, unused files, even when they were no longer needed. This release introduces a fix that discards all outdated files during pipeline execution.
2. Memory Usage: We received feedback on high memory consumption when storage was enabled. In response, we’ve fixed a regression to cap memory usage in case merging is the bottleneck.

These fixes are directly based on user reports, so please continue to share your experience and insights. Let us know how these improvements are working for you and where we can keep optimizing!

Outlook

More exciting features are coming to Feldera soon:

• Fault tolerance as a preview feature in our Enterprise edition—keep crashing the pods running your pipeline without losing any data
• Recursive queries—incrementally compute on graphs
• Improved error handling
• A UI redesign