Incremental Update 9

Today, we’re happy to announce the release of feldera v0.29! This release includes various bug-fixes for storage, the web-console and our compiler and a new feature, emit_final, which we will introduce in more detail in this post.

emit_final

When dealing with time series data, incremental updates can be a double-edged sword: while they provide a continuous stream of insights, they also result in a barrage of updates that may overwhelm downstream applications. Feldera's new emit_final feature brings a simple yet powerful way to deal with this problem, allowing you to receive only the final, stable value of a computation when it’s ready—no unnecessary noise, no handling of intermediate results.

Let’s illustrate this with a daily balance computation for transaction data. In the example below, we set up a transactions table and two materialized views. One view (daily_financial) tracks the ongoing daily balance for each account, updating continuously as transactions are added. The other view (daily_financial_final) uses the emit_final attribute, ensuring that only a single, finalized result is output at the end of each day. We also added a lateness annotation on transaction_timestamp. This is necessary for emit_final to know when it's safe to emit the final value. In our case, a lateness of 0 means as soon as we see a transaction for the next day, we can emit the final value for the previous day.

CREATE TABLE transactions (
    transaction_id INT NOT NULL PRIMARY KEY,
    transaction_timestamp TIMESTAMP NOT NULL LATENESS INTERVAL 0 DAY,
    account_id INT NOT NULL,
    amount INT NOT NULL
) WITH ('materialized' = 'true');

CREATE MATERIALIZED VIEW daily_financial
AS
SELECT
    TIMESTAMP_TRUNC(transaction_timestamp, DAY) as day,
    account_id,
    SUM(amount) AS daily_balance
FROM
    transactions
GROUP BY
    TIMESTAMP_TRUNC(transaction_timestamp, DAY), account_id;

CREATE MATERIALIZED VIEW daily_financial_final
WITH ('emit_final' = 'day')
AS
SELECT
    *
FROM
    daily_financial;

If you run this example in our sandbox, you can use the following ad-hoc queries to see emit_final in action:

// Insert two transactions for Oct 1
INSERT INTO transactions (transaction_id, transaction_timestamp, account_id, amount) VALUES (1, '2024-10-01 10:00:00', 101, 200.00);
INSERT INTO transactions (transaction_id, transaction_timestamp, account_id, amount) VALUES (2, '2024-10-01 14:30:00', 101, -50.00);

// Query the views
SELECT * FROM daily_financial;
(1 row)
SELECT * FROM daily_financial_final;
(0 rows) // Oct 10 has not been finalized yet

// Insert a new transaction for Oct 2, seeing a transaction for Oct 2 finalizes Oct 1 because lateness is 0
INSERT INTO transactions (transaction_id, transaction_timestamp, account_id, amount)
VALUES (2, '2024-10-02 09:30:00', 101, 70.00);

// Query the views again
SELECT * FROM daily_financial; // Contains result for Oct 1 and Oct 2 
SELECT * FROM daily_financial_final; // Contains final result for Oct 1

You can play around with the lateness annotation too, e.g., setting it to 1 day instead of 0 would mean there is an additional settlement period of one day for transactions to arrive, hence only an insert of a transaction for Oct 3 would finalize the transactions of Oct 1.

If you want to learn more about emit_final and time series analysis in general you should also check out our new tutorial in the documentation about it.

Outlook

Ben and Simon have been working hard behind the scenes on making fault tolerant pipelines a reality. A big PR got merged for this feature during the release window. Stay tuned for fault tolerant pipelines coming soon to the Feldera Enterprise offering.