In previous blog posts we have described at a high level how the
Feldera incremental query engine
works. An incremental query
engine continuously evaluates a set of queries (in our system these
queries are described as database views). The queries depend on some
data sources that continuously change.
The Feldera query engine (named DBSP) receives as inputs the changes
applied to the data sources, and produces as outputs the changes of
the results of the queries. The query engine uses the same data
structure to represent all changes; this representation is used for
inputs, outputs, and the internal data structures. This
representation is the Z-set, and its close relative,
the Indexed Z-set.
In today’s Apparel and Footwear markets, customers desire personalized
experiences, trends shift overnight, and supply chains need constant
optimization. In addition, traditional analytics infrastructure can no longer
provide the necessary decision support as it did in the past. Integration of
near real-time data synchronization, correct and timely analysis, and reduced
inventory costs have become indispensable. Efficient data-driven decisions
require efficient processing of data.
introduced Z-sets as a
representation for database tables and changes to database tables.
In this blog post we explain how an incremental query engine like
Feldera actually operates as seen by a user. We start by motivating
the need for incremental computation in databases. Future blog posts
will describe the internal operations in detail.
Incremental computation is a computation that reuses results that
were previously computed.
Let us assume you have a program P that is given an input I and
produces an output O:
Now let's change the input by modifying it with some change, which
we denote by ΔI:
We would like to avoid computing the entire output again, and ideally
reuse the previously computed O and only adjust it by a change
ΔO. This would be great especially if the output doesn't