Every system that processes data eventually has to answer one question. When is the data complete enough to be moved to the compute stage?. A program adding up a day’s sales needs to know whether all of today’s sales have actually arrived.
For data stored in a file, the answer is trivial because the file has an end. However, for data that arrives continuously and never stops, there is no clean answer, and how a system resolves that gap is the difference between batch processing and streaming.. Batch processing waits for completeness.
It collects data up to a natural boundary, a closing time, or a finished file, and then computes over the whole set at once. Streaming prioritizes completeness for speed. It produces answers continuously from data that is still arriving, which means it has to estimate when enough data has come in and handle the cases where that estimate is wrong.
This trade-off between completeness and latency is the key consideration when dealing with streaming and batch.. In this article, we will cover the strategies on each side and what each one costs.. On the batch side, that means full and incremental loads and large-window aggregation, with micro-batch sitting in between..
On the streaming side, the territory runs through tumbling, sliding, and session windows, watermarks and late data, the lambda and kappa architectures, and the often-misunderstood meaning of exactly-once processing.
