Your data team doesn't need streaming (yet)
Most teams reach for streaming before a batch job has ever failed them. Here is how to tell when the time has come.
Every data team hits the moment when somebody says “we need real time”. This post is about what to do when that happens.
Ask what “real time” means
Ask the person who wants real time how they use the data. Most answers fall into one of three groups:
- A dashboard that somebody looks at once a day.
- An alert that must fire within an hour.
- A product feature that needs data within seconds.
Only the last group needs streaming. The first two run well on a batch job that runs every 15 minutes.
The cost of streaming
A streaming pipeline has more moving parts than a batch job:
| Concern | Batch | Streaming |
|---|---|---|
| Backfill | Re-run the job | Replay the topic, handle duplicates |
| Schema change | Update the model | Coordinate producers and consumers |
| Debugging | Read the log | Trace events across services |
The costs are not a reason to avoid streaming. They are a reason to wait until a use case pays for them.
A cheaper first step
Shorten the batch interval first. A job that ran nightly can run hourly with a WHERE updated_at > @last_run filter:
SELECT *
FROM source.orders
WHERE updated_at > @last_runIf nobody notices the change, nobody needed streaming.