Streaming Database: 6 Key Components, Uses & Design Principles to Keep in Mind

Sam Erickson
Updated yesterday
0

Key Takeaways

A streaming data architecture connects continuous events to timely analytics and downstream actions; a streaming database can implement part of that architecture.
Separate six responsibilities: event production, ingestion, data contracts, processing, durable storage, and serving.
Correct results require explicit policies for event time, late arrivals, duplicates, recovery, and replay.
Streaming improves freshness, not every workload. Batch remains useful for delayed analysis and large-scale recomputation.
Scaling depends on keys, partitions, processing state, and sink capacity—not just adding workers.

A streaming data architecture ingests and processes continuously arriving events so applications and analysts can act without waiting for the next scheduled batch. A user's click on a webpage and a sensor's temperature reading are concrete examples: each becomes an event that can update a result or trigger an action. A streaming database is one possible component, not another name for the entire pipeline.

What Is a Streaming Database?

Here, streaming database means a database that ingests changing data and incrementally maintains queryable results. Materialize, for example, continuously updates materialized views as new data arrives rather than requiring you to rerun the full query for every change.

Keep the roles distinct. A broker transports and retains events; a processor transforms them; a streaming database exposes continuously maintained results. An operational database typically serves application reads and writes, while a warehouse supports analytical queries. Products can combine these capabilities, but accepting streaming input alone doesn't make them interchangeable. See OLTP versus OLAP for the transactional–analytical distinction.

When Streaming Data Architecture Is Worthwhile

Choose streaming when the value of a result declines while you wait: detecting repeated authentication failures, alerting on equipment temperatures, updating advertising metrics, or refreshing product-usage dashboards. Continuous processing can update patterns and aggregates as events arrive; it doesn't make batch processing inherently bad at finding patterns.

Design question	Streaming	Batch
When do results change?	Continuously as input arrives	When a bounded job runs
When is input complete?	Often requires a window and lateness policy	The job operates on a defined dataset
What must you operate?	Long-running processing, state, and recovery	Scheduled jobs, retries, and backfills
What should you optimize?	Freshness alongside correctness and cost	Completion time alongside correctness and cost

These are processing models, not mutually exclusive platform choices. A practical architecture can serve live results while retaining the same events for batch reconciliation and historical recomputation.

Six Key Components of a Streaming Data Pipeline

These are six logical responsibilities, not six mandatory products. A streaming database or managed platform may implement several together.

1. Event Producers and Source Connectors

Sources include application events, web logs, security logs, advertising events, and device readings. Database changes can enter through change data capture (CDC): capturing inserts, updates, and deletes as changes rather than repeatedly scanning the whole database.

Define an event identifier, business key, occurrence timestamp, and payload contract. If an application must update its database and publish a corresponding event, consider a transactional outbox: record the event with the business change, then publish it through CDC. This avoids relying on two independent writes succeeding together.

2. Ingestion, Transport, and Retention

Apache Kafka and Amazon Kinesis Data Streams provide ingestion, transport, and retained event data. They decouple producers from consumers and let consumers catch up within the available retention period. Kafka also includes processing capabilities through Kafka Streams, but its broker and that processing library are separate roles.

Choose partition keys around the entity whose events need related processing, such as a device or account. Kafka orders records within a partition, not across an entire multi-partition topic. The Kafka architecture guide explains that foundation.

3. Data Contracts, Serialization, and Validation

Serialization turns an event into transferable bytes; its schema defines the structure consumers expect. Avro, Protobuf, and JSON Schema have different evolution rules. A schema registry can store schema versions and check proposed changes against a configured compatibility policy.

Choose compatibility for your deployment and replay requirements, not just today's consumers. Test old retained events against new readers. Quarantine invalid records with an error reason and ownership; don't silently discard them or retry permanently invalid input forever.

4. Stream Processing and Transformation

Processing can validate, filter, enrich, join, aggregate, or route events. It doesn't have to produce analytics. ETL transforms before loading the target; ELT loads first and transforms there. Continuous routing may need neither a large transformation framework nor a database.

Kafka Streams: a processing library embedded in Java applications, particularly useful for Kafka-centered pipelines.
Apache Flink: a distributed engine with explicit event-time and stateful-processing facilities.
Spark Structured Streaming: incremental processing through Spark SQL's DataFrame and Dataset APIs, using micro-batches by default.

These choices differ in programming model and operations; benchmark your actual state, joins, latency requirements, and sinks. Spark's older DStream-based Spark Streaming is a legacy project; use Structured Streaming for new pipelines. For background, read what Apache Spark does.

5. Durable Storage and Recovery State

Distinguish retained source events, processor state, checkpoints, and analytical storage. A checkpoint records recoverable processing progress and state; it isn't a substitute for the input history needed for replay. Stateful engines can restore a checkpoint and reread subsequent input after failure.

For a lake, distinguish columnar files such as Parquet from table formats such as Apache Iceberg, which organize files into tables with additional table-management guarantees. Plan partitioning, compaction of small files, retention, and backfills. Frequent tiny writes can undermine query performance; object storage alone doesn't guarantee fast analytical access.

6. Serving, Analytics, and Downstream Actions

A sink is the destination receiving processed output: a serving database, warehouse, lake, another topic, or application integration. Choose it for the access pattern—point lookups, dashboards, historical SQL, or actions—not simply because it accepts events.

Make the AWS paths explicit. Land records in S3 and query them with Athena SQL; Athena isn't the windowing processor in that path. Alternatively, Redshift streaming ingestion can ingest Kinesis Data Streams or Amazon MSK data into a configured materialized view. Manual refresh is the default; enable automatic refresh explicitly when required.

A Reference Streaming Architecture

Application events / sensors / database CDC
                    |
                    v
          Broker / retained event log
             |                  |
             v                  v
       Stream processor      Raw event archive
             |                  |
             v                  v
     Serving database /     Batch reconciliation
     streaming database /   and historical backfills
     warehouse / actions

Replay: retained log or archive -> controlled reprocessing
Across the pipeline: schemas, access control, monitoring

The archive and replay path let you separate immediate results from later corrections. Don't replay directly into irreversible actions such as notifications without a deduplication or suppression policy.

Design Principles for Correct Stream Processing

Use Event Time and Define the Late-Data Policy

Event time is when something happened; processing time is when your system handles it. Network delays, retries, and disconnected devices can make arrival order differ from occurrence order. Use event time when the question is about when activity actually occurred.

Windows group events for calculation. Tumbling windows don't overlap; sliding windows do; session windows group activity separated by inactivity gaps. A watermark expresses progress in event time and helps an engine decide when to finalize results or release state. It isn't proof that every earlier event has arrived.

Decide whether very late data is dropped, quarantined, or incorporated through corrections. In Spark, a watermark delay is measured against observed event-time progress, not simply elapsed wall-clock time. Data within the configured delay is protected from lateness-based dropping; older data may or may not be processed.

Specify Guarantees Across the Whole Pipeline

At-most-once: failures can lose events, without delivery retries producing redelivery.
At-least-once: recovery can redeliver events, so processing must tolerate duplicates.
Exactly-once effects: retries don't create additional committed effects within the stated guarantee's boundary.

Exactly-once processing doesn't necessarily mean an event's code executes only once. Recovery may repeat execution while restoring a consistent result. End-to-end guarantees require replayable sources and cooperating sinks; a broker transaction doesn't automatically cover an external HTTP request.

Use stable event IDs and idempotent writes—repeating the write produces the same intended result. Where duplicates would change an aggregate, deduplicate before aggregation. Set a deduplication horizon that covers the retries and replays you intend to support.

Budget for State, Replay, and Recovery

Counts, joins, sessions, and deduplication retain information between events: processing state. More keys, longer windows, and longer lateness allowances can increase that state. Checkpoints preserve recovery progress; retained input enables replay.

Size retention for the outage and catch-up period you must survive. Keep checkpoints on durable storage, test restoration, and reserve capacity for backfills. Replaying through a changed enrichment dataset may produce different results, so version or retain reference data when reproducibility matters.

Design for Hot Keys and Backpressure

A hot key concentrates disproportionate work on one processing partition. More workers won't necessarily split that key's state. Consider pre-aggregation or carefully designed key splitting only when the operation permits it. Backpressure occurs when downstream work can't keep up and slows upstream processing.

Measure sink capacity as well as processor throughput. Batching can improve throughput at the cost of additional waiting time; increasing concurrency can merely move the bottleneck downstream.

A Small Event-Time Pipeline You Can Run

This local example publishes JSON files atomically, counts device events in one-minute windows, and writes finalized results to Parquet. Files replace the broker so you can examine timing behavior without deploying infrastructure. Install PySpark and configure its supported Java runtime using the official installation instructions, save this as stream_demo.py, and run python stream_demo.py.

import json
from pathlib import Path
from tempfile import TemporaryDirectory

from pyspark.sql import SparkSession
from pyspark.sql.functions import window

with TemporaryDirectory(prefix="stream-demo-") as directory:
    root = Path(directory)
    source = root / "input"
    source.mkdir()
    spark = (
        SparkSession.builder.master("local[2]")
        .appName("event-time-demo")
        .config("spark.sql.session.timeZone", "UTC")
        .config("spark.sql.shuffle.partitions", "2")
        .getOrCreate()
    )
    query = None
    try:
        events = spark.readStream.schema(
            "device_id STRING, event_time TIMESTAMP"
        ).json(str(source))
        counts = (
            events.withWatermark("event_time", "2 minutes")
            .groupBy(window("event_time", "1 minute"), "device_id")
            .count()
        )
        query = (
            counts.writeStream.format("parquet")
            .outputMode("append")
            .option("checkpointLocation", str(root / "checkpoint"))
            .start(str(root / "output"))
        )
        # Out of order, then advance event time, then send an old event.
        for index, time in enumerate(
            ["12:00:50", "12:00:10", "12:04:00", "12:00:20"]
        ):
            payload = {
                "device_id": "sensor-1",
                "event_time": f"2026-10-17T{time}Z",
            }
            staged = root / "pending.json"
            staged.write_text(json.dumps(payload) + "\n", encoding="utf-8")
            staged.replace(source / f"{index}.json")
            query.processAllAvailable()  # Testing helper, not a service loop.

        spark.read.parquet(str(root / "output")).show(truncate=False)
    finally:
        if query is not None:
            query.stop()
        spark.stop()

The finalized 12:00–12:01 UTC window has a count of two. The first two events belong together despite arriving out of order; advancing event time allows finalization, and the last event arrives after that window has closed. processAllAvailable() waits for available input to be processed and committed and is intended for testing.

Modern Streaming Architecture and Production Operations

A managed deployment could connect Kinesis Data Streams to Amazon Managed Service for Apache Flink, then write serving results and an S3 archive. Managed infrastructure doesn't remove partitioning, state, or downstream capacity constraints: scaling is still bounded by application configuration and service limits.

Before choosing a platform, ask which failures it handles, how it restores state, which connectors support your guarantees, what happens during scaling, and how you export data or migrate away.

Production Checklist

Freshness: monitor consumer lag, consumer health, event-to-query delay, and sink visibility—not just process uptime. Kafka's monitoring reference documents broker, consumer, and connector metrics.
Correctness: test duplicate events, malformed records, clock errors, late arrivals, and replay against known expected results.
Recovery: kill a worker, interrupt the sink, restore a checkpoint, and verify catch-up before retention expires.
Security: require encrypted transport, authenticated service identities, least-privilege authorization, and appropriate protection for archives and checkpoints. Consult the platform's security capabilities.
Capacity and cost: budget for retained bytes, state, checkpoint traffic, data transfer, idle capacity, backfills, file compaction, and analytical scans. Test peak load with realistic key skew and sink throttling.

FAQ

What is a streaming data architecture?

A streaming data architecture ingests and processes continuously arriving events to update results or trigger downstream actions. It connects sources, transport, processing, storage, and serving systems rather than describing a single database.

What are the main components of a streaming data pipeline?

Six useful logical components are event producers, ingestion and transport, data contracts, stream processing, durable storage, and serving or analytics. One product can implement several responsibilities; you don't necessarily need six separate services.

What is the difference between a message broker and a stream processor?

A message broker transports and retains events so producers and consumers can work independently. A stream processor computes over those events through operations such as filtering, enrichment, joins, and windowed aggregation.

How do streaming systems handle late or out-of-order events?

They can use event timestamps, windows, and watermarks to track progress and retain state for delayed arrivals. You must define what happens beyond the accepted lateness: dropping records, quarantining them, or correcting previously published results.

When should I use a streaming database instead of a data warehouse?

Consider a streaming database when applications repeatedly query results that must stay fresh as source data changes. A warehouse may be a better fit for broad historical analysis and ad-hoc queries; the two can coexist in the same architecture.