Batch and Streaming: From Architecture to Configuration

Most teams still treat the choice between batch and streaming as a foundational architecture decision. You pick a side at design time, build your entire stack around it, and accept that changing your mind later requires a full pipeline rewrite.

That framing is collapsing. Before you commit a quarter of engineering resources to either side, it is worth understanding why.

The practitioners already see it. In a June 2026 thread on r/dataengineering debating where micro-batching ends and streaming begins, the top answer simplified the entire debate: “At the end of the day a stream is a batch of 1. You are still conceptually optimizing latency… the difference is in the orders of magnitude.”

This is a more accurate reflection of modern infrastructure than most architecture diagrams suggest.

Engines no longer force a choice

The primary processing engines have converged.

Apache Flink 2.0, released March 24, 2025, introduced two critical shifts. First, state moved to remote storage by default, decoupling a job’s state size from the physical machine. Second, it introduced Materialized Tables, allowing users to “declaratively manage both real-time and historical data through a single pipeline,” removing the need for separate codebases for batch and stream.

Spark reached the same destination from a different direction. Databricks previewed Real-Time Mode for Structured Streaming on August 19, 2025, which later landed in Spark 4.1.0 (SPARK-53736). This enables continuous, sub-second latency processing—dropping to single-digit milliseconds for stateless tasks.

Crucially, this wasn’t a new engine or API. It was a trigger type. The same query now runs at different latencies simply by changing a setting.

The collapse of the log and the table

The more significant shift is in storage: the queue (log) and the warehouse (table) are merging into a single object.

Confluent’s Tableflow reached GA on March 19, 2025, turning Kafka topics directly into Iceberg tables without intermediary ETL jobs. It expanded on October 29, 2025 to support Delta Lake and Unity Catalog. Similarly, Redpanda shipped Iceberg Topics GA on April 7, 2025, allowing a topic to materialize as a table via a simple property. Aiven claims this removes roughly 60% of Kafka sink connector use cases, a plausible figure given the redundancy of sink connectors in this new model.

In June 2025, Alibaba donated Fluss to the Apache incubator, a columnar streaming storage layer designed specifically to bridge the gap between Kafka (streaming) and Iceberg (analytics). Its 0.8 release added Flink Materialized Tables with declared freshness targets.

This is the core of the argument: you no longer choose an architecture; you define a freshness target. The engine then decides how to run. Batch and streaming have become configurable numbers, much like how Iceberg stopped being a bet worth agonizing over.

The real cost of streaming

If the technology has converged, why does the cost still feel different?

KIP-1150, which proposes writing partition data directly to object storage instead of broker disks, passed its vote on March 2, 2026. The justification reveals the true pain point: cross-availability-zone (AZ) replication traffic. This is “the most substantial infrastructure cost for Apache Kafka operators on hyperscalers today,” with AWS charging $0.02 per GiB and Google Cloud $0.01 per GiB.

Streaming isn’t inherently expensive because of the continuous processing; it’s expensive because of the network egress tax. This is a solvable infrastructure problem, which is why so many vendors now offer direct-to-object-storage Kafka.

However, caution is required. As Jack Vanlightly noted in October 2025, S3 is not simply a “cheaper disk.” Object storage introduces its own latency and per-request charges. The bill doesn’t disappear; it just moves.

Auditing the “savings” claims

Diskless Kafka vendors frequently advertise massive cost reductions: WarpStream claims 80%, AutoMQ claims 90%+, and Confluent Freight claims up to 90%.

Claimed cost savings from diskless Kafka, and who is doing the claiming

Every figure below was published by a party selling the alternative. None has been independently reproduced.

Item	Claimed saving versus self-hosted Kafka
WarpStream vendor-reported	80%
AutoMQ vendor-reported	90%+
Confluent Freight vendor-reported	up to 90%
AWS with AutoMQ vendor co-published	94%
Independently reproduced no such figure exists	none

These numbers are marketing artifacts. Stanislav Kozlovski, a former Confluent engineer, audited WarpStream’s public cost calculator in December 2024. He found that the calculator overstated self-hosted Kafka costs by 2.8x in the base case—and up to 5.6x when realistic optimizations were applied.

Kozlovski identified several distortions: using SSD pricing where HDD was appropriate, recommending instances 5–10x larger than necessary, and applying a compression ratio that inflated Kafka’s costs while leaving the vendor’s figure untouched. His conclusion: self-hosted Kafka was actually cheaper on GCP and Azure.

The diskless architecture itself is sound—the KIP’s reasoning on cross-zone traffic is correct. But the headline savings percentages should be viewed with extreme skepticism.

The data gaps: What vendors aren’t showing you

Three critical absences in the current discourse are more telling than any vendor chart:

1. No like-for-like cost comparisons. There is no credible, independent study comparing the cost of batch, micro-batch, and true streaming on the same workload. Most “evidence” consists of contradictory blog posts. If a vendor quotes a cost multiplier for streaming, ask for the source.
2. No industry base rates. There is no independent survey showing the actual share of batch vs. streaming pipelines or the latency teams genuinely require. Most reports (like Confluent’s) survey people already familiar with streaming, creating a selection bias that ignores the teams who evaluated streaming and decided against it.
3. Unproven AI latency requirements. The claim that AI agents require sub-second data freshness is almost exclusively made by vendors selling streaming or CDC tools. We have found no independent study proving that agent task success degrades when data is minutes old rather than milliseconds old. Response time (latency of the LLM) is a constraint; data freshness (latency of the table) is a different matter entirely.

Finally, the common trope that data “loses its value within seconds” is folklore. The closest real evidence is Jordan Tigani’s 2023 observation that week-old data is about 20 times less likely to be queried than same-day data. That is a measure of query frequency in one company’s logs, not a universal law of data value.

Real-world implementation: The PostHog example

The shift from “philosophy” to “infrastructure property” is best seen in a pull request.

In July 2026, PostHog’s data warehouse team discovered their job queue couldn’t distinguish between batch and stream. One PR from July 13 noted that running compaction unconditionally after every final batch worked for scheduled syncs but failed for CDC schema ticks. A second PR from July 17 quantified the impact: CDC streaming merges represent 52% of queue batches for 12 teams, with a p95 duration 5.5x longer than the rest.

This is the reality: the distinction is no longer an architecture diagram; it is a workload profile that your scheduler must model correctly.

Aeolus view — Asking whether a platform should be “batch or streaming” usually triggers a philosophical argument. Asking “what freshness does this specific table owe its consumer” produces an answer—and usually a cheaper one. Apply that question table by table. For the majority, the honest answer is hours; for a small few, it is seconds. Build for those few and leave the rest alone. The teams that struggle are those who pick an architecture first and then try to force their tables to fit it.

Bottom line

Ben Rogojan’s heuristic remains a solid starting point: if a system needs to react, think real-time; if it needs to be analyzed, think batch.

What has changed in the last eighteen months is that the cost of being wrong is falling. Engines, storage layers, and table formats are converging, allowing you to adjust your freshness requirements without rebuilding your entire stack.

Two takeaways:

1. Treat freshness as a per-table requirement, not a platform identity.
2. Audit every cost percentage, especially those that support the choice you already want to make.

If you are weighing these trade-offs for a specific pipeline and want a second opinion on whether your latency requirements are genuine, we are happy to help. Often, a cheaper table is the right answer. If your data isn’t ready for this yet, we’ll help you get there.