Object-Storage-First: S3 Is Becoming the New Disk

The biggest line item on a self-managed Kafka bill at scale is not the brokers, and it is not the disks. It is the network — specifically, the replication traffic you never look at. A three-AZ cluster with replication factor 3 pushing 1 GiB/s sends every byte across at least two zone boundaries, and AWS charges $0.01/GB on each side of every crossing. That works out to roughly $138k a month in transfer fees for a cluster whose EC2 line might be a tenth of that. Confluent's own analysis of self-managed Kafka puts networking at up to ~88% of infrastructure cost at scale. You are paying enterprise-database money to move bytes between buildings that are, in most regions, a few miles apart.

Meanwhile S3 will store the same 1 GiB/s stream — replicated across three availability zones, erasure-coded, with 11 nines of designed durability — for about $15k a month at 7-day retention, and moving data from EC2 into S3 within a region is free. The Kafka community noticed: KIP-1150 "Diskless Topics," accepted in March 2026, redirects replication from broker disks to object storage entirely.

And Kafka is the loud example of something quieter and bigger. The warehouse made this move a decade ago. Observability stores made it over the last five years. Serverless Postgres made it for durability, if not for reads. One mechanism, applied one system at a time: put the durable bytes in object storage, make compute stateless, cache aggressively. This article is about the shared math that makes the mechanism work, the write path every one of these systems converged on, and the judgment call about what still needs a local disk.

Renting Amazon's replication

Start with what S3 actually is, because "cheap blob store" undersells it. S3 Standard stores every object redundantly across a minimum of three availability zones, erasure-coded, continuously scrubbed, designed for 99.999999999% durability. Since December 2020 it has strong read-after-write consistency. Since 2024 it supports conditional writes — If-None-Match on PUT, then compare-and-swap with If-Match — which turned it from a dumb bucket into something you can build coordination primitives on without a side database. (If you want the internals of how such a system is built, we design one from scratch in design an object storage system.)

The pricing has two axes, and the second one is where designs live or die:

• Capacity: ~$0.023/GB-month for S3 Standard (us-east-1, mid-2026). Replication is included. Compare EBS gp3 at ~$0.08/GB-month for a single-AZ volume — triple-replicate it yourself and you're at ~$0.24/GB-month plus the transfer fees to keep the replicas in sync.
• Requests: ~$0.005 per 1,000 PUTs, ~$0.0004 per 1,000 GETs. Bytes are cheap; operations are not.

The comparison that matters is not S3 versus disk on price per gigabyte. It is S3 versus running your own replication protocol — quorums, leader election, rebalancing, and the cross-AZ bandwidth that replication generates forever. That's the thing S3 lets you delete.

Tier	Access latency	Durability model	Price (us-east-1, mid-2026, illustrative)	Who runs replication
Local NVMe (instance store)	~50–100 µs	Dies with the instance	Bundled with instance	You
EBS gp3	~0.5–1 ms	Single AZ, replicated within it	~$0.08/GB-month per copy	You (across AZs)
S3 Express One Zone	single-digit ms	Single AZ	~$0.11/GB-month	AWS (within one zone)
S3 Standard	~20–200 ms first byte	3+ AZs, 11 nines	~$0.023/GB-month	AWS

Read that table bottom-up and the trade is stark: the most durable tier is also the cheapest per gigabyte — by 3–10× — and the slowest by 2–4 orders of magnitude. Every object-storage-first system is an exercise in engineering around exactly that gap.

A WAL made of PUTs: the object-storage-first write path

Every system that made this move converged on the same write path, because there is basically one correct answer. You cannot PUT every record individually (request costs, see below), and you cannot ack before the bytes are durable. So: buffer briefly, write big, commit metadata, ack.

sequenceDiagram
    participant Prod as Producer
    participant Brk as Stateless broker
    participant Obj as Object store
    participant Coord as Batch coordinator
    Prod->>Brk: produce(records)
    Note over Brk: buffer ~100-250 ms — mix batches from many partitions
    Brk->>Obj: PUT shared segment of several MiB
    Obj-->>Brk: 200 OK — durable across 3 AZs
    Brk->>Coord: commit segment metadata
    Coord-->>Brk: offsets assigned
    Brk-->>Prod: ack

This is a write-ahead log made of PUTs. The segment is the unit of durability; a small metadata commit makes it visible and orders it. Three consequences fall out immediately.

First, the ack latency has a floor: buffer interval + PUT latency + metadata commit. On S3 Standard, a multi-MiB PUT lands in tens to low hundreds of milliseconds at the median and worse at the tail, so realistic produce acks sit around 350–600 ms — Aiven's diskless work quotes ~350 ms with S3 Standard. Fine for pipelines, ETL, and observability ingest. Fatal for anything interactive.

Second, brokers become genuinely stateless. There is no partition data on the node, so there is no leader in the disk sense: any broker can serve any partition. KIP-1150's diskless topics drop the partition leader for exactly this reason. No rebalancing data when a node dies, no StatefulSets, no "broker 4 is running out of disk" pages. WarpStream's founding pitch — zero disks is better — is that agents scale like a web app: a Deployment, not a database.

Third, producers and consumers can always talk to a broker in their own AZ, because every broker is equivalent. Combined with free in-region EC2-to-S3 transfer, the cross-AZ line item does not shrink. It goes to zero.

S3 Express One Zone moved the breakeven. Until late 2023, "hundreds of milliseconds" was the only produce latency object storage could offer. Express One Zone brought single-digit-millisecond PUTs (AWS claims up to 10× faster access than Standard), and the April 2025 price cut — storage down 31% to $0.11/GB-month, PUTs down 55%, GETs down 85% — made it economical for hot write paths. WarpStream's benchmark after the cut: p99 produce of 169 ms and median of 105 ms at 268 MiB/s, roughly 3× better than Standard — with the Express-backed cluster still ~63% cheaper than equivalent self-hosted Kafka; their later "Lightning Topics" work reports 33 ms median produce. Aiven's numbers for the same 1 GiB/s reference workload: ~$15.2k/month on S3 Standard at ~350 ms, ~$46.3k/month on Express at ~60 ms — against ~$159k/month for classic replicated Kafka. The latency floor dropped by an order of magnitude in eighteen months, and each drop moves more workloads over the line.

Request pricing is the real design axis

Here's the math everyone gets wrong the first time. Storage is the visible cost; requests are the one that produces the incident-review-worthy bill.

Workload: 1 GiB/s produce, 3x consumer fan-out, 7-day retention
AWS us-east-1 list prices, mid-2026. Illustrative, not a quote.

Classic Kafka (RF=3, 3 AZs, fetch-from-follower enabled):
  producer -> leader crossing     2/3 x 1 GiB/s  = 0.67 GiB/s
  leader -> 2 followers           2 x 1 GiB/s    = 2.00 GiB/s
  total cross-AZ                                   2.67 GiB/s
  monthly volume     2.67 x 86,400 x 30          = ~6.9M GiB
  at $0.02/GB effective (out + in)               = ~$138k/month  <- network alone

Object-storage-first (S3 Standard):
  storage    1 GiB/s x 604,800 s retention = ~590 TiB -> ~$14.5k/month
  PUTs       8 MiB segments -> 128/s -> 332M/month    -> ~$1.7k/month
  GETs       3 GiB/s reads at 8 MiB -> 384/s          -> ~$0.4k/month
  cross-AZ   ~zero (same-AZ brokers, in-region EC2->S3 is free)
  total                                                ~$17k/month

The naive version that bankrupts you:
  2,000 partitions, each flushed to its own object every 50 ms
  = 40,000 PUT/s = ~104B PUTs/month x $0.005/1k       = ~$518k/month

That last block is the entire design constraint in one number. Per-partition objects at low latency cost 30× more than the workload's storage. So every shipped system does two things: it shares segments across partitions — one PUT carries interleaved batches from hundreds of partitions — and it compacts in the background, rewriting many small mixed segments into fewer, larger, better-sorted ones so that reads don't degrade into thousands of tiny GETs. If that sounds exactly like LSM-tree compaction, it is. The economics of LSM trees moved from amortizing disk seeks to amortizing request fees, and the algorithms came along unchanged.

Request pricing also explains why the batching interval is a pricing knob wearing a latency costume. Halve the flush interval and you double the PUT bill. Confluent Freight leans one way (acks up to a second or two, up to 90% cheaper than self-managed Kafka, aimed at telemetry and feeds that never cared); Express-backed low-latency modes lean the other. Same architecture, different point on the same curve.

The same movie, four screens

Streaming is the noisy one because the incumbent was so expensive. WarpStream shipped the clean-sheet version in 2023 — stateless agents, S3 as the log, a managed metadata plane — and was acquired by Confluent within about a year. AutoMQ reworked Kafka's storage layer against object storage with a low-latency WAL in front. Confluent built Freight; Aiven built Inkless and pushed KIP-1150 upstream, and its acceptance in March 2026 means the pattern is no longer a vendor differentiator — it's Kafka's own roadmap. If you're designing a distributed message queue in an interview and retention or cross-AZ cost comes up, this is now the expected answer, not the exotic one.

The warehouse did it first, and so thoroughly that we stopped noticing. Snowflake's 2014-era architecture separated compute from S3-resident storage; BigQuery ran on Colossus from the start. What changed recently is standardization: Apache Iceberg makes the table itself — data files, manifests, snapshot metadata — an open format on object storage, so Spark, Trino, Flink, DuckDB, and Snowflake can all be stateless engines over the same bytes. The warehouse stopped being a database and became a protocol over S3. Every modern data pipeline inherits that shape.

Observability is the perfect-fit workload: append-mostly, high volume, read rarely and mostly recently, retained for compliance. ClickHouse Cloud's SharedMergeTree keeps table data solely in object storage, with stateless compute nodes coordinating through Keeper — compute that "spins up, runs your query, and disappears." Quickwit (whose team joined Datadog in 2025) runs sub-second log search with indexes living directly on S3; Grafana Loki stores chunks the same way. When you design a log aggregation system today, local-disk indexing is the position you have to justify, not the default.

OLTP is the frontier, and the honest story is more nuanced. Neon decomposes Postgres: compute streams WAL to a Paxos quorum of safekeepers, pageservers materialize pages, and object storage holds the durable, immutable history — but reads go shared buffers, then local NVMe cache, then pageserver, and S3 is never on the query path. Commits ack against quorum disks in milliseconds, not against S3. At the other end of the spectrum, SlateDB — an embedded LSM engine that writes its entire tree to object storage, leaning on S3's conditional writes for coordination — accepts hundreds of milliseconds of write latency in exchange for zero-disk operation. OLTP moved its durability and history into object storage. Its hot path stayed on silicon nearby.

Squint and all four are the same diagram:

flowchart TD
    CU[Clients] --> CN["Stateless compute nodes\n(any node serves any data)"]
    CN --> RAM["RAM + local NVMe cache\n(ephemeral, rebuildable)"]
    RAM -->|"cache miss"| OS[("Object store\nthe only durable data tier")]
    CN -->|"commit metadata"| MD["Coordination layer\n(batch coordinator / Keeper / catalog)"]
    CMP["Background compactor"] -->|"merge small segments"| OS
    style OS fill:#15803d,color:#fff
    style MD fill:#ffaa00,color:#111
    style CN fill:#0e7490,color:#fff

Domain	Systems	In object storage	Still local	Write ack
Streaming	WarpStream, AutoMQ, Freight, KIP-1150	The log itself (shared segments)	Buffers, hot-read cache	~100–600 ms Standard; ~35–170 ms Express
Warehouse	Iceberg + Snowflake/Trino/Spark	Everything: Parquet + manifests	Scratch space, result caches	Seconds (batch commits)
Observability	ClickHouse Cloud, Quickwit, Loki	Segments and indexes	Query-node caches	Seconds (ingest batches)
OLTP	Neon, SlateDB	WAL history, pages, snapshots	NVMe page cache, quorum WAL tier	ms (quorum disks) — S3 is downstream

What breaks

The failure modes are not hypothetical; they are the reasons this architecture took the better part of two decades to arrive after S3 launched in 2006.

The latency floor is a distribution, not a number. S3's median is honest; its tail is not your friend. A p99.9 PUT can take multiple seconds, and a produce path that acks after a PUT inherits that tail directly. Mature implementations hedge — fire a second PUT if the first hasn't returned in ~200 ms — and set client timeouts assuming the tail, not the median. If your producers were tuned for 5 ms acks against local Kafka, moving them to a diskless topic without raising request.timeout.ms and buffer sizes will manufacture a retry storm on the first slow afternoon.

The small-object death spiral. Under-batch and you pay triple: request fees (the $518k block above), metadata bloat (every segment is a row the coordinator must track forever), and read amplification when a historical scan touches 40,000 objects instead of 400. Compaction is supposed to fix the last two — which means compaction lag is now a cost and latency incident, not a background curiosity. Alert on it like you'd alert on replication lag.

You didn't delete the state. You moved it. The log is in S3, but the thing that says which segments exist, in what order, with which offsets — that's a strongly-consistent, high-write-rate database, and now it's the whole ballgame. WarpStream's managed metadata plane is the product; KIP-1150's batch coordinator is the genuinely hard part of the implementation; ClickHouse Cloud leans on Keeper. When the coordination layer degrades, every stateless node degrades with it, simultaneously. You traded a hundred medium-hard broker problems for one very hard metadata problem. That's usually a good trade. It is not a free one.

Cold reads become GET storms. A consumer replaying seven days at 3 GiB/s hits objects no cache has seen, and every stateless node fetches them from S3 at first-byte latencies of 30–100 ms. Without request coalescing, ten nodes serving the same replay fetch the same segments ten times — a self-inflicted stampede that shows up as both a latency regression and a five-figure GET line. Shared distributed caches (ClickHouse Cloud built one) and single-flight fetch logic are load-bearing here, not optimizations.

S3 Express One Zone is one zone. The name says it and people still miss it: single-digit milliseconds comes from giving up the multi-AZ replication that justified this whole architecture. Systems using Express for the hot write path either write to two Express buckets in different zones or age data into Standard within seconds-to-minutes. If a design doc says "Express" without a zone-failure story, that's the review comment.

And a quiet one: immutable shared segments make per-record deletion a rewrite problem. A GDPR erasure request against a segment that interleaves 500 partitions means compacting the segment, not deleting a row. Plan the tombstone-and-rewrite pipeline before the first erasure request arrives, not after.

The decision in practice

The whole judgment call compresses into one question — what is the p99 budget of the ack path and the read path? — followed by an honest audit of whether your workload ever needed disks at all.

flowchart TD
    Q{"p99 budget for the\nack or read path?"} -->|"under 10 ms"| LOCAL["Local NVMe / EBS + your own replication"]
    Q -->|"10-100 ms"| EXP["S3 Express One Zone\n+ explicit cross-zone story"]
    Q -->|"100 ms is fine"| STD["S3 Standard — batch hard,\nzero cross-AZ traffic"]
    LOCAL --> ARCH["...and still archive history\nto object storage"]
    style STD fill:#15803d,color:#fff
    style EXP fill:#ffaa00,color:#111
    style LOCAL fill:#0e7490,color:#fff

Some workloads never needed local disks; they had them because that's what the software assumed in 2011. Append-mostly logs and event streams. Analytics scans. Observability and audit data. Anything consumed by a pipeline that's already seconds behind real time. For these, object-storage-first is now simply the correct default, and on AWS the cross-AZ math makes it a near-mandatory one — the KIP-1150 lineage exists because the community concluded the same thing. If your streaming bill has a transfer line you can't explain, start there.

Some workloads still need disks and will for years. OLTP hot paths reading pages at sub-millisecond latencies. Hot search indexes serving interactive queries. Low-latency KV serving. The tell is that even the systems that moved durability to S3 — Neon most explicitly — kept NVMe caches on the read path and quorum disks on the commit path. Object storage became the source of truth; it did not become the working set. When you're choosing a database, that distinction — where truth lives versus where reads land — is now a first-class question to ask of any vendor.

And the line between the two camps is not fixed. It has moved four times in six years — strong consistency in 2020, Express One Zone in 2023, conditional writes in 2024, the up-to-85% Express price cut in 2025 — and each move converted another tier of workloads. The direction of travel is one-way. So the practical stance for a design review in 2026: object-storage-first is the default for any new append-heavy or analytical system, S3 Express plus caching is the default question to ask for the 10–100 ms tier, and every local disk in the architecture should carry a one-line justification for why it exists. Most of them will have one. Fewer than you think.

Further reading

• KIP-1150: Diskless Topics — the accepted proposal itself
• Aiven: Diskless, unified, zero-copy Apache Kafka — the cost/latency numbers behind the KIP lineage
• WarpStream: Zero disks is better (for Kafka) and the S3 Express One Zone benchmark
• AWS: up to 85% price reductions for S3 Express One Zone — the pricing change that moved the breakeven
• ClickHouse: the architecture behind stateless compute
• Neon: architecture overview — safekeepers, pageservers, and S3 as the durable history