<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Streambed – Blog</title><link>https://streambed.dev/blog/</link><description>Recent content in Blog on Streambed</description><generator>Hugo -- gohugo.io</generator><language>en-us</language><atom:link href="https://streambed.dev/blog/index.xml" rel="self" type="application/rss+xml"/><item><title>Stream Postgres changes from a replica</title><link>https://streambed.dev/blog/postgres-replica-cdc/</link><pubDate>Sun, 27 Sep 2026 00:00:00 +0000</pubDate><guid>https://streambed.dev/blog/postgres-replica-cdc/</guid><description>
&lt;p>Streambed can now read Postgres changes from a hot-standby replica instead of connecting directly to the primary.&lt;/p>
&lt;p>Postgres usually has one writable primary. Every application write already passes through that node, so it is the last place you want to add avoidable CPU, I/O, and WAL-decoding work.&lt;/p>
&lt;p>Postgres 16 added support for logical decoding on a standby. Streambed can use that standby as its CDC source while the primary remains responsible for operations that require a writable connection.&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">Application → Postgres primary → hot standby → Streambed → Parquet&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;p>This does not remove the cost of CDC. It moves logical decoding and the Streambed replication connection away from the primary&amp;rsquo;s critical path.&lt;/p>
&lt;h2>How to use it&lt;span class="hx:absolute hx:-mt-20" id="how-to-use-it">&lt;/span>
&lt;a href="#how-to-use-it" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>Point &lt;code>--source-url&lt;/code> at the replica and &lt;code>--primary-url&lt;/code> at the primary:&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">./streambed sync &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> --source-url&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;postgres://user:pass@replica-host:5432/mydb&amp;#34;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> --primary-url&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;postgres://user:pass@primary-host:5432/mydb&amp;#34;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> --s3-bucket&lt;span class="o">=&lt;/span>my-bucket &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> --s3-prefix&lt;span class="o">=&lt;/span>streambed&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;p>Streambed splits the work between the two connections:&lt;/p>
&lt;ul>
&lt;li>&lt;code>--source-url&lt;/code>: creates or reuses the logical replication slot and streams WAL from the standby.&lt;/li>
&lt;li>&lt;code>--primary-url&lt;/code>: creates the publication and reads database metadata.&lt;/li>
&lt;/ul>
&lt;p>If &lt;code>--primary-url&lt;/code> is omitted, Streambed continues to use &lt;code>--source-url&lt;/code> for everything. Existing primary-only deployments do not need to change.&lt;/p>
&lt;h2>Requirements&lt;span class="hx:absolute hx:-mt-20" id="requirements">&lt;/span>
&lt;a href="#requirements" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>This feature requires &lt;strong>Postgres 16 or newer&lt;/strong>. The primary and standby must be configured for logical replication, and the standby should use &lt;code>hot_standby_feedback&lt;/code> so logical decoding retains the catalog information it needs.&lt;/p>
&lt;p>As with any replication setup, choose an appropriate WAL-retention strategy for the physical primary-to-standby connection. Streambed does not create or manage that topology.&lt;/p>
&lt;p>See the &lt;a href="https://github.com/viggy28/streambed/blob/main/docs/replica.md"target="_blank" rel="noopener">replica setup guide&lt;/a> for configuration and troubleshooting details.&lt;/p>
&lt;p>Thanks to &lt;a href="https://github.com/Kartik1397"target="_blank" rel="noopener">Kartik Prajapati&lt;/a> for contributing this feature.&lt;/p></description></item><item><title>Logical Replication Is Postgres's Most Important Feature</title><link>https://streambed.dev/blog/logical-replication-is-postgres-most-important-feature/</link><pubDate>Sat, 26 Sep 2026 00:00:00 +0000</pubDate><guid>https://streambed.dev/blog/logical-replication-is-postgres-most-important-feature/</guid><description>
&lt;p>Logical replication might be one of the most important features in Postgres.&lt;/p>
&lt;p>But not because it gives you another copy of your database.&lt;/p>
&lt;p>Because it gives you a way out.&lt;/p>
&lt;p>&lt;img src="https://streambed.dev/images/logical-replication-escape-hatch.png" alt="Logical replication as Postgres’s escape hatch" loading="lazy" />&lt;/p>
&lt;p>I came to this view after building and leading the Postgres platform at Cloudflare from 2019 to 2023, and co-founding &lt;a href="https://github.com/omnigres/omnigres"target="_blank" rel="noopener">Omnigres&lt;/a>, which explored Postgres extensions as an application runtime. I spent a lot of time thinking about a related problem: how can you build a sustainable company in the Postgres ecosystem without first persuading customers to replace the database at the center of their architecture?&lt;/p>
&lt;p>Your Postgres database may start as a single AWS, Azure, or Google Cloud resource. Over time, the provider&amp;rsquo;s identity system, private network, and operational tooling grow around it.&lt;/p>
&lt;p>Five years later, moving the database means moving half your architecture.&lt;/p>
&lt;p>A backup proves that you possess the data. It does not prove that you can move a live system built around that data. Restore a backup somewhere else and it begins going stale as soon as production receives its next write.&lt;/p>
&lt;p>Logical replication changes that constraint. It lets another system begin with the current state of the database and then continue receiving changes while production remains online.&lt;/p>
&lt;p>That is more than replication. It is optionality.&lt;/p>
&lt;h2>The production database is almost impossible to replace&lt;span class="hx:absolute hx:-mt-20" id="the-production-database-is-almost-impossible-to-replace">&lt;/span>
&lt;a href="#the-production-database-is-almost-impossible-to-replace" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>In a &lt;a href="https://www.buzzsprout.com/1783651/episodes/16211813-fivetran-ceo-george-fraser-data-for-the-ai-revolution"target="_blank" rel="noopener">conversation with Auren Hoffman&lt;/a>, Fivetran CEO George Fraser put it bluntly. &lt;a href="https://www.linkedin.com/posts/vigneshravichandran28_the-only-way-to-change-your-production-database-activity-7494086027423154176-Y1Mh"target="_blank" rel="noopener">I wrote about the line when I first heard it&lt;/a>:&lt;/p>
&lt;blockquote>
&lt;p>The only way to change your production database is to start a new company.&lt;/p>
&lt;/blockquote>
&lt;p>The statement is deliberately extreme, but the underlying point is right. A production database is not just software that stores rows. It sits underneath application behavior, operational procedures, compliance controls, and years of assumptions nobody has written down.&lt;/p>
&lt;p>This creates an awkward problem for a database startup. A new system can be dramatically better at one workload and still fail because adopting it requires a risky migration. The technical product may be good, but the first step asks the customer to move the most sensitive component in their architecture.&lt;/p>
&lt;p>Logical replication offers a different path: do not replace Postgres on day one. Attach to it.&lt;/p>
&lt;p>The existing database remains the system of record. The new product reads a consistent snapshot and then follows committed changes from the write-ahead log (WAL). It can prove its value on real production data without asking the application to switch databases or dual-write every transaction.&lt;/p>
&lt;p>For an early-stage infrastructure company, this is not a minor implementation detail. It changes the economics of adoption.&lt;/p>
&lt;h2>One Postgres feature created an entire product surface&lt;span class="hx:absolute hx:-mt-20" id="one-postgres-feature-created-an-entire-product-surface">&lt;/span>
&lt;a href="#one-postgres-feature-created-an-entire-product-surface" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>Every durable Postgres change is recorded in WAL. Physical replication interprets that log in terms of Postgres storage. Logical decoding interprets it as changes to data: a row was inserted, updated, or deleted. Native logical replication uses this mechanism between Postgres databases; tools such as Debezium and PeerDB consume the underlying logical change stream.&lt;/p>
&lt;p>A replication slot remembers how far a consumer has read. An output plugin such as Postgres&amp;rsquo;s built-in &lt;code>pgoutput&lt;/code> turns WAL records into a stream the consumer can understand. The consumer can then apply those changes to another Postgres database or translate them into a completely different system.&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl"> ┌──▶ another Postgres
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> │
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">Postgres WAL ──▶ logical decoding ──▶ Kafka / event consumers
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> │
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> ├──▶ warehouse or query engine
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> │
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> └──▶ Parquet + open table format&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;p>This is the important architectural boundary. The source continues to behave like Postgres. The destination does not need to share its storage format, query engine, release cycle, or cloud provider.&lt;/p>
&lt;p>That boundary creates room for products that are not trying to become the next primary database:&lt;/p>
&lt;ul>
&lt;li>analytical systems can remove expensive queries from production;&lt;/li>
&lt;li>search systems can maintain an index without polling tables;&lt;/li>
&lt;li>event platforms can turn committed changes into streams;&lt;/li>
&lt;li>security products can inspect a durable record of data changes;&lt;/li>
&lt;li>migration products can keep a destination current until cutover;&lt;/li>
&lt;li>developer tools can build derived views without changing application writes.&lt;/li>
&lt;/ul>
&lt;p>The startup gets a narrow entry point into an existing architecture. The customer gets to evaluate the product without making an irreversible decision.&lt;/p>
&lt;h2>Debezium, PeerDB, and the products built on this boundary&lt;span class="hx:absolute hx:-mt-20" id="debezium-peerdb-and-the-products-built-on-this-boundary">&lt;/span>
&lt;a href="#debezium-peerdb-and-the-products-built-on-this-boundary" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>&lt;a href="https://debezium.io/documentation/reference/stable/connectors/postgresql.html"target="_blank" rel="noopener">Debezium&lt;/a> turns Postgres changes into an event stream. It takes an initial consistent snapshot, continues from the corresponding WAL position, and publishes row-level changes into Kafka-compatible infrastructure. The applications consuming those events do not need to understand Postgres WAL.&lt;/p>
&lt;p>&lt;a href="https://github.com/PeerDB-io/peerdb"target="_blank" rel="noopener">PeerDB&lt;/a> uses the same opening to replicate Postgres data into analytical systems. Its product value is not merely reading WAL. It is everything required to make that stream useful at the destination: initial copies, type mapping, batching, and applying updates and deletes efficiently.&lt;/p>
&lt;p>&lt;a href="https://github.com/viggy28/streambed"target="_blank" rel="noopener">StreamBed&lt;/a> uses logical replication to write Postgres changes as Parquet with Iceberg or DuckLake metadata. It is one more example of the pattern: Postgres remains the transactional system, while the same data becomes available in an open analytical representation on object storage.&lt;/p>
&lt;p>These projects produce very different things: events, warehouse tables, and lakehouse files. But they share the same adoption model:&lt;/p>
&lt;ol>
&lt;li>meet the customer where their data already lives;&lt;/li>
&lt;li>begin as a downstream system rather than a replacement;&lt;/li>
&lt;li>prove value using real workloads;&lt;/li>
&lt;li>avoid requiring an application rewrite.&lt;/li>
&lt;/ol>
&lt;p>Logical replication is the distribution channel hidden inside the database.&lt;/p>
&lt;h2>An escape hatch from the cloud data stack&lt;span class="hx:absolute hx:-mt-20" id="an-escape-hatch-from-the-cloud-data-stack">&lt;/span>
&lt;a href="#an-escape-hatch-from-the-cloud-data-stack" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>Cloud lock-in is rarely one proprietary API. It is the accumulation of small dependencies until leaving requires every component to move at once.&lt;/p>
&lt;p>The data layer has the strongest gravity. Compute can often be redeployed elsewhere, but a database is large, stateful, and continuously changing. Logical replication weakens that gravity by allowing the data plane to cross the provider boundary before the rest of the architecture moves.&lt;/p>
&lt;p>A company can keep its managed Postgres instance running while it creates a live copy in another provider. It can send changes to an independently operated analytics system instead of adopting the provider&amp;rsquo;s warehouse. It can materialize data into an open format on object storage. It can test self-hosted infrastructure against current production data rather than a stale benchmark dump.&lt;/p>
&lt;p>The move becomes gradual:&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">One provider owns the whole path
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">Application ──▶ Managed Postgres ──▶ Provider analytics
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> ↓ logical replication
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">Application ──▶ Managed Postgres ──▶ Independent or open systems
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> │
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> └──▶ another cloud, later&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;p>Logical replication does not migrate IAM policies, networks, extensions, or operational knowledge. What it changes is timing. Those pieces no longer have to move in the same maintenance window as the data.&lt;/p>
&lt;p>It also changes negotiating power. An exit does not need to begin with a high-risk cutover. It can begin with a replication slot and a consumer.&lt;/p>
&lt;h2>Open source is not enough if the data cannot leave&lt;span class="hx:absolute hx:-mt-20" id="open-source-is-not-enough-if-the-data-cannot-leave">&lt;/span>
&lt;a href="#open-source-is-not-enough-if-the-data-cannot-leave" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>We often describe Postgres as open because the source is available and the database can run almost anywhere. That is necessary, but it is not the complete test.&lt;/p>
&lt;p>The meaningful test arrives years later, after the database contains terabytes of data and sits at the center of the company. Can another system consume that data continuously? Can you run the old and new architectures side by side? Can you leave without stopping the business for the duration of a bulk copy?&lt;/p>
&lt;p>Logical replication does not eliminate the hard work of migration. It creates the period of coexistence in which that work becomes possible.&lt;/p>
&lt;p>It also gives new infrastructure startups a realistic way to enter the Postgres ecosystem. They can complement the production database before asking to replace any part of the stack. Some may never replace it at all. They can build value downstream while preserving Postgres as the stable transactional core.&lt;/p>
&lt;p>Perhaps one of the most important properties of an open database is not simply that you can run it anywhere.&lt;/p>
&lt;p>It is that you, and the products built around your data, have a way out.&lt;/p></description></item><item><title>Parquet is to file formats what Postgres is to databases</title><link>https://streambed.dev/blog/parquet-postgres/</link><pubDate>Fri, 25 Sep 2026 00:00:00 +0000</pubDate><guid>https://streambed.dev/blog/parquet-postgres/</guid><description>
&lt;p>The more I work with modern data infrastructure, the more I believe this:&lt;/p>
&lt;blockquote>
&lt;p>Parquet is to file formats what Postgres is to databases.&lt;/p>
&lt;/blockquote>
&lt;p>&lt;img src="https://streambed.dev/images/parquet-postgres.png" alt="Postgres and Parquet as ecosystem defaults" loading="lazy" />&lt;/p>
&lt;p>When a new database launches, one of the first questions is whether it speaks the Postgres wire protocol. One of the first demos often involves connecting with &lt;code>psql&lt;/code>. Existing clients and tools already understand that interface.&lt;/p>
&lt;p>That does not make every Postgres-compatible database PostgreSQL. Compatibility can stop at the wire protocol, the SQL dialect, or a subset of behavior. But Postgres has become the center of gravity. New databases explain themselves by describing how close they are to it.&lt;/p>
&lt;p>Parquet increasingly plays the same role for warehouses and lakehouses.&lt;/p>
&lt;p>A new query engine is expected to read it. DuckDB reads it. Spark reads it. ClickHouse reads it. Snowflake can load it and query it through external tables. Object stores are full of it. If a system works with analytical data, not supporting Parquet now requires an explanation.&lt;/p>
&lt;p>Postgres became the database ecosystem&amp;rsquo;s common language. Parquet is becoming the data ecosystem&amp;rsquo;s common file format.&lt;/p>
&lt;h2>Defaults create ecosystems&lt;span class="hx:absolute hx:-mt-20" id="defaults-create-ecosystems">&lt;/span>
&lt;a href="#defaults-create-ecosystems" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>Postgres did not reach this position by being the perfect database for every workload.&lt;/p>
&lt;p>It became a dependable default. It runs almost everywhere. Every major language has mature drivers. Frameworks support it without ceremony. Cloud providers offer it. Extensions can turn it into a geospatial database, a vector store, a time-series system, or something its original authors never planned.&lt;/p>
&lt;p>That creates a flywheel:&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">More users
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> ↓
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">More drivers, tools, and integrations
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> ↓
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">Lower cost for the next system to support Postgres
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> ↓
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">More users&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;p>Eventually, compatibility becomes a product feature by itself. A database does not need to persuade users to adopt a new client protocol if it can use the one they already have.&lt;/p>
&lt;p>Parquet has entered a similar flywheel.&lt;/p>
&lt;p>The format is supported across languages, engines, warehouses, and table formats. Producers write Parquet because consumers already understand it. Consumers add Parquet support because so much data is already stored in it.&lt;/p>
&lt;p>The result is bigger than any individual engine.&lt;/p>
&lt;p>A Spark job can write a Parquet dataset. DuckDB can inspect it locally. ClickHouse can ingest it. A warehouse can load it. A lakehouse table format can use it as its data layer. Each system may have its own execution model, catalog, and preferred query language, but the bytes have a common shape.&lt;/p>
&lt;h2>The useful property is not ownership&lt;span class="hx:absolute hx:-mt-20" id="the-useful-property-is-not-ownership">&lt;/span>
&lt;a href="#the-useful-property-is-not-ownership" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>We see this while building Streambed.&lt;/p>
&lt;p>Streambed reads changes from PostgreSQL&amp;rsquo;s write-ahead log and publishes them into DuckLake or Iceberg. The table metadata differs between those targets, but the durable data layer is built around Parquet on object storage.&lt;/p>
&lt;p>Conceptually, the path is PostgreSQL WAL → Streambed → Parquet data files, with DuckLake or Iceberg managing the table metadata.&lt;/p>
&lt;p>That boundary matters.&lt;/p>
&lt;p>The data is not encoded in a private Streambed file format that only Streambed can read. Standard tools already know how to inspect Parquet. Iceberg is an open table format, and DuckLake keeps its table metadata in a queryable SQL catalog.&lt;/p>
&lt;p>This does not mean that a pile of Parquet files is automatically a table. Parquet describes the data inside each file. It does not, by itself, define which files are currently active, which rows were deleted, how schema changes should be interpreted, or which historical snapshot a query should read. DuckLake and Iceberg provide those table-level semantics.&lt;/p>
&lt;p>There is a useful parallel here:&lt;/p>
&lt;blockquote>
&lt;p>Speaking the Postgres wire protocol does not reproduce all of PostgreSQL.&lt;/p>
&lt;p>Reading Parquet does not reproduce all of a lakehouse table.&lt;/p>
&lt;/blockquote>
&lt;p>The common format gets a system into the ecosystem. The layers above it still matter.&lt;/p>
&lt;h2>Boring infrastructure compounds&lt;span class="hx:absolute hx:-mt-20" id="boring-infrastructure-compounds">&lt;/span>
&lt;a href="#boring-infrastructure-compounds" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>The most valuable standards often disappear into the architecture. Teams want existing database clients to connect and new query engines to read the files already in object storage. Every integration makes the next system cheaper to build and safer to adopt.&lt;/p>
&lt;p>Neither standard is perfect. Postgres compatibility varies, while Parquet has sharp edges around logical types, timestamps, nested data, compression codecs, and engine-specific conventions. A strong ecosystem learns those edges and builds the tests and adapters needed to live with them.&lt;/p>
&lt;p>A successful standard does not need to solve the entire stack. It needs to provide a stable meeting point: Postgres at the database boundary, and Parquet at the analytical data boundary. Boring standards tend to win.&lt;/p></description></item><item><title>Why time travel and branching are first-class lakehouse operations</title><link>https://streambed.dev/blog/time-travel-lakehouse-snapshots/</link><pubDate>Fri, 25 Sep 2026 00:00:00 +0000</pubDate><guid>https://streambed.dev/blog/time-travel-lakehouse-snapshots/</guid><description>
&lt;p>Time travel sounds like an advanced database feature. In an immutable-file system, it is closer to following an old pointer.&lt;/p>
&lt;p>A lakehouse normally publishes changes by writing new data files and new metadata rather than modifying already-published files in place. If the old metadata and files are retained, the previous table state is still addressable. Give that state an ID and a query syntax, and you have time travel. Give it a name and allow new commits from it, and you have a branch.&lt;/p>
&lt;p>None of this makes history free. It makes history explicit.&lt;/p>
&lt;h2>Why Can&amp;rsquo;t PostgreSQL do that?&lt;span class="hx:absolute hx:-mt-20" id="why-cant-postgresql-do-that">&lt;/span>
&lt;a href="#why-cant-postgresql-do-that" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>Coming from Postgres background, my instinct reaction is MVCC is very similar to this concept. So, why can&amp;rsquo;t Postgres do the same.&lt;/p>
&lt;p>An &lt;code>UPDATE&lt;/code> does not simply overwrite the existing tuple. It creates a new tuple version and marks the old version as superseded by the updating transaction.&lt;/p>
&lt;p>Conceptually:&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">Before UPDATE
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">(id=42, status=&amp;#39;pending&amp;#39;)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> xmin=100
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> xmax=—
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">After transaction 120 updates the row
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">(id=42, status=&amp;#39;pending&amp;#39;)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> xmin=100
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> xmax=120
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">(id=42, status=&amp;#39;shipped&amp;#39;)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> xmin=120
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> xmax=—&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;p>A transaction snapshot determines which version is visible. A transaction that began before transaction 120 committed may still see &lt;code>pending&lt;/code>; a newer transaction sees &lt;code>shipped&lt;/code>.&lt;/p>
&lt;p>So PostgreSQL already retains old physical versions. Why does it not expose a general query such as this?&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-sql" data-lang="sql">&lt;span class="line">&lt;span class="cl">&lt;span class="k">SELECT&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="o">*&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="k">FROM&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">orders&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="k">AS&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="k">OF&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="k">TIMESTAMP&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s1">&amp;#39;yesterday&amp;#39;&lt;/span>&lt;span class="p">;&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;p>Because those versions exist for &lt;strong>concurrency&lt;/strong>, not as a durable historical product contract.&lt;/p>
&lt;p>An MVCC snapshot describes transaction visibility for a running transaction. It is not normally a permanent, named database version. Once no active transaction can see an old tuple, PostgreSQL expects &lt;a href="https://www.postgresql.org/docs/current/routine-vacuuming.html"target="_blank" rel="noopener">&lt;code>VACUUM&lt;/code>&lt;/a> to reclaim its space.&lt;/p>
&lt;p>PostgreSQL can export a snapshot so that several live transactions share the same view, but the exporting transaction must remain open. The snapshot is not a token that can be saved and queried next week.&lt;/p>
&lt;p>PostgreSQL also supports point-in-time recovery using a base backup and archived WAL. That can reconstruct an earlier database state, but it is a recovery workflow: restore a cluster, replay WAL to a target, start the recovered database, and extract the required data. It is not an ordinary query against the running primary.&lt;/p>
&lt;h2>Could PostgreSQL MVCC be turned into time travel?&lt;span class="hx:absolute hx:-mt-20" id="could-postgresql-mvcc-be-turned-into-time-travel">&lt;/span>
&lt;a href="#could-postgresql-mvcc-be-turned-into-time-travel" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>In principle, yes—but retaining tuples is only the first piece.&lt;/p>
&lt;p>A complete temporal query system would also need to provide:&lt;/p>
&lt;ul>
&lt;li>durable historical snapshot identifiers;&lt;/li>
&lt;li>a mapping from timestamps to transaction visibility;&lt;/li>
&lt;li>retention rules that stop vacuum from removing required tuples;&lt;/li>
&lt;li>historical index behavior;&lt;/li>
&lt;li>transaction-ID lifecycle handling;&lt;/li>
&lt;li>garbage collection after a historical state expires.&lt;/li>
&lt;/ul>
&lt;p>Simply disabling vacuum would not provide this. It would mostly provide table and index bloat, increasingly unhappy transaction IDs, and a future operator asking who turned the database into an archaeological site.&lt;/p>
&lt;p>This distinction is important:&lt;/p>
&lt;blockquote>
&lt;p>PostgreSQL&amp;rsquo;s old tuple versions are implementation state retained while active transactions may need them.&lt;/p>
&lt;/blockquote>
&lt;p>A temporal extension or application-level history table can build durable history on top of PostgreSQL. But that is an additional data model and lifecycle, not a free consequence of MVCC.&lt;/p>
&lt;h2>The lakehouse starts from a different unit of change&lt;span class="hx:absolute hx:-mt-20" id="the-lakehouse-starts-from-a-different-unit-of-change">&lt;/span>
&lt;a href="#the-lakehouse-starts-from-a-different-unit-of-change" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>Parquet itself does not provide time travel, transactions, or branches. It is a file format.&lt;/p>
&lt;p>The relevant property is how lakehouse systems use Parquet: once a data file is published, it is normally treated as immutable. Changes produce new files and publish a new metadata state describing which files are active.&lt;/p>
&lt;p>That gives the system two layers:&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">Data layer: Parquet data files and delete files
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">Metadata layer: the file set visible in each table or catalog version&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;p>Suppose the current snapshot is &lt;code>S1&lt;/code>:&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">S1 ──▶ orders-001.parquet
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> └─▶ orders-002.parquet&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;p>An update affects rows stored in &lt;code>orders-002.parquet&lt;/code>. A copy-on-write implementation can write a replacement and publish &lt;code>S2&lt;/code>:&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">S1 ──▶ orders-001.parquet
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> └─▶ orders-002.parquet
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">S2 ──▶ orders-001.parquet
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> └─▶ orders-003.parquet&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;p>The unchanged file is shared. The changed file is replaced in the new snapshot. As long as &lt;code>S1&lt;/code> and &lt;code>orders-002.parquet&lt;/code> remain retained, both versions are queryable.&lt;/p>
&lt;p>A merge-on-read implementation may instead add a new data file and a delete file that hides the older row. The physical mechanism is different, but the principle is the same: a new metadata state is published without destroying the state required by retained snapshots.&lt;/p>
&lt;h2>Time travel is snapshot selection&lt;span class="hx:absolute hx:-mt-20" id="time-travel-is-snapshot-selection">&lt;/span>
&lt;a href="#time-travel-is-snapshot-selection" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>A historical query does not rebuild the table by replaying every change from the beginning. It selects a retained metadata state and reads the files referenced by it.&lt;/p>
&lt;p>For a requested time &lt;code>T&lt;/code>, the usual timestamp rule is:&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">select the newest retained snapshot whose commit time is &amp;lt;= T&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;p>In Streambed, that looks like:&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-sql" data-lang="sql">&lt;span class="line">&lt;span class="cl">&lt;span class="k">SELECT&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">id&lt;/span>&lt;span class="p">,&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">status&lt;/span>&lt;span class="p">,&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">total&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="k">FROM&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">orders&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="k">AT&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="k">TIMESTAMP&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="o">=&amp;gt;&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">TIMESTAMPTZ&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s1">&amp;#39;2026-08-10 12:00:00&amp;#39;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="p">)&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="k">WHERE&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">id&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="mi">42&lt;/span>&lt;span class="p">;&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;p>The timestamp is a lakehouse commit cutoff. It is not a predicate on an &lt;code>updated_at&lt;/code> column, and it is not necessarily the time of the original PostgreSQL transaction.&lt;/p>
&lt;p>Streambed consumes PostgreSQL WAL, buffers changes, and commits them according to its flush thresholds and interval. Two source transactions may appear in one lakehouse snapshot. A large PostgreSQL transaction may also be split across lakehouse snapshots if a table reaches its flush threshold before that transaction finishes.&lt;/p>
&lt;p>The historical state is therefore:&lt;/p>
&lt;blockquote>
&lt;p>What had Streambed committed to the lakehouse by this time?&lt;/p>
&lt;/blockquote>
&lt;p>It is not a reconstruction of PostgreSQL&amp;rsquo;s exact MVCC snapshot at that wall-clock instant.&lt;/p>
&lt;h2>Branching is the next metadata operation&lt;span class="hx:absolute hx:-mt-20" id="branching-is-the-next-metadata-operation">&lt;/span>
&lt;a href="#branching-is-the-next-metadata-operation" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>Once snapshots have durable identities, a branch is a natural extension.&lt;/p>
&lt;p>Imagine that &lt;code>main&lt;/code> points to &lt;code>S2&lt;/code>:&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">S1 ──▶ S2 ← main&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;p>Creating an experiment from &lt;code>S1&lt;/code> does not require copying every Parquet file. A branch can initially point to the same snapshot:&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl"> ┌──▶ S2 ← main
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">S1 ────┤
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> └─────── experiment&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;p>When the experiment writes data, it creates new files and metadata on a separate lineage:&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl"> ┌──▶ S2 ──▶ S3 ← main
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">S1 ────┤
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> └──▶ E2 ──▶ E3 ← experiment&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;p>Both lineages can continue sharing unchanged files. The cost grows with the data each branch changes and the history it retains, not necessarily with the full size of the original table.&lt;/p>
&lt;p>This is similar to Git: creating a branch is cheap because it starts as another reference to an existing commit. New commits add objects and move that reference. Old objects can be removed only when no retained reference needs them.&lt;/p>
&lt;p>Apache Iceberg exposes this model directly through &lt;a href="https://iceberg.apache.org/docs/latest/branching/"target="_blank" rel="noopener">branches and tags&lt;/a>. Other table formats and catalogs provide different APIs and guarantees, so branching should not be assumed merely because files are immutable. The point is that immutable files plus versioned metadata make branching a natural capability to implement.&lt;/p>
&lt;p>Streambed currently exposes timestamp-based historical reads; it does not yet expose user-created lakehouse branches. Branching here describes the architectural consequence, not a current Streambed command.&lt;/p>
&lt;h2>First-class does not mean free&lt;span class="hx:absolute hx:-mt-20" id="first-class-does-not-mean-free">&lt;/span>
&lt;a href="#first-class-does-not-mean-free" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>Time travel and branching have real costs.&lt;/p>
&lt;p>Consider small-file compaction:&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">Snapshot S1 → files A + B&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;p>A compaction job combines them into a larger file:&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">Snapshot S2 → file C&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;p>Current queries against &lt;code>S2&lt;/code> can read only &lt;code>C&lt;/code>, but retaining &lt;code>S1&lt;/code> means the object store must retain all three files:&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">S1 needs A + B
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">S2 needs C&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;p>Deleting &lt;code>A&lt;/code> and &lt;code>B&lt;/code> would break time travel to &lt;code>S1&lt;/code>. They become reclaimable only after &lt;code>S1&lt;/code> expires and no branch or tag references them.&lt;/p>
&lt;p>Historical retention therefore creates several forms of amplification:&lt;/p>
&lt;h3>Storage amplification&lt;span class="hx:absolute hx:-mt-20" id="storage-amplification">&lt;/span>
&lt;a href="#storage-amplification" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h3>&lt;p>Replaced data and delete files remain while retained snapshots reference them. More history and more branches generally mean more stored objects.&lt;/p>
&lt;h3>Metadata amplification&lt;span class="hx:absolute hx:-mt-20" id="metadata-amplification">&lt;/span>
&lt;a href="#metadata-amplification" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h3>&lt;p>Every commit adds snapshot, manifest, or catalog metadata. Long-running streaming workloads can accumulate many versions and require metadata maintenance.&lt;/p>
&lt;h3>Read amplification&lt;span class="hx:absolute hx:-mt-20" id="read-amplification">&lt;/span>
&lt;a href="#read-amplification" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h3>&lt;p>An older snapshot may reference small files that were later compacted. Current queries benefit from compaction while historical queries still pay the cost of the older layout.&lt;/p>
&lt;h3>Write amplification&lt;span class="hx:absolute hx:-mt-20" id="write-amplification">&lt;/span>
&lt;a href="#write-amplification" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h3>&lt;p>An update may rewrite a complete file or create both new data and delete files. Immutability moves work away from in-place modification; it does not remove the work.&lt;/p>
&lt;h3>Garbage-collection complexity&lt;span class="hx:absolute hx:-mt-20" id="garbage-collection-complexity">&lt;/span>
&lt;a href="#garbage-collection-complexity" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h3>&lt;p>A file is safe to delete only when it is unreachable from every retained snapshot, branch, and tag. Snapshot expiration and orphan-file cleanup must be conservative, especially while writers are active.&lt;/p>
&lt;p>The trade-off resembles PostgreSQL MVCC more than it first appears:&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">PostgreSQL:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">long-lived snapshots → old tuples cannot be vacuumed → heap/index bloat
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">Lakehouse:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">long retention/branches → old files cannot be deleted → object/metadata growth&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;p>Both systems must decide when an old physical representation is no longer observable. The difference is where that decision appears in the product.&lt;/p>
&lt;h2>What makes history first-class?&lt;span class="hx:absolute hx:-mt-20" id="what-makes-history-first-class">&lt;/span>
&lt;a href="#what-makes-history-first-class" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>The defining property is not that old bytes happen to exist. Old PostgreSQL tuples, WAL segments, temporary files, and object-store orphans may all exist without being safely queryable.&lt;/p>
&lt;p>History becomes first-class when the system provides:&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Identity&lt;/strong> — a snapshot ID, timestamp, tag, or branch names a state.&lt;/li>
&lt;li>&lt;strong>Visibility semantics&lt;/strong> — the system defines exactly which rows and files belong to that state.&lt;/li>
&lt;li>&lt;strong>Queryability&lt;/strong> — normal reads can select the state without restoring another database.&lt;/li>
&lt;li>&lt;strong>Retention policy&lt;/strong> — operators control how long the state remains available.&lt;/li>
&lt;li>&lt;strong>Safe garbage collection&lt;/strong> — data is removed only after retained states no longer reference it.&lt;/li>
&lt;li>&lt;strong>Lineage&lt;/strong> — the system records how snapshots and branches relate.&lt;/li>
&lt;/ol>
&lt;p>Lakehouse formats do not eliminate the costs of history. They promote these concerns into the table and catalog model.&lt;/p>
&lt;p>That is the useful distinction between MVCC history and lakehouse history:&lt;/p>
&lt;blockquote>
&lt;p>MVCC keeps old versions so concurrent transactions can finish correctly.&lt;/p>
&lt;p>A lakehouse keeps named snapshots so future queries can deliberately return to them.&lt;/p>
&lt;/blockquote>
&lt;h2>History is a storage policy, not magic&lt;span class="hx:absolute hx:-mt-20" id="history-is-a-storage-policy-not-magic">&lt;/span>
&lt;a href="#history-is-a-storage-policy-not-magic" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>Immutable files make time travel and branching natural because new states can share most of their physical data with old states. Metadata chooses which state a reader sees, while retention determines how long that choice remains available.&lt;/p>
&lt;p>But every retained possibility has a cost. Old files consume storage. Old layouts may be slower to query. Branches delay garbage collection. Compaction may temporarily increase rather than decrease total storage.&lt;/p>
&lt;p>So “first-class” should not be read as “free.” It means something more useful:&lt;/p>
&lt;blockquote>
&lt;p>History is explicit, addressable, queryable, and disposable according to policy.&lt;/p>
&lt;/blockquote>
&lt;p>That is the foundation Streambed uses when it turns PostgreSQL changes into lakehouse snapshots—and why a historical query can be a metadata selection instead of a database restore.&lt;/p></description></item><item><title>Time travel for Postgres: What did this table look like at noon?</title><link>https://streambed.dev/blog/time-travel-ducklake-iceberg/</link><pubDate>Thu, 24 Sep 2026 00:00:00 +0000</pubDate><guid>https://streambed.dev/blog/time-travel-ducklake-iceberg/</guid><description>
&lt;p>Streambed now supports time travel queries over your replicated Postgres data.&lt;/p>
&lt;p>Pick a timestamp, add an &lt;code>AT&lt;/code> clause to the table, and query a retained lakehouse snapshot through the same Postgres-compatible query server. Time travel works with both DuckLake and Iceberg targets, including joins across historical tables.&lt;/p>
&lt;p>&lt;img src="https://streambed.dev/images/streambed-time-travel.png" alt="How Streambed selects a historical snapshot" loading="lazy" />&lt;/p>
&lt;h2>Time travel to an earlier table state&lt;span class="hx:absolute hx:-mt-20" id="time-travel-to-an-earlier-table-state">&lt;/span>
&lt;a href="#time-travel-to-an-earlier-table-state" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>Suppose an order has changed, but you need to see what Streambed had committed at noon:&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-sql" data-lang="sql">&lt;span class="line">&lt;span class="cl">&lt;span class="k">SELECT&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">id&lt;/span>&lt;span class="p">,&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">status&lt;/span>&lt;span class="p">,&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">total&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="k">FROM&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">orders&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="k">AT&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="k">TIMESTAMP&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="o">=&amp;gt;&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">TIMESTAMPTZ&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s1">&amp;#39;2026-08-10 12:00:00&amp;#39;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="p">)&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="k">WHERE&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">id&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="mi">42&lt;/span>&lt;span class="p">;&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;p>The query selects the newest retained snapshot committed at or before the requested time. It never substitutes a later snapshot.&lt;/p>
&lt;p>Normal queries still return the latest state:&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-sql" data-lang="sql">&lt;span class="line">&lt;span class="cl">&lt;span class="k">SELECT&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">id&lt;/span>&lt;span class="p">,&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">status&lt;/span>&lt;span class="p">,&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">total&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="k">FROM&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">orders&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="k">WHERE&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">id&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="mi">42&lt;/span>&lt;span class="p">;&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;p>This makes historical reads opt-in. Existing queries do not need to change.&lt;/p>
&lt;h2>Historical joins&lt;span class="hx:absolute hx:-mt-20" id="historical-joins">&lt;/span>
&lt;a href="#historical-joins" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>Time travel is also available when a query needs more than one table:&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-sql" data-lang="sql">&lt;span class="line">&lt;span class="cl">&lt;span class="k">SELECT&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">o&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="n">id&lt;/span>&lt;span class="p">,&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">o&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="n">status&lt;/span>&lt;span class="p">,&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="k">c&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="n">name&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="k">FROM&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">orders&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="k">AS&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">o&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="k">AT&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="k">TIMESTAMP&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="o">=&amp;gt;&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">TIMESTAMPTZ&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s1">&amp;#39;2026-08-10 12:00:00&amp;#39;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="p">)&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="k">JOIN&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">customers&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="k">AS&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="k">c&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="k">AT&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="k">TIMESTAMP&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="o">=&amp;gt;&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">TIMESTAMPTZ&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s1">&amp;#39;2026-08-10 12:00:00&amp;#39;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="p">)&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="k">ON&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="k">c&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="n">id&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="n">o&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="n">customer_id&lt;/span>&lt;span class="p">;&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;p>Aliases appear before the &lt;code>AT&lt;/code> clause.&lt;/p>
&lt;p>V1 requires every historical table in one query to use the same instant. This keeps the query model clear and avoids accidentally joining tables from unrelated points in time.&lt;/p>
&lt;p>A timestamp without an offset is interpreted as UTC. Explicit offsets are accepted too, so these refer to the same instant:&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-sql" data-lang="sql">&lt;span class="line">&lt;span class="cl">&lt;span class="n">TIMESTAMPTZ&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s1">&amp;#39;2026-08-10 12:00:00&amp;#39;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="n">TIMESTAMPTZ&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s1">&amp;#39;2026-08-10 14:00:00+02:00&amp;#39;&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;h2>What can this help with?&lt;span class="hx:absolute hx:-mt-20" id="what-can-this-help-with">&lt;/span>
&lt;a href="#what-can-this-help-with" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>Historical queries are useful when you need to:&lt;/p>
&lt;ul>
&lt;li>investigate how data looked before a deployment, update, or delete&lt;/li>
&lt;li>reproduce a report from an earlier committed state&lt;/li>
&lt;li>debug related tables without restoring a database backup&lt;/li>
&lt;/ul>
&lt;p>The history comes from snapshots that Streambed already creates while flushing CDC changes. There is no separate audit-table schema to maintain.&lt;/p>
&lt;h2>Find available snapshots&lt;span class="hx:absolute hx:-mt-20" id="find-available-snapshots">&lt;/span>
&lt;a href="#find-available-snapshots" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>The new &lt;code>snapshots&lt;/code> command lists retained history for a table.&lt;/p>
&lt;p>For DuckLake:&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">./streambed snapshots &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> --target-format&lt;span class="o">=&lt;/span>ducklake &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> --table&lt;span class="o">=&lt;/span>public.orders&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;p>For Iceberg:&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">./streambed snapshots &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> --target-format&lt;span class="o">=&lt;/span>iceberg &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> --table&lt;span class="o">=&lt;/span>public.orders &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> --s3-bucket&lt;span class="o">=&lt;/span>streambed &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> --s3-prefix&lt;span class="o">=&lt;/span>test&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;p>The output includes snapshot timestamps and IDs, plus operation and WAL flush information when available. This gives you a concrete timestamp to use in a query and shows which historical states are still retained.&lt;/p>
&lt;h2>How Streambed time travel works&lt;span class="hx:absolute hx:-mt-20" id="how-streambed-time-travel-works">&lt;/span>
&lt;a href="#how-streambed-time-travel-works" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;ol>
&lt;li>&lt;code>psql&lt;/code> sends the query through Streambed&amp;rsquo;s existing PostgreSQL-compatible query server.&lt;/li>
&lt;li>Streambed validates the timestamp. DuckLake handles the &lt;code>AT&lt;/code> clause natively; for Iceberg, Streambed finds the metadata version containing the newest snapshot at or before the cutoff.&lt;/li>
&lt;li>DuckDB reads the selected snapshot&amp;rsquo;s Parquet data and returns the historical rows.&lt;/li>
&lt;/ol>
&lt;h2>One syntax, two targets&lt;span class="hx:absolute hx:-mt-20" id="one-syntax-two-targets">&lt;/span>
&lt;a href="#one-syntax-two-targets" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;ul>
&lt;li>&lt;strong>DuckLake:&lt;/strong> tables at the requested time come from a shared catalog snapshot.&lt;/li>
&lt;li>&lt;strong>Iceberg:&lt;/strong> each table independently selects its newest snapshot at or before that time, so a historical join is not one atomic cross-table commit.&lt;/li>
&lt;/ul>
&lt;h2>Three details to remember&lt;span class="hx:absolute hx:-mt-20" id="three-details-to-remember">&lt;/span>
&lt;a href="#three-details-to-remember" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;ul>
&lt;li>The timestamp is Streambed&amp;rsquo;s lakehouse commit time, not the original PostgreSQL transaction time or a row&amp;rsquo;s &lt;code>updated_at&lt;/code> value.&lt;/li>
&lt;li>Only retained snapshots are queryable. Once a snapshot and its files expire, that state is gone.&lt;/li>
&lt;li>In the current Iceberg implementation, deleting or truncating every row clears that table&amp;rsquo;s accessible snapshot history.&lt;/li>
&lt;/ul></description></item><item><title>The small-file problem gets worse with CDC</title><link>https://streambed.dev/blog/ducklake-target-support/</link><pubDate>Sun, 30 Aug 2026 00:00:00 +0000</pubDate><guid>https://streambed.dev/blog/ducklake-target-support/</guid><description>
&lt;p>&lt;em>Why Streambed is adopting DuckLake for Postgres-to-DuckDB analytics.&lt;/em>&lt;/p>
&lt;p>The &lt;a href="https://www.dremio.com/blog/apache-iceberg-small-files-problem-causes-fixes-and-prevention/"target="_blank" rel="noopener">small-file problem&lt;/a> is well known in the lakehouse world, but it gets worse for CDC workloads.&lt;/p>
&lt;p>Every flush to a lakehouse table has a cost. It creates more files, more metadata, and more commit work. Updates and deletes make it worse, especially with copy-on-write.&lt;/p>
&lt;p>Streambed started with Iceberg as the target format. Iceberg is mature, widely supported, and a good choice when you want an open table format across many engines.&lt;/p>
&lt;p>But Streambed has a narrower path:&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">Postgres WAL → Streambed → DuckDB query server&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;p>So the question became simpler:&lt;/p>
&lt;blockquote>
&lt;p>If DuckDB is already the query engine, is Iceberg still the best default target for small, repeated CDC flushes?&lt;/p>
&lt;/blockquote>
&lt;p>We started testing DuckLake to answer that.&lt;/p>
&lt;p>&lt;img src="https://streambed.dev/images/streambed-ducklake.png" alt="What one CDC flush costs" loading="lazy" />&lt;/p>
&lt;h2>Why CDC makes this painful&lt;span class="hx:absolute hx:-mt-20" id="why-cdc-makes-this-painful">&lt;/span>
&lt;a href="#why-cdc-makes-this-painful" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>CDC is rarely one large clean batch. It is usually a steady stream of small changes:&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">small batch → commit
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">small batch → commit
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">small batch → commit&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;p>Small commits are good for freshness. They are bad when each commit creates a lot of lake metadata.&lt;/p>
&lt;p>With Iceberg, small commits can mean more data files, manifests, metadata files, and snapshots. For updates and deletes, copy-on-write can also mean reading old files, removing changed rows, and writing new files.&lt;/p>
&lt;p>That work adds up quickly.&lt;/p>
&lt;h2>Why DuckLake fits Streambed better&lt;span class="hx:absolute hx:-mt-20" id="why-ducklake-fits-streambed-better">&lt;/span>
&lt;a href="#why-ducklake-fits-streambed-better" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>DuckLake still stores table data as Parquet. The difference is where the catalog metadata lives.&lt;/p>
&lt;p>Instead of storing all table metadata as object-store files, DuckLake keeps catalog metadata in a database such as DuckDB, SQLite, or Postgres.&lt;/p>
&lt;p>For Streambed, that matters because DuckDB is already in the hot path. Streambed writes Postgres changes, and the query server uses DuckDB to read them back.&lt;/p>
&lt;p>So DuckLake gives us a shorter path:&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">Postgres changes → DuckDB/DuckLake writer → DuckDB query server&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;p>It is not magic. The data still has to be written. But the commit path fits the shape of CDC better.&lt;/p>
&lt;h2>What the benchmark showed&lt;span class="hx:absolute hx:-mt-20" id="what-the-benchmark-showed">&lt;/span>
&lt;a href="#what-the-benchmark-showed" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>We added an initial benchmark comparing:&lt;/p>
&lt;ul>
&lt;li>Iceberg COW&lt;/li>
&lt;li>DuckLake with SQLite catalog&lt;/li>
&lt;li>DuckLake with DuckDB catalog&lt;/li>
&lt;/ul>
&lt;p>The benchmark uses:&lt;/p>
&lt;ul>
&lt;li>baseline tables: 100k and 1M rows&lt;/li>
&lt;li>mutation workload: 10% of the table&lt;/li>
&lt;li>flush sizes: 100, 500, 1,000, and 10,000 buffered units&lt;/li>
&lt;li>read queries: full aggregate, range filter, point lookup, and TopN&lt;/li>
&lt;/ul>
&lt;p>The question was not “is DuckLake faster in every possible setup?”&lt;/p>
&lt;p>The question was narrower:&lt;/p>
&lt;blockquote>
&lt;p>Does DuckLake handle small, repeated CDC-style flushes better than Streambed’s Iceberg COW path?&lt;/p>
&lt;/blockquote>
&lt;p>The initial answer is yes.&lt;/p>
&lt;p>For example, with a 1M-row baseline, 100k updates, and &lt;code>flush=1,000&lt;/code>:&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">Iceberg COW: 269,344 ms, 806 objects
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">DuckLake SQLite: 7,075 ms, 405 objects
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">DuckLake DuckDB: 6,576 ms, 405 objects&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;p>That is not a tiny difference. That is “go get coffee” vs “blink a few times.”&lt;/p>
&lt;p>The benchmark bypasses PostgreSQL logical replication and psql-wire overhead. It measures the writer/catalog commit path and DuckDB reads over local MinIO. So treat it as a lakehouse target benchmark, not an end-to-end Streambed benchmark.&lt;/p>
&lt;p>Full benchmark notes and raw results are here:&lt;/p>
&lt;p>&lt;a href="https://github.com/viggy28/streambed/blob/main/docs/benchmarks/ducklake-vs-iceberg.md"target="_blank" rel="noopener">DuckLake vs Iceberg benchmark&lt;/a>&lt;/p>
&lt;h2>What this means for Streambed&lt;span class="hx:absolute hx:-mt-20" id="what-this-means-for-streambed">&lt;/span>
&lt;a href="#what-this-means-for-streambed" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>Iceberg is not going away. It is still useful if you need broad lakehouse interoperability across engines like Spark, Trino, Snowflake, Flink, and others.&lt;/p>
&lt;p>But for Streambed, DuckLake is becoming the preferred direction. Streambed&amp;rsquo;s goal is:&lt;/p>
&lt;blockquote>
&lt;p>Make Postgres data available as a DuckDB-powered analytical replica, while preserving as much Postgres client and query compatibility as possible.&lt;/p>
&lt;/blockquote>
&lt;p>The table format should mostly stay in the background. Users should care that their Postgres data is fresh, queryable, and easy to connect to.&lt;/p>
&lt;p>So the next big Streambed work is on the Postgres-facing layer:&lt;/p>
&lt;ul>
&lt;li>Postgres clients&lt;/li>
&lt;li>SQL compatibility&lt;/li>
&lt;li>functions and operators&lt;/li>
&lt;li>&lt;code>information_schema&lt;/code> and &lt;code>pg_catalog&lt;/code>&lt;/li>
&lt;li>prepared statements and parameters&lt;/li>
&lt;li>type fidelity for JSON, arrays, UUID, numeric, bytea, timestamps, time zones, and TOAST-heavy columns&lt;/li>
&lt;/ul>
&lt;p>DuckLake helps because it makes the default write/read path simpler for the system Streambed is actually building.&lt;/p>
&lt;p>Streambed started with Iceberg because it was the obvious open lakehouse answer. For a DuckDB-backed analytical replica of Postgres, DuckLake looks like the better default.&lt;/p></description></item><item><title>Compaction in Streambed</title><link>https://streambed.dev/blog/introducing-compaction/</link><pubDate>Sun, 09 Aug 2026 00:00:00 +0000</pubDate><guid>https://streambed.dev/blog/introducing-compaction/</guid><description>
&lt;p>Streambed is getting more mature. Big data has a small problem or more precisely small file problem.&lt;/p>
&lt;p>Every flush (not every single DML) creates a new parquet file. Typically that means hundreds and thousands of small files for a single table. Two challenges with that:&lt;/p>
&lt;ol>
&lt;li>It impacts the latency (specifically query read)&lt;/li>
&lt;li>Maintenance (S3 scales but it&amp;rsquo;s not like infinite)&lt;/li>
&lt;/ol>
&lt;p>To solve that we introduce, &lt;a href="https://streambed.dev/docs/maintenance/#snapshot-expiration"target="_blank" rel="noopener">&lt;code>maintenance&lt;/code>&lt;/a> and &lt;a href="https://streambed.dev/docs/maintenance/#small-file-compaction"target="_blank" rel="noopener">&lt;code>maintenance compact&lt;/code>&lt;/a> command which does metadata maintenance, small file compaction.&lt;/p>
&lt;h2>Implementation&lt;span class="hx:absolute hx:-mt-20" id="implementation">&lt;/span>
&lt;a href="#implementation" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>Reads multiple small files and rewrites them as a large parquet file. Using sqlite for compare and swap. Explored S3 conditional rewites but we will introduce once streambed support multiple hosts. For single host, sqlite row lock is sufficient. Implemented as a separate process to keep the &lt;code>sync&lt;/code> process clean and also reliable (avoid co-ordination b/w multiple routine). Eventually should merge it as part of the &lt;code>sync&lt;/code> daemon.&lt;/p>
&lt;h2>What&amp;rsquo;s next&lt;span class="hx:absolute hx:-mt-20" id="whats-next">&lt;/span>
&lt;a href="#whats-next" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>Support compaction in MOR &lt;a href="https://github.com/viggy28/streambed/issues/32"target="_blank" rel="noopener">equality delete&lt;/a>, &lt;a href="https://github.com/viggy28/streambed/issues/36"target="_blank" rel="noopener">optimize COW with key-range pruning&lt;/a>,&lt;/p></description></item><item><title>Why I Built Streambed</title><link>https://streambed.dev/blog/why-streambed/</link><pubDate>Sun, 12 Apr 2026 00:00:00 +0000</pubDate><guid>https://streambed.dev/blog/why-streambed/</guid><description>
&lt;p>Every Postgres database eventually hits the same wall: analytical queries that are too slow for production but too important to ignore.&lt;/p>
&lt;p>The usual answer is to build an ETL pipeline — set up Spark, Airflow, or some managed service to extract data from Postgres, transform it, and load it into a data warehouse. By the time you&amp;rsquo;re done, you have a dozen moving parts, a multi-hour data delay, and a new class of bugs where your warehouse diverges from your source of truth.&lt;/p>
&lt;p>Streambed takes a different approach. It connects to Postgres as a logical replication subscriber — the same mechanism Postgres uses for read replicas — and streams WAL changes directly into Apache Iceberg tables on S3. No intermediate queue. No batch jobs. No Spark.&lt;/p>
&lt;p>The result: your analytical data is seconds behind production, stored in an open format (Parquet + Iceberg), and queryable from &lt;code>psql&lt;/code> using the built-in DuckDB query server.&lt;/p>
&lt;h2>The benchmark that convinced me&lt;span class="hx:absolute hx:-mt-20" id="the-benchmark-that-convinced-me">&lt;/span>
&lt;a href="#the-benchmark-that-convinced-me" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>pgbench schema, 1M accounts, 500K history rows. A correlated subquery that joins across tables:&lt;/p>
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code">
&lt;div>&lt;pre>&lt;code>Postgres (port 5432): ~45s
Streambed (port 5433): ~2s&lt;/code>&lt;/pre>&lt;/div>&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0">
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
>
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4">&lt;/div>
&lt;/button>
&lt;/div>
&lt;/div>
&lt;p>Same query. Same &lt;code>psql&lt;/code>. Same results. DuckDB&amp;rsquo;s columnar engine on Parquet files is simply a better fit for analytical workloads than Postgres&amp;rsquo;s row-oriented storage.&lt;/p>
&lt;h2>What&amp;rsquo;s next&lt;span class="hx:absolute hx:-mt-20" id="whats-next">&lt;/span>
&lt;a href="#whats-next" class="subheading-anchor" aria-label="Permalink for this section">&lt;/a>&lt;/h2>&lt;p>This blog will be a place to share the deeper technical challenges — WAL decoding edge cases, copy-on-write merge strategies, Iceberg metadata management, and more. Stay tuned.&lt;/p></description></item></channel></rss>