Database Schema Migration Safety in Continuous Deployment Workflows
Schema changes in automated deployments need their own gated pipeline to prevent outages.
A CHECK constraint on a phone_number column took down login for 12.4 million users. A single ALTER TABLE statement, adding an index to a table that had grown past 500 gigabytes, locked that table for eight hours during peak traffic. Neither incident involved a bug in the traditional sense. Both were schema changes that behaved exactly as the database engine promised they would, which turns out to be the problem: continuous deployment pipelines that automate application releases with real discipline still tend to treat schema migrations as an afterthought, and the database is what remembers that mistake far longer than anyone would like.
The team that added the phone_number constraint tested it against a small copy of production, saw it complete quickly, and shipped it. In full production, the same statement had to check every existing row for compliance before it could commit, and while it did that, it held a lock that made the table untouchable. Four hours and twelve minutes later, the checkout API had been timing out on every single request the entire time. The 500 GB table incident followed the same shape on a different axis: a routine index addition, run during business hours, that the storage engine could not complete without holding the table hostage for the better part of a workday.
These are the predictable output of a mechanism that most teams understand in the abstract and misjudge in the specific. They are the predictable output of a mechanism that most teams understand in the abstract and misjudge in the specific.
The mechanical reason schema changes cause outages: how table locks cascade
PostgreSQL and MySQL both use locking to keep schema changes safe from concurrent access, and that locking is what turns a three-second migration into a five-minute outage. In Postgres, many common DDL operations require what's called an AccessExclusiveLock. That lock doesn't just block other writers. It blocks reads, writes, and any other schema change on the table, all at once, for as long as the operation needs to hold it.
The real damage is not the migration's runtime; it's the queue that forms behind it. It's the queue that forms behind it. Once a DDL statement is waiting to acquire its lock, every query that comes in afterward on that same table lines up behind the DDL, not in front of it. A long-running transaction ahead of the migration in the lock queue can block every new application query that arrives after the migration starts waiting. Users don't experience the migration. They experience the wait.
That gap between "the migration took three seconds" and "the site was down for five minutes" is the single most common failure pattern in schema deployment, and it shows up because engineers benchmark the DDL statement in isolation and never account for what's sitting in front of it in the lock queue. MySQL isn't structurally different here: ALTER TABLE requires a lock that blocks concurrent access, and any transaction already reading or writing that table, even a slow one that has nothing to do with the migration, can block the ALTER from proceeding.
The foundational rule: schema changes must be a separate, gated pipeline stage
The discipline that prevents both incidents above is simple to state and consistently hard to enforce: a breaking schema change should never ship inside the same deployment artifact as the application code that depends on it. Bundle them together, and you've coupled two failure domains that behave nothing alike. Application code is disposable. A bad container gets replaced, a feature flag gets flipped off, and the failure disappears. A schema change acts directly on persistent state, and there's no equivalent of "redeploying away" a table that's been locked for four hours or a column that's already been dropped.
The fix is positional. Migrations belong in one dedicated pipeline stage, run after application tests pass but before any new application container gets provisioned. Running the migration inside container startup, which is the default behavior in a lot of frameworks, puts the control point in exactly the wrong place. It also creates a race condition in rolling deployments: if a rolling deploy spins up several replicas near-simultaneously, each one may attempt the same migration against the same database at the same time. A dedicated migration stage that runs to completion before any replica starts eliminates that race by construction, because there's only ever one thing attempting the migration.
Treat that stage as a gate. The schema change should execute only after the pipeline has already confirmed that the new application version and the previous one can coexist against the same schema. That confirmation is the entire point of the next section.
How expand-and-contract works
Expand-and-contract is the pattern that makes the gate in the previous section actually enforceable, and it's become the dominant approach for one reason: it never asks two incompatible schema versions to exist for the app at the same moment. It runs in three phases. Expand introduces new structures without touching or removing the old ones, so both the old and new application versions can read and write safely against the same database. Migrate keeps the old and new structures in sync while traffic gradually shifts from one version to the other. Contract removes the deprecated structures, but only after every bit of traffic has fully moved onto the new version.
A column rename is the clearest way to see the mechanics. The naive approach renames the column in one migration and ships it alongside the code that expects the new name, which works fine until the previous application version, still running somewhere in a rolling deploy, tries to read a column that no longer exists. The expand-and-contract version breaks that single step into four separate ones. First, deploy code that writes to both the old column and the new column at the same time. Second, run a background backfill that populates the new column from whatever's already sitting in the old one. Third, deploy code that switches reads over to the new column and stops writing to the old one. Fourth, once the new application version is fully rolled out and has proven stable, deploy one final, separate migration that drops the old column.
Keeping the two representations consistent during that migrate window is its own engineering problem, and there are several accepted ways to solve it: database triggers, change data capture, transactional dual writes, or event-driven synchronization pipelines. None of them is automatically safe by virtue of being on this list. Dual writes handled purely at the application layer are a particularly common trap, because a failure between the two writes, a retry that fires only one of them, or a transaction that partially completes can all quietly introduce divergence between the old and new columns, and that divergence often isn't detected until the contract phase, when it's much more expensive to fix.
Done properly, this pattern scales past the size where anyone would trust a single ALTER statement. One documented case involved migrating 2 terabytes of data across 50 tables using expand-and-contract with zero user impact. The lesson from that case wasn't really about the tooling. It was that designing for zero downtime from the outset is consistently easier than retrofitting it onto a schema that was never built with a migrate phase in mind.
Online schema change tools for tables where locking is not acceptable
Expand-and-contract handles the sequencing. It doesn't remove the underlying fact that some DDL operations, run directly against a large or high-write table, will still lock it long enough to matter. That's the 500 GB, eight-hour incident from the introduction: a table too large and too active for a plain ALTER TABLE to be an option, expand-and-contract or otherwise. This is the specific gap that online schema change tools exist to close.
gh-ost, GitHub's Online Schema Transmogrifier, Translator, Transformer, Transfigurator for MySQL, works by reading the binary log stream instead of relying on triggers to track changes while it copies data into a shadow table. That triggerless design is the whole advantage: trigger-based approaches carry their own overhead and can trigger lock escalation under write-heavy load, and gh-ost avoids that entirely by watching the binlog instead of instrumenting the table itself. Percona's pt-online-schema-change is the commonly cited alternative, and it takes the opposite approach: a synchronous, trigger-based method that's been in production use for a long time and remains a reasonable choice depending on the workload.
On the Postgres side, pgroll, built by Xata, takes expand-and-contract and pushes it down into the engine layer instead of leaving it as an application-level discipline. Its headline feature is making two schema versions, the old one and the new one, simultaneously available to client applications, which is expand-and-contract implemented directly rather than assembled by hand across four separate deploys. It also adds automatic retry with exponential backoff when a lock acquisition fails, which addresses a real gap Postgres leaves open: when lock_timeout fires, Postgres just gives up, and pgroll's retry behavior is built to cover that failure mode instead of leaving it to the operator.
Migration versioning tools: what Flyway, Liquibase, Atlas, and others give you
Sequencing the migration and protecting the table from a bad lock only solves part of the problem. Someone still has to track which migrations have run, in what order, against which environment, and versioning tools are what answer that question. The landscape splits into four working groups: versioned migration CLIs like Flyway, Liquibase, Sqitch, goose, golang-migrate, and dbmate; declarative schema-as-code tools like Atlas and Skeema; ORM-integrated migrators like Alembic and Prisma; and platforms such as Bytebase, which layer review and approval workflows on top of the migration process itself.
Flyway, from Redgate, runs on convention over configuration. Naming a file V2__add_orders_table.sql causes Flyway to run it after V1 and record the result in its own history table. That naming discipline is close to the entire learning curve. The engine is currently at version 13.3.0, released August 13, 2026. Redgate closed the Flyway Teams tier to new customers on May 14, 2025, though existing Teams customers can still renew or add licenses, leaving Community and Enterprise as the two tiers going forward. Flyway remains the default answer for teams that want plain SQL migrations running in CI/CD without pulling in a framework-specific dependency.
Liquibase has taken a more complicated licensing turn. The Community edition, now on the 5.x line, moved to the Functional Source License, which is not an OSI-approved open source license, starting with version 5.0 in 2025. The commercial tier was renamed from Liquibase Pro to Liquibase Secure, and Liquibase Secure 5.1 added PII policy checks and stricter pre-deployment validation aimed squarely at regulated industries. Pricing is $500 for Pro according to one source, though a separate source lists $1,259 per target per year. Whatever the number, Liquibase's actual footprint is broad: support for over 60 databases, migration formats spanning SQL, XML, YAML, and JSON, and integrations across Jenkins, GitLab, GitHub, Azure, CircleCI, Bamboo, and TeamCity. On TrustRadius, it holds an 8.1 out of 10 across 115 reviews.
Atlas takes a genuinely different approach from either of those. It's declarative, schema-as-code, closer in spirit to how Terraform manages infrastructure than to how Flyway manages a sequence of numbered scripts. Instead of writing a migration file, you describe the schema you want, and Atlas diffs that description against the live database, then generates and lints a migration plan to close the gap. It can take ORM models directly as input, including Prisma, GORM, Drizzle, and Entity Framework Core. No manually numbered script has to exist. Atlas also ships CI integration that surfaces the generated migration plan during code review, and atlas migrate lint is built specifically to catch destructive changes before they reach a reviewer's eyes.
The full pipeline: trigger method, ordering, and guard rails
Choosing a migration tool answers what runs. It doesn't answer when, and that second question, how a migration actually gets triggered inside a deployment, is where a lot of teams quietly reintroduce the exact race conditions expand-and-contract was supposed to eliminate.
One well-structured pattern is a dedicated Kubernetes Job, wired in as a Helm hook. GitLab runs its migration as a post-deploy hook. Airflow, Superset, and Temporal all use hook points as well, Airflow and Superset on post-install/upgrade, Temporal on pre-deploy, and Kong and Jaeger both run theirs pre-install or pre-deploy. The shared property across all of them is that the migration runs exactly once, cleanly, at a defined point relative to the deploy, rather than being left to whichever pod happens to start first.
Gitea takes a slightly different but equally deliberate approach: it runs gitea migrate inside an init container, which by definition completes before the main application container ever starts, so migration is finished before the pod can accept any traffic.
A larger group of projects, including Ghost, Backstage, Keycloak, Grafana, Mattermost, Odoo, Parse Server, Appsmith, and Rocket.Chat, lets the application process itself run the migration on pod startup. It's the most convenient option to wire up, since there's no separate job or hook to configure, but it's also the pattern most exposed to the exact race condition described earlier: a rolling deploy that starts several replicas close together can end up with more than one process attempting the same migration against the same database simultaneously.
And then there's the deliberately manual camp. Zipkin and APISIX both leave migration as an explicit operator action rather than automating it into the pipeline. That's not a gap in tooling. For teams operating under strict change-control requirements, where a human sign-off before a schema change is a compliance requirement rather than a convenience, manual triggering is the appropriate choice.
What separates teams that deploy continuously without database outages from teams for whom the database stays the last manual bottleneck isn't the specific tool on the list above. It's whether schema change is treated, structurally, as its own artifact class, gated, sequenced, and synchronized on its own terms, rather than as one more line item in the same deploy as the code that happens to depend on it.
