Day 095 — Migrations with golang-migrate

Month 4 · Week 2 · ⬅ Day 094 · Day 096 ➡ · Journal index

🎯 Learning Objective

Version a database schema with golang-migrate: write ordered up/down migrations, apply them idempotently, embed them in the binary, and recover from a "dirty" failure.

📚 Topics

• Migration file naming, ordering, and the schema_migrations table
• up/down reversibility; running via CLI and as a library
• Embedding migrations with embed.FS; the dirty-state trap

📖 Reading / Sources

• golang-migrate — README
• golang-migrate — Migration files & best practices
• golang-migrate — Postgres driver
• embed package docs

📝 Notes

• A migration tool keeps the schema as an ordered list of versioned change scripts and records which have run. golang-migrate uses paired files: 000001_create_users.up.sql and 000001_create_users.down.sql. The up applies a change; the matching down reverses it → [[schema-migrations]]. I modelled the runner stdlib-only in examples/month-04/migrate.
• Versions are a monotonic prefix (timestamp or sequence). The tool tracks the current version in a schema_migrations table, so applying is idempotent — running up again does nothing once you're current.
• Never edit a shipped migration. Every environment replays the same ordered scripts to converge on an identical schema; editing history makes environments diverge. Add a new migration instead.
• Reversibility: write a real down for each up so you can roll back in dev. In prod, prefer forward-only fixes; many teams treat down as best-effort.
• Dirty state: if a migration fails partway, golang-migrate marks the version dirty and refuses to continue. You must fix the database by hand, then force the version back to a clean number. This is why each migration should be small and, where possible, run in a transaction (Postgres DDL is transactional — most statements can be wrapped; CREATE INDEX CONCURRENTLY cannot).
• Two ways to run it: the CLI (migrate -path ./migrations -database "$DATABASE_URL" up) in CI/deploy, or the library (migrate.New(...), m.Up(), m.Steps(n), m.Down()) at app startup.
• Embed the SQL with //go:embed migrations/*.sql + the iofs source driver, so the binary carries its migrations — no files to ship alongside it.
• Keep migrations separate from data access: golang-migrate owns the schema (DDL); sqlc/pgx run the queries (DML). They meet only in that sqlc reads the same DDL as its schema.

💻 Code Examples

golang-migrate is third-party; this snippet shows embedded migrations applied at startup. The ordering/idempotence logic is runnable here: examples/month-04/migrate · Run: go run ./examples/month-04/migrate.

import (
    "embed"
    "errors"

    "github.com/golang-migrate/migrate/v4"
    _ "github.com/golang-migrate/migrate/v4/database/postgres" // db driver
    "github.com/golang-migrate/migrate/v4/source/iofs"
)

//go:embed migrations/*.sql
var migrationsFS embed.FS

func runMigrations(databaseURL string) error {
    src, err := iofs.New(migrationsFS, "migrations")
    if err != nil {
        return err
    }
    m, err := migrate.NewWithSourceInstance("iofs", src, databaseURL)
    if err != nil {
        return err
    }
    defer m.Close()

    // Up applies all pending migrations. ErrNoChange means "already current" —
    // that is success, not failure, so treat it as nil.
    if err := m.Up(); err != nil && !errors.Is(err, migrate.ErrNoChange) {
        return err
    }
    return nil
}

migrations/
  000001_create_users.up.sql     -- CREATE TABLE users (...);
  000001_create_users.down.sql   -- DROP TABLE users;
  000002_add_orders.up.sql       -- CREATE TABLE orders (...);
  000002_add_orders.down.sql     -- DROP TABLE orders;

🏋️ Exercises / Practice

Exercise	Status	Link
Migration planner: pending versions, dedupe, ordering	✅	exercises/month-04/week-2/migrationplan
Idempotent migration runner (concept)	✅	examples/month-04/migrate

🐛 Mistakes Made

• A migration half-failed and left the version dirty; the next deploy refused to run. Fixed the DDL by hand, then migrate force <version> to clear it.
• Treated migrate.ErrNoChange as an error and crash-looped a healthy service. It means "nothing to do" — ignore it.
• Edited an already-applied migration to "fix" it; staging and prod diverged. Reverted and added a new migration instead.

❓ Open Questions

• Best practice for zero-downtime schema changes (expand/contract) when old and new app versions run simultaneously during a rollout?

🧠 Active Recall (answer without looking)

1. Q: Why must you never edit a migration that has already been applied?

A

Every environment replays the same ordered scripts to reach an identical schema. Editing applied history makes environments diverge — add a new migration instead.

1. Q: What is a "dirty" migration and how do you recover?

A

A migration that failed partway; golang-migrate marks the version dirty and refuses to proceed. Fix the database manually, then `force` the version to a clean number.

🪶 Feynman Reflection

Migrations are version control for your schema. Each numbered up file is a commit; the schema_migrations table is the HEAD pointer. Running up fast-forwards the database to the latest version, skipping anything already applied. Like git, you append history; you don't rewrite it.

🕳️ Knowledge Gaps

• Transactional DDL boundaries — which Postgres statements can't run inside a migration transaction.

✅ Summary

I can author ordered up/down migrations, apply them idempotently via CLI or library, embed them with embed.FS, and recover from a dirty failure.

⏭️ Next Steps / Prep for Tomorrow

• Day 096: organising data access behind the repository pattern.

---

Time spent	Difficulty	Confidence
90 min	🟦🟦⬜⬜⬜	🟦🟦🟦⬜⬜

Suggested commit: docs(journal): schema migrations with golang-migrate (day 095)