MSK backup without a practiced restore is theatre

A production Kafka cluster failure is the sort of incident people invent runbooks for and never rehearse. We had never needed to recover a managed Kafka cluster in anger. That was not a reason to skip backups — it was a reason to treat restore as the acceptance test, not a wiki page.

Automated backups

A Terraform module provisions the backup path: an MSK Connect S3 sink connector and a bucket for topic data. Messages land as JSON with an envelope, flushed on size and interval so S3 does not become a per-message object storm.

Topic selection uses a regex that captures application topics and skips AWS-managed canary and connect internals. Regex on the backup side is fine. Assuming the same trick works on restore is where we got stuck.

KCQL for the sink is powerful and awkward. Getting the exclusion regex right took more iteration than the Terraform surrounding it. The module made the infrastructure repeatable; it did not make the query language pleasant.

What restore actually required

We provisioned a fresh cluster with the same shape as production and an S3 source connector to replay from the backup bucket. The goal was not elegance — it was wall-clock time to usable topics.

Backup can say “all topics matching this pattern.” Restore, with the connector and query language we had, could not expand a topic name from the S3 path back into the target topic in one regex pass. Maintainers confirmed the gap. So restore queries had to name topics explicitly — fine for a drill with a known set, painful as a general DR procedure.

Rough timings from the exercise:

• New MSK cluster plus connectors: on the order of 40 minutes
• Topic provisioning and pointing applications at new bootstrap endpoints: another ~15 minutes on top

Call it about an hour of downtime as a planning number, not a promise. That number is why “we have backups in S3” is not the same as “we can fail over.”

Lessons

Backup without restore is incomplete. The first restore taught more than the module README.

Regex asymmetry matters. Tools that let you select broadly on write and force enumeration on read will dominate your RTO once topic counts grow.

Offsets and internals are a deliberate choice. Skipping managed connect topics keeps backup noise down; it also drops offset-related data other teams might want. Document the omission.

Warm standby is the next cost conversation. A pre-provisioned cluster in another region buys time and spends money whether you need it or not. After a timed restore, that trade-off is a number instead of a slogan.