Encrypted mysqldump → S3 with restic vs openssl + aws-cli: why one survives a corrupted backup file and the other doesn’t

I had a single mysqldump file go partially-corrupted on S3 last year — exactly one byte flipped somewhere in the middle of a 1.2 GB nightly backup. The kind of thing that should never happen, but did, probably during transit to S3 over a flaky home connection. The interesting part: my restic backup of the same database recovered cleanly. My openssl enc + aws s3 cp backup of the same database did not. They were the same SQL bytes going in. The difference was in the encryption layer.

This post is about why those two pipelines have wildly different recovery characteristics, and why I now run restic for everything that matters and reserve openssl + aws-cli for the things I’m OK losing.

The two pipelines

The naive openssl version, which I’d been running for years:

mysqldump --single-transaction --routines --triggers mydb \
  | gzip -9 \
  | openssl enc -aes-256-cbc -pbkdf2 -pass file:/etc/backup-pass \
  | aws s3 cp - s3://my-backups/mydb-$(date +%F).sql.gz.enc

The restic version that replaced it:

mysqldump --single-transaction --routines --triggers mydb \
  | restic backup --stdin --stdin-filename mydb.sql --tag mysql --tag mydb

Both produce encrypted backups in S3. Both are roughly the same size for the first run. Both decrypt with a passphrase. From the outside they look interchangeable. They are not.

Why one corrupted byte killed the openssl backup

AES-256-CBC chains every block to the previous one. If a single byte gets corrupted in the middle of the file, every byte that follows is now also corrupted — the chain breaks and the rest of the plaintext is garbage. openssl enc doesn’t add any error correction, doesn’t chunk the file into independently decryptable pieces, and doesn’t include any integrity check beyond “did the passphrase decrypt block 1 to something plausible”.

So when I tried to restore: openssl decrypted the first ~600 MB cleanly, hit the corrupted byte, and from that point on emitted gzip-format garbage. gzip -d failed at the same offset. I had ~50% of my SQL dump and no way to get the rest.

I could have salvaged the recovered half with head + manual SQL replay, but the corruption hit during the middle of a long INSERT block, so even the recovered SQL was syntactically broken at the truncation point.

Why restic survived

restic doesn’t store your file as one encrypted blob. It chunks the input using content-defined chunking (rolling hash, ~1 MB chunks on average), encrypts each chunk independently with AES-256-CTR + Poly1305 MAC, and stores them as separate objects in the S3 “pack” files. Each chunk has its own MAC; if one chunk is corrupted, the rest are still valid.

So when restic tried to restore the same backup: it noticed one chunk failed MAC verification, logged a warning, and produced a SQL file with one ~1 MB hole in it where that chunk would have gone. The rest was perfectly readable. I lost a single block of one INSERT — recoverable from the previous night’s backup, with five minutes of merging.

The architectural difference is the whole story:

• openssl enc: one giant encrypted stream. One bad byte ruins everything after it.
• restic: many independent encrypted chunks. One bad chunk ruins ~1 MB. Everything else is fine.

The other things restic gives you for free

• Deduplication. A 5 GB nightly mysqldump that’s 99% identical to last night’s takes ~50 MB of new storage in S3, not 5 GB. Over a month, this is the difference between paying $30 and paying $1.
• Snapshots. restic snapshots shows you every backup, when it ran, what tags it had. aws s3 ls shows you a directory of opaque .sql.gz.enc files.
• Verification. restic check --read-data downloads every chunk and verifies its MAC. You’ll know within hours if a backup has gone bad, not when you need it.
• Forget + prune. Retention policy is one command (restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune). With openssl + aws-cli you’re writing your own lifecycle policy and hoping it’s right.

When the openssl path is fine

If your backup is small (a few hundred MB), runs over a stable LAN to a local NAS, and the consequence of a corrupted backup is “we restore yesterday’s instead”, the openssl pipeline is genuinely fine. It’s two commands, no daemon, no dependency on a Go binary. The simplicity is real.

But for anything that crosses the public internet, anything sized in GB, anything where “yesterday” isn’t an acceptable RPO — restic. Every time. The chunked-encryption-with-MAC architecture isn’t an optimisation; it’s the difference between a backup that survives transit corruption and one that doesn’t.

Cover photo: Cookiecutter on Pexels.