Redis MISCONF: Permanent Fix for Failed RDB Snapshots

Error Pattern

Redis can return MISCONF errors when background RDB persistence fails and stop-writes-on-bgsave-error is enabled. The server blocks writes to avoid accepting data it cannot safely persist.

MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk. Commands that may modify the data set are disabled.

Immediate Recovery

• Confirm whether reads still work and whether only write commands fail.
• Check Redis logs before changing configuration.
• If business impact requires temporary write restoration, document the risk of disabling stop-writes-on-bgsave-error and time-box the change.
• Do not treat CONFIG SET stop-writes-on-bgsave-error no as the permanent fix.

Disk, Inode and Filesystem Checks

df -h
 df -i
 mount | grep ' ro,'
 journalctl -u redis --since '30 min ago'

Low disk, exhausted inodes, a read-only filesystem, wrong directory ownership or an invalid working directory can all produce the same application symptom.

Persistence Path and Permissions

• Inspect dir and dbfilename in Redis configuration.
• Confirm the Redis process user can write to the configured directory.
• Check whether a container volume is mounted at the expected path.
• Verify AppArmor/SELinux or systemd sandboxing is not blocking writes.

redis-cli CONFIG GET dir
redis-cli CONFIG GET dbfilename
namei -l /var/lib/redis
sudo -u redis test -w /var/lib/redis && echo writable

Permanent Fix

Repair the underlying storage or permission failure, restore the safe stop-writes behavior, and force a background save only after the write path is healthy.

redis-cli CONFIG SET stop-writes-on-bgsave-error yes
redis-cli BGSAVE
redis-cli INFO persistence | egrep 'rdb_last_bgsave_status|rdb_last_save_time'

Validation and Monitoring

• RDB status reports ok after BGSAVE.
• Application writes succeed without disabling protection.
• Disk and inode alerts exist before exhaustion.
• Redis persistence failures are logged and monitored.
• Rollback plan is documented if storage repair changes mounts or volumes.

Read the Persistence State Before Acting

Start with INFO persistence and the Redis log. rdb_last_bgsave_status tells you whether the latest child completed; rdb_last_bgsave_time_sec shows whether saves are becoming slow; rdb_changes_since_last_save shows how much accepted data is waiting for a durable snapshot. A failed child process, a fork failure, and a filesystem write failure require different repairs.

redis-cli INFO persistence
redis-cli LASTSAVE
redis-cli CONFIG GET save
redis-cli CONFIG GET appendonly
journalctl -u redis-server -n 200 --no-pager

Separate fork pressure from storage failure

BGSAVE forks the Redis process and relies on copy-on-write memory while the child serializes the dataset. On a memory-constrained host, the kernel can refuse the fork or kill the child. Check dmesg for OOM events, compare used_memory_rss with the container or cgroup limit, and inspect vm.overcommit_memory. Do not increase limits blindly: first account for dataset size, fragmentation, replication buffers and copy-on-write growth during a busy save.

redis-cli INFO memory
dmesg -T | egrep -i 'oom|killed process'
sysctl vm.overcommit_memory
systemctl show redis-server -p MemoryMax

Verify the exact write path

The configured dir is evaluated inside the Redis runtime namespace. In a container, a host directory that looks healthy proves nothing unless it is mounted at that exact path. Resolve symlinks, inspect every parent directory with namei, check the mount is read-write, and create then remove a test file as the Redis UID. A successful root write is not a valid permission test.

Choose Containment Deliberately

Disabling stop-writes-on-bgsave-error restores availability by accepting writes without the RDB guarantee. That can be reasonable for a rebuildable cache, but it is a material durability decision for sessions, queues, rate-limit state or primary application data. Record the timestamp, dataset role, last successful save and owner who accepted the risk. Keep the window short and avoid restarting until persistence is understood.

Repair and Prove the Durable Path

• Free space without deleting the last known-good snapshot.
• Correct ownership on the configured directory, not only the dump file.
• Repair or remount a read-only filesystem only after checking kernel and storage errors.
• Raise memory only from measured fork and copy-on-write demand.
• Run BGSAVE, wait for completion, and verify the snapshot timestamp and non-zero size.

redis-cli BGSAVE
watch -n 1 "redis-cli INFO persistence | egrep 'rdb_bgsave_in_progress|rdb_last_bgsave_status|rdb_changes_since_last_save'"
stat /var/lib/redis/dump.rdb
redis-check-rdb /var/lib/redis/dump.rdb

Validate Restart Recovery, Not Only BGSAVE

A successful BGSAVE proves that one snapshot completed; it does not prove that the file can be consumed during recovery. In a disposable replica or isolated environment, start the same Redis version against a copy of the RDB and verify representative keys, TTLs and application reads. This is especially important after disk errors or an interrupted copy.

Prevent the Next MISCONF

• Alert on rdb_last_bgsave_status and time since rdb_last_save_time.
• Alert on disk bytes, inodes and read-only remount events.
• Capacity-plan memory for fork and copy-on-write peaks.
• Test snapshots by loading them outside production.
• Document whether Redis is a cache or a system of record and choose RDB/AOF policy accordingly.

A Safe Incident Runbook

During the first five minutes, preserve the exact MISCONF response, current persistence INFO, Redis role, configured save policy and host or pod identity. Confirm whether the affected endpoint is primary, replica or standalone and whether clients can fail over. Stop automated restarts: they can replace a useful error with a longer outage if Redis cannot load its persistence file. Ask the application owner whether losing writes is worse than rejecting them. That answer determines whether temporary write enablement is even an option.

During the next fifteen minutes, correlate the first failed save with infrastructure events. Look for disk growth, inode exhaustion, a deployment that changed volume mounts, container rescheduling, permission changes, kernel I/O errors and memory pressure around the same timestamp. Compare the configured persistence directory with the process mount namespace and service unit restrictions. If Redis is replicated, check replication offset and lag before promoting anything; a replica can contain a newer in-memory dataset while sharing the same broken persistence assumptions.

RDB, AOF and Backup Boundaries

RDB snapshots trade some recent-write exposure for compact files and fast restarts. AOF records write operations and can reduce the loss window, but rewrite operations still require disk and memory headroom. Enabling AOF in the middle of an incident is not a substitute for repairing storage. Decide persistence mode from recovery point and recovery time objectives, then test it under production-sized data. External backups remain necessary because both RDB and AOF can faithfully preserve accidental deletion or application corruption.

For Kubernetes, inspect the PersistentVolumeClaim, StorageClass, access mode, node attachment and pod security context together. An init container can change ownership successfully and a later security-context change can still remove write access. EmptyDir is unsuitable for durable persistence across pod replacement. For managed Redis, the provider owns the filesystem but not capacity choices, backup policy or application durability expectations; use provider events and metrics instead of host commands.

Change Control and Rollback

Make one repair at a time and record its expected observation. A permission change should make the Redis UID write test pass. A storage expansion should change available bytes and allow a snapshot file to complete. A memory adjustment should permit fork without OOM or cgroup failure. If the expected observation does not occur, revert the change before stacking another hypothesis on top. Keep copies of the prior configuration and persistence file, and never overwrite the only recoverable snapshot during validation.

Close the incident only after a scheduled save and an operator-triggered BGSAVE both succeed, the protection setting is restored, application writes remain healthy, and a copied artifact passes redis-check-rdb or an isolated load test. Add the failure timeline to the runbook, including the leading metric that would have warned earlier. The permanent outcome is not merely “Redis accepts writes”; it is a measured persistence path with known capacity, tested recovery and an alert that reaches an owner.

Questions to Answer Before the Incident Closes

Confirm which data written since the last successful snapshot would be lost in a restart, whether replicas contain a more recent dataset, and whether any operator disabled persistence protection. Record the exact filesystem, volume and Redis configuration repair. Check scheduled save rules under representative write load, because an immediate BGSAVE on a quiet system may not reproduce copy-on-write pressure. Verify backup retention and restore ownership, not just snapshot creation. Finally, make the service owner state whether Redis is a disposable cache, a recoverable derived store or a primary data store; that classification drives the acceptable recovery point, persistence mode and alert urgency.

A post-incident review should also ask why the storage or memory warning did not reach an operator before stop-writes activated. Check whether dashboards use host, container or volume metrics matching the actual persistence boundary. Set a warning far enough below capacity to allow one full snapshot plus temporary files and normal growth. If log rotation, package caches or another tenant consumed the same filesystem, separate those failure domains. Re-run the restore procedure with a production-sized copy and time it. The useful control is a tested recovery objective with named ownership, not a green BGSAVE at the end of the call.