Challenge
Multiple applications needed a primary-loss plan that could be tested without turning a disaster recovery design into a collection of unverified diagrams.
Evidence
- Application dependencies were mapped across App Services, SQL, storage and external DNS.
- Front Door routing behavior was reviewed for regional failover.
- SQL failover group and storage replication choices were validated against application recovery needs.
Fix
The design separated traffic failover, database role changes, storage access and application validation into runbook steps that operators could rehearse.
Outcome
The platform gained a clearer primary-loss plan with validation and failback checkpoints. No private application names or business metrics are included in this public summary.
Context and Architecture
The platform contained several web applications with different release cadences, regional App Service deployments, Azure Front Door, SQL data, storage dependencies and external integrations. A single traffic switch could not make every dependency ready in the secondary region.
Business Risk
A regional loss could produce a misleading partial recovery: Front Door might route successfully while an application used the wrong database role, lacked a storage object, or called a regional integration that had not failed over. The key risk was declaring recovery from edge health alone.
Observable Evidence
- Application-to-database and application-to-storage dependency maps.
- Front Door origin health and routing configuration.
- SQL failover-group listener and replication state.
- Storage redundancy mode and secondary-read constraints.
- Configuration and secret differences between regions.
Hypotheses Considered
The design compared active-passive and active-active application tiers, operator-initiated versus automatic database failover, RA-GRS storage reads, and whether some applications should remain unavailable rather than return inconsistent results.
Root Cause of the Readiness Gap
The original plan treated regional routing as the recovery mechanism. Dependency review showed that traffic, data role, storage access, secrets and application validation were separate state transitions with different rollback consequences.
Permanent Design
The runbook established ordered gates: confirm incident scope, freeze risky deployments, verify secondary capacity and configuration, change the SQL role where approved, validate storage behavior, warm application instances, run synthetic transactions, then adjust Front Door routing.
Validation and Failback
Exercises used application-specific checks rather than a shared 200 response. Operators verified authentication, representative reads and writes, background jobs and integrations. Failback required replication health, a new change window and the same gates in reverse; it was not automatic after the primary region returned.
Operational Lessons
Regional disaster recovery is a dependency choreography, not a routing feature. Recovery objectives must be assigned per application and backed by rehearsed data and configuration transitions.