Show Notes
Redundancy on Paper Is Not Resilience
Many technology environments look protected until a real outage exposes the dependencies no one tested. This episode examines “redundancy theater”: systems that appear to have backups, mirrored hardware, or diverse service paths but cannot restore service when conditions matter most.
The discussion opens with a multi-tenant office building hit by a midday storm. The building’s specifications called for two independent internet feeds and mirrored network hardware. Yet point-of-sale terminals went dark, access-control panels stopped responding, and elevators shifted into a slower backup mode. The apparent redundancy did not restore service because both internet feeds shared an underground conduit, and the switchover scripts had never been run under load.
The central lesson is simple: redundancy on paper is not resilience in practice. Decision makers need to verify how systems behave during a real failure, not merely confirm that a specification contains the word “redundant.”
The Anatomy of False Redundancy
False redundancy often comes from hidden shared dependencies. A second component does not create meaningful backup if both components depend on the same physical, electrical, operational, or human resource.
- Shared physical paths: Two fiber routes may both run through the same manhole, conduit, room, or building entrance.
- Shared power dependencies: Duplicate switches may be connected to the same power strip, or a second UPS may sit in the same room as the first.
- Untested switchover logic: Backup systems may rely on manual steps, factory-default scripts, or procedures that have never been executed end to end.
- Human process single points: One technician may understand vendor customizations, or only one access card may open a critical panel.
These issues are easy to miss when teams evaluate redundancy as a procurement checkbox. Engineering may install mirrored equipment, while operations inherits responsibility for proving the environment works under real conditions.
A Carrier-Diversity Lesson
An anonymized campus example shows why coordination and verification are as important as carrier contracts. The telecom team had contracted two carriers, with what appeared to be separate paths on paper. However, both carriers terminated in the same IDF, and both used a new conduit that had not been inspected after drawings changed.
When a construction crew cut that one conduit, both feeds failed simultaneously. The problem was not a failure by either carrier. It was a failure to confirm the physical route, termination location, and actual diversity of the installation. For tenants, the promised redundancy did not help. For the property manager, it became an operational and verification failure.
Prioritizing Redundancy Investments
Not every system requires the same level of redundancy. Owners must balance capital expense and operating expense against the business impact of downtime. The recommended starting point is to map impact: determine which services create immediate tenant harm when they fail.
Elevators, HVAC supporting critical floors, point-of-sale systems, and access control can create immediate operational consequences. These services may justify full physical and logical separation. Lower-impact services may be appropriately protected through partial redundancy, faster recovery processes, local spare equipment, or rapid vendor service commitments.
The key is to avoid vague requirements. Instead of simply asking whether a system is redundant, define measurable outcomes:
- Mean time to recover.
- Allowable downtime per year.
- Expected performance while operating in failover.
- Manual actions required to restore service.
- Ownership for each recovery step.
Make Acceptance Criteria Testable
Meaningful acceptance criteria focus on outcomes that can be measured. For example, a requirement could state that, after loss of a primary feed, the secondary feed must sustain full tenant traffic within a defined number of seconds and maintain packet loss below an agreed percentage for a specified duration.
For power resilience, an acceptance test could require the on-site generator to support life-safety loads within 30 seconds and critical HVAC zones within 60 seconds after utility power loss. Each requirement should include a defined test procedure and explicit pass/fail metrics.
If a team can script a test and measure it with simple tools, ambiguity is reduced. Vendors have less room for handwaving, and owners gain evidence that the system performs as intended.
Keep Failover Real Through Operations
Resilience degrades when failover is treated as a one-time commissioning event. Teams should make testing part of routine operations.
- Run end-to-end failover drills at least annually and after any significant change.
- Assign ownership for each failure mode, including notification, rollback, and tenant communication responsibilities.
- Instrument every drill with timestamps, logs, and measured service impact.
- Document lessons learned and convert them into configuration baselines.
A practical drill can be straightforward: schedule an off-peak window, simulate loss of the primary path, observe failover, record restoration time, and capture every manual intervention. If restoring service requires a vendor to run a 45-minute console script, that is a process risk that belongs in the SLA and staffing plan.
Three Questions to Ask Now
- Do both paths traverse separate physical routes?
- Can the backup restore service under production load within an agreed time?
- Who is responsible for failover, and do they have the required access and training?
Finally, make verification part of procurement. Require vendors to demonstrate failover in situ before final acceptance. If the demonstration cannot produce measurable results, final payment should remain tied to successful verification. That approach aligns incentives and helps ensure that redundancy is more than a paper promise.
Redundancy Theater: Why Backup Systems Fail When They Are Needed Most
Redundancy is often treated as a simple purchasing decision. Add a second internet connection, mirror network hardware, install another UPS, and the environment is resilient. In reality, those measures may only create the appearance of protection.
That gap between an impressive specification and a working recovery capability is redundancy theater. It happens when systems look redundant on paper but cannot restore service during a real outage. For commercial properties and business environments, the result can be immediate: interrupted tenant operations, unavailable access control, slowed elevator traffic, impaired HVAC for critical areas, and lost point-of-sale capability.
The difference between redundancy and resilience is verification. A resilient environment has independent paths, testable recovery criteria, clear ownership, and evidence that failover works under real conditions.
How Redundancy Theater Develops
Redundancy theater rarely begins with bad intentions. Facilities, IT, procurement, vendors, and tenants often have different priorities. Procurement may seek a checkbox requirement. Engineering may focus on mirrored hardware. Operations may inherit the environment after the project is complete and be expected to prove it works during a failure.
The problem emerges when no one asks the most important question: what breaks if this dependency fails?
That question reveals whether seemingly separate systems share a point of failure. Two fiber feeds may appear diverse, but both can run through the same underground conduit. Two switches may be installed, but both may plug into the same power strip. A second UPS may exist, but it may be located in the same room as the first. Hardware duplication does not provide resilience if one physical incident can disable both sides.
The same principle applies beyond facilities and network paths. A system can have redundant hardware while remaining operationally fragile if only one technician understands the vendor customization, only one person has authorization to access a panel, or recovery requires a script that has never been tested beyond its factory defaults.
Physical Diversity Must Be Verified, Not Assumed
A campus example illustrates the danger of relying on paperwork. A telecom team had contracted with two carriers for path diversity. On paper, the arrangement looked complete: path A and path B. But site drawings had not been updated, and the new conduit used by both providers had not been inspected.
Both carriers ultimately terminated at the same IDF and shared the same conduit. When a construction crew cut that single conduit, both services went down at the same time. Tenants understandably questioned why promised redundancy had not protected them.
The lesson is not that two carriers are ineffective. It is that carrier contracts alone do not prove physical diversity. The failure was one of coordination and verification. Teams must inspect actual routes, entry points, conduit paths, termination locations, power dependencies, and intermediate infrastructure. A logical diagram is not enough when physical infrastructure can defeat the design.
Start With Business Impact
Full redundancy is not always necessary for every system. Owners must make choices based on capital costs, operating costs, risk tolerance, and the consequences of service loss. The right starting point is impact mapping.
Ask which failures create immediate tenant harm. Elevators, access control, HVAC for critical floors, and point-of-sale systems may deserve full physical and logical separation because their outage has direct operational impact. For lower-impact systems, a different approach may be appropriate: partial redundancy, local spare equipment, faster recovery procedures, or a rapid vendor response commitment.
This approach avoids overspending on unnecessary duplication while making sure that systems with serious operational consequences receive meaningful protection. It also produces a clearer conversation about risk. Instead of declaring every service “redundant,” stakeholders can define how long each service may be unavailable and what performance must be maintained while failover is active.
Replace Vague Requirements With Measurable Outcomes
The word “redundant” does not establish a service level. Contracts, specifications, and acceptance tests should define outcomes that can be observed and measured.
For a network connection, a useful requirement might state that, after loss of the primary feed, the secondary feed must sustain full tenant traffic within a specified number of seconds. It can also require packet loss to remain below a defined percentage for a stated period. That requirement describes what success looks like under failure conditions rather than merely confirming that a second circuit exists.
Power requirements can be equally specific. An acceptance test could require an on-site generator to support life-safety loads within 30 seconds after utility power loss and critical HVAC zones within 60 seconds. The exact targets will vary by environment, but the structure matters: define the event, the expected response, the time limit, the performance standard, and the pass/fail measure.
Measurable criteria make vendor accountability practical. If a test can be scripted and evaluated with simple tools, the owner has clear evidence of whether the system meets the agreed outcome. That reduces ambiguity during commissioning and strengthens future conversations about budget, renewal, remediation, and risk acceptance.
Failover Must Be Part of Operations
A backup that has never been exercised is an assumption, not a proven capability. Failover drills should be built into commissioning and maintenance calendars, conducted end to end at least annually, and repeated after significant changes.
Each drill should answer operational questions that a design drawing cannot answer:
- How long did restoration take?
- Did service remain usable during failover?
- What manual steps were required?
- Who owned the call, rollback, and tenant communication?
- Did the responsible people have the access and training needed to act?
Testing should produce artifacts: timestamps, logs, measured service impact, documented interventions, and lessons learned. Those records become valuable configuration baselines. They also provide evidence when an owner needs to defend a budget request, negotiate a vendor commitment, or decide whether an accepted risk is still acceptable.
A basic drill can be highly revealing. Schedule an off-peak window, simulate loss of the primary path, observe the switchover, record the restoration time, and document every intervention. A manual process may be acceptable, but it must be represented honestly in the service-level expectation. If a vendor must run a 45-minute console script to restore service, that recovery time and dependency should be included in staffing plans and SLAs.
Make Verification a Procurement Requirement
Procurement should not end with delivery of equipment or activation of a service. Vendors should demonstrate failover in situ as a condition of final acceptance. The test should use agreed scenarios, measurable results, and clear pass/fail criteria.
That requirement aligns incentives. It moves the conversation from paper claims to operational outcomes and gives every stakeholder a shared definition of success. It also makes hidden dependencies visible before an outage affects tenants.
A Practical Redundancy Checklist
Before approving a design, accepting a project, or reviewing an existing system, ask three direct questions:
- Do both paths traverse genuinely separate physical routes?
- Can the backup restore service under production load within an agreed time?
- Who is responsible for executing failover, and do they have the access and training to do it?
Resilience is not created by mirrored lights, duplicate equipment, or contract language alone. It is created through independent design, outcome-focused requirements, routine testing, and accountable operations. For a deeper discussion of practical failover criteria and verification, listen to this episode of Built, Wired & Secured.