Show Notes
Why Redundancy Fails in the Real World
In this episode of Built, Wired, and Secured, Alex Morgan and Michael Harrington take on a problem that shows up in buildings, campuses, and multi-tenant environments far too often: systems that look redundant in a spec sheet but fail when they are actually needed. The conversation opens with a storm-related outage in a multi-tenant office building where internet service and some building controls go down even though the property supposedly had two independent internet feeds and mirrored network hardware. Tenants lose point-of-sale capability, access control panels stop responding, and elevators fall back to a slower operating mode. The problem was not a lack of equipment labeled as backup. The problem was that the two feeds shared the same underground conduit and the switchover scripts had never been tested under load.
That setup becomes the central lesson of the episode: redundancy on paper is not the same as resilience in practice. Decision makers can buy duplicate hardware, add secondary feeds, and check every procurement box, but if those systems share hidden dependencies or fail when production traffic is on the line, the organization still carries real outage risk.
What "Redundancy Theater" Actually Means
Michael breaks down the anatomy of false redundancy by focusing on the places where hidden single points of failure remain. Teams often assume a second component automatically means protection, but that assumption can be dangerously misleading. The discussion highlights how easy it is to create the appearance of resilience without building the conditions required for real failover.
- A second UPS in the same room may still be exposed to the same environmental or access issue.
- Duplicate switches connected to the same power strip are not operationally independent.
- Two fiber routes that run through the same manhole or conduit can fail from a single cut.
- Switchover logic may exist, but if it has never been executed end to end, there is no proof it will work.
- Hardware redundancy can still collapse if only one person knows the vendor customizations or only one access credential can reach the critical gear.
The recurring question in the episode is simple and useful: what breaks if this goes down? As soon as teams ask that question honestly, shared dependencies become easier to spot.
An Example of Redundancy That Wasn't Real
Michael shares an anonymized campus example where a telecom team had contracted with two carriers to create path diversity. On paper, the design appeared strong. Path A and path B were both present. In practice, both carriers terminated at the same IDF because site drawings had not been updated, and both used the same new conduit because no one had physically verified the route. When a construction crew cut that conduit, both feeds failed at the same time.
The important takeaway is that the carriers were not the core problem. The failure came from coordination and verification. The organization believed it had diversity because the contract language suggested it. But no one confirmed whether the physical design actually supported the resilience goal.
How Owners Should Think About Redundancy Investments
The episode does not argue that every system deserves full duplication at any cost. Instead, it frames redundancy as a prioritization exercise tied to operational impact. Owners have to weigh capital budgets, operating budgets, maintainability, and risk. The right answer depends on what hurts tenants or operations most when a service fails.
Michael recommends starting with impact mapping. Identify the systems whose failure causes immediate tenant harm or building disruption. The episode specifically calls out elevators, HVAC for critical floors, point of sale, and access control as services that people feel immediately. For those systems, full physical and logical separation is often worth the investment.
For lower-impact systems, the conversation suggests other options may be sufficient:
- Partial redundancy instead of full duplication
- Faster recovery procedures
- Locally stocked spares
- Vendor SLAs that support acceptable recovery times
The key is to avoid vague language. Rather than asking for something to be "redundant," owners should define measurable operational outcomes.
What Good Acceptance Criteria Looks Like
One of the most practical parts of the episode is the discussion of acceptance criteria. Alex asks what realistic contract language or construction acceptance testing should look like, and Michael answers with outcome-focused examples that can be tested and measured.
- On loss of primary feed, the secondary feed must sustain full tenant traffic within a defined number of seconds.
- During failover, packet loss must remain below an agreed threshold for a specified period.
- Upon utility power loss, the on-site generator must support life safety loads within 30 seconds.
- Critical HVAC zones must recover within 60 seconds.
This matters because measurable criteria reduce ambiguity. If the test can be scripted, observed, and recorded with simple tools, there is less room for vendor handwaving and fewer surprises at turnover.
How to Keep Redundancy Real After Commissioning
The episode also makes it clear that resilience is not something teams buy once and forget. Systems change. Staff changes. Vendors change. Configurations drift. If failover is not part of routine operations, redundancy degrades into theater over time.
Michael outlines three practical steps for operationalizing resilience:
- Run realistic end-to-end failover drills at least annually and after major changes.
- Define ownership for each failure mode, including rollback, communications, and tenant notifications.
- Instrument tests with timestamps, logs, and measured service impact so results can be repeated and compared.
He also gives a straightforward drill example: schedule an off-peak window, simulate loss of the primary path, observe failover, record restoration time, and note any manual interventions. That final point matters. If service can only be restored after a vendor spends 45 minutes running a console script, that process risk belongs in staffing assumptions and SLA discussions.
Three Immediate Checks You Can Apply Now
The episode promises practical next steps, and it delivers them. Michael and Alex summarize a short checklist listeners can use on existing systems or new projects right away.
- Map dependencies end to end, including physical paths, power sources, network design, and human roles.
- Define measurable acceptance criteria and test them during commissioning.
- Establish routine, documented failover drills with clear ownership.
They also offer three questions that quickly expose redundancy theater:
- Do both paths actually traverse separate physical routes?
- Can the backup restore service under production load within an agreed time?
- Who is responsible for executing failover, and do they have the access and training to do it?
Why Procurement Needs to Be Part of Verification
Michael closes with an operational recommendation that stands out: make failover verification part of procurement. Require vendors to demonstrate failover in situ before final acceptance. If there is no measurable demonstration, reserve final payment until there is. That single step aligns incentives and pushes resilience from a marketing claim into a verifiable deliverable.
For building teams, IT leaders, property managers, and anyone responsible for tenant-facing systems, this episode is a strong reminder that resilience has to be demonstrated, not assumed. Duplicate equipment means very little if shared conduits, untested scripts, or single-person knowledge dependencies still remain. Real redundancy is built through physical separation, tested procedures, clear ownership, and measurable outcomes.
Redundancy Theater: Why Backup Systems Fail When It Matters Most
Many organizations believe they are protected from outages because their environment includes duplicate hardware, secondary circuits, or vendor language that promises resilience. But when a real failure hits, those same systems often reveal a hard truth: backup is not the same as failover, and redundancy is not the same as resilience.
In this episode of Built, Wired, and Secured, Alex Morgan interviews Michael Harrington about what they call redundancy theater. The phrase describes systems that appear resilient in design documents and procurement language but break down under actual stress. The discussion is especially relevant for commercial real estate, multi-tenant buildings, and any environment where building operations and IT systems overlap. Elevators, access control, internet connectivity, HVAC, and tenant-facing services all depend on infrastructure that has to perform when conditions are least forgiving.
The episode starts with a vivid scenario. A multi-tenant office building loses internet service and some critical building controls during a storm. The design spec claimed two independent internet feeds and mirrored network hardware. Yet point-of-sale systems go dark, access control panels stop responding, and elevators fall back to a slower backup mode. The reason is simple and costly: both internet feeds shared the same underground conduit, and the switchover scripts had never been executed under load. Everything looked redundant until it had to work.
That opening example frames the central point of the conversation. Redundancy on paper can create false confidence. Real resilience depends on whether systems can maintain service under real failure conditions, not whether they satisfy a checklist at purchase time.
What False Redundancy Looks Like
Michael breaks false redundancy into a few recurring patterns. The first is shared dependencies. Teams install secondary equipment or alternate service paths, but the backup still relies on the same room, the same conduit, the same power source, or the same human access point. On paper, the system has duplication. Operationally, it still has a single point of failure.
Examples from the episode include a second UPS located in the same room, duplicate switches plugged into the same power strip, and dual fiber paths that both pass through the same manhole. These details matter because a single event can still take both paths down at once. If nobody asks where the physical routes converge, the organization may not discover the weakness until an outage is already happening.
The second pattern is untested switchover logic. A system may include scripts, automation, or mirrored components that imply continuity, but those mechanisms often sit unused for months or years. No one runs a full end-to-end failover drill. No one validates behavior under real traffic. No one confirms what happens when timing, latency, or production load enters the picture. In that environment, backup capability becomes an assumption rather than a verified outcome.
The third pattern is human-process fragility. Even when hardware is duplicated, operations may still depend on a single technician, a single credential, or undocumented vendor customizations. If one person is unavailable or one access issue appears during an incident, the organization learns that its redundancy was technical in theory but not operational in practice.
The Coordination Problem Behind Many Failures
One of the strongest examples in the episode involves a campus that had contracted with two separate carriers for diversity. Contractually, the redundancy case looked solid. There was a path A and a path B. But because site drawings had not been updated and the new conduit both carriers used had not been inspected, both services terminated through the same route. When a construction crew cut that conduit, both feeds failed together.
This is an important distinction. The failure was not primarily about the carriers. It was a coordination and verification failure. Facilities, telecom, engineering, procurement, and operations each owned part of the picture, but no one verified the end-to-end reality. That is one of the most useful lessons from the conversation: resilience breaks down when responsibilities are fragmented and assumptions are left unchallenged.
How to Prioritize Redundancy Without Overspending
The episode does not argue that every system must receive full duplication regardless of cost. Instead, Michael recommends prioritizing based on impact. Start by identifying which service failures cause immediate tenant harm or operational disruption. In the discussion, elevators, HVAC for critical floors, point-of-sale systems, and access control are highlighted as services where downtime is felt immediately.
For those systems, full physical and logical separation is often worth the cost because the business impact of failure is immediate and visible. For lower-impact systems, owners may reasonably choose partial redundancy, faster recovery procedures, local spares, or stronger vendor response commitments instead of full duplication.
That is where the episode becomes especially practical. It shifts the redundancy conversation away from labels and toward measurable business outcomes. Instead of asking whether a system is "redundant," owners should ask what level of disruption is acceptable, how quickly service must recover, and what degraded performance is tolerable during failover.
Why Better Acceptance Criteria Matter
Vague language is one of the biggest reasons redundancy theater survives procurement and commissioning. When a requirement simply says "redundant," vendors can satisfy the letter of the request without proving meaningful resilience. Michael recommends outcome-focused acceptance criteria that can be tested and measured.
In the episode, he gives examples such as requiring the secondary feed to sustain full tenant traffic within a defined number of seconds after loss of the primary feed, with packet loss remaining below a specified percentage for a set duration. He also points to power scenarios, such as requiring a generator to support life safety loads within 30 seconds and critical HVAC zones within 60 seconds after utility loss.
These examples are useful because they define success in operational terms. They also create a basis for repeatable testing. If a test can be scripted and measured with timestamps, logs, and simple monitoring tools, the team can evaluate performance objectively instead of debating interpretations after a failure.
Commissioning Is Not the Finish Line
Another major lesson in the episode is that resilience must be maintained, not just installed. Even a well-designed system can drift into fragility if failover is never retested after a change, if documentation goes stale, or if staffing assumptions no longer hold. That is why Michael recommends making failover drills part of the commissioning and maintenance calendar.
His practical guidance is straightforward: run end-to-end failover drills at least annually and after significant changes. Define who owns each failure mode. Clarify who initiates rollback, who contacts vendors, and who notifies tenants. Instrument the exercise with logs, timestamps, and measured service impact. Then turn the results into updated baselines.
That documentation has value beyond technical operations. It supports contract renewals, budget requests, and risk conversations with leadership because it replaces assumptions with evidence.
A Short Checklist for Real-World Resilience
The episode closes with a concise framework that building teams and IT leaders can use immediately.
- Map dependencies end to end across physical routes, power, network design, and human roles.
- Define measurable acceptance criteria and test them during commissioning.
- Run routine, documented failover drills with clear ownership.
Alex and Michael also offer three questions that quickly reveal redundancy theater:
- Do both paths actually traverse separate physical routes?
- Can the backup restore service under production load within an agreed time?
- Who is responsible for the failover, and do they have the training and access required to execute it?
Michael adds one final recommendation that deserves attention: make verification part of procurement. Require vendors to demonstrate failover in situ before final acceptance, and if they cannot provide a measurable demonstration, reserve final payment until they can. That step changes incentives and makes resilience a delivered outcome rather than a sales claim.
The Bigger Takeaway
This episode is ultimately about accountability. Real redundancy is not created by mirrored gear alone. It comes from separated physical paths, tested switchover behavior, documented procedures, trained personnel, and measurable acceptance standards. For owners, operators, and technology leaders, the message is simple: if failover has not been demonstrated under realistic conditions, it should not be treated as proven.
If you are responsible for technology environments that support tenants, staff, and daily operations, this conversation is a strong reminder to move beyond paper promises. Listen to the full episode for the complete discussion and use it as a practical starting point for reviewing how your organization defines, verifies, and maintains resilience.