Why failover becomes theater

This episode of Built, Wired & Secured takes on a problem that many commercial buildings quietly carry until an outage exposes it: failover that looks solid on paper but falls apart in real conditions. The opening scenario is familiar and uncomfortable at the same time. The utility blips, the lights return, the elevator stops, and suddenly part of the building is inaccessible. What failed was not just one component, but the assumptions behind how power, networking, automation, and people were expected to respond together.

Alex Morgan is joined by Michael Harrington and James Rogers for a grounded discussion on realistic failover design. Instead of treating redundancy as a checklist item, they focus on what actually happens when systems restart out of sequence, when dependencies are hidden, and when manual procedures exist but have never been practiced under stress.

What usually breaks first

One of the first issues discussed is sequencing. A short delay in generator start or transfer switch timing can be enough to trigger a much larger disruption. The group explains that even a 10- to 15-second miss can reboot PLCs, flap network devices, and create cascading retries that consume CPU and hide the original fault.

They also point to common choke points that make outages spread faster than expected:

• Undersized UPS systems that only sustain load for 10 to 15 minutes
• A single core switch carrying traffic for access control, elevators, and BAS
• Shared authentication or DNS services hosted in the primary data closet
• Systems left in maintenance mode without clear communication

The episode makes a simple but important point: many failures are not exotic. They come from ordinary dependencies that were never mapped clearly enough to be protected.

Where automation helps and where it hurts

A major theme in the conversation is the tradeoff between automatic failover and manual intervention. James argues for automating predictable, repeatable, and easily monitored sequences. Generator start and life-safety load transfer are good candidates because they can and should be validated regularly.

At the same time, the group avoids treating manual steps as automatically inferior or automation as automatically better. Michael pushes on the idea that humans should always handle edge cases, noting that manual intervention can be slow and inconsistent when tenants are already feeling the impact.

The practical conclusion is more nuanced. Automation works best when it is:

• Deterministic
• Visible
• Testable
• Capable of exposing intermediate states

Opaque automation that silently retries in the background can make outages longer and more confusing. If automation hides failure, the team argues, it can be worse than having no automation at all.

The real cost of redundancy

The episode also frames redundancy as an operational commitment, not just a capital expense. Redundant equipment adds cost in hardware, power, maintenance, testing, and lifecycle planning. That investment only delivers value if both sides of the redundant design are kept current and validated.

Michael emphasizes using a risk-based lens. The right question is not whether redundancy sounds good in theory, but whether the cost of downtime justifies testable automation and disciplined upkeep. If a site cannot maintain both redundant systems on compatible schedules, the added complexity may become its own liability.

That idea comes through clearly in one of the episode’s strongest lines: redundancy without maintenance is just expensive shelfware.

One success story and one failure

The conversation becomes especially useful when it shifts from principles to examples.

In the success case, a campus had separate power feeds to access panels and a local UPS sized for about 20 minutes at full load. Generator transfer for those feeds was automated, while manual override procedures for high-impact doors were rehearsed quarterly and lower-priority systems were reviewed annually. When utility power dropped, access stayed live and the manual step became a verification step rather than a rescue move. The result: zero tenant complaints.

In the failure case, a mid-rise property had access control and elevator PLCs sharing a telecom switch. That switch depended on a UPS rated for only 10 to 15 minutes, and the battery bank was already degraded. During the outage, the UPS failed, both systems went down, and the front desk was overwhelmed. Manual keys and procedures existed, but staff had not practiced under pressure. The procedures were technically available, but operationally unreliable.

The contrast reinforces a core lesson: failover design is only as strong as the maintenance, testing, and training behind it.

How to validate assumptions this week

For teams that do not have a simulation lab, the episode offers practical next steps. Michael recommends choosing one subsystem, defining a success criterion, scheduling a low-impact window, and running a focused test. Examples include transferring generator power for one floor and confirming BAS and access control stay functional for five minutes, or testing network uplink failover and measuring downtime in seconds.

Just as important, the team recommends a 30-minute tabletop exercise to establish roles and escalation paths. Who calls whom at 2 a.m.? What happens if the first contact does not answer? Those answers should be written into a short runbook with simple decision logic and contact names.

Five rules of thumb from the episode

• Map dependencies across power, network, and authentication services
• Automate only what you can reliably test in a controlled way
• Assign ownership and backups for after-hours decisions
• Maintain redundancy through battery checks, firmware updates, and planned replacements
• Verify real-world behavior so tenant communications match actual impact

The speakers close with a reminder that small issues often reveal large risks. An undocumented VLAN, a single power strip, or an ignored UPS alarm may be the difference between a brief hiccup and a building-wide disruption. The goal is not perfect resilience everywhere. It is disciplined, rational prioritization based on what really breaks when something goes down.

If there is one practical takeaway from this episode, it is this: start small, test one assumption, document what happens, and build confidence from there. That loop of mapping, testing, and documenting moves teams further than a year of disconnected upgrades.

Failover Theater: Designing Realistic Failover for Building Systems

In commercial buildings, failover is often treated as a comfort phrase. It sounds reassuring in project meetings, appears clean in diagrams, and gives teams confidence that redundancy exists if something goes wrong. But when a real interruption happens, many buildings discover that their failover design is less a disciplined system and more a collection of assumptions.

That is the core issue explored in this episode of Built, Wired & Secured. Alex Morgan sits down with Michael Harrington and James Rogers to talk through what realistic failover actually looks like in modern buildings. The discussion stays practical throughout, focusing on the systems that matter most in day-to-day operations: power, network infrastructure, building automation, access control, and the human procedures that connect them.

The opening scenario sets the tone. A utility blip hits. The lights come back, but the elevator stops and a section of the building is suddenly inaccessible. The problem is not just the outage itself. It is the chain reaction that follows when systems restart in the wrong order, dependencies are hidden, and manual workarounds have never been rehearsed together.

Failover problems usually begin with sequencing

One of the first points raised in the episode is that failover failures often start with timing. A generator may start, but not quickly enough. A transfer switch may do its job, but not in the sequence the rest of the building expects. Those small timing gaps matter because downstream systems are not always tolerant of short interruptions.

The speakers explain that even a 10- to 15-second miss can be enough to trigger a larger event. PLCs reboot. Network devices flap. Retry storms begin. CPU gets consumed by recovery activity, and the original root cause becomes harder to identify. In other words, a short interruption can create a long troubleshooting cycle because the systems layered on top of one another respond differently under stress.

This is a useful reminder for facilities and IT leaders alike: uptime is not only about whether power returns. It is about whether interconnected systems resume in a stable and predictable sequence.

Shared dependencies are where isolated outages become building-wide problems

The conversation then turns to hidden choke points. These are the places where teams believe they have separation, but real-world dependency mapping tells a different story.

Several examples stand out in the transcript. An undersized UPS may provide only 10 to 15 minutes of runtime under actual load. A single core switch may be carrying VLANs for elevators, access control, and BAS at the same time. Authentication or DNS may be hosted in the primary data closet, making multiple downstream systems dependent on a single room and a single service path. Even a contractor leaving a system in maintenance mode can become part of an outage chain if the rest of the team is unaware.

These examples are valuable because they are not edge cases. They are common operational realities. They show how a building can appear redundant at the component level while still being fragile at the system level.

For business and property leaders, that matters because the tenant experience does not care which technical layer caused the interruption. Occupants only experience the result. A locked door, a stopped elevator, or a front desk flooded with confusion feels like a building failure, not a networking failure or a battery issue.

Automation should be visible, not mysterious

A strong part of the episode is the discussion around when to automate failover and when to keep people in the loop. James argues that predictable, repeatable, and easily monitored sequences should be automated. Generator start and life-safety load transfer are the obvious examples. These are not processes teams should be improvising during a live event.

At the same time, the conversation does not drift into blind faith in automation. Michael pushes back on the idea that manual handling is always best for unusual scenarios, noting that people can be slow and inconsistent under pressure. The group lands on a more useful standard: automation is best when it is deterministic, transparent, testable, and able to expose intermediate states.

That distinction matters. Good automation makes the building easier to operate because teams can see what it is doing and validate its behavior in controlled tests. Bad automation hides its failure modes behind silent retries and unclear system states. In those cases, automation does not reduce risk. It delays visibility.

For building operators, that means the right question is not “Should we automate this?” The better question is “Can we test this repeatedly, understand its states, and escalate visibly when it fails?”

Redundancy is not a purchase. It is a maintenance program.

The transcript makes a point that deserves more attention across commercial real estate and facilities operations: redundancy has an ongoing cost structure. It is not just the upfront spend on duplicate hardware. It includes power draw, testing windows, firmware coordination, battery management, spare parts, and lifecycle planning.

Michael frames this as a risk-based decision. Teams should automate where the cost of downtime is high and where the automation can be validated. But that investment only works if the organization can keep both sides of the redundant design current. Two redundant components with mismatched maintenance schedules can create operational drag instead of resilience.

This is where realistic failover design starts to look less like engineering theory and more like operational discipline. A redundant path that is never tested is not a reliable path. A backup battery with degraded capacity is not true runtime. A manual key procedure that no one has practiced is not a meaningful fallback.

The success case and failure case make the lesson clear

The episode does an excellent job of contrasting what worked with what failed.

In the success story, a campus separated power feeds to access panels and used a local UPS sized for about 20 minutes at full load. Automated generator transfer handled the main event, while manual override procedures for high-impact doors were rehearsed quarterly and lower-priority systems annually. When the utility dropped, access stayed online and the manual process was used only to verify. The outcome was simple and measurable: zero tenant complaints.

In the failure story, access control and elevator PLCs shared a telecom switch supported by a UPS rated for only 10 to 15 minutes. The battery bank was already degraded. During the outage, the UPS failed, both systems dropped, and the front desk became the operational bottleneck. Manual keys and procedures existed, but staff had not practiced under stress. On paper, the building had fallback options. In practice, those options were not dependable.

These stories highlight a business truth that extends beyond buildings. A backup strategy is only real if it works under pressure, with the actual people, timing, and dependencies involved.

A practical way to test failover without a simulation lab

One of the most useful takeaways from the conversation is how approachable the next step can be. Teams do not need a full simulation environment to start improving failover confidence. Michael recommends choosing one subsystem, defining a success criterion, scheduling a low-impact test window, and running a focused exercise.

That might mean transferring generator power for one floor and verifying that BAS and access control remain functional for five minutes. It might mean testing network uplink failover and measuring the interruption in seconds. The point is to move from assumption to evidence.

The team also recommends a 30-minute tabletop exercise to clarify roles and escalation. Who makes the call at 2 a.m.? Who is the backup? What is the first communication step? Those answers should live in a short runbook with simple “if X, then Y” logic, current contacts, and defined escalation times.

That kind of documentation may not look sophisticated, but it is often what turns a stressful event into a manageable one.

Five rules worth applying now

The episode closes with five practical rules of thumb that translate well into action:

• Map dependencies so you know which systems share power, network, or authentication services
• Automate only the sequences you can reliably prove in controlled testing
• Name owners and backups for after-hours decisions and escalation
• Maintain redundancy with regular battery checks, firmware discipline, and planned replacement cycles
• Verify real-world behavior so tenant impact and communications are accurate

Those rules are effective because they are testable. They do not require a major capital project to begin. They require visibility, discipline, and follow-through.

Why this matters for building operations

For facilities teams, property managers, and technology leaders, realistic failover is not just a technical concern. It affects tenant confidence, incident response, safety posture, and how rationally budgets get prioritized. If you can explain in plain language what breaks when a given service goes down, you are in a far better position to invest wisely.

That is what this episode gets right. It does not chase perfection. It advocates for practical resilience: visible automation, practiced manual fallback, clear ownership, and evidence-driven testing.

If your team has not validated even one failover assumption recently, this episode offers a straightforward place to start. Pick one subsystem. Define what success looks like. Run a 30-minute tabletop. Document what you learn. Then expand from there.

For listeners responsible for keeping buildings operational under pressure, that approach is far more valuable than another year of disconnected upgrades. And if you want to hear the full conversation, this episode is a strong listen for anyone trying to make building systems more predictable when conditions are not.