Show Notes
Why Small Building Failures Become Big Operational Problems
In this episode of Built, Wired, and Secured, Alex Morgan and Michael Harrington examine a problem that many commercial property teams underestimate until it causes real disruption: cross-system cascades. The conversation starts with a realistic scenario that shows how a single faulty motion sensor can create far-reaching consequences across building operations. A sensor begins sending false occupancy signals. The building automation system responds by increasing ventilation. That added load pushes an already stressed distribution panel too far, trips an upstream breaker, and knocks out network equipment in the communications room. From there, tenants lose badge access, elevators move into recall behavior, and front desk staff are forced into manual workarounds.
The point is simple but important. Buildings are not collections of isolated systems. They are webs of dependencies. When one component misbehaves, the effect can spread into power, networking, controls, access control, and tenant operations faster than many teams expect.
The Two Patterns Behind Cascading Outages
Michael identifies two recurring issues that sit behind most cascades: hidden dependencies and unclear ownership. Hidden dependencies are the connections teams do not fully see, document, or account for in day-to-day operations. Unclear ownership shows up when nobody is explicitly responsible for a dependency, a change, or a recovery path.
The discussion breaks hidden dependencies into three practical categories:
- Shared physical infrastructure, such as a single communications room, single UPS, or single distribution panel supporting multiple critical systems
- Logical coupling, where separate platforms rely on the same network, VLAN, or authentication service
- Undocumented operational workarounds that quietly become part of the environment over time
Each of these can be manageable on its own. The real risk appears when they overlap. A building can look redundant on paper but still be fragile in practice if multiple systems depend on the same unseen foundation.
Why “Separate Systems” Are Often Not Truly Separate
One of the more useful insights from the episode is that different vendors or different control panels do not automatically mean independence. Michael gives the example of access control and building automation depending on the same authentication service or network VLAN. If that shared service has an issue, badge reads may fail at the same time HVAC schedules or emergency alert functions are affected.
That kind of logical coupling is dangerous because teams may assign different vendors or departments to each system while overlooking the shared backplane underneath them. In other words, the organizational chart suggests separation, while the infrastructure tells a different story.
The Tradeoff Between Redundancy and Complexity
The episode also tackles a common assumption: that more redundancy automatically solves resilience problems. Michael pushes back on that idea. Redundancy can absolutely reduce single points of failure, but only if it is supported operationally. Adding a second UPS or a parallel network segment means more assets to test, more configurations to maintain, and more recovery paths to understand under pressure.
If those additional paths are not exercised and managed, teams may discover in the middle of an incident that they traded simplicity for theoretical resilience. In that situation, recovery can actually slow down. The lesson is not to avoid redundancy. It is to match redundancy to the organization’s ability to maintain, document, and test it.
How Temporary Workarounds Turn Into Long-Term Risk
Another key topic is the danger of manual workarounds. Michael describes how temporary fixes often create new hidden dependencies. One example is routing access control through a guest VLAN because the main VLAN was unstable. The fix may solve an immediate problem, but it can also bypass monitoring, skip documentation, and shift risk into places nobody reviews later.
When that workaround is forgotten, future troubleshooting becomes slower and more error-prone. Teams no longer know the real topology, and repairs can trigger unintended side effects. This is one of the clearest themes in the episode: informal fixes may look small in the moment, but they often increase the blast radius of the next failure.
An Incident Example: Broadcast Traffic and Building Disruption
To make the risk concrete, Michael shares an anonymized example involving a contractor replacing a lighting control panel. Management traffic from that panel was not routed away from the tenant VLAN. After a firmware update, the panel generated excessive broadcast traffic that saturated the switch stack. Core network services degraded, phones and badge readers lost connectivity, tenants were locked out of floors, and meetings were delayed.
The immediate recovery required isolating the broadcast source, rolling back the firmware, and manually restoring key network paths. But the more valuable outcome came after the incident. The team mapped devices to VLAN ownership, added a change control checklist, and implemented monitoring that would have surfaced the broadcast storm earlier.
That example captures the heart of the episode: the outage was recoverable, but the operational cost and reputational damage were real. The bigger win would have been preventing the cascade altogether.
A Practical Playbook for Property and IT Teams
Rather than staying at the theory level, the conversation ends with practical actions teams can take right away.
- Map dependencies. Document which systems rely on which panels, VLANs, and shared services. The speakers stress that even a simple spreadsheet is better than assumptions.
- Assign ownership and handoffs. Every dependency needs a clear owner. Change control should be formalized so devices are not moved or reconfigured without visibility.
- Run tabletop exercises. Short scenario-based reviews help teams uncover brittle interfaces before an actual incident exposes them.
Michael also highlights several low-effort, high-return improvements: clearly label power feeds, network jacks, and patch panels; add basic monitoring for broadcast storms and power anomalies; and define a critical path list of the systems that must stay online for tenant operations.
Key Takeaways from the Episode
- Map shared dependencies across power, networking, and control systems before a failure exposes them for you.
- Treat ownership as an operational control. If nobody owns a dependency, recovery will take longer.
- Use tabletop exercises and simple monitoring to catch brittle interactions early.
- Be cautious with temporary fixes that bypass normal design, documentation, or monitoring.
- Focus on the systems that are essential to tenant operations and build manual contingencies around them.
This episode stays vendor-neutral and highly practical. Its message is especially relevant for property teams, facilities leaders, and IT stakeholders responsible for keeping commercial buildings functional under stress. A small fault does not have to become a building-wide incident. But avoiding that outcome requires visibility, discipline, and coordination across systems that are more connected than they appear.
The Quiet Dominoes Inside Commercial Buildings
Commercial buildings rarely fail all at once. More often, they fail in sequence. A small issue appears in one system, the impact spreads into another, and what began as a manageable fault becomes a broader operational event. That is the central theme of this episode of Built, Wired, and Secured, where Alex Morgan speaks with Michael Harrington about preventing cross-system cascades in commercial buildings.
The discussion is grounded in a scenario that feels uncomfortably realistic. A motion sensor in a tenant suite starts generating false occupancy signals. The building automation system interprets that as a change in demand and increases ventilation. The additional electrical load stresses a distribution panel that was already close to capacity. An upstream breaker trips, network equipment in the communications room goes down, badge access fails, elevators move into recall behavior, and staff shift into manual operations just to keep the property functioning.
That chain matters because it shows how modern buildings actually operate: not as independent systems, but as interdependent ones. Access control, HVAC, networking, power, lighting, elevators, and tenant-facing services may be procured separately, managed by different teams, or maintained by different vendors. But in practice, they often share power paths, communications infrastructure, and operational assumptions. When one part of that environment is brittle, the damage can spread faster than expected.
The Real Problem Is Not Just Failure. It Is Hidden Dependency.
Michael identifies two recurring drivers behind these incidents: hidden dependencies and unclear ownership. Those two problems do not always create outages on their own, but they make small failures far more likely to escalate.
Hidden dependencies are the shared elements teams either do not know about, do not document, or stop noticing over time. The episode breaks them into three categories.
- Shared physical infrastructure, such as a single communications room, a single UPS, or a single distribution panel supporting several critical functions
- Logical coupling, where systems depend on the same network segment, VLAN, or authentication service even if they appear separate on the surface
- Undocumented workarounds, where temporary fixes become part of the production environment without proper visibility or control
That framework is useful because it shifts the conversation away from isolated device failures and toward systemic fragility. A single component can malfunction in any environment. The more important question is whether the environment around it is designed and operated in a way that contains the issue or allows it to cascade.
Why Different Vendors Do Not Guarantee Independence
One of the strongest points in the episode is that separate systems are often not truly separate. Michael gives a practical example involving access control and building automation tied to the same authentication service or VLAN. If that shared service has a problem, teams may lose badge reads while also affecting HVAC scheduling or emergency alerts.
This is exactly the kind of dependency that surprises stakeholders during an incident. On paper, each system may have its own vendor, support process, and maintenance schedule. But if the underlying network or service layer is shared, those systems can fail together. The result is not just technical disruption. It creates confusion about who owns the issue, who should respond first, and which restoration path matters most.
In commercial buildings, that confusion carries operational consequences quickly. Tenants feel the effects before teams finish diagnosing the root cause. Access delays, uncomfortable spaces, missed meetings, and manual staffing burdens all add up. The technical event becomes a business event almost immediately.
Redundancy Helps, But Only If the Team Can Operate It
The episode also addresses a common instinct in resilience planning: add redundancy. The guests do not dismiss redundancy. In fact, they acknowledge that a second UPS or a parallel network segment can absolutely reduce single points of failure. But they make an important distinction between installed redundancy and operationalized redundancy.
Every additional recovery path introduces additional operational burden. More assets must be tested. More configurations must stay aligned. More assumptions need to be validated. If the team does not maintain those paths actively, the organization may discover during an outage that its backup design exists only in theory.
That is a critical lesson for property teams and building operators. Complexity has a cost. A more resilient design is not just one with more components. It is one the organization can understand, document, exercise, and recover through under pressure. Otherwise, added redundancy can slow response instead of accelerating it.
Temporary Fixes Often Become the Next Incident
Another major takeaway is the danger of workarounds that quietly become permanent. Michael calls out the example of routing access control through a guest VLAN because the primary VLAN was unstable. In the moment, that kind of decision can seem practical. The system comes back up, tenants move on, and the issue feels closed.
But that shortcut may bypass monitoring, avoid formal documentation, and alter the real topology in a way future responders do not understand. Then, when another problem occurs later, the team is troubleshooting a system that no longer matches the diagrams or assumptions people rely on.
This is how mean time to recovery grows. Not necessarily because the root fault is unsolvable, but because the environment around it has drifted away from visibility and control. The episode is clear on this point: undocumented fixes are not neutral. They often become hidden dependency shifts that increase the cost of future incidents.
A Broadcast Storm That Became a Tenant Problem
To illustrate how quickly building issues can widen, Michael shares an anonymized incident involving a contractor replacing a lighting control panel. Its management traffic was not separated from the tenant VLAN. After a firmware update, the panel generated excessive broadcast traffic that saturated the switch stack. Core network services entered a degraded state. Phone systems and badge readers lost connectivity. Tenants were locked out of floors, and meetings were delayed.
The recovery steps were straightforward in hindsight: isolate the broadcast source, roll back the firmware, and manually restore key network paths. But the larger lesson came after the event. The team improved VLAN ownership mapping, implemented a change control checklist, and added monitoring that would have surfaced the broadcast storm sooner.
That progression matters because it shows the difference between technical recovery and operational learning. Restoring service ends the incident. Learning from the incident reduces the chance of the same failure chain repeating.
What Teams Can Do This Week
The best part of the episode is that it stays practical. The recommendations are not product-driven and do not depend on major capital projects. Instead, they focus on actions teams can take now.
First, map dependencies. Document which systems run on which power panels, VLANs, and shared services. The speakers explicitly note that even a simple spreadsheet is valuable if it replaces assumptions with visible relationships.
Second, define ownership and handoffs. Every dependency needs a clear owner, and every material change needs a documented process. If devices can be moved, reconfigured, or repathed without notice, the building is accumulating future recovery risk.
Third, run tabletop exercises. Short scenario-based sessions reveal brittle interactions before tenants experience them in real time. These exercises also help facilities and IT teams align on what happens first, who decides what, and which systems matter most during a disruption.
High-ROI Improvements That Do Not Require a Large Budget
Michael also highlights some of the simplest and most effective improvements available to property teams with limited budgets.
- Label power feeds, network jacks, and patch panels clearly
- Add simple monitoring for abnormal traffic patterns such as broadcast storms
- Monitor for power anomalies that may indicate upstream stress
- Create a critical path list of the handful of systems that must remain online for tenant operations
- Define manual workarounds for those systems before an incident occurs
These are not glamorous steps, but they directly improve recovery speed and decision-making quality. In many buildings, the fastest way to improve resilience is not to add another platform. It is to make the current environment easier to understand and operate.
The Broader Business Impact
One reason this episode stands out is that it never treats cascades as merely technical annoyances. The conversation keeps returning to tenant experience, operational cost, and reputational damage. When badge access fails, HVAC behaves unpredictably, elevators enter restricted states, or communications systems degrade, the issue is visible. Tenants notice. Staff improvise. Confidence erodes.
That is why resilience planning in commercial buildings should be treated as a cross-functional discipline rather than a collection of isolated maintenance tasks. It requires facilities, IT, operations, and vendors to understand where systems intersect and how they fail together.
Contain the First Failure Before It Becomes Five
The episode closes with a simple challenge: audit the critical path systems this week, label infrastructure, bring facilities and IT into the same room for one tabletop exercise, and add one monitoring rule for abnormal network or power behavior. Those are modest actions, but they reflect the larger point of the conversation. Cascading failures are often preventable when organizations understand their dependencies, assign ownership clearly, and test their response before a crisis forces them to.
If your building environment depends on connected systems, shared services, and multiple stakeholders, this episode is worth your time. It offers a grounded reminder that resilience does not begin during the outage. It begins in the visibility, discipline, and coordination you build beforehand. Listen to the full episode for the complete discussion and use it as a prompt to evaluate where the quiet dominoes may be hiding in your own property operations.