Show Notes
Why Small Failures Turn Into Big Building Problems
Modern buildings rarely fail in dramatic ways at first. More often, the first issue is small, ordinary, and easy to dismiss: a sensor freezes, a handoff is unclear, a backup path is assumed to work but has not been tested. In this episode of Built, Wired & Secured, Alex Morgan talks with Michael Harrington about the quiet cascades that turn those small issues into tenant-facing outages.
The episode opens with a vivid example. In a downtown office tower on a hot August afternoon, a single temperature sensor freezes at 72 degrees. The building automation system believes a zone is stable. A VAV damper drives open. The economizer cycles oddly. Thirty minutes later, the chilled water plant trips because systems have been working against each other. What should have been a simple part replacement becomes emergency overtime, tenant complaints, and a growing service backlog.
That scenario captures the core point of the conversation: many outages are not caused by one catastrophic event. They are caused by a chain of small failures, hidden dependencies, and unclear accountability.
The Three Common Starting Points for Quiet Cascades
Michael breaks the problem down into three categories that start most cascades:
- Bad data from sensors and field devices
- Power handling issues, including imperfect UPS transfers
- Human and contractual handoff failures, such as unlabeled panels or unclear scopes of responsibility
Any single issue can often be contained. The real risk appears when two or three of them line up at the same time. That is when a minor fault becomes a chain reaction.
One of the most useful takeaways here is that tenants feel the outage first, while owners feel the financial hit later. The customer impact shows up immediately in comfort complaints, access issues, or lost productivity. The business impact follows in overtime, emergency service calls, operational disruption, and damaged trust.
Why These Cascades Happen So Easily
The discussion makes it clear that quiet cascades are not just an operations problem. They are often built in through a mix of design, procurement, and day-to-day management decisions.
- Design choices can create hidden dependencies when systems are integrated without clear isolation.
- Procurement decisions aimed at the lowest first cost often skip spares, labeling, and practical resiliency measures.
- Operational gaps show up when nobody can answer a simple 2 a.m. question: who owns this if it fails?
That combination is what makes small issues so dangerous. A building may look redundant on paper while still depending on one undocumented transfer switch, one untested backup sequence, or one vendor assumption that falls apart during an event.
Redundancy Helps Only When It Is Real
One of the strongest parts of the episode is the pushback on a common assumption: more redundancy always means less risk.
Michael argues that redundancy is a tool, not a cure-all. If two controllers still rely on one unreliable transfer switch, the system has not really become resilient. It has simply become more complex. Worse, poorly documented redundancy can create false confidence. Teams assume the backup will work, stop testing it, and end up discovering the weakness only during an actual outage.
The practical message for owners and operators is simple: backup systems only reduce risk when their dependencies are understood, their failovers are validated, and their ownership is clear.
Operational Controls Teams Can Start Using Now
This episode is especially useful because it stays concrete. Instead of offering broad advice, Michael outlines practical controls teams can begin using right away.
- Telemetry sanity checks: Do not just collect data. Watch for impossible values and compare independent measurements of the same physical condition.
- A simple trigger: If two independent measurements disagree by more than 10%, investigate it.
- Spare part strategy: Keep the top three low-voltage parts that fail most often on site, tested and clearly labeled.
- Validated failovers: Schedule low-impact failover tests quarterly and document who performed them, what happened, and how long recovery took.
- Visible ownership maps: Make sure both facilities and technical teams can quickly see who owns which critical component.
- Low-disruption testing: A planned 30-minute window is far better than a surprise outage.
These ideas are not expensive or theoretical. That is part of the point. The biggest reduction in outage risk often comes from process discipline, simple checks, and a modest investment in preparedness.
Case Studies That Show the Difference
Michael shares two short examples that make the contrast clear.
In the first, a hospital campus had a chilled water plant that kept tripping. Telemetry sanity checks revealed that a supply temperature sensor was intermittently flatlining. Because the team had a labeled spare sensor kit on site and a low-impact maintenance window already scheduled, they were able to swap the sensor, rerun the failover test, and stop the tripping. No major capital project. No broad disruption. Just better diagnostics, a tested spare, and clear ownership.
In the second example, an office park had two vendors sharing access control and network responsibilities. Each assumed the other handled firmware updates on edge devices. During a power event, an authentication bug surfaced. Neither team stepped in quickly because the ownership line was blurry. Access was lost for hours. The lesson is direct: clarity in ownership and contract language matters just as much as the technology itself.
Three Actions to Take This Week
For listeners who want to act immediately, the episode closes with three concrete steps:
- Run a 5-minute ownership map review for the top 10 components that would create tenant impact if they failed.
- Identify the sensor type that fails most often and confirm there is at least one tested spare on site.
- Schedule a low-impact failover test this quarter and document results and lessons learned.
These are not abstract best practices. They are practical operating habits that can reduce the odds that a minor issue becomes a visible outage.
The Larger Lesson for Owners and Operators
A recurring theme throughout the conversation is that complexity is not the same as resilience. Owners may feel pressure to add redundancy for marketing reasons or risk expectations, but redundancy should follow operational readiness, not replace it.
Michael’s rule of thumb is especially useful: only add redundancy when the people, processes, and budget exist to validate and maintain it. If a team has not yet invested in telemetry checks, tested spares, ownership maps, and scheduled failovers, adding more layers can simply hide failure modes rather than reduce them.
That is a valuable framing for property leaders, facilities teams, and IT stakeholders. Resilience is not just about buying more infrastructure. It is about understanding dependencies, assigning responsibility, and proving that recovery paths work before tenants ever notice a problem.
Final Takeaway
This episode is a reminder that building outages often begin quietly. A frozen sensor, an imperfect UPS transfer, or an unlabeled vendor handoff may look minor in isolation. But when those issues stack up, they can disrupt comfort, access, and business operations very quickly.
The good news is that the controls that interrupt those chains are achievable: sanity checks, tested spares, visible ownership, and validated failovers. As Michael puts it, the right question to ask every quarter is this: what is the smallest failure that would cause the biggest impact? Fix that first.
The episode also points listeners to practical resources in the notes, including a one-page cascade checklist, an ownership map template, a spare part starter list, and a 30-minute failover test plan. If you are responsible for building operations, tenant experience, or infrastructure reliability, this conversation offers a clear place to start.
Quiet Cascades: How Small Building Failures Become Major Outages
Most building outages do not begin with a dramatic failure. They begin quietly.
A sensor freezes. A transfer does not happen as cleanly as expected. A vendor assumes someone else owns a firmware update. Nothing looks catastrophic at first. Then multiple systems start reacting to bad information or incomplete action, and a routine issue becomes a tenant-facing event.
That is the central theme of this episode of Built, Wired & Secured, where Alex Morgan speaks with Michael Harrington about “quiet cascades” and what owners, facilities teams, and technical operators can do to stop them before they become expensive outages.
The conversation is highly practical because it focuses less on theory and more on the operational controls that actually reduce risk. The takeaway is clear: resilience usually improves faster through ownership clarity, telemetry checks, tested spares, and scheduled failovers than through adding complexity for its own sake.
A Simple Failure Scenario With Expensive Consequences
The episode starts with a realistic scenario in a downtown office tower on a hot August afternoon. A temperature sensor freezes at 72 degrees. The automation system thinks the zone is stable. A VAV damper drives open. The economizer cycles in a way it should not. Within 30 minutes, the chilled water plant trips because systems are effectively fighting one another.
That progression matters because it shows how disconnected-seeming issues are often linked in practice. A frozen sensor is a small part failure. The building outage that follows is not small at all. Now tenants are uncomfortable, teams are scrambling, emergency labor is involved, and the service backlog grows.
For owners and operators, that is the real cost of cascade risk. The triggering event may be a low-cost part, but the business consequences are measured in downtime, complaints, delayed work, emergency response, and reputation damage.
Where Quiet Cascades Usually Start
Michael identifies three categories that start most of these events.
- Bad data from sensors and field devices
- Power handling hiccups, including imperfect UPS transfers
- Human and contractual handoff failures such as unlabeled panels, unclear scopes, or ambiguous ownership
Each one is manageable on its own. A drifting sensor can be replaced. A transfer issue can be tested and corrected. A handoff gap can be clarified in documentation or contract language. The problem is that real outages often involve more than one of these categories at once.
That is when the risk multiplies. Bad data leads to the wrong equipment response. A power event stresses a weak dependency. A vendor assumption slows resolution because no one can immediately say who owns the fix. What looked minor at noon can become a major outage by mid-afternoon.
One of the strongest lines in the episode is that tenants feel the outage first, while owners feel the balance-sheet impact later. That framing is useful because it ties operations to business outcomes. Quiet cascades are not just engineering problems. They are service, cost, and trust problems too.
Why Design, Procurement, and Operations All Matter
The episode does not place blame on a single team or discipline. Instead, it explains how design, procurement, and operations often combine to create the conditions for cascade failure.
On the design side, integration can create hidden dependencies when systems are linked without clear isolation. That means a fault in one area can influence another area more easily than expected.
On the procurement side, lowest-first-cost decisions may cut exactly the items that help teams recover quickly later. Spares are not stocked. Labeling is incomplete. Practical resiliency measures are deferred because they do not look urgent during purchasing.
On the operations side, teams may not have a clear ownership map. When an issue happens at 2 a.m., responsibility starts bouncing between departments or vendors instead of moving directly to action.
Put together, those choices create fertile ground for quiet cascades. None of them seem severe in isolation. Together, they make a minor problem much harder to detect, contain, and resolve.
The Redundancy Trap
Another valuable part of the discussion is the challenge to a common assumption: if a system is redundant, it must be safer.
Michael’s point is that redundancy is helpful only when it is designed, documented, tested, and maintained in a way that truly removes shared dependencies. If two controllers still depend on the same unreliable transfer switch, then the outage risk has not been meaningfully reduced. The environment may look more robust on paper, but it still has a hidden single point of failure.
There is also an operational danger in redundancy that is assumed rather than proven. Teams can become overconfident. They believe the backup will engage when needed, so they stop validating it. Then the first real test of that failover path happens during an actual incident.
This is an important message for property leaders and operators evaluating capital requests. More infrastructure does not automatically create more resilience. In some cases, it creates more complexity to manage. If the organization is not prepared to validate and maintain that complexity, it may be buying uncertainty instead of protection.
The Controls That Actually Interrupt a Cascade
What makes this episode especially useful is how specific it gets about prevention.
The first recommendation is telemetry sanity checks. Teams should not settle for collecting data. They should also look for values that are impossible, suspicious, or inconsistent with independent measurements. Michael offers a simple rule that listeners can apply immediately: if two independent measurements of the same physical condition disagree by more than 10 percent, trigger an investigation.
That kind of rule is practical because it does not require a complete technology overhaul. It is a straightforward way to catch frozen or drifting sensors before they drive bad automated behavior.
The second control is a basic spare parts strategy. Keep the top three low-voltage parts that fail most often on site, tested and clearly labeled. When a common part fails, time matters. The difference between an outage and a fast correction may simply be whether a known-good replacement is immediately available.
The third control is validated failovers. Schedule low-impact failover tests quarterly. Record who did what, what happened, and how long the process took. A recovery path is only real if it has been exercised under controlled conditions.
The fourth control is ownership visibility. Both technical and facilities teams should be able to see who owns critical components and decisions. That visibility should exist before a problem occurs, not be assembled in the middle of one.
Finally, Michael emphasizes low-disruption testing. A planned 30-minute maintenance window is almost always cheaper and less painful than an unexpected building-wide issue.
What Good Control Looks Like in Practice
The hospital case study in the episode shows how these controls work together. A chilled water plant had been tripping repeatedly. Telemetry sanity checks showed a supply temperature sensor intermittently flatlining. Because the team had a labeled spare sensor kit on site and already had a low-impact window scheduled, they were able to replace the part, rerun the failover test, and stop the tripping.
That example matters because it was not solved by a large capital project. It was solved by disciplined operations: better detection, a tested spare, and a defined window to act.
The office park example shows the opposite outcome. Two vendors shared access control and network duties, and each assumed the other handled firmware updates for edge devices. During a power event, an authentication bug appeared, and the gap in ownership became obvious. Access was lost for hours.
The technology issue was real, but the core failure was organizational. An ownership map and a contractual clause assigning firmware responsibility could have prevented the confusion.
Three Practical Steps for This Quarter
For leaders who need a starting point, the episode closes with three actionable steps.
- Review the top 10 components that would cause tenant impact if they failed and assign a clear owner to each.
- Identify the sensor type that fails most often and make sure at least one tested spare is on site.
- Schedule a low-impact failover test this quarter and document the results.
These are not glamorous projects, but that is exactly why they work. They reduce operational risk without waiting on a major redesign or budget cycle.
Resilience Before Complexity
The broader lesson from this conversation is that owners should prioritize controls before complexity. There may be valid reasons to add redundancy, including marketing expectations or risk requirements, but redundancy should follow operational readiness. If a team has not yet invested in telemetry checks, spares, ownership maps, and testing, more complexity may only hide more failure modes.
That perspective aligns closely with how strong infrastructure programs are built in the real world. Reliable environments are not created by assumptions. They are created by clear accountability, practical diagnostics, tested recovery procedures, and attention to small details that prevent larger disruption later.
Listen With a Checklist in Hand
If you manage building systems, support critical infrastructure, or oversee tenant experience, this episode offers a practical framework for reducing avoidable outages. It is a reminder that the smallest failure can create the biggest impact when dependencies are unclear and controls are weak.
The good news is that many of the best interventions are feasible this week, not next year. Start with the smallest failure that could cause the biggest impact. Fix that first. Then prove your backups, clarify ownership, and keep the parts and processes in place that let teams respond fast.
That is how quiet cascades get interrupted before anyone notices.
Listen to the full episode for the complete discussion and the practical resources mentioned in the notes, including the ownership map template, spare part starter list, cascade checklist, and 30-minute failover test plan.