Graceful Degradation: Designing Building Systems to Fail Safely

Graceful Degradation Starts Before the Outage

Buildings rarely fail in a clean, all-or-nothing way. More often, a brief power disturbance, network interruption, cloud-service issue, or overlooked single point of failure creates a chain reaction across systems that were expected to work together. In this episode of Built, Wired & Secured, Alex Morgan and Michael Harrington examine how building teams can design for graceful degradation: preserving essential functions during partial failures instead of allowing a small technical disruption to become an operational crisis.

The episode opens with a practical scenario. A 15-second power dip takes down network switches, access control reboots in the middle of transactions, elevator control reports a fault, HVAC coordination is lost on multiple floors, security radio traffic becomes congested, and tenants begin calling the lobby. What began as a short electrical event becomes an evacuation because the team did not know which services had to remain available or how to operate systems in a degraded state.

The central point is straightforward: resilience is not only about preventing every outage. It is about deciding in advance what happens when prevention fails.

What Graceful Degradation Means

Graceful degradation means systems move into predictable, safer, and operationally manageable states when part of the environment is unavailable. That differs from brittle failure, where one interruption causes unrelated functions to stop, leaves operators without visibility, or forces improvised decisions under pressure.

For property operations, the question is not simply whether a platform is online. It is what breaks when a component goes down, who needs to respond, what services must continue, and whether people have a practical fallback.

• Essential services should remain available or move to an intentional safe state.
• Operators should be able to recognize the degraded condition quickly.
• Manual procedures should exist where automation or redundancy is not justified.
• Staff should know who has authority to make decisions during the incident.

Common Causes of Cascading Problems

Michael identifies several recurring triggers that lead to cascading degradation in buildings. Power disturbances are particularly deceptive because they may not produce a full outage, yet they can interrupt devices during transactions or reboot connected equipment at different times. Network and communications interruptions can disconnect systems from central management. Third-party vendor or cloud outages can remove access to management consoles even when field devices remain operational. And design blind spots can leave a single component controlling multiple critical functions without local failover.

A cloud outage illustrates the difference between a technical problem and an operational problem. In one example, an access-control vendor update led to cloud-authentication timeouts for several hours. Badge readers defaulted to a locked state. The immediate operational questions were not about the vendor’s infrastructure. They were whether manual entry could be authorized, whether temporary badges existed, and whether property and security staff knew the fallback process. Without a pre-agreed process, the lobby became a bottleneck.

Prioritize What Must Stay On

Graceful degradation begins with prioritization. Teams need to identify services that must remain operational, including life safety, emergency communications, essential tenant access, and critical tenant functions. The goal is not to apply the same level of resilience to every building service. It is to invest deliberately where a failure would create an immediate safety, access, or business-impact problem.

• Protect life-safety functions and emergency communications first.
• Identify essential access requirements, especially for emergency services and tenants.
• Document critical tenant functions that cannot tolerate disruption.
• Map dependencies between field devices, networks, cloud platforms, and management consoles.
• Identify single points of failure that control more than one essential function.

Choose the Right Mix of Redundancy and Fallbacks

Not every system needs full active-active redundancy. Michael recommends building tiers of redundancy and using local fallback modes for essential services. For systems where safety or immediate access is at stake, redundancy and automation are appropriate investments. For lower-risk building services, the better decision may be a clearly documented and tested manual fallback.

More redundancy also creates management overhead and testing obligations. The practical rule offered in the episode is to protect the essentials with automation and redundancy, then make sure everything else has a reliable, documented manual path that has been tested.

Local control is especially important when cloud platforms or central networks are unavailable. A building automation system may lose cloud connectivity while local controllers continue running safe schedules. This keeps the building operational even when central visibility is reduced.

Make Degraded States Observable

Operators cannot act effectively if they do not know what state a system is in. Building teams should make degraded modes visible in user interfaces or dashboards. A predictable, observable degraded state allows facilities, IT, security, and vendors to respond quickly rather than spending critical minutes determining whether a problem is isolated, widespread, or safe to operate through.

The episode also distinguishes between losing control and losing function. A vendor or cloud outage may prevent access to a management console while field devices continue operating. That is still a serious control-plane problem, but it requires a different response than a complete field-device outage.

Test Without Creating Tenant Disruption

Testing must be intentional and low impact. Michael recommends beginning with tabletop scenarios that walk through roles, communications, decisions, and recovery steps without touching production systems. Teams can then progress to staged tests, such as isolating a management console while leaving field devices intact to simulate loss of central visibility.

• Run a tabletop exercise to evaluate roles, communications, and decisions.
• Notify tenants before staged tests and define rollback steps.
• Practice manual fallbacks during off-hours.
• Confirm staff can complete critical tasks under time pressure.
• Use a quarterly tabletop cadence and targeted staged tests for high-risk systems.

Lessons From Actual Degraded States

One successful example followed a utility substation hiccup. The building’s primary BAS lost cloud connectivity, but local controllers defaulted to safe schedules and emergency ventilation kept critical labs stable. Tenants experienced slight temperature drift, but there was no work stoppage. That outcome was designed through local control and priority schedules.

In contrast, another site used a shared appliance for visitor management and access control. When it failed, elevators went to recall and an entire floor was evacuated. Manual override policies existed, but the runbook had not been practiced, extending the disruption.

Action Checklist for This Quarter

• Identify and document must-stay-on services.
• Map single points of failure and control-plane dependencies.
• Define local fallback behavior and make degraded states observable.
• Assign ownership for each degraded-mode decision through joint facilities and IT leadership.
• Run a tabletop exercise within 30 days.
• Schedule a low-impact staged test during the quarter.
• Maintain simple one-page runbooks with rollback steps.

Facilities typically owns day-to-day building operations, but IT must co-own resilience decisions because networks and cloud dependencies directly affect building systems. Vendors should document failure modes and provide executable manual fallbacks. Those requirements should be written into RFPs and service agreements, including how degraded states will be observed and tested.

Design Building Systems to Fail Safely, Not Spectacularly

A building can experience a major operational interruption without ever suffering a major outage. A short power dip can reboot network switches. An access-control platform can restart while a credential transaction is underway. A cloud service can become unavailable while field devices continue operating without central visibility. A shared appliance can fail and unexpectedly affect visitor management, access control, elevator behavior, and the people responsible for responding.

That is why graceful degradation matters. The goal is not a promise that nothing will ever fail. The goal is to ensure that when a component does fail, essential building services remain available, operators understand the condition, and the response is predictable rather than improvised.

In a Built, Wired & Secured episode focused on graceful degradation, Alex Morgan and Michael Harrington translate this idea into practical decisions for property operations, facilities, IT, security, and vendors. The discussion centers on a simple operational question: what must stay on when the normal environment is no longer fully available?

The Difference Between a Partial Failure and a Full Operational Crisis

Consider a short electrical disturbance on a weekday morning. Network switches drop for 15 seconds. Access control reboots during transactions. Elevator control reports a fault. HVAC coordination is lost across several floors. Security radio traffic is congested, tenants call the lobby, and a minor incident grows into an evacuation.

The electrical event was short. The operational consequences were not. The gap was not merely technical. The team had not established which systems were essential, what their fallback states should be, or how staff should operate when normal tools were unavailable.

Brittle systems have hidden dependencies and unclear failure behavior. A disruption to one part of the environment creates unexpected failures elsewhere. Gracefully degrading systems are designed so that a failure moves affected services into an intentional state: perhaps reduced capability, perhaps local operation without cloud management, perhaps a documented manual process. The essential difference is predictability.

Identify the Failure Modes That Matter

Resilience planning should begin with realistic causes of degradation. The episode identifies four common categories: power disturbances, network or communications interruptions, third-party service outages, and single points of failure that were not obvious during installation.

Power issues can be especially difficult because they may not look like complete outages. A brief dip can interrupt devices in the middle of a transaction, reset components at different times, and leave dependent systems in inconsistent states. The impact may be far greater than the duration of the power event.

Network and communications failures can disconnect systems from central management. A cloud or vendor interruption can create a control-plane problem: field devices may still be running, but operators cannot see status, make changes, or use the usual management console. That distinction matters. A loss of central control is not always a loss of function, but it still requires a practiced operational response.

Single points of failure are often the most consequential because they can affect several functions at once. A single remote I/O rack controlling multiple critical functions without local failover is one example. A shared appliance supporting visitor management and access control is another. These dependencies should be surfaced before an incident, not discovered during one.

Start With What Must Remain Operational

Graceful degradation is not a technology shopping list. It is a prioritization exercise. Teams should decide what must remain available when power, communications, cloud platforms, or management systems are impaired.

The priority list discussed in the episode includes life safety, emergency communications, secure access for emergency services, and critical tenant functions. Each building will have its own operating requirements, but the process is consistent: identify essential services, understand their dependencies, then choose appropriate continuity measures.

That sequence is important. Without a clear priority list, organizations can spend heavily on redundancy for lower-risk services while overlooking the functions that create immediate safety, access, or tenant-impact consequences. The question is not whether every system deserves the same architecture. It is which failure would create an unacceptable outcome.

Use Redundancy Where It Matters Most

Not every building function needs full active-active redundancy. The episode makes a practical distinction between essential systems and lower-risk services. When tenant safety or immediate access is affected, redundancy and automation are appropriate investments. When the operational impact is lower, a clear manual fallback may be a better use of budget.

This approach also recognizes that redundancy has costs beyond the initial purchase. Additional components increase management overhead. They add dependencies of their own. They also create a test burden. A redundant design that is never tested may provide a false sense of confidence.

The practical rule is to protect the essentials with automation and redundancy, then ensure that other services have a reliable, documented manual path. The manual path must be realistic. It should identify who can authorize action, what tools or credentials are needed, how the action is communicated, and how the system is returned to normal.

Preserve Local Control When Central Systems Are Unavailable

A central theme of graceful degradation is separation of control planes where practical. Local control allows a building system to continue performing essential functions even if the cloud platform or central network becomes unavailable.

The episode describes a utility substation hiccup in which a building’s primary BAS lost cloud connectivity. Local controllers defaulted to safe schedules, and emergency ventilation kept critical labs stable. Tenants noticed slight temperature drift, but there was no work stoppage. The outcome was not accidental. It resulted from local control and priority schedules that had been designed to operate through loss of cloud connectivity.

This is a useful model for other building systems. A loss of central visibility should not automatically mean a loss of safe local behavior. Teams should ask whether field devices can continue operating in a defined safe mode, what decisions are still possible locally, and how operators will know the system has shifted into that mode.

Make Failure States Visible and Understandable

Operators need more than technology that keeps running. They need to know when it is operating normally, when it is operating in a degraded state, and what action is required. Observable failure modes make that possible.

Dashboards and interfaces should communicate degraded behavior clearly. If a management console is unavailable but field devices remain active, that status should be visible. If a system has shifted to local schedules, staff should understand which functions continue, which functions are unavailable, and where the next decision belongs.

Clear visibility shortens response time and reduces unnecessary escalation. It also helps teams avoid treating every alert as either a complete failure or a nonissue. A predictable degraded state gives operators useful options.

Turn Fallbacks Into Operating Practice

A manual fallback that only exists in a document is not a dependable fallback. The episode describes an access-control vendor outage in which cloud authentication timed out for hours and badge readers defaulted to locked. The operational response depended on questions that should have already been answered: who authorizes manual entry, are temporary badges available, and do security and property staff know the process?

Without those decisions, the lobby became a bottleneck. The lesson is that technology behavior and human response must be designed together. A secure default may be technically correct, but it can still create operational chaos if the building team has no practiced procedure for safe manual access.

One-page quick-reference runbooks are a practical tool. They should include the degraded condition, immediate steps, decision authority, communications guidance, manual procedures, and rollback steps. They should be simple enough to use under time pressure.

Test Degraded States Without Disrupting Tenants

Testing does not need to mean taking production systems offline during business hours. Michael recommends a progression from tabletop exercises to staged, low-impact tests.

Start with a tabletop scenario. Simulate the event and walk through roles, communication paths, authority, procedures, and recovery decisions. This exposes ambiguity before a real incident creates urgency.

Then perform targeted staged tests for high-risk systems. One example is isolating a management console while leaving field devices intact, allowing the team to practice loss of central visibility without disrupting normal building operation. Tests should include tenant notification where appropriate, clear rollback steps, and off-hours practice for manual procedures.

A quarterly tabletop cadence paired with targeted staged tests provides a manageable rhythm. The purpose is not to generate paperwork. It is to confirm that people can complete essential tasks under time pressure.

Assign Joint Ownership Across Facilities, IT, and Vendors

Graceful degradation is a building-operations responsibility, but it cannot sit with facilities alone. Facilities typically operates the building day to day. IT must co-own the decisions because network, cloud, and communications dependencies influence how building systems behave. Security teams may have direct responsibilities for access and incident response. Vendors need to be accountable for documenting their systems’ failure modes and providing executable manual fallbacks.

Those expectations should be included in RFPs and service agreements. Ask vendors to describe the degraded states their systems can enter, how those states are observed, what remains functional, what manual fallback exists, and how the behavior can be tested.

A Practical Resilience Checklist

• Identify and document must-stay-on services, including life safety, emergency communications, and essential tenant access.
• Map single points of failure and all control-plane dependencies.
• Define local fallback behavior for essential services.
• Make degraded states visible in interfaces or dashboards.
• Assign a joint facilities and IT owner for each degraded-mode decision.
• Run a tabletop exercise within 30 days.
• Schedule a low-impact staged test within the quarter.
• Maintain simple runbooks with rollback steps.

The strongest resilience decisions are made before the incident. Listen to the full Built, Wired & Secured episode for the operational examples and practical framework behind this checklist. The objective is simple: when systems fail, essential services should remain safe, staff should know what to do, and a small interruption should not become a building-wide surprise.