Show Notes
Why Building Technology Outages Turn Into Bigger Problems Than They Should
This episode of Built, Wired & Secured makes a practical case for running short tabletop drills before a real building technology incident forces the issue. The opening scenario is familiar to anyone responsible for commercial property operations: a vendor pushes a firmware update to a door controller, the device reboots, the BAS flags a sensor fault, chilled water valves close, and part of the building loses conditioned air at the worst possible moment. The technical trigger may be small, but the operational fallout grows fast when nobody is clear on authority, escalation, rollback, communications, or vendor ownership.
The conversation stays focused on one core idea: many outages become painful not because the equipment fails, but because decision paths were never rehearsed. A short, focused tabletop drill can expose those handoff problems in 60 to 90 minutes without requiring a live outage or expensive test event.
What Tabletop Drills Actually Reveal
The guests draw an important distinction between testing infrastructure and testing operations. A load test proves whether a generator can carry load. A tabletop reveals who approves the switch to generator power, who calls the vendor, and who communicates with tenants. That difference matters.
- Tabletops surface unclear ownership and missing authority.
- They reveal communication breakdowns between facilities, IT, security, and vendors.
- They show where vendor assumptions and internal assumptions do not match.
- They often identify low-cost operational fixes before a capital project is needed.
- They help teams separate technical capacity questions from decision-making questions.
One example from the episode involved simulating a single switch failure and discovering that the failover path had not been tested in months. What started as a one-hour outage scenario turned into a two-week capital fix. That is exactly the kind of hidden risk a tabletop is meant to uncover before a live event does it for you.
Who Needs to Be in the Room
For a first drill, the guidance is intentionally simple: bring together the people who make decisions and the people who do the work. The recommended mix includes facilities, IT, security or access control, primary vendors, and someone responsible for communications. When tenant-facing systems are in play, a tenant representative can add real value by pressure-testing expectations and message clarity.
The discussion also acknowledges that teams can start smaller if needed. BAS, IT, and property management may be enough for an initial session. The point is not to create a perfect simulation on day one. The point is to get the right people talking through a realistic disruption and documenting how they would respond.
Best First Scenarios to Run
The episode gives listeners a strong starter list of scenarios for a first exercise. These are practical, believable, and closely tied to cross-trade building operations.
- Localized network failure that takes access control and BAS offline.
- Partial power loss where UPS protects servers but not field devices.
- Primary vendor unreachable during a maintenance window.
- Generator failover with unclear signoff and tenant notification responsibilities.
- BAS controller reboot or vendor maintenance window gone wrong.
- Degraded performance scenarios such as slowly failing valves or intermittent chilled water flow.
The degradation point is especially useful. Not every major incident begins with a dramatic hard failure. Slowly degrading systems often expose weak monitoring and vague service level objectives long before they produce a fully visible outage.
How Often Teams Should Drill
The recommendation is a simple rhythm: run a short tabletop quarterly and one deeper annual exercise that includes vendors and tenant representatives. Keeping the quarterly drill under 90 minutes preserves attention and makes it easier to maintain momentum. The annual session can go broader and test coordination, vendor dependencies, and capital assumptions.
Just as important, the episode stresses that a drill only matters if the outcomes stay visible. Teams should publish an action register with owners and deadlines and give brief status updates in staff meetings. Without that discipline, the meeting becomes a conversation instead of an operational improvement tool.
How to Handle Tenant Participation
The conversation tackles a common concern directly: should tenants be included at all? The answer is yes, but with control and purpose. A tenant should not be handed a technical vulnerability review. But leaving tenant communication out of the rehearsal can create a different risk: silence, confusion, and loss of trust during a real event.
The better approach is to practice messaging rather than technical minutiae. A deidentified tenant representative or a communications-only exercise can help teams refine the timing, tone, and clarity of outage updates. One example in the episode described an HVAC interruption drill that included practicing a 60-minute notification. In the next real event, escalations dropped because tenants received clearer communication.
A 60-Minute Pilot You Can Run This Quarter
One of the most useful parts of the episode is the practical outline for a first pilot drill. The structure is intentionally compact so teams can put it on the calendar quickly.
- 10 minutes: setup, objectives, rules of engagement, and participant roles.
- 30 minutes: scenario play with two pauses for direct questions on immediate actions and communications.
- 10 to 15 minutes: capture decisions, owners, and required follow-up.
- 10 minutes: review and prioritize the action register.
Participants should have a simple one-page template that lists the scenario, presumed cause, impacted systems, decision triggers, owner, and recommended mitigation. Updating that document live turns the exercise into an accountability artifact immediately.
How Drills Change Operations and Capital Planning
The episode closes the loop by showing how tabletop findings can lead to real improvements. In one case, a drill uncovered an overlooked UPS distribution rack. A modest capital fix reduced risk to core systems and cut false alarms. In another, a BAS vendor was found to rely on a single dial-out modem for remote access. Adding a second path and testing both monthly created inexpensive redundancy that could prevent a multi-day outage.
These examples reinforce the broader point: a tabletop does not just reveal problems. It helps teams prioritize what deserves a policy fix, what needs better monitoring, what requires a communication standard, and what warrants capital investment.
The 30-60-90 Plan
- Within 30 days, choose one realistic scenario and schedule a 60-minute pilot with core stakeholders.
- Within 60 days, run the drill and publish an action register with named owners.
- Within 90 days, review progress, close quick wins, and schedule the next drill with a broader group.
- Define one realistic SLO for each critical system and test whether operations can actually support it.
The final takeaway is clear: do not wait for a live outage to discover who owns the next move. Run a short, realistic drill that tests decisions, not just devices. Make the outcomes visible, assign owners, and follow up. That simple discipline can reduce surprises, strengthen tenant communication, and turn building technology from a reactive burden into a more reliable operating system for the property.
Tabletop Drills for Building Tech: Why Short Rehearsals Prevent Bigger Outages
Most building teams understand the value of maintenance. They schedule inspections, coordinate vendor windows, and try to keep core systems stable. But this episode of Built, Wired & Secured argues that maintenance alone is not enough. If teams never rehearse how they will respond when building technology fails across trades, even a small event can become a larger operational and reputational problem.
The episode opens with a highly believable scenario. A vendor pushes a firmware update to a door controller. The device reboots. The BAS flags a sensor fault. Chilled water valves close. Half of an east wing loses conditioned air during a critical tenant meeting. The front desk gets flooded with calls. The help desk starts opening tickets. And nobody is sure who has authority to roll back the change.
That is the central lesson of the conversation. In many building incidents, the original technical issue is not the main reason the event becomes painful. The real damage comes from unpracticed escalation paths, unclear ownership, weak communications, and untested vendor dependencies.
Why a Tabletop Beats Waiting for a Real Incident
The guests make a useful distinction between a live technical test and a tabletop drill. If you want to prove that a generator can carry full load, run a load test. If you want to know who signs off on switching to generator power, who informs tenants, and who calls the vendor, that is a tabletop problem.
That distinction matters because many organizations default to thinking only in technical terms. They ask whether equipment works, whether redundancy exists, or whether failover is configured. Those are important questions, but they are incomplete. Building operations live at the intersection of technology, facilities, security, property management, and outside vendors. The friction is often in the handoff.
A tabletop brings those handoffs into the open quickly and cheaply. In as little as 60 to 90 minutes, teams can expose unrealistic assumptions, overlapping responsibilities, missing authority, and communication gaps that would otherwise surface during a real outage.
What These Drills Are Designed to Expose
The episode stays focused on decisions rather than devices. That framing is especially valuable for commercial real estate teams and operators responsible for multiple building systems at once.
Short tabletop drills can reveal:
- Who owns the first response when a failure crosses multiple systems.
- Whether rollback authority is clear during vendor-driven changes.
- How quickly tenants will be informed and by whom.
- Whether vendors can actually meet their assumed response commitments.
- Where monitoring is noisy, incomplete, or too ambiguous to support fast action.
- Which systems have hidden single points of failure.
One example from the discussion illustrates the point well. A simulated switch failure uncovered that the failover path had not been tested in months. That discovery changed the conversation from a one-hour outage assumption to a two-week capital issue. Without the exercise, that hidden weakness could have remained invisible until it caused a real disruption.
Who Should Be at the Table
The advice for a first drill is practical and refreshingly straightforward. Invite both decision makers and doers. That typically means facilities, IT, security or access control, key vendors, and someone responsible for communications. If the incident would affect tenants directly, a tenant representative can be useful as well.
There is also an important note of realism here: teams do not need to wait until they can gather everyone. A smaller pilot with BAS, IT, and property management is a perfectly acceptable starting point. The goal is not to design an enterprise-grade exercise program before doing anything. The goal is to start rehearsing cross-functional decisions now.
The Best First Scenarios to Use
The scenarios recommended in the episode are specific enough to be useful and broad enough to apply in many buildings. A strong first exercise should focus on something plausible, not theatrical.
- A localized network failure that takes access control and BAS offline.
- A partial power loss where servers stay up on UPS but field devices do not.
- A vendor being unreachable during a maintenance window.
- A BAS controller reboot that triggers wider operational confusion.
- A generator failover scenario with unclear signoff and notification steps.
- A degraded-performance issue such as intermittent chilled water flow or slowly failing valves.
The degraded-performance suggestion is especially smart. Not every operational problem arrives as a hard outage. Slow failures often create the most confusion because the symptoms are inconsistent. That uncertainty exposes weak monitoring and unrealistic service expectations in ways a binary up-or-down scenario may not.
Why Tenant Communication Deserves Rehearsal Too
One of the more nuanced points in the episode is the discussion around tenant participation. Some teams hesitate to include tenants in any exercise because they worry it will reduce confidence. The guests do not dismiss that concern, but they argue that the greater risk is often poor communication during a real event.
The answer is controlled participation. Do not turn a tenant exercise into a technical vulnerability briefing. Instead, use a deidentified representative or run a communications-focused scenario that tests timing, clarity, and escalation language. In the episode, one HVAC interruption drill included practicing a 60-minute tenant notification. In the next real event, escalations dropped because the communication was clearer.
That is a powerful reminder that operational trust is built as much through message discipline as it is through technical resilience.
A Simple 60-Minute Pilot Structure
For teams that want to act immediately, the episode lays out a compact structure that can fit into a normal workweek.
- First 10 minutes: define the objective, set rules of engagement, and identify participants.
- Next 30 minutes: play through the scenario, pausing twice to ask what actions happen now and who communicates what.
- Next 10 to 15 minutes: capture decisions, owners, and follow-up items.
- Final 10 minutes: review and prioritize the action register.
Participants should work from a one-page template that includes the scenario, presumed cause, impacted systems, decision triggers, owner, and recommended mitigation. Updating it live is important. The document becomes the first version of accountability, not just a meeting note.
How Often to Run Them
The recommended cadence is also sensible: short quarterly tabletops with a slim roster, plus one deeper annual exercise that includes vendors and tenant representatives. Quarterly sessions keep decision paths fresh without becoming a burden. Annual sessions test broader coordination and may help validate bigger assumptions tied to capital planning and major dependencies.
But frequency alone is not enough. The episode emphasizes that findings must remain visible. Teams should publish an action register, assign owners, set deadlines, and give brief status updates during regular staff meetings. That follow-through is what turns a tabletop into a repeatable operational discipline.
How Drills Influence Budgeting and Capital Planning
One of the strongest business takeaways from the episode is that tabletop drills can improve capital prioritization. The exercises do not just identify risks. They help teams understand which problems can be solved with process, which need monitoring changes, and which require targeted spend.
The examples shared are instructive. In one case, a drill uncovered an overlooked UPS distribution rack. A modest capital fix reduced risk to core systems and cut false alarms. In another, a BAS vendor depended on a single dial-out modem for remote access. Adding a second path and testing both monthly created low-cost redundancy that could prevent a multi-day outage.
For owners and operators, that matters. A drill can help separate expensive assumptions from actual priorities and support smarter conversations about where dollars should go first.
Use SLOs to Turn Discussion Into Standards
The conversation also recommends defining one realistic service level objective for each critical system. Badge access restored within 30 minutes is one example given in the episode. That kind of target forces operational clarity. If the team cannot meet it under current conditions, leadership has a clear choice: adjust the expectation or invest to meet it.
That is where tabletop drills become especially valuable as a management tool. They do not just expose risk. They create a bridge between operations, communications, and budgeting.
A 30-60-90 Plan to Get Started
The episode ends with a direct action plan:
- In 30 days, pick one realistic scenario and schedule a 60-minute pilot with core stakeholders.
- In 60 days, run the drill and publish the action register with owners.
- In 90 days, review progress, close quick wins, and schedule the next session with a broader roster.
The final advice is equally important: keep the tone constructive. Drills are not about blame. They are about reducing friction, making responsibilities clearer, and building repeatable responses before a live event forces the issue.
If your team supports networks, BAS, access control, power dependencies, or tenant-facing building systems, this episode offers a practical starting point. The smartest next step is not a perfect plan. It is a short, realistic rehearsal that tests decisions, documents ownership, and turns what you learn into visible action. If you want the full conversation and the sample starter template mentioned in the episode, listen to the episode at https://builtwiredsecured.com/episodes/tabletop-drills-for-building-tech-outages.