Show Notes
Why building system updates are different from normal IT patching
In this episode of Built, Wired, and Secured, Alex Morgan sits down with Michael Harrington to examine a problem that looks routine on paper but can become costly in real operations: firmware updates and configuration changes in building systems. The conversation opens with a realistic scenario. An overnight firmware push is applied to access control controllers, the update is recommended by the vendor, and by morning tenants cannot badge into the building. Meetings are missed, deliveries are delayed, and the property team has no rollback path.
That example sets up the core issue of the episode. In traditional IT, teams often have testing pipelines, snapshots, backups, and short maintenance windows that make patching manageable. In buildings, the environment is far less forgiving. Embedded devices, HVAC sequences, access control systems, and related operational technology all affect the physical experience of the property. A bad change does not just break software. It can lock doors, disrupt schedules, and create broad tenant impact very quickly.
- Building devices are often single-purpose and deeply embedded
- Many of these systems do not support simple rollback or snapshot-based recovery
- Operational failures are immediately visible to tenants and staff
- What works for server patching often does not translate cleanly to building systems
The operational realities behind update risk
Michael outlines three realities that decision makers need to understand upfront. First, many building devices lack the transactional rollback capabilities IT teams take for granted. Second, ownership is fragmented. Facilities, IT, vendors, and integrators may all touch the same environment without sharing one disciplined change process. Third, the downstream effects are immediate and public. A misbehaving door or elevator affects an entire floor, not just a subset of application users.
That combination changes the risk profile. Teams cannot assume they will be able to reverse a change quickly. They also cannot assume that responsibility is obvious when something goes wrong. The result is a class of outages that feels avoidable in hindsight but is often created by weak coordination long before the incident occurs.
What causes configuration drift in commercial properties
The episode spends substantial time on the root causes of configuration drift and unmanaged updates. Michael describes several recurring failure modes that property and IT leaders should recognize.
- Heterogeneous stacks with different vendors and different upgrade cadences
- Unclear ownership of firmware inventories and change responsibility
- Deferred hygiene, where drift accumulates because daily operational work feels more urgent
- Vendor-driven updates presented as recommended without enough context
- Parallel work streams across construction, security, and IT with no shared change calendar
These issues create brittle environments. Over time, systems diverge from intended baselines, and teams lose confidence in what is actually running in production. Then a recommended update, applied without adequate review, exposes hidden dependencies that nobody fully documented.
One of the strongest points in the discussion is that drift is not always caused by neglect in the obvious sense. It can also be the byproduct of multiple competent teams working independently. When no single process connects their decisions, even small changes can stack into a larger operational risk.
Security versus uptime is not a theoretical trade-off
The heart of the episode is the trade-off between security and availability. Patching reduces exposure to known vulnerabilities, but applying updates too casually can create outages. Waiting reduces immediate operational risk but may leave known weaknesses in place. The answer, according to Michael, is not to choose one side emotionally. It is to make the trade-offs explicit, risk-based, and auditable.
He recommends categorizing devices based on impact. Which systems are safety critical? Which directly affect tenants? Which are lower-impact devices that can wait for scheduled maintenance? Once those categories are defined, teams can apply different patch cadences rather than pretending every device should follow the same timetable.
- Use risk categories instead of a one-size-fits-all update policy
- Coordinate change windows around tenant impact
- Require approvals that include both facilities and IT
- Document rollback steps before making the change
- Communicate clearly with tenants and frontline staff
Michael also emphasizes a point many teams skip: a rollback plan on paper is not enough. It needs to be tested. If the team has never validated the rollback path, they should not assume it will work under pressure.
Lightweight controls that reduce real risk
A major strength of the episode is that it stays practical. Rather than advocating a heavy process overhaul, Michael recommends lightweight controls that provide meaningful risk reduction.
The first is a firmware and configuration inventory. Teams need to know device type, firmware version, last update date, and responsible owner. Without that baseline, every future change is harder to assess.
The second is an immutable baseline for critical systems. In other words, define the intended configuration and only change it through a formal request process. This helps limit drift and gives teams a reference point when troubleshooting.
The third is staging. If a full lab is not possible, use a staged rollout to a subset of non-critical units before rolling out broadly. The episode includes a real operational example where staged deployment caught a vendor firmware change that reset schedules. Because only a limited set of non-critical units was affected, the team avoided a building-wide disruption and the vendor was able to issue a patch within 24 hours.
Additional recommended controls include explicit change windows, a one-page rollback checklist technicians can follow under pressure, and regular tabletop exercises for likely failure scenarios such as lost access or HVAC schedule issues.
Why capital planning matters here
One of the more useful business-oriented sections of the conversation focuses on capital planning. Michael argues that spare controllers and small test benches should not be viewed as unnecessary extras. They are inexpensive insurance compared to the cost of tenant downtime, emergency response, and reputational damage.
Having spare hardware creates options. It allows teams to test firmware before building-wide deployment, reduces dependence on vendor timelines, and gives owners more leverage when vendors push aggressive upgrade schedules. It also supports a longer-term ownership mindset rather than a reactive one.
The broader message is clear: modest upfront investment often lowers total cost of ownership because it reduces crisis-driven work later.
The five-step checklist from the episode
To close the discussion, Michael offers a compact checklist listeners can apply right away:
- Build or update a firmware and configuration inventory and assign owners
- Define critical versus non-critical devices and document patch cadence by tier
- Create an approval and communication process that includes facilities, IT, and tenant-facing teams
- Implement staged rollouts and a one-page rollback checklist for every change
- Schedule a quarterly tabletop exercise to rehearse common failure scenarios and validate assumptions
The episode’s value is its practicality. It does not suggest that property teams need to become DevOps experts. Instead, it shows how a small amount of operational discipline can prevent large, visible failures. For owners, operators, and technology leaders responsible for modern buildings, that is the real takeaway: update control is not just an IT hygiene issue. It is a continuity, tenant experience, and risk management issue.
Listeners looking for templates, checklists, and supporting resources are directed in the episode to the Built, Wired, and Secured page at GDS Technology for a firmware inventory template, a sample rollback checklist, and tabletop exercise scenarios.
Patch now or preserve uptime? Why building system updates demand a different playbook
Firmware updates and configuration changes are normal parts of technology operations. In most IT environments, that statement barely raises concern. Teams patch servers, update applications, reboot devices, and move on. But in building systems, the same kind of change can produce a very different outcome. A recommended firmware update can turn into badge access failures, broken HVAC schedules, delayed deliveries, and frustrated tenants before the business day has fully started.
That is the central message of this episode of Built, Wired, and Secured, where Alex Morgan speaks with Michael Harrington about firmware management, configuration drift, and update governance in commercial buildings. The conversation is practical, but the implications are serious. Building technology sits at the intersection of uptime, physical operations, tenant experience, and risk management. That means patching decisions cannot be handled with server-room assumptions alone.
Why building systems behave differently than traditional IT
One of the clearest insights from the episode is that building environments are not simply another branch of IT. They include embedded controllers, access control, HVAC logic, and other operational systems that are tightly tied to real-world conditions. When a server update goes wrong, the damage may be limited to part of an application stack or a subset of users. When a building system update goes wrong, people may not be able to enter spaces, temperature schedules may fail across floors, and frontline staff may be forced into manual workarounds immediately.
Michael explains that many of these devices are single-purpose and deeply embedded. Unlike modern server infrastructure, they often lack straightforward rollback capabilities, snapshots, or transaction-style recovery. That matters because it removes one of the biggest safety nets IT teams rely on when they deploy changes.
Just as important, ownership in building environments is usually fragmented. Facilities, IT, vendors, and integrators can all interact with the same systems. Yet they often do not share one disciplined change process or one source of truth. That fragmentation makes it easier for risky updates to slip through and harder to assign accountability when something fails.
How configuration drift quietly creates brittle operations
The episode makes a strong case that big outages are often built slowly. Configuration drift is a prime example. Teams rarely wake up and decide to let their systems become inconsistent. More often, drift accumulates because immediate operational priorities crowd out maintenance, documentation, and version control.
Michael identifies several recurring causes:
- Mixed vendor environments with different assumptions and update schedules
- No clear owner for the firmware and configuration inventory
- Deferred housekeeping because daily issues feel more urgent
- Recommended vendor updates applied without enough context
- Construction, security, and IT work happening in parallel without a unified change calendar
Each of those factors may seem manageable on its own. Together, they create a brittle estate where nobody is fully confident about the true production baseline. At that point, even a small firmware push can trigger a chain reaction because hidden dependencies were never fully mapped or reviewed.
That is why configuration drift is not just a technical cleanliness issue. In buildings, it becomes an operational risk issue. If the environment has drifted far enough from the intended state, troubleshooting takes longer, rollback becomes less reliable, and tenant-facing impact becomes more likely.
The real trade-off: security exposure versus operational disruption
Many teams frame update decisions emotionally. Patch immediately to stay secure, or delay changes to protect uptime. The episode argues for a more mature approach. The trade-off is real, but it should be made explicitly, based on risk, and documented.
Michael recommends starting with categorization. Not every device should follow the same patching policy. Some systems are safety critical. Some are tenant impacting. Others are lower risk or less time sensitive. Once those categories exist, teams can define different cadences and approval expectations for each tier.
That kind of segmentation changes the conversation. Instead of debating patching in the abstract, leaders can ask better questions. Does this change affect a system tied directly to building access? Is there a known exploit that justifies a faster maintenance window? Can this update wait until validation is complete? What communication needs to happen before the change?
Michael’s recommended control set is straightforward:
- Use change windows aligned to tenant impact
- Require approvals that include facilities and IT
- Document rollback plans for every change
- Use communication templates for tenants and frontline staff
- Validate rollback procedures instead of assuming they work
That last point deserves emphasis. A rollback plan that has never been tested is only a theory. In building systems, where recovery paths may already be limited, treating rollback as a checked box instead of a verified capability creates unnecessary exposure.
The most practical controls are also the most repeatable
Another strength of the episode is that it does not ask listeners to adopt heavyweight engineering processes that are unrealistic for property teams. Instead, it focuses on simple controls that reduce risk without creating excessive overhead.
The first is a firmware and configuration inventory. Teams need a living record of device type, firmware version, last update date, and named owner. Without that information, there is no foundation for disciplined change management.
The second is an immutable baseline for critical systems. If the intended configuration is documented and only changed through a defined request process, drift becomes easier to detect and easier to contain.
The third is staged deployment. A small test environment is ideal, but even a staged rollout to a handful of non-critical units is better than pushing an update everywhere at once. Michael shares a real-world example where staged deployment caught a vendor firmware change that reset schedules. Because the change was limited to a small non-critical subset, the team avoided a broader outage and the vendor delivered a patch within 24 hours.
The fourth is having a one-page rollback checklist that technicians can use under pressure. Stress changes how well people execute. Short, clear recovery steps improve the odds of fast containment.
The fifth is tabletop exercises. Rehearsing common scenarios such as lost access, HVAC schedule failures, or bad rollback attempts helps teams validate assumptions before an actual incident forces the lesson.
Why spares and test benches belong in capital planning
One of the most practical business points in the discussion is the argument for modest capital planning. Spare controllers and small test benches may look optional when budgets are tight, but the episode frames them correctly: they are low-cost insurance compared to tenant downtime, emergency labor, tenant credits, and reputational damage.
More than that, testing capability gives operators leverage. If a vendor or integrator pushes aggressive upgrade timing, a team with staging capacity can evaluate the change on its own terms rather than accept unnecessary risk. That reduces dependence on outside timelines and helps avoid vendor lock in practice.
This is a useful reminder for owners and operators: resilience is rarely just about buying more technology. Often it comes from funding the ability to validate, document, and recover.
A five-step checklist to apply this week
Michael closes the episode with a concise set of actions property and IT leaders can start on immediately:
- Build or refresh the firmware and configuration inventory and assign ownership
- Define which devices are critical and non-critical and set patch cadence by tier
- Create a shared approval and communication process across facilities, IT, and tenant-facing teams
- Use staged rollouts and require a one-page rollback checklist for every change
- Run a quarterly tabletop exercise to rehearse failure scenarios and validate assumptions
These are not abstract governance ideas. They are operating disciplines designed to reduce surprise failures and improve decision quality. That is what makes the episode useful. It gives listeners a realistic path to better control without pretending every property team needs to become a software release organization.
Final takeaway
The biggest lesson from this conversation is simple: in building systems, update management is not just an IT maintenance task. It is a business continuity issue. It affects tenant experience, operational stability, and the credibility of the teams responsible for the property.
If your environment includes access control, HVAC, and other embedded building systems, the safest path is not blind patching or indefinite deferral. It is disciplined change management built around visibility, staged validation, tested rollback, and shared ownership.
For listeners who want a practical starting point, this episode points to templates, checklists, and tabletop scenarios on the Built, Wired, and Secured page at GDS Technology. And if this topic is already creating friction inside your operations meetings, the episode is worth a full listen because it puts structure around a problem many teams are already feeling but have not yet formalized.