Telemetry Debt: Taming Building Data Overload

When Building Data Stops Helping and Starts Hurting

Modern buildings produce a constant stream of operational data. Access control systems generate door events and reader warnings. HVAC platforms send fault alarms and performance signals. UPS hardware reports aging batteries and power anomalies. Network gear and carriers add their own logs, health checks, and status updates. On paper, all of that visibility should make operations stronger. In practice, as this episode explains, unmanaged telemetry can create confusion instead of clarity.

In this conversation, Alex Morgan and Michael Harrington define telemetry debt as the point where too much unstructured, poorly owned, or overly noisy telemetry becomes an operational liability. Instead of helping teams respond faster, it creates alert fatigue, unclear priorities, and slower decisions during real incidents. The core issue is not that teams lack data. It is that signal quality, ownership, and triage discipline are often missing.

What Telemetry Debt Looks Like in the Real World

The episode opens with a familiar operational scenario: an HVAC fault alarm, a forced door alert, several vendor health pings, and an aging UPS warning all arrive around the same time. Phones start ringing. Tenants call the front desk. Multiple teams see the noise, but nobody can agree on what deserves attention first.

That is the practical cost of telemetry debt. Teams are not necessarily blind. They are overwhelmed.

• Too many alerts arrive at once, with no clear priority
• Tenants feel the disruption before operations teams can explain it
• Teams lose time deciding who owns the issue
• Real failures get buried under transient warnings and low-value notifications

Michael offers a simple framing question that helps cut through the noise: What breaks if this goes down? That question pushes teams to evaluate telemetry based on business and tenant impact, not just technical visibility.

Why Telemetry Debt Happens

The discussion makes it clear that telemetry debt is rarely caused by a single bad tool or vendor. It usually grows from a mix of technical defaults and organizational habits.

• Vendors often ship systems with high-verbosity default alerts
• Every sensor, status bit, or warning may be flagged because it demonstrates platform capability
• Retention settings often preserve months of raw logs that nobody actually reviews
• Ownership is unclear across IT, facilities, and outside vendors
• No triage workflow exists to guide first response and escalation

One example in the episode is especially concrete: a badge reader reporting low RSSI every five minutes. On its own, that warning may be harmless and transient. But multiply it across hundreds of readers and the result is hundreds of warnings a day. Teams become conditioned to ignore the stream, which makes it much easier to miss the alerts that truly matter.

The Right Goal Is Not Silence

A key point in this episode is that telemetry cleanup is not about turning alarms off for convenience. It is about making alarms useful. Alex raises the concern that if teams prune too aggressively, they may lose early warning signs. Michael agrees that there is a real tradeoff, but argues that the answer is disciplined risk-based filtering, not keeping everything forever.

The practical standard is straightforward: preserve the signals that, if missed, would lead to tenant impact, safety risk, or extended outages. Those deserve intact visibility. Lower-value data can often be rolled up, sampled, delayed, compressed, or handled with less aggressive alerting.

The additional requirement is escalation. If circumstances change, teams need a way to raise priority quickly instead of assuming a low-severity signal will always remain low risk.

Three Low-Effort Moves That Reduce Noise Quickly

For listeners looking for immediate action, the episode outlines three practical moves that do not require a long transformation project.

• Set sensible thresholds so teams can distinguish a short-lived blip from a trend that needs intervention
• Create an ownership matrix with a primary owner, a secondary contact, and vendor involvement defined up front
• Use tiered retention so critical systems keep full fidelity while lower-risk devices are sampled or compressed

These changes matter because they address the main sources of operational drag: too many alerts, no clear owner, and no discipline around what data is worth preserving in detail.

The Ownership Matrix and Triage Playbook

One of the strongest operational ideas in the episode is the ownership matrix. Michael keeps the concept simple. For each alert category, teams should identify the primary owner, the secondary owner, and the vendor. But that assignment alone is not enough. The primary owner also needs clear first actions.

The episode recommends a simple triage structure:

• Verify: confirm the alert or check one reliable data point that supports it
• Identify ownership: determine who is primary and who is next on call
• Contain: take the smallest action that reduces tenant impact while the team investigates

If containment fails, the issue should move to the vendor with a packaged set of logs and context. That structure helps prevent alerts from bouncing between teams without progress.

Retention Strategy Without Losing Forensic Value

The episode also pushes back on a common objection: the belief that every system needs identical log retention in case of future forensic work. Michael argues that forensics matter, but uniform retention is not required to support them.

Instead, teams should use tiered retention:

• Keep detailed logs for systems with high forensic or operational value
• Reduce fidelity for low-risk systems that generate large volumes with limited investigative benefit
• Document the rationale so leadership understands the tradeoffs

This is where a targeted pilot becomes useful. Rather than arguing in the abstract, teams can test a narrower approach, measure results, and use those findings to settle concerns.

What Success Looks Like

The episode recommends measuring progress with metrics that reflect meaningful operational improvement, not vanity numbers. Instead of focusing only on mean time to acknowledge, teams should watch mean time to meaningful action. They should also measure repeated alerts per device and tenant impact events, including how often alert storms correlate with calls from tenants.

When cleanup works, the expected results are practical:

• Fewer repetitive alerts
• Faster containment
• Less confusion during incidents
• Fewer tenant escalations

Michael shares two examples that make those results tangible. In one anonymous campus environment, a two-week pilot across access control and HVAC used an ownership matrix, adjusted thresholds, and reduced debug retention for non-critical devices. Daily alerts dropped by about 70 percent, and mean time to meaningful action improved from four hours to under 90 minutes. In another managed office portfolio, teams added a delay-and-confirm rule for transient sensors. If an alarm cleared within five minutes, no ticket was created. That cut midnight wake-up calls in half without missing outages.

What to Do This Week

The episode closes with three immediate actions building leaders and operations teams can start now:

• Run a seven-day audit of alert volume and map who touches each alert
• Agree on retention windows for critical and non-critical telemetry, then document why
• Create a one-page triage playbook and test it on one building or one system first

The bigger message is simple: telemetry becomes valuable when teams can trust it, route it, and act on it. Cleaning up telemetry debt is not an exercise in convenience. It is an operational decision tied directly to uptime, tenant experience, and faster response when something actually goes wrong.

Telemetry Debt Is Becoming a Building Operations Problem

Buildings generate more operational data than most teams can realistically use well. Access control systems produce warnings, event logs, and device health signals. HVAC systems add faults, performance alerts, and trend data. UPS infrastructure reports aging components and power conditions. Network devices, carriers, and vendors layer on health checks, logs, and notifications of their own.

The promise behind all of that telemetry is better visibility. The reality, as discussed in this episode of Built, Wired, and Secured, is that more data does not automatically create better operations. When alerting is noisy, ownership is unclear, and retention is unmanaged, telemetry becomes debt. Instead of helping teams act faster, it slows them down at the worst possible moment.

That is the central idea Alex Morgan and Michael Harrington unpack in this conversation: telemetry debt is what happens when useful operational data turns into operational drag.

What Telemetry Debt Actually Means

The episode starts with a scenario many property and facilities teams will recognize immediately. An HVAC fault alarm appears. A forced door alert comes in. Vendor health pings pile up. An aging UPS warning is added to the mix. Tenants start calling the front desk. Multiple alerts are technically visible, but nobody agrees on what matters first.

That is telemetry debt in action.

The problem is not a total lack of visibility. It is the absence of signal quality and decision structure. Teams are not dealing with a clean stream of prioritized issues. They are dealing with a cluttered inbox of mixed importance, mixed ownership, and mixed urgency.

Michael offers a useful way to frame the issue: ask, What breaks if this goes down? That question forces teams to think about impact before reacting to volume. In building environments, the cost of getting that wrong is not theoretical. It shows up in tenant frustration, delayed containment, and preventable outages.

Why So Much Building Telemetry Becomes Noise

One of the strengths of the episode is that it does not blame telemetry debt on a single bad platform or a single careless team. It comes from a combination of technology defaults and operational habits.

First, vendors often ship alerting at a very high verbosity level. Every warning, status bit, and transient condition is exposed because it demonstrates capability. That may look good in a product demo, but in live operations it creates noise that teams must sort through.

Second, organizations often keep months of raw logs and debug data simply because they can. The assumption is that more retention is always safer. In reality, large volumes of low-value data can make it harder to identify what matters.

Third, ownership is often fuzzy. A network ping may be technically related to IT, but operationally relevant to facilities. An HVAC issue may trigger tenant complaints first, even if the root cause involves a vendor system. Without a clear ownership model, alerts move sideways instead of forward.

Finally, many teams never formalize triage. Alerts arrive, but there is no standard first move, no predefined primary responder, and no disciplined path to escalation.

The example shared in the episode about badge readers makes this especially concrete. If a reader reports low RSSI every five minutes and the vendor classifies that as a warning, a single device may not be a problem. Hundreds of devices doing it every day will bury the real failures. Teams start ignoring warnings because too many of them are harmless, which makes the truly important ones easier to miss.

The Goal Is Better Signals, Not Fewer Responsibilities

A critical distinction in the episode is that telemetry cleanup is not about making life easier by turning alarms off. It is about making operational data usable.

Alex raises the tension many teams feel: if you reduce noise, do you risk missing early warning signs? Michael’s answer is balanced and practical. Start with risk. Keep the signals that, if missed, would create safety issues, tenant impact, or extended outages. Those need to remain visible. Lower-value telemetry can be rolled up, sampled, compressed, or delayed, provided there is still a clear escalation path when context changes.

That matters because many organizations frame telemetry decisions the wrong way. They argue about whether to keep or remove alerts. The more useful question is whether each alert helps someone make a better decision in time to reduce impact.

Three Practical Moves Teams Can Make Quickly

The episode does a good job staying operational instead of theoretical. Rather than proposing a large transformation effort, it outlines a few low-friction changes teams can make quickly.

The first is setting sensible alert thresholds. Teams need to distinguish between a temporary blip and a trend that requires attention. Without that distinction, every event feels urgent, which makes none of them truly urgent.

The second is building an ownership matrix. For each alert category, teams should define a primary owner, a secondary owner, and the vendor relationship. That structure reduces the all-too-common problem of alerts bouncing between departments with no action taken.

The third is creating tiered retention. Critical systems keep full-fidelity logs. Lower-risk systems keep less detail through compression or sampling. This gives teams a more rational way to preserve what matters without drowning in long-term low-value data.

A Simple Triage Model That Prevents Alert Ping-Pong

One of the most useful parts of the conversation is the triage model. The episode recommends a simple three-step response for primary owners:

• Verify the alert by reproducing it or checking one confirming data point
• Identify ownership by confirming who is primary and who is on call next
• Contain the issue with the smallest action that reduces tenant impact during investigation

If containment does not work, the issue should then escalate to the vendor with a packaged set of logs and context. That last point is important. Vendor handoffs often fail because internal teams escalate without enough information, which restarts the diagnostic process instead of speeding it up.

What the playbook really does is reduce indecision. It gives the first responder a starting point and keeps the organization from losing time on the question of who should move first.

You Do Not Need the Same Retention Everywhere

Another practical insight in the episode is the argument against identical retention across every system. Teams often defend excessive telemetry retention by pointing to post-incident forensics. That concern is legitimate, but the episode makes the case that forensics do not require one universal rule.

Some systems have high forensic value and deserve detailed historical logs. Others produce large amounts of low-risk or low-actionability data that can be sampled or compressed without meaningful loss. Tiered retention lets organizations align their storage and review habits with operational reality.

Just as important, the team recommends documenting the rationale. That helps leadership see that telemetry cleanup is a controlled decision tied to risk and outcomes, not a convenience project.

How to Measure Whether a Cleanup Is Working

The conversation also avoids a common measurement trap. It does not focus only on mean time to acknowledge. Instead, it recommends tracking mean time to meaningful action. That is a stronger operational metric because it reflects when the team actually begins doing something useful, not just when someone clicked on an alert.

The episode also points to two other indicators worth monitoring:

• Repeated alerts per device
• Tenant impact events, including calls that correlate with alert storms

If cleanup is working, teams should see fewer repetitive notifications, faster containment, and fewer tenant escalations.

What Real Improvements Can Look Like

The episode includes two strong examples. In one anonymous campus environment, a two-week pilot across access control and HVAC used an ownership matrix, threshold adjustments, and reduced debug retention for non-critical devices. The result was a roughly 70 percent drop in daily alerts and an improvement in mean time to meaningful action from four hours to under 90 minutes.

In another managed office portfolio, teams implemented a delay-and-confirm rule for transient sensors. If an alarm cleared within five minutes, no ticket was created. That reduced midnight wake-up calls by half with no missed outages. The operational benefit was obvious, but so was the human one: less staff disruption and fewer tenant complaints during off-hours.

A Good First Pilot Is Small, Measurable, and Operational

For teams ready to act, the episode closes with three immediate next steps: run a seven-day alert audit, agree on retention windows for critical and non-critical telemetry, and build a one-page triage playbook for one building or one system.

That advice is practical because it avoids overdesign. A cleanup pilot does not need to solve every telemetry problem in the portfolio. It needs to prove that better signal management leads to faster response, less confusion, and a better tenant experience.

The most valuable takeaway from the episode may be the simplest one: do not frame the conversation as turning off alarms. Frame it as making alarms useful. That keeps leadership attention where it belongs—uptime, operational clarity, and tenant impact.

If your building systems are generating more noise than action, this episode offers a clear operational starting point. Listen to the full conversation for the examples, framework, and pilot ideas discussed in detail at https://builtwiredsecured.com/episodes/telemetry-debt-taming-building-data-overload.