Proactive IT support: fixing tickets before anyone files one

A content caching server disappeared from a data center rack and nobody noticed for two weeks.

Another team clearing out old hardware pulled it and stacked it with gear headed for recycling. It came back by luck, because someone walked past and noticed one machine looked newer than everything around it.

Here is the part that should bother you. Not a single user filed a ticket. When a cache host vanishes, clients silently fall back to pulling updates straight from the vendor, and everything keeps working, slightly worse, for as long as you let it.

The admin who owned it was blunt about the cause: there was no uptime monitoring. He put a ping and a port check on it that week, then switched caching off deliberately to see how fast the alert would fire. Under five minutes.

That gap between "broken" and "somebody complained" is the whole subject of proactive IT support. If your users are your monitoring, you find out last, and from the person least equipped to describe what happened.

Discovery before decisions

Most monitoring projects fail at the first step, deciding what to watch. People pick a template, inherit 40 checks they do not understand, then spend a year tuning noise.

Walk the tree first. Point snmpwalk at the device, or a subtree of it, purely to discover what the thing exposes before you have any opinion about what matters. Only then narrow to an snmpget on the specific value you will poll on a schedule.

One caution that has ruined afternoons: walking from the root OID on some devices will spike the SNMP daemon to 100% CPU and take the whole device down with it. Always test against a live scope, narrow it, and pipe through more rather than firing a full dump at a production core during business hours.

The volume will startle you. A small router can expose 70,000 individual values, and finding the six that matter is the actual work.

Match the interval to the physics. A certificate expiry check needs to run once a day. Not every five minutes. The value cannot change faster than that, so polling faster buys load and log volume and no information. Every check should have an interval you can justify out loud. For scale, one environment ran 6 monitoring servers watching roughly 13,000 to 15,000 endpoints.

Inside the machine is not the outside world

This is the single most important thing in this article, and it took me embarrassingly long to internalize.

Internal monitoring tells you what is happening inside a machine. It does not tell you whether the outside world can reach it. Two different questions, with different answers more often than anyone is comfortable with.

The example that fixed my thinking: a web server whose listener had silently died kept reporting its process as running, happily, for two hours. Every check green. The process existed. Nothing was being served to anyone.

Real monitoring checks both. Internally, is the process alive and the queue draining. Externally, an actual port probe from somewhere that is not the machine, connecting the way a user would. Skip the second half and you have built a reliable system for feeling good.

50,000 emails

A router interface started flapping, up and down 10 to 20 times per second. Each transition fired a trap. Each trap generated an email.

50,000 emails in 5 to 10 minutes.

The instructive part is what the team concluded while it was happening. From the flood, the obvious diagnosis was a mail server outage, because mail was visibly on fire. The root cause was a chattering port on one switch. Monitoring had turned a small hardware fault into a large outage of an unrelated service, then aimed the investigation at the wrong building.

I built a system like that once and defended it for two years. My reasoning was that filtering alerts risked missing something, so everything should page everyone. That is not caution. It is refusing to decide and calling it thoroughness, and it trained a team to delete alerts unread.

The fix is alert-processing logic: deduplication, flap detection, thresholds that require a condition to persist before it counts, suppression of downstream alerts when an upstream dependency is already known bad.

An alert hygiene standard worth stealing. Three tiers, and every check gets assigned to exactly one, in writing:

• Pages a human at 3am if it is affecting people who are awake, or will be by morning, and a human can act on it right now. Site down, identity provider unreachable, storage under 5%. Keep this list short enough to recite from memory.
• Posts to a channel if someone should look today but nothing is burning. Backup job failed, a device has not checked in for a week, a caching host went offline, disk at 80% and climbing.
• Weekly digest for trends rather than events: certificate expiries 30 days out, patch compliance, license counts, printer consumables.

If a check does not clearly belong in one tier, do not deploy it yet. If a page fires twice and nobody acts, demote it. Ignored alerts are worse than none, because they teach a habit that later gets applied to a real one.

And what to monitor first, if you are starting from nothing:

1. External reachability of everything users depend on, probed from outside it.
2. Disk capacity on anything that fills.
3. Backup and job completion, alerting on the absence of success rather than the presence of failure.
4. Certificate, token, and license expiry, checked daily, digested weekly.
5. Device check-in recency, which catches theft, breakage, and departure at once.
6. Authentication failure rates.
7. Environmental and power monitoring, if you own physical space.
8. Trends compared against peers, once you have enough history for the comparison to mean anything.

Everything after that is refinement. Most teams have items 2 and 3 and nothing else, then wonder why ticket volume never falls.

The horn

Two stories are why I treat monitoring as an evidence-gathering practice rather than an alerting one.

The HVAC unit that told on itself. Temperature and airflow output on one unit trended down while its siblings on the same system stayed flat. No threshold had been crossed and nothing was broken yet, but the divergence from its peers was enough to flag it, and it got replaced before it failed and cooked a server room. Trending only works if you record peers alongside each other.

The 1929 horn. "Computers keep losing power every morning" came in as a facilities complaint. The utility was asked, looked at its side, and said incoming power was fine.

Reboot count monitoring across the desktop fleet said otherwise. Roughly 800 desktops were all powering on inside the same 5-minute window every morning. A building-wide horn schedule, inherited from a company culture dating to 1929, told everyone when the workday started, and everyone hit their power button on cue. That synchronized inrush overloaded the supply the utility insisted was healthy, until the monitoring data made the argument for them.

The utility upgraded the incoming power. A chronic undiagnosable ticket category became a one-time infrastructure fix, which is a more powerful use of monitoring than paging somebody.

Three more that pay for themselves fast. Printer counters give you an independent record when a vendor claims you used a certain amount of ink and you are fairly sure you did not. Process count monitoring catches runaway retry storms early, the way a historical mail client bug spawned hundreds of stuck connection attempts per machine. And authentication failure alerts double as an intrusion indicator, because if your devices are firing them, somebody is probing without valid credentials.

Where this gets grim. A 25-year-old HVAC unit had a network card supporting only Telnet and old unencrypted protocols. No HTTPS, no path forward. The card could not be replaced without swapping the entire multi-ton chiller, a $50,000 job involving a crane, welding, and several weeks. So it stays. Building management gear becomes a permanent security liability rather than a Tuesday patch, and the well-known retail breach that walked in through an HVAC system is why that segment deserves its own VLAN and ACLs.

Also worth knowing: at least one major platform runs its monitoring daemon as root, so a malicious script entry in its config can do arbitrary root-level damage. Reported, and the vendor's response amounted to not thinking it was bad. The same vendor will interrupt a user three times over whether a text editor may read their Documents folder.

Silent failure is the house style

Modern endpoints are built to fail politely, which is a problem.

An app blocked by policy at the endpoint security layer bounces once in the Dock and does nothing. No dialog, no explanation. The user files a ticket saying their Mac is broken, and they are not wrong to. Plugins whose dependencies vanished in an OS upgrade produce no error, just blank output, which is how a virtual camera ends up feeding a black rectangle into a client call.

So build a readiness audit instead of waiting for reports. Before a forced platform change lands, inventory which apps and devices will break using architecture detection and usage data. Launch counts and last-launch dates matter as much as presence: an app opened once a year ago is a different problem from one opened forty times last month, and that ordering lets a big fleet remediate by real impact.

Vendor deprecations follow a template. A documentation notice, then infrequent user notifications, then escalating nags, then removal on a published date. It has run that way for several technology generations, so the deadline is knowable months out. You can staff for a known deadline. What wrecks teams is discovering it during the nag phase.

The stampede. Push a fleet-wide update and every online device hits the vendor CDN at once. One site with 500 devices online took its own network down that way, and the network team's first question was who did it. Three things fix it in production: local content caching, sharding the update commands by subnet or geographic location, and scheduling the push for off-peak local hours.

Where proactive work goes to be seen

The tooling half of this is solvable. The hard part is political: proactive work is invisible by construction, so it is the first thing defunded.

Nobody thanks you for the outage that did not happen. No ticket to close, no survey, no number that goes up. Meanwhile the person who heroically fixed the outage that did happen gets visible credit, so incentives end up pointing toward letting things break. The only counter I have found is to make prevented work loud.

A scheduled scan checks machines for outdated applications, updates them, and posts both successes and failures into an internal channel a human watches. Problems get caught before the affected person notices. The channel does double duty: an operational surface, and a record that the work happened.

Devices that have not checked in for some number of days trigger an automatic message to the person holding it, with a link showing live status, so they can plug it in, confirm it is checking in again, and never contact the help desk.

A management platform's notification catalog runs to roughly 65 types. Forwarding all of them rebuilds the 50,000-email problem in a nicer font. One team curated theirs down to a handful a human can act on (expiring push certificates, expiring VPP tokens, unsigned vendor terms) and posts those to a shared channel on a schedule.

And the one I like most, because it inverts the usual purpose of reporting: a monthly health report scraped across 40 to 50 separate tenant instances and dropped into per-client folders, covering encryption status, recovery key presence, and managed versus unmanaged counts. It exists specifically to seed support tickets, which is what separates it from a monthly PDF.

The last word

The best support team I ever worked alongside looked underemployed. Short queue, silent phone, and I remember privately wondering what they did all day, which is a thought I am not proud of.

They were reading. Reports, counters, trend graphs, a channel of automated output nobody outside the team ever saw. They found things on Tuesday that would have been Friday emergencies.

Measure a support team by the requests that never arrive and you get a very different team than you get by measuring tickets closed. Only one of those numbers is easy to put in a slide, which is why this keeps being hard.

---

Foqal gives IT teams a place in Slack and Microsoft Teams where automated alerts and human requests land side by side, so proactive work is visible to the people funding it.