The environment I inherited had about sixty scripts in it. No categories, no notes, nothing to indicate who wrote any of them or when.
One was named "test. Do not use."
It was live in three policies.
I don't think my predecessor was careless. He was busy, he had no version control, and he had no particular reason to believe anyone would ever read those names back to him. But I found a second thing that week that stuck with me longer. Inside a script that ran at every device enrollment sat a comment that said "don't touch this part."
I touched it.

Say the situation out loud and it sounds indefensible. No version history. No attribution, so nobody knows who changed what or when. No review gate anywhere in the path between an idea and sixty thousand devices.
Anyone with admin access could edit any script in that console right now. Instantly. No approval, no notification, no record that it happened.
We also had a test instance, which is how I know that keeping a test and a production instance in sync is a manual process that everyone intends to do, but nobody actually does.
I resisted fixing this for about two years, and my reasoning was that pipelines are for software teams. We were three people managing endpoints. Adding CI felt like buying a forklift to move a chair. I was wrong in a specific way: what I was calling overhead was the only mechanism that would have let me answer a question I got asked twice a year and could never answer.
The architecture fits on an index card. The repo is the source of truth. CI is the automation layer. The management platform is a deployment target and nothing else, which is the part people find hardest to accept.
It is push-based and one-directional. The pipeline pushes into the platform. The platform never pulls, and it never sends anything back up.
Which means any edit you make directly in the platform's UI gets silently overwritten by the next pipeline run. The phrase one team I traded notes with uses for this is "Git wins. Always." They say it in the tone of people who learned it the hard way.
The branching model is three branches and takes ten minutes to set up. A working branch where the pipeline never fires, so you can commit half-finished thoughts without deploying them. A protected test branch that deploys to the test instance. A protected prod branch where merge is restricted to maintainers.
Here is the part I underestimated for years. The protected-branch mechanism is where the security actually lives. CI variables marked protected only inject on protected branches. So a fork, or a branch somebody spun up on a Tuesday afternoon, literally cannot reach the credentials. There is nothing in it to steal. Not because a policy forbids it. Because the values never appear.
One consequence to write on a sticky note: if you hotfix directly in the console during an incident, and eventually you will, you have to go back and replicate that change in Git afterward. Otherwise the next scheduled run reverts your fix and does not tell anybody.
Scope the API role to read, create, and update scripts, plus read, create, and update extension attributes. That is the complete list. It does not need anything else, and the temptation to give it "just admin, we'll tighten it later" is how pipelines end up holding the keys to a whole fleet.
One API client per environment, test and prod kept separate. Never a person's account. If the human whose credentials are baked into the pipeline leaves, the pipeline breaks, and whoever debugs it three weeks later will have no idea why.
Two small habits I picked up from people more careful than me. Scope the secret to the environment, so the production credential is locked to the production branch and a dev branch cannot read it at all. And have the script read secrets from the environment rather than accepting them as command-line arguments, because arguments can be captured by process logging while the script runs.
Reconcile everything on every run. Fetch every deployed object into a dictionary keyed by name. Walk every file in the repo. If a name exists and the content differs, update it. If the name does not exist, create it. Anything deployed with no matching file gets marked for deletion.
That costs extra API calls and buys correctness that does not depend on pipeline history. A run that gets skipped, or dies halfway through, or happens during a platform outage, gets cleaned up by the next one. No state to reconstruct.
Deletion happens twice, 72 hours apart. The pipeline never deletes anything. It appends a delete marker with a timestamp set 72 hours out. A separate nightly job at 12:15am sweeps for expired markers and does the actual deletion.
The reasoning is worth quoting directly: hard-deleting a script attached to 200 policies is a great way to have a very bad morning. A deleted script that a policy still references fails silently on every device that policy next touches. Nobody gets an alert. The policy simply stops doing its job, and you find out from a compliance report six weeks later.
The rename will get you. The filename is the object's identity, matched exactly, spaces included. Rename a file in Git without renaming the object in the platform first and the pipeline reads it as one new file plus one missing file. So it creates a duplicate and marks your original for deletion.
I have heard of people losing scripts this way. I met them in the mirror.

One Mac engineer tells a version of this that I suspect a lot of readers will recognize. He hardcoded an API token into a script while testing. Told himself he would clean it up before committing. Got distracted, because that is what happens, added features across several more commits over the following days, and pushed the lot. He found out days later, when an automated scanner caught it.
The token itself was recoverable. Rotate it, move on, feel stupid for an afternoon.
What he could not do was answer the questions that came next. Which version of that script was running on which machines. When it last changed. Whether anyone had ever reviewed it.
Sit with that for a second, because it describes most fleet-management scripting honestly. These are unversioned production software artifacts that happen to have a .sh extension. They frequently run as root. They fire on every enrollment and every update check, across every machine you own, with no sandbox and no staged rollout. If your platform team shipped server code under those conditions, somebody would stage an intervention. Blast radius is as big as your fleet.
The fix that engineer built has two gates, and the second one matters more than the first.
Gate one is local: pre-commit hooks running a secret scanner and a linter, which refuse the commit outright. Nothing leaves the machine, nothing enters commit history, and the token never becomes a thing you have to purge from a repo's past.
Gate one can be bypassed with git commit --no-verify. Under deadline pressure, it will be. So gate two is a CI check that re-runs the same scans on push and blocks the merge whether or not the local hook ever ran. The bypassed commit can exist in a branch. It cannot reach production. The real enforcement point lives on the remote, where your good intentions at 6pm on a Friday cannot reach it.
This is the objection I hear every time, and it is a fair one.
Two answers, both less satisfying and more effective than they sound. Get a non-technical manager to do an idiot check. They do not need to read bash. A person who has never written a line of shell can still notice a recursive delete with a variable in the path and ask what that does. Or borrow a scripting-literate colleague from an adjacent team, a Windows admin or somebody in app dev, as an informal reviewer.
Deep code review is beside the point here. Nobody is doing static analysis on your battery-health extension attribute. What you are buying is a chance to catch the catastrophic mistake before it reaches tens of thousands of machines, and a second pair of eyes clears that bar at a surprisingly low level of expertise.
There is also a governance shape from the software-packaging side worth stealing. Some teams have each app's version bump land as its own independently mergeable pull request. Chrome ships this morning, Firefox waits until somebody has time to test it against the finance team's plugin, and neither decision blocks the other or produces a merge conflict with an unrelated update. Approval granularity that matches how you actually make decisions.
And then there is the payoff, which is impossible to appreciate until the first time you need it. Somebody asks who broke that script last Tuesday, and you have an actual answer: it is in the commit history, with a commit message attached to a merge request that somebody approved.
Two footnotes on hosting. Somebody counted 257 platform incidents on one major CI host across a twelve-month window: 48 classified as major outages, 57 specific to the Actions product, roughly one significant disruption a week. I do not read that as an argument against hosted CI, which I still use happily. I read it as an argument for keeping the sync logic in a plain script you can run from a laptop on the morning the pipeline is down and you need to ship a fix.
The other footnote is money. macOS runners bill at roughly ten times the Linux rate, which is why one team refactored their scripts to be OS-agnostic and moved the runner to Linux, and then wondered why they had not done it a year earlier.
Change control earned its bad reputation fairly. A Wednesday meeting. A form with fourteen fields. A board of people signing off on work they cannot evaluate, three days after the window when the change would have been useful. Endpoint teams mostly skipped all of it, and I do not feel much guilt about that.
What the rest of the software industry replaced it with is smaller and much less ceremonious. A branch, a diff, one person who looks, a merge. That version fits a team of two. It survives somebody leaving. It produces an audit trail as a side effect of how the work already happens, rather than as a separate document nobody updates.
The mess I inherited was not evidence of a bad administrator. Any environment turns into that when changes leave no trace behind them, and mine had started heading the same way again within a year of cleaning it up, until the pipeline made leaving a trace the default instead of a chore.
Somebody is going to inherit your environment eventually. Leave them commit messages instead of a comment that says don't touch this part.
Foqal runs IT support inside Slack and Microsoft Teams, so the request that kicks off a change keeps a record of its own.
Get the latest insights on IT operations, AI, and workplace productivity delivered to your inbox.
See how Foqal can help your team deliver faster, smarter support.
Start Free TrialWhy category ambiguity, self-diagnosis and missing screenshots slow every help desk down, with a copy-paste request template. Plus the reciprocal list of what IT needs to stop doing.
An honest split between the AI IT support tasks that genuinely work now (classification, log triage, knowledge base answers) and the ones that fail badly, with the failure data and real incidents.
One org cut two hours of IT prep per device to zero and about 30 minutes of user setup. Then the hard problems moved into people's homes: bandwidth, missing cables, and old laptops that never came back.