Adopting machine-learning assistance in a security operations centre is less a purchase than a sequence of controlled experiments. This guide lays out that sequence for AIOps for Security — anomaly detection, alert scoring, LLM triage agents — as six steps, each with an evidence gate that must pass before the next one opens. The gates follow the autonomy ladder now used across the field, and the arithmetic underneath them is decades old and routinely skipped; the Ops Atlas page on AIOps for Security collects those gates in one place. Teams that skip them pay later, usually in the currency of a missed intrusion buried inside a cheerful automation percentage.
Step 1: Inventory the actions before the models
Automation decisions attach to actions, not to products. List every move your team makes in a week: closing phishing tickets, disabling accounts, isolating endpoints, resetting tokens. Each action class earns its own autonomy level and its own evidence file. Where intelligent operations already exist for infrastructure work — the pattern we described in The Blueprint for Intelligent IT Operations: AIOps for SRE and DevOps Engineers — borrow the habit of naming the action before naming the tool. At this stage everything stays at L0: the machine observes, humans decide.
Step 2: Count the base rate for each category
Base rates govern whether any of this can work. Take two categories and count genuine positives against total volume. The canonical teaching example: a stream of 50 million daily events holding 50 genuinely malicious items. Let the false-alarm rate sit at 0.0001 and the detector flags roughly 5,000 innocent items daily, so alerts are correct only 0.9% of the time — about 112 reads per real attack. Tighten the false-alarm rate to 0.000001 and precision leaps to 47.4%. Axelsson showed the same cliff twenty-five years earlier: at a base rate of 0.00002, moving the false-alarm rate from 0.00001 to 0.001 drags precision from 66.7% down to about 1.4%. If your category’s base rate is worse than the detector’s noise floor, no amount of tuning rescues it.
Step 3: Shadow-score with blind labels
Run the candidate model beside the queue in its first month, dashboards read-only. Humans label a blind set the model never sees; the model’s verdicts get graded against them afterwards. Days 1 through 30 of a standard plan look exactly like this: export prior dispositions, pick the categories, build the labelled set, score quietly, and write down what the counts say about volume and noise. Resist the urge to let the ranker choose what it trains on — a model that shapes its own training data is grading its own homework, and no gate downstream can repair that.
Step 3b: Budget for drift, not just launch
Day 31 onward shifts from building to watching: per-category evaluation on a monthly rhythm, gate values written down and signed, drift monitors on the features that move most. Day 61 onward adds the adversarial side — purple-team canaries seeded into the queues, incident-level recall checks, and a cost baseline so the program can prove its savings are real rather than transferred onto someone else’s queue. It is not glamorous work, and it is precisely why the automation still functions in month nine.
Step 4: Grant recommendations first (L1)
The first earned privilege is a suggestion, not an action. The threshold is explicit: at least 200 labelled alerts evaluated, of which 30 or more are true positives, before the system may recommend a disposition to a human. Below that sample size, the numbers carry almost no information — which is precisely why vendors rarely mention them. Recommendations also make an honest interface: every accepted or rejected suggestion becomes new labelled evidence at almost zero collection cost.
Step 5: Auto-close only behind evidence (L2)
Closing tickets without a human is the step where savings live and where mistakes compound. The gate: a demonstrated miss rate no worse than 2% across a minimum of 150 true positives, canaries in the flow, a blind holdout untouched, and one-click reopen available to every analyst. Sample-size mathematics deserves respect here. With zero observed errors across n cases, the true rate still sits near 3 divided by n; demonstrating a 2% ceiling with a single miss in hand takes 236 cases, and two misses require 313. Auditing a random 5% slice of about 1,845 quarterly closes and finding nothing wrong bounds the miss rate only near 38% — which is why the canary contract matters: plant fifty decoys a quarter, and pull auto-close the moment a third one disappears.
Step 6: Reversible actions, then stop (L3 and L4)
Acting on infrastructure — disabling a service account, quarantining a host — opens the next gate under stricter terms: wrong actions held to 1% or tighter across a minimum of 300 verdicts where the system wanted to act, a rollback procedure that has been rehearsed rather than merely documented, and hard rate limits. The ladder ends there. Irreversible moves are never handed to software; that tier stays human by design, not by temporary caution.
Where AIOps for Security programs stall
Three places, in our experience. The calendar: Sentinel’s Fusion engine is being retired in favour of Defender XDR correlation, and every workspace must complete the move by 31 March 2027 — programs that treat migration as an afterthought lose their correlation history mid-flight. The metric: reporting “percent automated” as the headline number rewards shutting tickets, not catching attackers. And the sample: precision quoted at 99.8% on closes says nothing about the 2 true positives missed out of 71 in one documented blind trial — a 2.8% miss rate whose 95% upper bound near 8.6% failed that very 2% gate. Vendors quote the first number; you live with the second.
What AIOps for Security asks of governance
Gate values are board-level artefacts once money and breaches ride on them, and the evaluation cadence — monthly, per category, never pooled — deserves the same standing as access reviews. Security leadership frameworks such as the CISM body of knowledge already cover this ground; our overview in Why CISM Matters for Enterprise Security maps the overlap. Publish the register of which action classes hold which autonomy level, who signed the evidence, and when it expires. Models drift, tenants change, and a gate earned in March says little about October.
Nothing in this sequence is exotic. It is measurement, patience, and the willingness to keep a model on a leash one notch longer than the vendor recommends — which, historically, is exactly where the saves happen.
The gates at a glance
- Inventory action classes; everything starts at L0, observe-only.
- Count base rates per category before believing any precision claim.
- Shadow-score against blind labels for a full month.
- L1 recommendations: 200+ labelled alerts, 30+ true positives.
- L2 auto-close: miss bound 2% or better on 150+ true positives, canaries live, one-click reopen.
- L3 reversible actions: wrong-action bound held to 1% across 300+ verdicts, rehearsed rollback, rate limits.
One more gate fits no checklist: the decision score some teams apply before buying at all. Weighted 0-to-3 answers across evidence, exposure, and reversibility top out at 27; scores above 14 argue for shadow mode only, 9 to 14 for a time-boxed pilot, and anything under 9 for a polite decline.
