Every growing company ends up paying for software that watches its other software. The bill arrives as storage, licences or engineers’ hours, and it tends to grow faster than the products it observes. ObservabilityOps is the name some teams use for the work of running that watching layer deliberately: deciding what gets collected, how long it stays, who pays and how anyone knows the layer itself is healthy. Below are eight evidence-led ideas, each with the numbers behind it, for managers and engineers who would like fewer surprises on the invoice.
1. The practice is real even though the label is not standard
It helps to be honest about terminology. The Ops Atlas page covering ObservabilityOps states plainly that no standards body, certification or vendor programme defines the term. Its reading is narrow: operating the observability system itself. The work nonetheless exists under other names, from dedicated observability teams at software firms to a government platform team that offers logging, metrics and alerting to other departments. Judge the idea by what it does, not by whether the badge is famous.
2. Monitoring and observability answer different questions
Monitoring checks conditions you wrote down in advance. Observability, as OpenTelemetry phrases it, is the ability to understand a system’s internal state from the outputs it produces. DORA research makes a similar split, describing monitoring as predefined and observability as covering properties not defined ahead of time. For a budget holder the practical consequence is that you pay for breadth of data in the second case, which is exactly why somebody has to govern it.
3. Spend scales with how much you run
A survey by Grafana in 2025 covering 1,255 respondents reported mean observability spend at 17 percent of compute spend, with a median of 10 percent. Averages hide wide variation, but the figures suggest the layer is far from a rounding error. If your own ratio is unknown, that is the first gap to close, and the cheapest fix is a monthly report that divides the observability bill by something the business understands, such as requests served. The Atlas works one example: a monthly cost of 18,000 against 900 million requests comes to 20 per million.
A sensible reading is to treat the survey median as a loose benchmark and the mean as a warning about the long tail. If your ratio sits far above either figure, ask whether the cause is genuine coverage or accumulated clutter such as dashboards nobody opens and debug logs kept for months. Finance teams like this framing because it turns a murky technical line into a percentage they can trend alongside other overheads.
4. Cardinality is the quiet multiplier
In metric systems, every unique combination of label values becomes its own time series. Add a customer identifier with fifty thousand values to a counter and a single metric can balloon into roughly 1.8 million series, according to the Atlas’s budget exercise in which a team’s total reached 1,946,880 against an allowance of 200,000. Removing that one label brought the total down to 146,916. It is the sort of arithmetic that turns an abstract warning into a decision you can defend in a planning meeting.
The lesson for non-engineers is that growth in a bill is not always growth in usage. A one-line code change by a well-meaning developer can add more series than a year of new customers would. That is why the Atlas pairs the budget exercise with limits applied at the point of ingestion, so a team that goes over its allowance sees a failure in a pre-release check and not a shock at month end.
5. Retention is a purchasing decision, not a default
Prometheus keeps data for 15 days unless told otherwise, and most teams never revisit that figure. The Atlas offers a rule of thumb for disk planning: retention in seconds, multiplied by samples per second, multiplied by one to two bytes per sample. In its example the same workload needed 84.1 to 168.2 GB with the heavy label in place against 6.3 to 12.7 GB without it. Tiering data into hot, warm and archive storage lets you keep expensive fast disks for what people actually query.
Ask a simple question of every data type: who looked at it after day seven, and what did they decide? Where the honest answer is nobody, shorter retention or cheaper storage is a straightforward saving. Where it is auditors or incident reviewers, keep the archive but move it off premium disks. Retention schedules are easy to explain to stakeholders because they read like any records policy.
6. Alerts have a cost too
An alert nobody acts on still consumes attention. The Atlas recommends paging on symptoms users can feel, and it cites Google’s SRE guidance that signals which are never used are candidates for removal. One of its sample measurements is page actionability: 15 useful pages out of 42 is 35.7 percent, meaning most interruptions were noise. Tracking that ratio quarterly gives managers a figure to improve rather than a vague sense that on-call is exhausting.
Some of the damage never shows up on an invoice. Engineers who are woken by pages that need no action learn to ignore pages, and the one real emergency then arrives in a crowd of false alarms. A quarterly review that deletes or rewrites low-value alerts is therefore both a cost measure and a reliability measure, and it is among the cheapest changes on this list.
7. Prove the alerting path works
A fire alarm that has silently lost power is worse than none. The Atlas describes a dead-man’s-switch pattern, often called a watchdog: a test alert that fires constantly, and an external check that raises the alarm if it ever stops. It also argues that one end-to-end test of the alert path beats a pile of per-component checks. For a business owner this is a cheap insurance policy and a good question to put to any team: how would we know if alerting broke?
The same idea extends to freshness. A small synthetic event sent through the pipeline on a schedule, with a check on how long it takes to appear, tells you whether the data on screen is current. Without it, a delayed pipeline produces graphs that look reassuring and are hours old, which is a trap for anyone reading them during a live problem.
8. Ownership beats tooling
Teams that grow quickly often buy another product when the real gap is accountability. The Atlas lays out a maturity ladder that climbs from ad hoc, through a single owner with a watchdog, to standardised practice with limits, to a governed internal product with cost showback, and finally to an optimised stage in which unit cost falls while coverage holds. If you are weighing options, the discussion of choosing business technology solutions sensibly is a useful companion, since the same questions about fit and ownership apply. For the planning side, this walkthrough on how digital growth planning is structured shows where a recurring review slot would sit.
One caution applies. The Atlas, echoing DORA, warns against a central team owning everyone’s alerts, because the people who know a service best should decide when it deserves a page. The owner of the watching layer sets the rules of the road and keeps the road in repair; the producing teams still drive on it.
Three moves for the coming week
No formal programme is required to begin. Choose three actions and give each a date:
- Name one accountable person for the watching layer and write down what they decide.
- Produce a single unit-cost figure from last month’s invoice.
- List the ten noisiest alerts and retire or rewrite the ones nobody acted on.
After thirty days, compare the numbers. If cost per unit is flat or falling and fewer pages are being ignored, you are moving in the right direction. If not, you have the data to ask a pointed question at the next budget review instead of a vague one.
A note on caution
Several of these figures come from vendor-neutral surveys and worked examples, not from controlled studies of your own estate, so treat them as prompts for measurement. The Atlas itself warns that phrases such as “AI observability” or “observability pipeline” are marketing until tested on your own data. A healthy scepticism, plus a few well-chosen numbers, is a better starting point than any promise of effortless insight.

