Taming observability cost at scale.

Telemetry bills grow quietly. Nobody decides to 10x the observability budget — it happens one new label, one new log line and one new service at a time, until a Prometheus-compatible backend is ingesting more data than the applications it's watching. The good news: most of that growth is reducible without losing the signal that actually matters during an incident.

Cardinality is the real cost driver, not volume

A metrics backend's cost scales with the number of unique time series, not the number of data points. A counter with a user_id or request_id label doesn't add one time series — it adds one per unique value, forever, until retention expires. The usual culprits:

• Unbounded labels: user IDs, trace IDs, raw URLs with path parameters left unparameterized.
• Per-pod labels on metrics that should be aggregated at the service level.
• Auto-instrumentation defaults that emit far more detail than any dashboard or alert consumes.

Before adding infrastructure to absorb the cost, audit the label set. A relabeling rule that drops or aggregates one high-cardinality label often cuts active series by an order of magnitude, with zero application code changes.

metric_relabel_configs:
  - source_labels: [request_id]
    regex: '.*'
    action: labeldrop

Logs: sample, don't silence

The instinct to cut logging costs is often "log less." A better instinct is "log selectively." Debug-level logs for a request that ends in a 5xx are worth keeping at full fidelity; the same debug logs for a clean 200 response rarely are. A sampling policy that keeps 100% of errors and a small percentage of successful requests preserves the ability to investigate incidents while cutting ingest volume substantially — often by more than half.

Structured logging matters here too: unstructured text is expensive to index and hard to query under pressure. A consistent JSON shape with a handful of indexed fields (service, severity, trace ID) is both cheaper and faster to search than free-text logs with a grep-and-pray query pattern.

Retention should not be one number for everything

Most teams set a single retention window — 30 days, 90 days — across every signal, because that's the default. In practice, different signals age differently:

• High-resolution metrics (10–15s scrape) are rarely queried past a few days except for the exact series involved in an incident.
• Downsampled metrics (5m/1h rollups) are what capacity planning and long-term trend dashboards actually use — keep these far longer, at a fraction of the storage cost.
• Logs and traces tied to a specific incident are valuable for postmortems; the rest ages out of usefulness quickly.

A tiered setup — short raw retention, longer downsampled retention, selective long-term archival for incident-linked data — routinely cuts storage cost well below a flat-retention policy, without anyone noticing a gap in what they can query.

Route before you store

Not every signal needs to reach the same backend. A pipeline (OpenTelemetry Collector, Vector, Fluent Bit) sitting in front of storage can apply cardinality limits, sampling and routing rules once, centrally, instead of depending on every service team to configure their own SDK correctly. That single control point is usually where the biggest, safest cost reductions happen — because it's enforced infrastructure-side, not a convention teams have to remember.