Observability

Scaling Prometheus metrics with Grafana Mimir.

When metrics storage grows from gigabytes to terabytes, it stops being a monitoring problem and becomes an architecture problem. Here's the shape of the fix, without throwing away Prometheus.

Article

Prometheus is a good default for Kubernetes and cloud-native monitoring, and it stays good for a long time. What changes is everything around it: more clusters, more services and exporters, more high-cardinality labels, more active time series, longer retention requirements, more historical data that someone eventually wants to query. None of that is a Prometheus problem exactly — it's what happens when a single-node time-series database keeps absorbing growth it was never architected to store long-term.

What Grafana Mimir actually is

Grafana Mimir is a horizontally scalable, highly available, multi-tenant time-series database built for long-term storage of Prometheus and OpenTelemetry metrics. It's not a replacement for Prometheus — it's what sits underneath it once a single Prometheus instance (or even Prometheus plus Thanos-style federation) stops being the right shape for the retention and query load you actually have.

What it adds

  • Prometheus-compatible. Existing remote_write configuration and PromQL queries keep working — this is additive, not a migration off the tooling teams already know.
  • Horizontal scalability. Ingestion and querying scale across multiple components instead of depending on the capacity of one metrics server.
  • Long-term storage. Metrics land in object storage — S3, GCS or Azure Blob — instead of local disk, which is what actually makes large-scale retention affordable.
  • Multi-tenancy. Multiple teams, clusters or customers get tenant isolation inside one platform rather than one Prometheus per tenant.
  • High availability. The ingestion and query paths are built to survive component failure, which a single Prometheus process isn't.
  • Distributed querying. Long time ranges run through distributed query execution instead of one process scanning everything itself.

What the architecture looks like

Metrics still originate the same way — Prometheus or an OpenTelemetry pipeline scraping Kubernetes, VMs and applications. What changes is where they land after that:

Kubernetes / VMs / Applications
              |
              v
      Prometheus / OTel
              |
       remote_write / OTLP
              |
              v
         Grafana Mimir
       +---------------+
       |  Distributor  |
       |       |       |
       |    Ingester   |
       |       |       |
       |   Compactor   |
       |       |       |
       | Store Gateway |
       +-------+-------+
               |
               v
        Object Storage
       S3 / GCS / Azure
               |
               v
      Query Frontend
               |
               v
             Grafana
             + PromQL

The distributor accepts incoming writes and routes them; ingesters hold recent data in memory before it's flushed; the compactor merges and optimizes blocks in object storage; the store gateway and query frontend serve PromQL reads back out — across whatever time range object storage retains, not just what fits on local disk.

Prometheus doesn't go away

The point of this architecture isn't to replace the Prometheus ecosystem — it's to stop treating every individual Prometheus instance as the permanent storage layer. Split three ways, the responsibilities are straightforward: Prometheus collects and forwards metrics, Mimir scales and retains them, Grafana visualizes and analyzes them. Each piece keeps doing the job it's already good at.

For a platform running one cluster, none of this matters yet. For a platform running many clusters with thousands of services and years of retention requirements, it's the difference between metrics storage being a solved problem and metrics storage being the thing that pages someone at 2am because a Prometheus disk filled up. We design observability platforms with this distinction in mind from the start, rather than retrofitting it once a single Prometheus instance has already become a bottleneck.

Let's engineer your platform.

Tell us what you are running today.Tell us where you want to go.

We'll help design the path between them.