# Metrics with OpenTelemetry

> Turn on OTLP metrics, what is stored, reading them by aggregation, filter and group, the series limits that keep cost predictable, privacy and metric alerts.

Traces show one request and logs show what the code said. **Metrics** show a number over time: requests per second, memory in use, jobs waiting in a queue, free connections in a pool. Belvador takes them from the standard OpenTelemetry SDKs and the Collector over OTLP/HTTP, stores one point a minute on your own node, and lets you chart them and alert on them.

Metrics are **off by default**. They cost requests and have their own monthly budget, so you turn them on where you want them.

## Turn them on

1. **Ingest key.** In **Project settings → Ingest keys**, tick **Metrics** on the key your services use (or create a new key with it). A key without it is answered `403`.
2. **Exporter.** In the [standard variables](https://belvador.dev/en/docs/otel/quickstart#2-set-the-standard-variables), switch the metric exporter on:

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.belvador.dev/otlp/<project-id>
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer%20<your-ingest-key>
export OTEL_TRACES_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_METRICS_EXPORTER=otlp
export OTEL_METRIC_EXPORT_INTERVAL=60000
export OTEL_SERVICE_NAME=my-service
export OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production
```

Export once a minute (60 s): Belvador keeps one point a minute, so exporting more often only adds requests. Both temporalities work — cumulative (the default of every SDK) and delta.

3. **Your node** stores metrics from agent 0.1.37 on; nodes update themselves. Until then, metrics wait at the edge, encrypted to your node's key, and are delivered after the update.

> **Tested on 2026-10-27** with otelcol-contrib 0.163.1, telemetrygen v0.163.0: traces passed, logs passed, metrics passed.

The Collector run sends one metric of each type below and reads it back from the node. OpenTelemetry SDKs send the same OTLP format, but Belvador has not run their metric exporters yet, so they are not marked tested for metrics. Recipes for host, queue, business and Supabase metrics: [Metrics recipes](https://belvador.dev/en/docs/otel/metrics-recipes).

## What is stored

| OTLP type | Stored as | Typical instruments |
|---|---|---|
| Gauge | a level | `ObservableGauge`, `system.cpu.load_average.1m` |
| Sum, monotonic | a counter (increments) | `Counter`, `system.network.io`, `app.signups` |
| Sum, not monotonic | a level (cumulative) or increments (delta) | `UpDownCounter`, `system.memory.usage` |
| Histogram | a distribution with its buckets | `Histogram`, `http.server.request.duration` |
| Exponential histogram | a distribution with its buckets | SDKs configured for base-2 buckets |

**Summary** points are not stored; they are counted as *Unsupported metric type* (every current SDK exports histograms instead). Cumulative counters are turned into per-minute increments on your node; a counter that restarts (after a deploy) counts on from its new value instead of showing a negative drop.

## Read them

The **Metrics** page lists the metrics your project sent, with their type, unit and number of series. Pick one, then:

- **Aggregation** — `avg`, `sum`, `min`, `max`, `count`, `rate` (per second, for counters and histograms), or `p50`/`p90`/`p95`/`p99` (histograms). A counter opens as a rate, a histogram as p95, a level as the average.
- **Filters** — up to 5 attribute conditions, the same as in Traces and Logs.
- **Group by** — up to 2 attribute keys; each value set is one line on the chart (the first 50 groups are drawn).
- **Step** — one point per minute up to one per day. One query covers at most 7 days and 10,000 points.

Histogram points carry **exemplars** when your SDK records them: the trace of one request in that bucket. Click one to open the trace.

## Limits that keep cost predictable

A **series** is one metric name with one set of attribute values. The number of series is what costs disk and memory, so your node caps it:

| | Default plan |
|---|---|
| Active series per node (seen in the last 24 h) | 10,000 |
| Metric names per node | 1,000 |
| Attributes per series | 32, values up to 256 characters |
| Kept for | 14 days, one point a minute |
| Monthly volume | 2 GB (compressed request bodies), separate from traces and logs |

Points of a **new** series above the cap are not stored and counted as *Over the series limit*; series that already exist keep working, and the Metrics page names the metrics that hit the cap. The usual cause is an attribute with unbounded values — a user id, an order id, a full URL. Keep those on spans and logs, and give metrics bounded attributes (`http.route`, not the URL).

The Metrics page shows how many series and names your node holds against these caps; retention and the monthly volume are in **Project settings → Traces, logs and metrics received**.

## Privacy

Metric attributes go through the same [privacy rules](https://belvador.dev/en/docs/otel/privacy) as span attributes **before** a series is formed: a removed attribute never becomes part of a series, and a hashed one groups by its hash. Values are stored only on your node. Belvador's database receives counts and sizes per day, never names, values or attribute values.

## Alert on a metric

Create a **Metric** monitor in **Monitors → Create monitor**, or press **Create monitor** on the Metrics page to carry the current query over. The monitor aggregates one metric over a window (for example *p95 of `http.server.request.duration` over 5 minutes above 0.8 s*) and alerts when it is above, below or outside a range. With **group by**, each value set is judged on its own, so one slow route alerts without the others. See [Signal monitors](https://belvador.dev/en/docs/monitors/signals#metric-monitors).

## From the API or an AI agent

`GET /v1/metrics/names`, `GET /v1/metrics/series` and `GET /v1/metrics/query` read the same data with a token that has `metrics:read`; the MCP tool `query_metrics` lists metrics without a name and queries one with a name. See the [API reference](https://belvador.dev/en/docs/reference/api) and [MCP](https://belvador.dev/en/docs/mcp).

## What it is not

- **No browser metrics.** Ingest keys are secret; web vitals come later through the browser SDK, not through OTLP.
- **No Prometheus endpoint.** Prometheus and other sources are read by the [Collector](https://belvador.dev/en/docs/otel/metrics-recipes#supabase-metrics) and forwarded as OTLP.
- **No long-term storage.** Points are kept for the plan's retention at one-minute resolution; there is no downsampling to keep years of data.