Aggregates

aggregate cuts a range into buckets of every and answers each with one exact value: an increase, a rate, an average, a percentile. It is what a dashboard draws: requests per minute by route, the p95 latency of each five minutes, the average queue depth of each hour.

const traffic = await store.metrics.aggregate({
	name: 'http_requests_total',
	since: '24h',
	every: '1h',
	op: 'increase',
	by: ['route'],
})
// [{ name, labels: { route: '/notes' }, buckets: [{ at, value: 1840, samples: 240, resets: 0 }, …] }, …]

const p95 = await store.metrics.aggregate({ name: 'http_request_ms', since: '1h', every: '5m', op: 'p95' })
// [{ name, labels: {}, buckets: [{ at, lower: 25, upper: 50, samples: 3812 }, …] }]

traffic = await store.metrics.aggregate(name="http_requests_total", since="24h", every="1h", op="increase", by=["route"])
p95 = await store.metrics.aggregate(name="http_request_ms", since="1h", every="5m", op="p95")

traffic, err := metrics.Aggregate(ctx, store, metrics.Query{Name: "http_requests_total", Since: 24 * time.Hour,
	Every: time.Hour, Op: metrics.Increase, By: []string{"route"}})
p95, err := metrics.Aggregate(ctx, store, metrics.Query{Name: "http_request_ms", Since: time.Hour, Every: 5 * time.Minute, Op: metrics.Percentile(95)})

let stats = store.metrics()?;
let traffic = stats
    .query()
    .name("http_requests_total")
    .since(Duration::from_hours(24))
    .every(Duration::from_hours(1))
    .by(&["route"])
    .aggregate(Op::Increase)?;
let p95 = stats.query().name("http_request_ms").since(Duration::from_hours(1)).every(Duration::from_mins(5)).aggregate(Op::Percentile(95.0))?;

#Ops

Every value is exact and rounded once: a sum is the exact sum of its samples, rounded to the nearest float, whatever order or blocks they were kept in.

#Buckets add up to the range

Buckets are spans of every counted from 1970 in UTC, so 1h buckets begin on the hour. An increase, a rate and a delta count each step between two samples in the bucket where the step ends, and the first bucket takes its step from the newest sample before the range:

a counter growing by 1 a second, sampled every 15 s, read 10:00 to 11:00
1h buckets   3600
5m buckets   300 + 300 + 300 + … + 300 = 3600
1m buckets    60 +  60 +  60 + … +  60 = 3600

Buckets of any width add up to the range, and two adjacent ranges to the range they make. Nothing is interpolated or extrapolated to a bucket's edges, unlike Prometheus' increase, whose answer changes with the window.

#Partial buckets and gaps

http_requests_total   100 on 1 October, nothing since, 300 at 11:59 on 11 October
11:00 to 12:00        { value: 200, gap: true }

What a counter grew over a gap is counted in the bucket after it, since when it grew is not known. A bucket without samples is left out of the answer.

#Percentiles

• p95 of a gauge is the sample at that rank, nearest rank: a value that was stored, exact.
• p95 of a timer is the bucket of its spread that holds the rank: lower and upper, both exact, and no value, since any one number would be read as the percentile. Past the last bound upper is Infinity.
• estimate: true adds estimate, a point between them interpolated as Prometheus' histogram_quantile does, named so nobody takes it for a measurement.
• A timer without buckets has no percentile: invalid, naming it.
• p50 through p99.9 and any rank between; p99.9 ranks at 999 in 1,000 exactly.

#Groups

Without by, every series of the name is one group and one answer. by keeps the labels it names; without keeps all but those; without: [] keeps every series apart.

Each series' bucket is computed exactly, then joined, and the group's value rounded once: counts, sums and increases add; avg divides every sample's sum by every sample's count, never averaging averages; p95 ranks every sample of the group, or adds a timer's bucket counts before it ranks. last adds up too: the last of several queues' depths is their total at the bucket's end.

const backlog = await store.metrics.aggregate({ name: 'queue_depth', since: '1d', every: '1h', op: 'last' })

backlog = await store.metrics.aggregate(name="queue_depth", since="1d", every="1h", op="last")

backlog, err := metrics.Aggregate(ctx, store, metrics.Query{Name: "queue_depth", Since: 24 * time.Hour, Every: time.Hour, Op: metrics.Last})

let backlog = stats.query().name("queue_depth").since(Duration::from_hours(24)).every(Duration::from_hours(1)).aggregate(Op::Last)?;

#A heatmap

const heat = await store.metrics.aggregate({ name: 'http_request_ms_bucket', since: '1h', every: '1m', op: 'increase', by: ['le'] })

heat = await store.metrics.aggregate(name="http_request_ms_bucket", since="1h", every="1m", op="increase", by=["le"])

heat, err := metrics.Aggregate(ctx, store, metrics.Query{Name: "http_request_ms_bucket", Since: time.Hour, Every: time.Minute,
	Op: metrics.Increase, By: []string{"le"}})

let heat = stats.query().name("http_request_ms_bucket").since(Duration::from_hours(1)).every(Duration::from_mins(1)).by(&["le"]).aggregate(Op::Increase)?;

How many durations fell under each bound, each minute: a timer's buckets are counters, one a bound.

#Fast over long ranges

The store packs a series' samples into blocks where whole hours begin, each with an exact summary. A bucket that holds whole blocks answers from their summaries, so a month drawn by the hour decodes almost nothing; a bucket finer than a block, and a gauge's percentile always, decode its samples. explain says what a query will cost.

#Errors

#See also

• Metrics: the engine's overview.
• Instruments: what writes these series.
• How logs and metrics pack in the Handbook.