Skip to content

PromQL Quick Start

PromQL is the query language of the Prometheus monitoring system, known for its concise syntax and powerful capabilities. Unlike traditional SQL, PromQL has a unique syntax structure.

This article will introduce common query usages of PromQL.

Selecting Time Series with PromQL

In PromQL, selecting time series is straightforward—just write the series name directly. For example, the following query returns all time series with the name node_network_receive_bytes_total:

node_network_receive_bytes_total

This name corresponds to the node_exporter metric, which records the number of bytes received by various network interfaces. A simple query might return time series with the following labels, such as for eth0, eth1, and eth2 network interfaces:

node_network_receive_bytes_total{device="eth0"}
node_network_receive_bytes_total{device="eth1"}
node_network_receive_bytes_total{device="eth2"}

Different labels are enclosed in curly braces, such as {device="eth0"}, {device="eth1"}, {device="eth2"}.

Measurement

In Guance, all Metrics belong to a Measurement, and we manage the lifecycle of Metrics by Measurement. However, in Prometheus, there is no concept of Measurement. When reporting data through Datakit, you can manually configure the Measurement or generate it automatically based on prefixes.

Taking the node_network_receive_bytes_total metric as an example, assuming we generate it through Datakit's automatic rules, this metric will be split into Measurement and Field parts, which are node and network_receive_bytes_total, respectively.

The query format will also change slightly. Continuing with the example of different network interfaces:

node:network_receive_bytes_total{device="eth0"}
node:network_receive_bytes_total{device="eth1"}
node:network_receive_bytes_total{device="eth2"}

You can observe that the metric name filtering format has changed to measurement:field, connected by a colon.

Filtering by Labels

A single metric name may correspond to multiple time series with different label sets, as in the example above. How to select only the time series matching {device="eth1"}? Simply mention the desired label in the query:

node:network_receive_bytes_total{device="eth1"}

If you want to select all time series for devices other than eth1, just replace = with != in the query:

node:network_receive_bytes_total{device!="eth1"}

How to select time series for devices starting with eth? Just use a regular expression:

node:network_receive_bytes_total{device=~"eth.+"}

Filters can include any Go-compatible regular expressions (also known as RE2).

To select all time series for devices not starting with eth, simply replace =~ with !~:

node:network_receive_bytes_total{device!~"eth.+"}

Filtering by Multiple Labels

Label filters can be combined. For example, the following query will return only the time series on the node42:9100 instance for devices starting with eth:

node:network_receive_bytes_total{instance="node42:9100", device=~"eth.+"}

Label filters are combined using the and operator, meaning "return time series that match this filter and that filter." How to implement the or operator? Currently, PromQL lacks an or operator for combining label filters, but in most cases, it can be replaced with regular expressions. For example, the following query will return time series for eth1 or lo devices:

node:network_receive_bytes_total{device=~"eth1|lo"}

Filtering Metric or Measurement Names by Regular Expressions

Measurement and metric names are essentially ordinary labels with special names: __measurement__ and __field__. Therefore, you can apply regular expressions to these labels to filter the data you want.

For example, query all time series in the node Measurement with metric names network_receive_bytes_total or network_transmit_bytes_total:

{__measurement__="node", __field__=~"network_(receive|transmit)_bytes_total"}

Or query time series with metric names network_receive_bytes_total or network_transmit_bytes_total distributed across node1 and node2 Measurements:

{__measurement__=~"node1|node2", __field__=~"network_(receive|transmit)_bytes_total"}

Comparing Current Data with Historical Data

PromQL allows querying historical data and combining or comparing it with current data. Just add offset to the query. For example, the following query will return all time series with the name node:network_receive_bytes_total from one week ago:

node:network_receive_bytes_total offset 7d

The following query will return points where the current GC overhead exceeds 1.5 times the GC overhead from one hour ago:

go:memstats_gc_cpu_fraction > 1.5 * (go:memstats_gc_cpu_fraction offset 1h)

Calculating Rates

You may notice that the graphs for all the above queries plot continuously increasing lines:

The usability of such graphs is close to zero because they show ever-increasing counter values that are difficult to interpret, while we need to plot network bandwidth graphs. PromQL has a function rate that calculates the per-second rate for all matching time series:

rate(node:network_receive_bytes_total[5m])

Now such graphs become understandable:

What does [5m] in the query mean? It is the time duration (d).

In our case, it is 5 minutes—looking back when calculating the per-second rate for each graph point. The simplified rate calculation for each point is as follows: (Vcurr-Vprev) / (Tcurr-Tprev), where Vcurr is the value at the current point -Tcurr, and Vprev is the value at time Tprev = Tcurr-d.

If this seems too complicated, just remember that a higher d smooths the graph, while a lower d introduces more noise to the graph.

Guance uses a PromQL extension syntax MetricsQL (thanks to VictoriaMetrics for open-sourcing it!), in which case [d] can be omitted. In this case, it equals the duration between two consecutive points on the graph (also known as "step"):

rate(node:network_receive_bytes_total)

So when you are unsure about the duration to fill after rate, feel free to omit it directly.

Notes on rate

rate removes the metric name but retains all labels of the internal time series.

Do not apply rate to time series that may fluctuate up and down. Such time series are called Gauges. rate must only be applied to Counters, which always rise but may sometimes reset to zero (e.g., during service restarts).

Do not use irate instead of rate, as it cannot capture spikes, and it is not much faster than rate.

Arithmetic Operators

PromQL supports all basic arithmetic operations:

  • Addition (+)
  • Subtraction (-)
  • Multiplication (*)
  • Division (/)
  • Modulo (%)
  • Exponentiation (^)

This allows for various transformations. For example, converting bytes/second to bits/second:

rate(node:network_receive_bytes_total[5m]) * 8

Additionally, it enables calculations across time series. For example, a Flux query can be simplified to the following PromQL query:

co2 * (((temp_c + 273.15) * 1013.25) / (pressure * 298.15))

When combining multiple time series using arithmetic operators, you need to understand the matching rules. Otherwise, the query may fail or produce incorrect results. The basics of matching rules are simple:

  • The PromQL engine strips the metric names from all time series on both sides but retains the labels;
  • For each time series on the left, the PromQL engine searches for the corresponding time series with the same label set on the right, applies the operation to each data point, and returns the resulting time series with the same label set. If there is no match, the time series is removed from the result.

The matching rules can be enhanced with ignoring, on, group_left, and group_right modifiers. Although the use of these modifiers is complex, they are not needed in most cases.

PromQL Comparison Operators

PromQL supports the following comparison operators:

  • Equal to (==)
  • Not equal to (!=)
  • Greater than (>)
  • Greater than or equal to (>=)
  • Less than (<)
  • Less than or equal to (<=)

These operators can be applied to any PromQL expression, just like arithmetic operators. The result of a comparison operation is the time series of matching data points. For example, the following query will return only the time series where the bandwidth is less than 2300 bytes/second:

rate(node:network_receive_bytes_total[5m]) < 2300

This will result in the following graph with gaps where the bandwidth exceeds 2300 bytes/second:

rate(node:network_receive_bytes_total[5m]) < 2300

The result of the comparison operator can be enhanced with the bool modifier:

rate(node:network_receive_bytes_total[5m]) < bool 2300

In this case, the result will contain 1 for true comparisons and 0 for false comparisons:

rate(node:network_receive_bytes_total[5m]) < bool 2300

Aggregation and Grouping Functions

PromQL allows aggregating and grouping time series. Time series are grouped by the given label set, and then the given aggregation function is applied to each group. For example, the following query will return the total ingress traffic grouped by instance on all nodes where node_exporter is installed:

sum(rate(node:network_receive_bytes_total[5m])) by (instance)

Using Gauges

Gauges are time series that can fluctuate up and down at any time, such as memory usage, temperature, or pressure. When plotting graphs for dashboards, it is often necessary to display the minimum, maximum, average, and/or quantile values for each point. PromQL provides the following functions to meet these needs:

For example, the following query will plot the minimum free memory for each point on the graph:

min_over_time(node:memory_MemFree_bytes[5m])

MetricsQL adds rollup_* functions to PromQL, which automatically return min, max, and avg values when applied to Gauges. For example:

rollup(node:memory_MemFree_bytes)

Label Manipulation

PromQL provides two functions for modifying, beautifying, deleting, or creating labels:

Although the use of these functions is complex, they allow powerful dynamic manipulation of the labels of selected time series. MetricsQL extends these capabilities with more convenient label manipulation functions:

  • label_set: Set additional labels for time series
  • label_del: Delete given labels from time series
  • label_keep: Delete all labels from time series except the given ones
  • label_copy: Copy label values to other labels
  • label_move: Rename labels
  • label_transform: Replace all substrings matching the given regular expression with template replacements
  • label_values: Return values from the given label

Returning Multiple Results from a Single Query

Sometimes it is necessary to return multiple results from a single PromQL query. This can be achieved using the or operator. For example, the following query will return all time series with the names metric1, metric2, and metric3:

metric1 or metric2 or metric3

MetricsQL simplifies the process of returning multiple results by just listing them in ():

(metric1, metric2, metric3)

Note that arbitrary PromQL expressions can be placed there, not just metric names.

There is a common pitfall when combining expression results: results with duplicate label sets will be skipped. For example, the following query will skip sum(b) because both sum(a) and sum(b) have the same label set—they have no labels at all:

sum(a) or sum(b)

Conclusion

PromQL is an easy-to-use yet powerful query language for time series databases. Compared to SQL, InfluxQL, or Flux, it allows writing typical TSDB queries in a concise and clear manner.

This tutorial does not cover all features of PromQL, as some are not very commonly used:

You can continue learning PromQL using this PromQL cheat sheet.

The main content of this article is translated from PromQL tutorial for beginners and humans, and Guance is also using the MetricsQL engine implementation open-sourced by VictoriaMetrics. Thanks again to VictoriaMetrics!

Further Reading

Feedback

Is this page helpful? ×