Skip to content

PromQL Quick Start

PromQL is the query language for the Prometheus monitoring system, known for its concise syntax and powerful capabilities. Unlike traditional SQL, PromQL has a unique syntax structure.

This article will introduce common query usages of PromQL.

Selecting Time Series with PromQL

In PromQL, selecting time series is straightforward—simply write the series name directly. For example, the following query will return all time series with the name node_network_receive_bytes_total:

node_network_receive_bytes_total

This name corresponds to the node_exporter metric, which records the number of bytes received by various network interfaces. A simple query might return time series with labels such as eth0, eth1, and eth2:

node_network_receive_bytes_total{device="eth0"}
node_network_receive_bytes_total{device="eth1"}
node_network_receive_bytes_total{device="eth2"}

Different labels are enclosed in curly braces, such as {device="eth0"}, {device="eth1"}, {device="eth2"}.

Measurement

In Guance, all Metrics belong to a Measurement, and we manage the lifecycle of Metrics by Measurement. In Prometheus, there is no concept of Measurement. When reporting data through Datakit, you can manually configure the Measurement or generate it automatically based on prefixes.

Taking the node_network_receive_bytes_total metric as an example, assuming we use Datakit's automatic rules to generate it, this metric will be split into Measurement and Field, namely node and network_receive_bytes_total.

The query will also change slightly, continuing with different network interfaces as an example:

node:network_receive_bytes_total{device="eth0"}
node:network_receive_bytes_total{device="eth1"}
node:network_receive_bytes_total{device="eth2"}

You can observe that the metric name filtering format has now changed to measurement:field, connected by a colon.

Filtering by Labels

A single metric name may correspond to multiple time series with different label sets, as in the example above. How to select only the time series matching {device="eth1"}? Simply mention the desired label in the query:

node:network_receive_bytes_total{device="eth1"}

If you want to select all time series for devices other than eth1, just replace = with != in the query:

node:network_receive_bytes_total{device!="eth1"}

How to select time series for devices starting with eth? Simply use a regular expression:

node:network_receive_bytes_total{device=~"eth.+"}

Filters can include any Go-compatible regular expressions (also known as RE2).

To select all time series for devices not starting with eth, just replace =~ with !~:

node:network_receive_bytes_total{device!~"eth.+"}

Filtering by Multiple Labels

Label filters can be combined. For example, the following query will return only the time series on the node42:9100 instance for devices starting with eth:

node:network_receive_bytes_total{instance="node42:9100", device=~"eth.+"}

Label filters are combined using the and operator, meaning "return time series that match this filter and that filter." How to implement the or operator? Currently, PromQL lacks an or operator for combining label filters, but in most cases, it can be replaced with regular expressions. For example, the following query will return time series for eth1 or lo devices:

node:network_receive_bytes_total{device=~"eth1|lo"}

Filtering by Regular Expressions on Metric or Measurement Names

Measurement and metric names are essentially ordinary labels with special names: __measurement__ and __field__, so you can apply regular expressions to these labels to filter the data you want.

For example, query all time series in the node Measurement with metric names network_receive_bytes_total or network_transmit_bytes_total:

{__measurement__="node", __field__=~"network_(receive|transmit)_bytes_total"}

Or query time series with metric names network_receive_bytes_total or network_transmit_bytes_total distributed across node1 and node2 Measurements:

{__measurement__=~"node1|node2", __field__=~"network_(receive|transmit)_bytes_total"}

Comparing Current Data with Historical Data

PromQL allows querying historical data and combining or comparing it with current data. Simply add offset to the query. For example, the following query will return all time series with the name node:network_receive_bytes_total from one week ago:

node:network_receive_bytes_total offset 7d

The following query will return points where the current GC overhead is more than 1.5 times the GC overhead from one hour ago:

go:memstats_gc_cpu_fraction > 1.5 * (go:memstats_gc_cpu_fraction offset 1h)

Calculating Rates

You might notice that the graphs for all the queries above plot continuously increasing lines:

The usability of such graphs is close to zero, as they show ever-increasing counter values that are difficult to interpret, while we need to plot network bandwidth. PromQL has a function rate that calculates the per-second rate for all matching time series:

rate(node:network_receive_bytes_total[5m])

Now such graphs become understandable:

What does [5m] in the query mean? It is the time duration (d).

In our case, it is 5 minutes—looking back when calculating the per-second rate for each graph point. The simplified rate calculation for each point is: (Vcurr-Vprev) / (Tcurr-Tprev), where Vcurr is the value at the current point -Tcurr, and Vprev is the value at time Tprev = Tcurr-d.

If this seems too complicated, just remember that a higher d smooths the graph, while a lower d adds more noise to the graph.

Guance uses a PromQL extension syntax MetricsQL (thanks to VictoriaMetrics for open-sourcing it!), in which case, [d] can be omitted. In this case, it equals the duration between two consecutive points on the graph (also known as "step"):

rate(node:network_receive_bytes_total)

So when you're unsure how to fill in the duration after rate, feel free to omit it directly.

Notes on rate

rate removes the metric name but retains all labels of the internal time series.

Do not apply rate to time series that may fluctuate up and down. Such time series are called gauges. rate must only be applied to counters, which always rise but may sometimes reset to zero (e.g., during service restarts).

Do not use irate instead of rate, as it cannot capture spikes, and it is not much faster than rate.

Arithmetic Operators

PromQL supports all basic arithmetic operations:

  • Addition (+)
  • Subtraction (-)
  • Multiplication (*)
  • Division (/)
  • Modulo (%)
  • Exponentiation (^)

This allows for various transformations. For example, converting bytes/second to bits/second:

rate(node:network_receive_bytes_total[5m]) * 8

Additionally, it enables calculations across time series. For example, a Flux query can be simplified to the following PromQL query:

co2 * (((temp_c + 273.15) * 1013.25) / (pressure * 298.15))

When combining multiple time series using arithmetic operators, it's important to understand the matching rules. Otherwise, the query may fail or produce incorrect results. The basic matching rules are simple:

  • The PromQL engine strips the metric name from all time series on both the left and right sides but retains the labels;
  • For each time series on the left, the PromQL engine searches for the corresponding time series with the same label set on the right, applies the operation to each data point, and returns the resulting time series with the same label set. If there is no match, the time series is removed from the result.

Matching rules can be enhanced with ignoring, on, group_left, and group_right modifiers. Although the use of these modifiers is somewhat complex, they are not needed in most cases.

PromQL Comparison Operators

PromQL supports the following comparison operators:

  • Equal (==)
  • Not equal (!=)
  • Greater than (>)
  • Greater than or equal to (>=)
  • Less than (<)
  • Less than or equal to (<=)

These operators can be applied to any PromQL expression, just like arithmetic operators. The result of a comparison operation is a time series of matching data points. For example, the following query will return only the time series where the bandwidth is less than 2300 bytes/second:

rate(node:network_receive_bytes_total[5m]) < 2300

This will result in a graph with gaps where the bandwidth exceeds 2300 bytes/second:

rate(node:network_receive_bytes_total[5m]) < 2300

The result of comparison operators can be enhanced with the bool modifier:

rate(node:network_receive_bytes_total[5m]) < bool 2300

In this case, the result will contain 1 for true comparisons and 0 for false comparisons:

rate(node:network_receive_bytes_total[5m]) < bool 2300

Aggregation and Grouping Functions

PromQL allows aggregating and grouping time series. Time series are grouped by a given set of labels, and then a given aggregation function is applied to each group. For example, the following query will return the total ingress traffic grouped by instance across all nodes with node_exporter installed:

sum(rate(node:network_receive_bytes_total[5m])) by (instance)

Using Gauges

Gauges are time series that can fluctuate up and down at any time, such as memory usage, temperature, or pressure. When plotting graphs for dashboards, it is often necessary to display the minimum, maximum, average, and/or quantile values for each point. PromQL provides the following functions to meet these needs:

For example, the following query will plot the minimum value of free memory for each point on the graph:

min_over_time(node:memory_MemFree_bytes[5m])

MetricsQL adds rollup_* functions to PromQL, which automatically return min, max, and avg values when applied to Gauges. For example:

rollup(node:memory_MemFree_bytes)

Label Manipulation

PromQL provides two functions for modifying, beautifying, deleting, or creating labels:

Although the use of these functions is somewhat complex, they allow for powerful dynamic manipulation of labels on selected time series. MetricsQL extends these capabilities with more convenient label manipulation functions:

  • label_set: Set additional labels for time series
  • label_del: Delete given labels from time series
  • label_keep: Delete all labels except the given ones from time series
  • label_copy: Copy label values to other labels
  • label_move: Rename labels
  • label_transform: Replace all substrings matching the given regular expression with template replacements
  • label_value: Return numerical values from given labels

Returning Multiple Results from a Single Query

Sometimes, it is necessary to return multiple results from a single PromQL query. This can be achieved using the or operator. For example, the following query will return all time series with the names metric1, metric2, and metric3:

metric1 or metric2 or metric3

MetricsQL simplifies the process of returning multiple results by simply listing them in ():

(metric1, metric2, metric3)

Note that any PromQL expression can be placed there, not just metric names.

There is a common pitfall when combining expression results: results with duplicate label sets will be skipped. For example, the following query will skip sum(b) because both sum(a) and sum(b) have the same label set—they have no labels at all:

sum(a) or sum(b)

Conclusion

PromQL is an easy-to-use yet powerful query language for time series databases. Compared to SQL, InfluxQL, or Flux, it allows writing typical TSDB queries in a concise and clear manner.

This tutorial does not cover all features of PromQL, as some are not commonly used:

You can continue learning PromQL using this PromQL cheat sheet.

The main content of this article is translated from PromQL tutorial for beginners and humans, and Guance is also using the MetricsQL engine implementation from the open-source VictoriaMetrics, thanks again to VictoriaMetrics!

Further Reading

Feedback

Is this page helpful? ×