Fundamental workshop 1 — PromQL: one metric, one label, one function

Duration

20 min

Module

1/7

Prerequisites

the lab has been running for at least two minutes (etat displays 8/8 cibles up), Prometheus's Query tab open at http://localhost:9090

You will build

ten PromQL queries typed by hand, from the shortest (api_info) to the first real aggregation (sum by (code) (rate(…[1m]))), changing only one thing at a time

Deliverable

the output of step 10 as displayed in the Table tab, two lines, with one sentence saying what each measures

How to read this page. Ten steps, one query at a time. For each: the query to type, the lab's exact answer (Table tab), and what to look at in it. Type each query yourself (no copy-paste): it is by writing the braces, quotes and brackets that the grammar sinks in. The numbers will be different on your machine; the shapes (number of lines, labels, order of magnitude) must be the same. The "To understand better" blocks are optional. If the lab is not started, go back to the guided practice: the In short section gives the commands, kit included (https://github.com/hrhouma2/aiopsatlas-observabilite-labo-fr). Nothing is created or modified in this workshop: PromQL only reads.

Goal

The guided practice had you type twelve already-written queries. You saw the results, but if the sheet is taken away, can you write sum by (code) (rate(http_requetes_total{route="/cours"}[1m])) without misplacing a parenthesis? Here, you start again from the shortest possible query, a metric name, and you add one single piece at each step: a label, an operator, a function, a time range, a grouping. Two steps are deliberate traps: you will trigger the two error messages every beginner meets, so as to recognize them next time. At the end, you know what a metric, a label and a function are because you assembled the three pieces yourself.

The vocabulary in one image

Prometheus is a readings logbook. Every 15 seconds, it walks past each target, reads its /metrics page and notes each value with the time. A metric is the name of a column in the logbook (up, http_requetes_total). A label is a tag stuck on the line to say what we are talking about (job="api", code="200"); the same metric name with different tags makes different series. A function is an operation on what was read: count the lines, compute a slope, add up.

Where to type, and how to read an answer

Open http://localhost:9090. You are on the Query page. The input field accepts a query; Execute (or Enter) sends it. The result is displayed under the field, in the Table tab. Stay on Table for the whole workshop: that is where you see the labels written in plain text. The Graph tab draws the same thing over time; the Explain tab breaks down the query.

A result line always has the same shape: the name of the metric, then between braces the labels sorted alphabetically, then the value on the right:

up{instance="api:8000", job="api", service="api"}    1

When the query has melted the name away (a function, an aggregation), the braces remain, sometimes empty: {} 8. Under the tabs, Result series: N tells you how many lines you have. An empty result reads Empty query result; a badly written query displays a red Error executing query box followed by the message.

Step 1 — Read a metric

api_info

What the query asks: the last value of the api_info metric, for all its series.

api_info{instance="api:8000", job="api", service="api", version="1.0.0"}    1

What to look at: a single line, Result series: 1. The value is 1 and will never change: api_info is an information metric, everything it has to say is in its version="1.0.0" label. Three other labels the API did not write: instance, job and service were added by Prometheus at read time. On the page http://localhost:8000/metrics, the same line reads api_info{version="1.0.0"} 1.0.

Step 2 — Read a metric with several series

up

What the query asks: the last value of up, for all its series.

up{instance="localhost:9090", job="prometheus"}    1
up{instance="alloy:12345", job="alloy"}    1
up{instance="api:8000", job="api", service="api"}    1
up{instance="cadvisor:8080", job="cadvisor"}    1
up{instance="alertmanager:9093", job="alertmanager"}    1
up{instance="loki:3100", job="loki"}    1
up{instance="node-exporter:9100", job="node-exporter"}    1
up{instance="grafana:3000", job="grafana"}    1

What to look at: Result series: 8, the same name on every line, and what changes from one line to the next: the values of the job and instance labels. That is what a series is: a name plus a set of labels. Eight different sets, eight series. Notice that only the third line carries service="api": this label was added by hand in prometheus.yml, for the api job only. up exists on no /metrics page: Prometheus makes it itself, 1 if the read succeeded, 0 otherwise.

Step 3 — Pick a series with a label

up{job="api"}

What the query asks: the series of up whose job label is exactly api.

up{instance="api:8000", job="api", service="api"}    1

What to look at: a single line, the third of step 2. The braces after the name are a filter: they are called a selector. job is the label name, "api" its value, between double quotes, = exact equality. It is SQL's WHERE job = 'api', word for word. You can put several conditions separated by commas; all must be true.

Step 4 — The trap: no quotes, then the wrong case

Two wrong queries, on purpose. First, forget the quotes:

up{job=api}

Error executing query
invalid parameter "query": 1:8: parse error: unexpected identifier "api" in label matching, expected string

What to look at: parse error, Prometheus did not even search: the query is badly written. 1:8 is the position (line 1, character 8, just after up{job=). expected string: it expected a string between quotes. A label value is always a string, even when it looks like a number: {code=500} gives the same family of error, {code="500"} is the right form.

Then, put the quotes but change the case:

up{job="API"}

Empty query result

What to look at: no error, no line. This is the most vicious trap: the query is correct, it simply asks for a series that does not exist. Label values are case- and spelling-sensitive ("api " with a space does not work either). When you get Empty query result for no reason, retype step 2 and reread the exact values.

Step 5 — Pick several series with a pattern

up{job=~"a.*"}

What the query asks: the series of up whose job label matches the regular expression a.*: an a followed by anything.

up{instance="alloy:12345", job="alloy"}    1
up{instance="api:8000", job="api", service="api"}    1
up{instance="alertmanager:9093", job="alertmanager"}    1

What to look at: three lines, the three jobs starting with a. A single novelty compared to step 3: =~ instead of =. The regular expression must match the entire value: "a" alone would return nothing, you need "a.*". The four selection operators: = (equal), != (different: up{job!="api"} returns the other seven), =~ (matches), !~ (does not match). It is =~ that the guided practice used in {code=~"5.."} to catch all the 5xx.

Step 6 — Apply a function

count(up)

What the query asks: the number of series that up returns.

{}    8

What to look at: a single line, and the name has disappeared: empty {}, then 8. This is the first function of the workshop, and the result is no longer up, it is a number computed from up. count is an aggregation: it takes several series and makes one. Its cousins: sum (the sum of the values: sum(up) also gives 8 as long as everything is at 1, and 7 as soon as a target falls), min, max, avg. The kit's dashboard and the etat command count the targets exactly like that.

Step 7 — Ask for a time range

http_requetes_total{route="/cours", code="200"}[1m]

What the query asks: all the values of this series recorded during the last minute, not only the last one.

http_requetes_total{code="200", instance="api:8000", job="api", methode="GET", route="/cours", service="api"}
    2439 @1789505608.199
    2499 @1789505623.199
    2550 @1789505638.196
    2609 @1789505653.197

What to look at: one series, but four values, each followed by @ and a date in seconds. Fifteen seconds apart between two: that is the scrape_interval. The counter climbs from 2439 to 2609: 170 200 requests on /cours in 45 seconds. A single novelty: the [1m] brackets after the selector. They turn an instant vector (one value per series) into a range vector (a list of dated values per series). Click the Graph tab: it refuses this query (Error executing query then invalid expression type "range vector" for range query, must be Scalar or instant Vector). You do not draw a raw range, you give it to a function. That is step 8. Go back to Table.

Step 8 — Apply a function to the range

rate(http_requetes_total{route="/cours", code="200"}[1m])

What the query asks: the speed at which this counter increased, in units per second, computed over the range of the last minute.

{code="200", instance="api:8000", job="api", methode="GET", route="/cours", service="api"}    3.7779456864749545

What to look at: again a single value, and the name http_requetes_total has disappeared from the braces: it is no longer a counter, it is a speed. 3.78 requests per second. Check with step 7: 170 requests in 45 seconds make 3.78. A single novelty: the rate() function, which takes a range vector and returns an instant vector. It is the most important function in PromQL: a raw counter is never read, its slope is.

Step 9 — The trap: rate without a range

rate(http_requetes_total{route="/cours", code="200"})

Error executing query
invalid parameter "query": 1:6: parse error: expected type range vector in call to function "rate", got instant vector

What to look at: expected type range vector … got instant vector. You gave rate an instant vector (one value), it wanted a range (several dated values): without two points, no slope. The correct gesture is step 8, with [1m]. You will read this message often; it always means "a […] is missing".

A variant that gives no error but returns nothing:

rate(http_requetes_total{route="/cours", code="200"}[10s])

Empty query result

What to look at: a 10-second range contains at best one reading (they are 15 s apart), and rate needs at least two. Practical rule: the range must be at least twice the scrape_interval, hence [30s] minimum here; [1m] or [5m] in real life.

Step 10 — Group

sum by (code) (rate(http_requetes_total{route="/cours"}[1m]))

What the query asks: the speed of all the /cours series (all codes), added up keeping only the code label.

{code="200"}    3.7779456864749545
{code="500"}    0

What to look at: two lines, and only one label remains in the braces: code. All the others (instance, job, methode, route, service) melted into the sum. A single novelty: sum by (code) (…), the aggregation of step 6 with a by clause. The parentheses around code are mandatory (sum by code (…) gives parse error: … expected "("). The line {code="500"} 0 deserves a look: it is zero because no 500 fell on /cours during the last minute (the API produces about one every twelve seconds, all routes combined). Zero is not absent: the series exists, it just has a null slope. If you run .\labo.ps1 casser erreurs (or ./labo.sh casser erreurs) and retype this query a minute later, the second line climbs; reparer brings it back down.

This result is your deliverable: the two lines, and one sentence for each ("/cours serves 3.78 200 responses per second"; "no 500 response on /cours in the last minute").

Final check

Redo the ten queries from memory, in order, and tick:

• api_info returns 1 series, value 1, with version="1.0.0" in the labels.
• up returns 8 series, all at 1 (otherwise, a target has fallen: etat will tell you which one).
• up{job="api"} returns 1 series.
• up{job=api} returns parse error … expected string; up{job="API"} returns Empty query result.
• up{job=~"a.*"} returns 3 series: alloy, api, alertmanager.
• count(up) returns {} 8.
• http_requetes_total{route="/cours", code="200"}[1m] returns 1 series with 4 or 5 dated values @….
• rate(…[1m]) returns 1 value, between 3 and 4 requests per second on the course lab.
• rate(…) without brackets returns expected type range vector … got instant vector.
• sum by (code) (rate(http_requetes_total{route="/cours"}[1m])) returns 2 series, {code="200"} and {code="500"}.

Nothing to clean up: you created nothing. etat still displays 8/8 cibles up and the same number of series in memory, give or take a few dozen (Prometheus keeps collecting).

If it breaks