Guided practice — Start, verify, break and fix the lab

Lab kit: https://github.com/hrhouma2/aiopsatlas-observabilite-labo-fr

You clone the kit into a lab3 folder, check that Docker is ready, start the ten services, open the web pages, type the queries, then break and fix. At the end, etat must display Labo : 10/10 services, 8/8 cibles up, 0 alertes actives., the API must know 64 cours, and the webhook must have received two notifications for APIInjoignable: one firing, one resolved. Start by running this block.

Windows (PowerShell)

git clone https://github.com/hrhouma2/aiopsatlas-observabilite-labo-fr.git lab3
cd lab3
ls                       # explorer le contenu : docker-compose.yml, labo.ps1, labo.sh, api/, prometheus/, grafana/, modules/
.\labo.ps1 prerequis
.\labo.ps1 demarrer
.\labo.ps1 etat          # attendu : Labo : 10/10 services, 8/8 cibles up, 0 alertes actives.

Wait two minutes (long enough for Prometheus to have a few measurements), then open the pages in the browser:

Prometheus       http://localhost:9090          (Status → Target health : 8 cibles UP ; onglet Graph pour P1 à P12)
Grafana          http://localhost:3000          (utilisateur admin · mot de passe aiopsatlas2026 ; menu → Explore, source Loki pour G1 à G8)
Alertmanager     http://localhost:9093          (vide au départ)
API catalogue    http://localhost:8000/cours    ·   http://localhost:8000/metrics
Webhook          http://localhost:8090          (vide au départ)

Queries P1 to P12 are also in modules\01-le-labo\requetes.txt, and G1 to G8 in modules\01-le-labo\requetes-logql.txt: open them in an editor and copy-paste. Then the outage:

.\labo.ps1 casser api    # arrête le conteneur de l'API ; la charge continue de frapper dans le vide
.\labo.ps1 etat          # attendu : 9/10 services, 7/8 cibles up ; regarde aussi Targets, Alerts, 9093 et 8090
.\labo.ps1 reparer       # redémarre l'API
.\labo.ps1 etat          # attendu : Labo : 10/10 services, 8/8 cibles up, 0 alertes actives.

If PowerShell refuses .\labo.ps1 ("l'exécution de scripts est désactivée", script execution is disabled): Set-ExecutionPolicy -Scope CurrentUser RemoteSigned, answer O, retry. If port 3000 is already taken on your machine, $env:GRAFANA_PORT = '3001' before demarrer, and replace 3000 with 3001 in the Grafana URLs.

Linux, macOS, WSL 2, Git Bash

git clone https://github.com/hrhouma2/aiopsatlas-observabilite-labo-fr.git lab3
cd lab3
ls                       # explorer le contenu : docker-compose.yml, labo.sh, labo.ps1, api/, prometheus/, grafana/, modules/
./labo.sh prerequis
./labo.sh demarrer
./labo.sh etat           # attendu : Labo : 10/10 services, 8/8 cibles up, 0 alertes actives.

Wait two minutes, then open the pages in the browser:

Prometheus       http://localhost:9090          (Status → Target health : 8 cibles UP ; onglet Graph pour P1 à P12)
Grafana          http://localhost:3000          (utilisateur admin · mot de passe aiopsatlas2026 ; menu → Explore, source Loki pour G1 à G8)
Alertmanager     http://localhost:9093          (vide au départ)
API catalogue    http://localhost:8000/cours    ·   http://localhost:8000/metrics
Webhook          http://localhost:8090          (vide au départ)

The queries are in modules/01-le-labo/requetes.txt (P1 to P12) and modules/01-le-labo/requetes-logql.txt (G1 to G8). Then the outage:

./labo.sh casser api     # arrête le conteneur de l'API ; la charge continue de frapper dans le vide
./labo.sh etat           # attendu : 9/10 services, 7/8 cibles up ; regarde aussi Targets, Alerts, 9093 et 8090
./labo.sh reparer        # redémarre l'API
./labo.sh etat           # attendu : Labo : 10/10 services, 8/8 cibles up, 0 alertes actives.

If port 3000 is taken: GRAFANA_PORT=3001 ./labo.sh demarrer, then 3001 in the Grafana URLs.

Before typing a single query, look at what the lab observes. Everything revolves around a course catalog API: a small web service written in Python (FastAPI) that serves 64 courses and records enrollments. A second service, charge, plays the role of the users: it calls the API continuously, with successful requests, a few deliberate 404s and, one time in a hundred, a 500 error that the API manufactures itself. Everything you are going to read in Prometheus and Loki comes from these two services. The rest of the kit, plainly:

lab3/
├── api/
│   ├── app.py                     l'API catalogue (FastAPI) : 6 routes publiques, 3 routes /admin
│   └── donnees/cours.json         64 cours → servis par GET /cours et GET /cours/{id}
├── charge/charge.py               le générateur de trafic : GET /cours, /cours/{id}, POST /inscriptions, GET /lent, des 404
├── prometheus/
│   ├── prometheus.yml             8 cibles lues toutes les 15 s (scrape_interval: 15s)
│   └── regles/alertes.yml         10 règles d'alerte ; APIInjoignable est la première
├── alertmanager/alertmanager.yml  groupe les alertes et les envoie au webhook (group_wait: 10s)
├── alloy/config.alloy             lit les journaux des conteneurs et les pousse dans Loki
├── grafana/provisioning/          3 tableaux de bord et 3 sources de données, créés au démarrage
└── modules/01-le-labo/
    ├── requetes.txt               P1 à P12, à coller dans Prometheus
    └── requetes-logql.txt         G1 à G8, à coller dans Grafana Explore

Open the data yourself, it takes ten seconds:

# Windows (PowerShell), depuis le dossier lab3
Get-Content api\donnees\cours.json -TotalCount 16
Invoke-RestMethod "http://localhost:8000/cours?limite=2"
Invoke-RestMethod http://localhost:8000/cours/C0001
(Invoke-WebRequest http://localhost:8000/metrics -UseBasicParsing).Content -split "`n" | Select-String "^http_requetes_total"

# Linux, macOS, WSL 2, Git Bash, depuis le dossier lab3
head -n 16 api/donnees/cours.json
curl -s "http://localhost:8000/cours?limite=2"
curl -s http://localhost:8000/cours/C0001
curl -s http://localhost:8000/metrics | grep "^http_requetes_total"

The catalog API: 64 courses in a JSON file

api/donnees/cours.json is a JSON array of 64 objects, one per course, identifiers C0001 to C0064. The first one, as the API returns it on GET /cours/C0001:

{"id":"C0001","titre":"Introduction à Python","categorie":"programmation","niveau":"debutant","prix":89,"duree_heures":6,"professeur":"Karim Haddad","tags":["code","algorithmes"],"note":4.8,"inscrits":2319}

The API routes and what they answer on the course lab:

Each response carries an x-id-requete header (for example x-id-requete: c5c49525ea53): it is the same identifier as the id_requete field of the log line written for that request. It will serve you in query G7.

The metrics: what /metrics exposes, one real line per type

The page http://localhost:8000/metrics is text, one series per line, preceded by two comment lines # HELP (what the metric is for) and # TYPE (its type). On the course lab, it is about 270 lines. The four metrics you are going to query, copied from the page:

# HELP http_requetes_total Nombre de requêtes HTTP reçues, par méthode, route normalisée et code de réponse.
# TYPE http_requetes_total counter
http_requetes_total{code="200",methode="GET",route="/cours"} 3247.0
http_requetes_total{code="500",methode="GET",route="/cours"} 32.0

# HELP requetes_en_cours Nombre de requêtes HTTP en cours de traitement à cet instant.
# TYPE requetes_en_cours gauge
requetes_en_cours 1.0

# HELP http_duree_requete_seconds Durée de traitement des requêtes HTTP, en secondes, par route normalisée.
# TYPE http_duree_requete_seconds histogram
http_duree_requete_seconds_bucket{le="0.005",route="/cours"} 32.0
http_duree_requete_seconds_bucket{le="0.01",route="/cours"} 74.0
http_duree_requete_seconds_bucket{le="0.025",route="/cours"} 1566.0
http_duree_requete_seconds_bucket{le="0.05",route="/cours"} 3248.0
http_duree_requete_seconds_bucket{le="0.1",route="/cours"} 3276.0
http_duree_requete_seconds_bucket{le="0.25",route="/cours"} 3278.0
http_duree_requete_seconds_bucket{le="0.5",route="/cours"} 3279.0
http_duree_requete_seconds_bucket{le="1.0",route="/cours"} 3279.0
http_duree_requete_seconds_bucket{le="2.0",route="/cours"} 3279.0
http_duree_requete_seconds_bucket{le="+Inf",route="/cours"} 3279.0
http_duree_requete_seconds_count{route="/cours"} 3279.0
http_duree_requete_seconds_sum{route="/cours"} 85.42293146001248

# HELP api_info Informations sur l'API (toujours 1) ; la version est dans le label.
# TYPE api_info gauge
api_info{version="1.0.0"} 1.0

There is no summary in this API: Prometheus advises against it in favor of the histogram, which aggregates across instances.

The labels: the columns of your future tables

A series is a metric name plus a set of label="value" pairs. Two origins:

Notice the /cours/{id} route: the API normalizes the URL before counting. /cours/C0001 and /cours/C0043 fall into the same series. Without that, there would be 64 series per code instead of one, and 64 times more lines in /metrics.

The logs: one JSON line per request

The API writes one log line per processed request, in JSON. Alloy reads the output of each labo-* container and pushes it into Loki. A real line, read with .\labo.ps1 journal api (or ./labo.sh journal api):

{"horodatage": "2026-09-15T19:33:14.781+00:00", "niveau": "ERROR", "id_requete": "dc1c2ff189a6", "methode": "GET", "route": "/cours/{id}", "code": 500, "duree_ms": 0.1, "message": "GET /cours/C0019 -> 500"}

Loki does not read the JSON line by line at query time, unless you ask it to (| json, query G5). What it indexes are labels set by Alloy on arrival:

The common thread

Remember two things you are going to find everywhere: the series http_requetes_total{code="500",route="/cours"}, at 32 on /metrics at the time of the capture, which you will see again at 32 in P4; and the identifier dc1c2ff189a6, that of the error line above, which you will find again in G2 and then go and look for on your own in G7. Metrics count, logs tell; both talk about the same request.

Open http://localhost:9090. You land on the Query page (the top menu offers Query, Alerts, Status). Paste a query into the field, press Enter or click Execute. The result appears below in the Table tab (one line per series, the value on the right); the Graph tab draws the same series over time. Stay on Table for this section, unless stated otherwise. Under the tabs, a line such as Load time: 40ms Result series: 8 tells you how many series answered.

The image to keep in mind: Prometheus is a readings logbook. Every 15 seconds, it walks past each of its eight targets, reads their /metrics page and notes each value with the time. A PromQL query is a question asked to that logbook.

P1. Who answers?

up

up{instance="localhost:9090", job="prometheus"}               1
up{instance="alloy:12345", job="alloy"}                       1
up{instance="api:8000", job="api", service="api"}             1
up{instance="cadvisor:8080", job="cadvisor"}                  1
up{instance="alertmanager:9093", job="alertmanager"}          1
up{instance="loki:3100", job="loki"}                          1
up{instance="node-exporter:9100", job="node-exporter"}        1
up{instance="grafana:3000", job="grafana"}                    1

Result series: 8, all at 1.

What the query asks: "Give me the last value of the up metric for all targets."

Zero parameters: just a metric name. up is exposed by no target; Prometheus manufactures it at every scrape: 1 if the /metrics page answered, 0 otherwise. Eight series because prometheus.yml declares eight jobs. Each line reads: the metric name, then between braces the labels Prometheus set (job and instance on all, service in addition on the API), then the value. SQL equivalent: SELECT * FROM up. What Prometheus does that SQL does not: it produced this table itself by going to knock on eight doors.

P2. A single target

up{job="api"}

up{instance="api:8000", job="api", service="api"}             1

What the query asks: "The value of up, only for the series whose job label is api."

A single novelty: the selector {job="api"}. The braces filter on labels, like a WHERE job = 'api'. The quotes are mandatory around the value: up{job=api} is refused with parse error: unexpected identifier "api" in label matching, expected string. It is exactly this expression, up{job="api"} == 0, that the APIInjoignable rule watches; you will see it go to 0 in the appendix.

P3. The raw counter

http_requetes_total

http_requetes_total{code="200", instance="api:8000", job="api", methode="GET", route="/sante", service="api"}             91
http_requetes_total{code="200", instance="api:8000", job="api", methode="GET", route="/cours/{id}", service="api"}       2714
http_requetes_total{code="404", instance="api:8000", job="api", methode="GET", route="/cours/{id}", service="api"}        209
http_requetes_total{code="200", instance="api:8000", job="api", methode="GET", route="/lent", service="api"}              314
http_requetes_total{code="404", instance="api:8000", job="api", methode="GET", route="inconnue", service="api"}           312
http_requetes_total{code="200", instance="api:8000", job="api", methode="GET", route="/cours", service="api"}            3241
http_requetes_total{code="201", instance="api:8000", job="api", methode="POST", route="/inscriptions", service="api"}     855
http_requetes_total{code="404", instance="api:8000", job="api", methode="POST", route="/inscriptions", service="api"}      46
http_requetes_total{code="500", instance="api:8000", job="api", methode="GET", route="/cours", service="api"}              32
http_requetes_total{code="422", instance="api:8000", job="api", methode="POST", route="/inscriptions", service="api"}      51
http_requetes_total{code="500", instance="api:8000", job="api", methode="GET", route="/cours/{id}", service="api"}         27
http_requetes_total{code="200", instance="api:8000", job="api", methode="GET", route="/admin/etat", service="api"}         14
http_requetes_total{code="500", instance="api:8000", job="api", methode="GET", route="/lent", service="api"}                1
http_requetes_total{code="500", instance="api:8000", job="api", methode="POST", route="/inscriptions", service="api"}       7

Result series: 14 on the course lab (the exact number depends on the combinations charge has already produced; it climbs to 16 over time).

What the query asks: "All the series of the http_requetes_total counter, with their current value."

Nothing new in the syntax: a name, like P1. What is new is what you read. Compare with the /metrics page: the line http_requetes_total{code="200",methode="GET",route="/cours"} 3247.0 has become http_requetes_total{code="200", instance="api:8000", job="api", methode="GET", route="/cours", service="api"} 3241. Prometheus has added three labels (instance, job, service) and the value differs by a few units: the page was read at another instant. One series per combination (methode, route, code): this is what is called the cardinality of the metric, here 14.

The essential difference between /metrics and Prometheus. The /metrics page is the state of the API at the instant you open it, with no history. Prometheus keeps all the reads, one every 15 seconds, and that is what makes P5 possible: computing a speed requires at least two points.

P4. Only the 500 errors

http_requetes_total{code="500"}

http_requetes_total{code="500", instance="api:8000", job="api", methode="GET", route="/cours", service="api"}              32
http_requetes_total{code="500", instance="api:8000", job="api", methode="GET", route="/cours/{id}", service="api"}         27
http_requetes_total{code="500", instance="api:8000", job="api", methode="GET", route="/lent", service="api"}                1
http_requetes_total{code="500", instance="api:8000", job="api", methode="POST", route="/inscriptions", service="api"}       7

What the query asks: "The series of http_requetes_total whose code label is 500."

Nothing new: it is P2 applied to P3. Four series, one per affected route. The common thread is there: route="/cours" at 32, the value read on /metrics. The 500 is a string, not a number: http_requetes_total{code=500} is refused (parse error: unexpected character inside braces: '5'). These 500s are not an outage: charge deliberately triggers one error in a hundred (taux_erreurs: 0.01 in /admin/etat) so that the error curves are never empty.

Summary of P1 to P4: you have not computed anything yet. You have read instantaneous values and learned to filter them by label.

P5. The speed of a counter

rate(http_requetes_total[1m])

{code="200", instance="api:8000", job="api", methode="GET", route="/sante", service="api"}             0.11112345816201799
{code="200", instance="api:8000", job="api", methode="GET", route="/cours/{id}", service="api"}        3.000333370374486
{code="404", instance="api:8000", job="api", methode="GET", route="/cours/{id}", service="api"}        0.2666962995888432
{code="200", instance="api:8000", job="api", methode="GET", route="/lent", service="api"}              0.42226914101566837
{code="200", instance="api:8000", job="api", methode="GET", route="/cours", service="api"}             3.733748194243805
{code="201", instance="api:8000", job="api", methode="POST", route="/inscriptions", service="api"}     0.8667629736637403
{code="500", instance="api:8000", job="api", methode="GET", route="/cours", service="api"}             0.0222246916324036
…

Result series: 14, in requests per second.

What the query asks: "For each series of the counter, by how much did it increase per second, on average, over the last minute?"

A single novelty, in two inseparable pieces: [1m] turns the series into a range (all the values of the last minute, instead of only the last one), and rate() computes the slope of that range. Look at the result: the metric name has disappeared from the braces, because it is no longer http_requetes_total, it is a derived speed. /cours receives 3.7 requests per second, /sante 0.11 (one every 9 seconds: that is Docker's healthcheck). SQL equivalent: there is no simple one; you would need two reads, a subtraction and a division by the elapsed time. rate() does that for every series, and additionally corrects the resets to zero when the API restarts.

Without the range, Prometheus refuses: rate(http_requetes_total) gives parse error: expected type range vector in call to function "rate", got instant vector. You will read this message often; it means "a […] is missing".

P6. Add up by route

sum by (route) (rate(http_requetes_total[1m]))

{route="/sante"}          0.11112345816201799
{route="/cours/{id}"}     3.3114790532281364
{route="/lent"}           0.42226914101566837
{route="inconnue"}        0.3778197577508612
{route="/cours"}          3.7559728858762087
{route="/inscriptions"}   0.9334370485609511
{route="/admin/etat"}     0.0444493832648072

Result series: 7.

What the query asks: "Take the speeds from P5 and add them up keeping only the route label."

A single novelty: the aggregation sum by (route) (…). It melts away all the other labels (code, methode, instance…) and adds up what remains. /cours/{id} goes from three series (200, 404, 500) to one: 3.00 + 0.27 + 0.04 = 3.31. SQL equivalent: SELECT route, SUM(speed) FROM … GROUP BY route. The parentheses around route are mandatory: sum by route (…) is refused (parse error: unexpected identifier "route" in grouping opts, expected "(").

P7. Add up by code

sum by (code) (rate(http_requetes_total[1m]))

{code="200"}    7.311923547060784
{code="404"}    0.6445160573397044
{code="201"}    0.8667629736637403
{code="500"}    0.0888987665296144
{code="422"}    0.0444493832648072

What the query asks: "The same sum as P6, but grouped by HTTP code."

Nothing new: P6 with another label. This is what the « Réponses par code » (responses by code) panel of the « API catalogue — signaux dorés » dashboard does in Grafana. Five codes, five lines; the 500s at 0.09 per second, that is a bit more than one error every twelve seconds.

P8. Total traffic

sum(rate(http_requetes_total[1m]))

{}    8.956550727858652

What the query asks: "Add up all the speeds, without keeping any label."

A single novelty: sum(…) without by. Result: a single series, with an empty label set ({}), the value 8.96 requests per second. This is the first of the four golden signals, traffic. Check: the sum of the seven lines of P6 does make 8.96.

Summary of P5 to P8: you know how to turn a counter into a speed, then group that speed as you like. Three quarters of Prometheus dashboards do nothing but that.

P9. The error rate

sum(rate(http_requetes_total{code=~"5.."}[1m])) / sum(rate(http_requetes_total[1m]))

{}    0.009925558312655085

What the query asks: "The speed of responses whose code starts with 5, divided by the speed of all responses."

Two novelties, but small ones. First =~: a selector by regular expression, "5.." = a 5 followed by any two characters, hence all the 5xx. Then the division of two results: Prometheus divides the series that have the same labels, and here both sides have an empty set {}, so they pair up. Result: 0.0099, that is 1%; it is the value set in /admin/etat (taux_erreurs: 0.01). Second golden signal, errors. The kit's TauxErreursEleve alert rule fires when this same expression, computed over 5 minutes, exceeds 0.05.

P10. The p95 latency

histogram_quantile(0.95, sum by (le) (rate(http_duree_requete_seconds_bucket[5m])))

{}    0.09797705555555555

What the query asks: "From the buckets of the duration histogram, all routes combined, under which value do 95% of the requests of the last 5 minutes fall?"

A single novelty: histogram_quantile(0.95, …). It wants as input the _bucket buckets summed by le (that is why the sum by (le) is mandatory: without it, it computes one quantile per route and the result no longer has the meaning you expected). Result: 0.098 second, so 95% of requests are served in under 98 ms. This is the third golden signal, latency, and the metric the LatenceP95Elevee alert watches (threshold: 0.5 s). Look at the bucket distribution of /cours in the dataset: 3248 requests out of 3279 under 50 ms, but /lent (300 to 900 ms) pulls the global p95 upwards.

P11. A gauge, as is

requetes_en_cours

requetes_en_cours{instance="api:8000", job="api", service="api"}    0

What the query asks: "The last value of the requetes_en_cours gauge."

Nothing new in the syntax, it is P1. What is new is the type: a gauge is read as is, without rate(). Zero or one, depending on the instant: the API processes each request in a few milliseconds, it is rare to catch one in progress. This is the fourth golden signal, saturation: if this value climbed to 50, the API would be overwhelmed. rate(requetes_en_cours[1m]) triggers no error, but returns a number that means nothing; Prometheus does not protect you from this confusion.

P12. The alerts, as seen by Prometheus

ALERTS

Empty query result

What the query asks: "The alerts currently pending or firing."

Nothing new: a metric name, like P1. ALERTS is, like up, manufactured by Prometheus: one series per active alert, with the alertname and alertstate labels. On a healthy lab, the result is empty: Empty query result. It is not an error, it is the best possible answer. You will retype it during the appendix's outage and you will see ALERTS{alertname="APIInjoignable", alertstate="pending", …} appear, then alertstate="firing".

Summary of P9 to P12: the four golden signals (traffic P8, errors P9, latency P10, saturation P11) fit in four queries, and alerts are a metric like any other.

The message to get across. What SQL also does: filter by column ({job="api"} = WHERE), group and sum (sum by (route) = GROUP BY), divide two aggregates. What only Prometheus does: it went and fetched the data itself every 15 seconds from eight services, it turns any counter into a speed with one function (rate), it estimates a quantile from buckets (histogram_quantile), and it exposes its own alerts as a metric (ALERTS).

Open http://localhost:3000 (user admin, password aiopsatlas2026). In the main menu (icon at the top left), click Explore. At the top of the page, the data source selector offers Prometheus, Loki and Alertmanager: choose Loki. To the right of the query field, two modes: Builder (menus) and Code (you type). Switch to Code, paste the query, then Run query (or Shift+Enter). The logs appear at the bottom, the most recent line first. At the top right, the period selector is at Last 1 hour by default: keep it. These steps are identical on Windows and on Linux, it is the browser that does the work.

The image to keep in mind: Loki is a cabinet of logbooks, one binder per combination of labels. It does not read the content of the lines to file them, only the binder's label. A LogQL query therefore always starts by choosing a binder, between braces, then optionally by filtering the lines inside.

G1. The whole API log

{service="api"}

2026-09-15 19:33:30.416  {"horodatage": "2026-09-15T19:33:30.416+00:00", "niveau": "INFO", "id_requete": "875939d9cdac", "methode": "GET", "route": "/cours", "code": 200, "duree_ms": 11.5, "message": "GET /cours -> 200"}
2026-09-15 19:33:30.186  {"horodatage": "2026-09-15T19:33:30.186+00:00", "niveau": "INFO", "id_requete": "6d6e21e6d17b", "methode": "GET", "route": "/cours/{id}", "code": 200, "duree_ms": 14.3, "message": "GET /cours/C0001 -> 200"}
2026-09-15 19:33:30.142  {"horodatage": "2026-09-15T19:33:30.142+00:00", "niveau": "INFO", "id_requete": "42a64fb435bf", "methode": "GET", "route": "/cours", "code": 200, "duree_ms": 32.3, "message": "GET /cours -> 200"}
…

What the query asks: "All the log lines whose service label is api."

Zero novelty compared to P2: a selector between braces. The difference is the result: lines of text, not numbers. Grafana displays the time (converted to your time zone) then the raw line; click a line to unfold its labels: service="api", conteneur="labo-api", niveau="INFO", code="200", detected_level="info". About nine lines per second, as many as P8 announced: one request, one line. The braces are mandatory: service="api" alone is refused (parse error at line 1, col 1: syntax error: unexpected IDENTIFIER), and {service="api" without closing too (syntax error: unexpected $end, expecting } or ,).

G2. Only the errors

{service="api", niveau="ERROR"}

2026-09-15 19:33:28.931  {"horodatage": "2026-09-15T19:33:28.931+00:00", "niveau": "ERROR", "id_requete": "f5e09e6ed543", "methode": "POST", "route": "/inscriptions", "code": 500, "duree_ms": 0.0, "message": "POST /inscriptions -> 500"}
2026-09-15 19:33:16.381  {"horodatage": "2026-09-15T19:33:16.381+00:00", "niveau": "ERROR", "id_requete": "18f95992a8fa", "methode": "POST", "route": "/inscriptions", "code": 500, "duree_ms": 0.0, "message": "POST /inscriptions -> 500"}
2026-09-15 19:33:14.781  {"horodatage": "2026-09-15T19:33:14.781+00:00", "niveau": "ERROR", "id_requete": "dc1c2ff189a6", "methode": "GET", "route": "/cours/{id}", "code": 500, "duree_ms": 0.1, "message": "GET /cours/C0019 -> 500"}
…

What the query asks: "The API lines whose niveau label is ERROR."

A single novelty: two labels in the selector, separated by a comma, which is an AND. The niveau label is not in the line to begin with: Alloy extracted it from the JSON niveau field before sending to Loki, and that is what makes this query fast. The common thread is there, third line: id_requete: dc1c2ff189a6, the line from the dataset. Far fewer lines than in G1: about one every twelve seconds, as P7 said for the 500s.

G3. By HTTP code

{service="api", code="500"}

2026-09-15 19:33:28.931  {"horodatage": "2026-09-15T19:33:28.931+00:00", "niveau": "ERROR", "id_requete": "f5e09e6ed543", "methode": "POST", "route": "/inscriptions", "code": 500, …}
2026-09-15 19:33:16.381  {"horodatage": "2026-09-15T19:33:16.381+00:00", "niveau": "ERROR", "id_requete": "18f95992a8fa", "methode": "POST", "route": "/inscriptions", "code": 500, …}
2026-09-15 19:33:14.781  {"horodatage": "2026-09-15T19:33:14.781+00:00", "niveau": "ERROR", "id_requete": "dc1c2ff189a6", "methode": "GET", "route": "/cours/{id}", "code": 500, …}
…

What the query asks: "The API lines whose code label is 500."

Nothing new: G2 with another label. The same lines as in G2, because in this API every 500 is an ERROR and vice versa. This is the log version of P4: where Prometheus tells you "32 errors on /cours", Loki shows you which ones, with the real URL (/cours/C0019) and the request identifier. Notice that code is here a string ("500") because it is a label; in the line's JSON, it is a number (500). G6 will show you the difference.

G4. Filter on the text

{service="api"} |= "inscriptions"

2026-09-15 19:33:29.738  {"horodatage": "2026-09-15T19:33:29.738+00:00", "niveau": "INFO", "id_requete": "5afb73494ebd", "methode": "POST", "route": "/inscriptions", "code": 201, "duree_ms": 22.9, "message": "POST /inscriptions -> 201"}
2026-09-15 19:33:28.931  {"horodatage": "2026-09-15T19:33:28.931+00:00", "niveau": "ERROR", "id_requete": "f5e09e6ed543", "methode": "POST", "route": "/inscriptions", "code": 500, "duree_ms": 0.0, "message": "POST /inscriptions -> 500"}
2026-09-15 19:33:27.187  {"horodatage": "2026-09-15T19:33:27.187+00:00", "niveau": "INFO", "id_requete": "c4e29db13e12", "methode": "POST", "route": "/inscriptions", "code": 201, "duree_ms": 26.1, "message": "POST /inscriptions -> 201"}
…

What the query asks: "The API lines that contain the text inscriptions."

A single novelty: the line filter |= "…", which keeps the lines containing exactly this text. It is grep. Unlike a label, Loki must here open every line of the {service="api"} binder to look inside: slower, but you can search for anything. The variants: != (does not contain), |~ (regular expression), !~. Mixed result: 201s and 500s, everything that touches enrollments.

Summary of G1 to G4: two ways to filter, by label (fast, before opening the lines) and by text (flexible, after). The right query always starts with the narrowest possible label.

G5. Open the JSON

{service="api"} | json

2026-09-15 19:33:30.416  {"horodatage": "2026-09-15T19:33:30.416+00:00", "niveau": "INFO", "id_requete": "875939d9cdac", "methode": "GET", "route": "/cours", "code": 200, "duree_ms": 11.5, "message": "GET /cours -> 200"}
   labels : code="200" conteneur="labo-api" duree_ms="11.5" horodatage="2026-09-15T19:33:30.416+00:00" id_requete="875939d9cdac"
            message="GET /cours -> 200" methode="GET" niveau="INFO" route="/cours" service="api" …

What the query asks: "The API lines, and for each one, turn the JSON fields into labels."

A single novelty: the parser | json. The lines displayed are the same as in G1, but unfold one: it now has many more labels (route, methode, duree_ms, id_requete, message…), one per JSON field. Those labels are computed at query time, not stored: Loki still only indexes service, conteneur, niveau, code. You will also see code_extracted and niveau_extracted: when a JSON field bears the same name as a label already set by Alloy, Loki suffixes the copy rather than overwriting.

G6. Filter on a field

{service="api"} | json | duree_ms > 500

2026-09-15 19:33:27.838  {"horodatage": "2026-09-15T19:33:27.838+00:00", "niveau": "INFO", "id_requete": "416f1ab0eb41", "methode": "GET", "route": "/lent", "code": 200, "duree_ms": 587.4, "message": "GET /lent -> 200"}
2026-09-15 19:33:16.335  {"horodatage": "2026-09-15T19:33:16.335+00:00", "niveau": "INFO", "id_requete": "9d1cb1767846", "methode": "GET", "route": "/lent", "code": 200, "duree_ms": 737.8, "message": "GET /lent -> 200"}
2026-09-15 19:33:15.375  {"horodatage": "2026-09-15T19:33:15.375+00:00", "niveau": "INFO", "id_requete": "cab8f698f89b", "methode": "GET", "route": "/lent", "code": 200, "duree_ms": 526.2, "message": "GET /lent -> 200"}
…

What the query asks: "The API lines whose duree_ms field, once the JSON is opened, exceeds 500."

A single novelty: the label filter | duree_ms > 500, which compares an extracted label to a number. It is only possible after | json, otherwise duree_ms does not exist. Result: only /lent, the deliberately slow route (300 to 900 ms). This is the log version of P10: Prometheus says "the p95 is at 98 ms"; Loki shows the individual requests that exceeded a threshold, with their identifier. SQL equivalent: WHERE duree_ms > 500, except that the column did not exist before the query.

G7. Find a precise request

{service="api"} |= "dc1c2ff189a6"

2026-09-15 19:33:14.781  {"horodatage": "2026-09-15T19:33:14.781+00:00", "niveau": "ERROR", "id_requete": "dc1c2ff189a6", "methode": "GET", "route": "/cours/{id}", "code": 500, "duree_ms": 0.1, "message": "GET /cours/C0019 -> 500"}

A single line.

What the query asks: "The API line that contains the identifier dc1c2ff189a6."

Nothing new: it is G4 with another text. What changes is the use: on your machine, dc1c2ff189a6 does not exist; copy an id_requete seen in your own G2 output and paste it instead. This is the gesture you will make in production: a user gives you the identifier returned by the x-id-requete header of their failed response, and you find in one query the exact line, with the route, the code and the duration. A single line: the identifier is unique per request.

G8. Count the lines: a metric drawn from the logs

sum by (niveau) (count_over_time({service="api"}[1m]))

This time, Grafana displays a graph instead of lines: three curves, {niveau="INFO"} around 470 to 500 lines per minute, {niveau="WARNING"} around 40 to 50, {niveau="ERROR"} between 2 and 10, on the course lab. Hover over the graph to read the values.

What the query asks: "Count the API lines per one-minute slice, then add up keeping the niveau label."

A single novelty, in a piece you already know: count_over_time(…[1m]) counts the lines of a selector over a range, exactly as rate(…[1m]) computes a slope in P5. Around it, sum by (niveau) is the sum by (route) of P6, word for word. LogQL borrowed this grammar from PromQL on purpose: what you learned on one side serves on the other. Compare with P7: Prometheus counts 0.09 500 response per second, that is 5 per minute; Loki counts 5 ERROR lines per minute. Two tools, two paths, the same number.

The essential difference between Prometheus and Loki. Prometheus stores numbers already counted by the API (http_requetes_total), Loki stores the lines and can recount them on demand (count_over_time). The first is light and fast, and answers "how many"; the second is heavy but keeps the detail, and answers "which ones". The lab has both because neither replaces the other.

The message to get across. What grep also does: search for a text in lines (|=). What only Loki does: file the lines of ten containers by labels and read only the right binder, open the JSON on demand to filter on a numeric field (| json | duree_ms > 500), and turn logs into a curve with PromQL's grammar (count_over_time).

All commands are typed in PowerShell, from the lab3 folder. Docker Desktop must be running (green icon). If PowerShell refuses to run .\labo.ps1, type once Set-ExecutionPolicy -Scope CurrentUser RemoteSigned and answer O.

A.0 — Clone the kit

cd C:\Users\<toi>\Documents
git clone https://github.com/hrhouma2/aiopsatlas-observabilite-labo-fr.git lab3
cd lab3
ls

You must see docker-compose.yml, labo.ps1, labo.sh, README.md, and the folders alertmanager, alloy, api, charge, grafana, loki, modules, outils, prometheus, webhook. If you already cloned the kit in lesson 03, skip this step and just do cd lab3.

A.1 — Check the prerequisites

.\labo.ps1 prerequis

== Prérequis ==
  ✔ docker : Docker version 29.3.1, build c2be9cc
  ✔ le démon Docker répond
  ✔ docker compose : 5.1.1
  ✔ mémoire disponible pour Docker : 31 Go
  ✔ processeurs : 20
  ✔ port 9090 libre
  ✔ port 9093 libre
  ✔ port 3000 libre
  ✔ port 3100 libre
  ✔ port 12345 libre
  ✔ port 9100 libre
  ✔ port 8080 libre
  ✔ port 8000 libre
  ✔ port 8090 libre

Tout est prêt. Lancez : .\labo.ps1 demarrer

Checkpoint: the last line is Tout est prêt. (everything is ready). The versions, memory and number of processors are those of the course machine. If a port is marked ✘ … déjà occupé (already occupied), lesson 03 explains what to do; for 3000, $env:GRAFANA_PORT = '3001' is enough.

A.2 — Start

.\labo.ps1 demarrer

The first time, downloading the six public images takes one to five minutes depending on your connection. Expected end of output:

== Attente que chaque service soit prêt ==
  prometheus       prêt (0 s)
  alertmanager     prêt (0 s)
  loki             prêt (0 s)
  alloy           .. prêt (6 s)
  node-exporter    prêt (0 s)
  cadvisor         prêt (0 s)
  api              prêt (0 s)
  webhook          prêt (0 s)
  charge           prêt (0 s)
  grafana         ... prêt (9 s)

Le labo est prêt.
  Grafana        http://localhost:3000   (utilisateur admin · mot de passe aiopsatlas2026)
  Prometheus     http://localhost:9090   (Status → Target health, puis onglet Graph)
  Alertmanager   http://localhost:9093
  API catalogue  http://localhost:8000/cours   ·   http://localhost:8000/metrics
  Webhook        http://localhost:8090   (les alertes reçues)
  Loki           http://localhost:3100/ready   ·   Alloy   http://localhost:12345
  node-exporter  http://localhost:9100/metrics   ·   cAdvisor   http://localhost:8080

Étape suivante : .\labo.ps1 etat   (laissez tourner 2 minutes pour avoir des courbes)

Checkpoint: ten prêt (ready), then Le labo est prêt. (the lab is ready). On the course machine, images already cached, the command took 44 seconds. Lesson 04 comments on this output line by line.

A.3 — Read etat

.\labo.ps1 etat

== Conteneurs ==
NAME                 SERVICE         STATUS
labo-alertmanager    alertmanager    Up About a minute (healthy)
labo-alloy           alloy           Up 48 seconds (healthy)
labo-api             api             Up About a minute (healthy)
labo-cadvisor        cadvisor        Up About a minute (healthy)
labo-charge          charge          Up About a minute (healthy)
labo-grafana         grafana         Up 48 seconds (healthy)
labo-loki            loki            Up About a minute (healthy)
labo-node-exporter   node-exporter   Up About a minute (healthy)
labo-prometheus      prometheus      Up About a minute (healthy)
labo-webhook         webhook         Up About a minute (healthy)

== Supervision ==
  ✔ Prometheus répond — cibles up : 8/8
     séries en mémoire : 9038
     alertes : 0 active(s), 0 en attente (pending)
  ✔ Alertmanager répond (http://localhost:9093)
  ✔ Grafana répond (http://localhost:3000)
  ✔ Loki répond (http://localhost:3100)
  ✔ API catalogue répond — version 1.0.0, 64 cours
  ✔ Webhook répond — 0 alerte(s) reçue(s) (http://localhost:8090)

Labo : 10/10 services, 8/8 cibles up, 0 alertes actives.

Checkpoint: ten (healthy), 8/8, 64 cours, 0 alerte(s) reçue(s), and the last line Labo : 10/10 services, 8/8 cibles up, 0 alertes actives.. The number of series in memory climbs during the first minutes (9038 right after startup, 12,000 to 15,000 after an hour on the course machine). If alloy or grafana are still (health: starting), wait thirty seconds and retry.

A.4 — The eight targets in Prometheus

Open http://localhost:9090, Status menu, then Target health. Eight blocks, one per job, each with 1 / 1 up and a line:

api
1 / 1 up
Endpoint                    Labels                                        Last scrape   State
http://api:8000/metrics     instance="api:8000" job="api" service="api"   6.014s ago    UP

Checkpoint: eight UP, no DOWN. The Last scrape column never exceeds 15 seconds: that is the scrape_interval of prometheus.yml. On the command line, the same information:

(Invoke-RestMethod http://localhost:9090/api/v1/targets).data.activeTargets | Select-Object @{n='job';e={$_.labels.job}}, health, scrapeUrl | Sort-Object job

job           health scrapeUrl
---           ------ ---------
alertmanager  up     http://alertmanager:9093/metrics
alloy         up     http://alloy:12345/metrics
api           up     http://api:8000/metrics
cadvisor      up     http://cadvisor:8080/metrics
grafana       up     http://grafana:3000/metrics
loki          up     http://loki:3100/metrics
node-exporter up     http://node-exporter:9100/metrics
prometheus    up     http://localhost:9090/metrics

A.5 — Queries P1 to P12 and G1 to G8

Wait until demarrer has at least two minutes behind it, then follow the Prometheus, in the Graph tab and Grafana, in Explore sections above. The queries are ready to copy:

Get-Content modules\01-le-labo\requetes.txt
Get-Content modules\01-le-labo\requetes-logql.txt

Checkpoint: P1 returns 8 series at 1, P12 returns Empty query result, G1 returns JSON lines, G8 returns three curves.

A.6 — Break: stop the API

Before breaking, note the time (Get-Date -Format HH:mm:ss). Then:

.\labo.ps1 casser api

== Panne : arrêt de l'API ==
 Container labo-api Stopping
 Container labo-api Stopped
  ✔ API arrêtée. La charge continue de frapper dans le vide.
  À observer : .\labo.ps1 etat  ·  http://localhost:9090/targets (api → down)
               http://localhost:9090/alerts (APIInjoignable : pending puis firing après 30 s)
               http://localhost:9093 et http://localhost:8090 (l'alerte arrive ~10 s après firing)

Pour tout remettre en ordre : .\labo.ps1 reparer

The script did a docker compose stop api: the container is stopped cleanly, its data and its image are intact. Now, observe the outage through five paths, in order. You have about 70 seconds before the alert reaches the webhook: run etat right away.

Path 1, etat:

.\labo.ps1 etat

== Conteneurs ==
NAME                 SERVICE         STATUS
labo-alertmanager    alertmanager    Up 22 minutes (healthy)
labo-alloy           alloy           Up 22 minutes (healthy)
labo-api             api             Exited (0) About a minute ago
labo-cadvisor        cadvisor        Up 22 minutes (healthy)
labo-charge          charge          Up 22 minutes (healthy)
labo-grafana         grafana         Up 22 minutes (healthy)
labo-loki            loki            Up 22 minutes (healthy)
labo-node-exporter   node-exporter   Up 22 minutes (healthy)
labo-prometheus      prometheus      Up 22 minutes (healthy)
labo-webhook         webhook         Up 22 minutes (healthy)

== Supervision ==
  ✔ Prometheus répond — cibles up : 7/8
  ✘ cible api (http://api:8000/metrics) : down — Get "http://api:8000/metrics": dial tcp: lookup api on 127.0.0.11:53: no such host
     séries en mémoire : 15116
     alertes : 2 active(s), 0 en attente (pending)
  ✘ APIInjoignable [critique] — L'API catalogue ne répond plus
  ✘ TauxErreursEleve [critique] — Plus de 5 % des requêtes de l'API échouent
  ✔ Alertmanager répond (http://localhost:9093)
  ✔ Grafana répond (http://localhost:3000)
  ✔ Loki répond (http://localhost:3100)
  ✘ API catalogue ne répond pas (http://localhost:8000)
  ✔ Webhook répond — 2 alerte(s) reçue(s) (http://localhost:8090)

Labo : 9/10 services, 7/8 cibles up, 2 alertes actives.

What to read, top to bottom: labo-api is Exited (0) (code 0: deliberate stop, not a crash); Prometheus now reads only 7/8 targets and tells you why (lookup api … no such host: the name api no longer exists on the Docker network since the container is stopped); the APIInjoignable alert is active; the API does not answer on port 8000; the webhook received something. This output was captured on the course machine one minute after casser api, while a casser erreurs had just been played a few minutes earlier: that is why a second alert, TauxErreursEleve, also appears. On your lab, you will only have APIInjoignable, 1 alertes actives and 1 alerte(s) reçue(s). If you run etat within the first 30 seconds, the alert is still en attente (pending) and the webhook is still at 0: retry a minute later.

Path 2, the targets. Reload http://localhost:9090 → Status → Target health. The api block has switched to 0 / 1 up, state DOWN, and the Error column carries the same message as etat: Get "http://api:8000/metrics": dial tcp: lookup api on 127.0.0.11:53: no such host. The other seven remain UP. Retype up in Query: the line up{instance="api:8000", job="api", service="api"} is at 0, the other seven at 1. Then up == 0: a single line.

Path 3, the alerts in Prometheus. Alerts menu. The APIInjoignable rule changes state in three stages, timed on the course machine:

t+0 s   : APIInjoignable inactive      (Prometheus n'a pas encore rescrappé l'API)
t+40 s  : APIInjoignable pending       (up{job="api"} == 0 est vrai, le compte à rebours « for: 30s » tourne)
t+70 s  : APIInjoignable firing        (vrai depuis 30 s : Prometheus envoie à Alertmanager)
t+70 s  : APIInjoignable reçue par le webhook (firing)

Why 40 seconds before pending: Prometheus reads the API every 15 seconds, so it takes up to 15 seconds for a scrape to fail, then it evaluates the rules every 15 seconds. Why 30 more before firing: the rule says for: 30s. On another capture, pending arrived at 31 s and firing at 61 s: the order of magnitude is the same, the detail depends on the instant you broke it relative to the scrape cycle. Retype ALERTS in Query:

ALERTS{alertname="APIInjoignable", alertstate="pending", instance="api:8000", job="api", service="api", severite="critique"}    1

then, thirty seconds later, alertstate="firing". Note the time of the switch to firing: it is the first of the three lines of your deliverable.

Path 4, Alertmanager. Open http://localhost:9093. The Alerts page shows a group alertname="APIInjoignable" service="api" (that is the group_by: [alertname, service] of alertmanager.yml) with the alert, its labels (instance="api:8000", job="api", labo="observabilite", severite="critique"), its summary L'API catalogue ne répond plus (the catalog API no longer answers) and its description. The labo="observabilite" label was not in the rule: it is the external_labels of prometheus.yml, added to everything that leaves Prometheus. On the command line:

(Invoke-RestMethod http://localhost:9093/api/v2/alerts) | Select-Object @{n='alerte';e={$_.labels.alertname}}, @{n='etat';e={$_.status.state}}, startsAt

alerte          etat   startsAt
------          ----   --------
APIInjoignable  active 2026-09-15T19:41:11.496Z

Path 5, the webhook. Open http://localhost:8090. The page « Alertes reçues d'Alertmanager » (alerts received from Alertmanager) is no longer empty: a line APIInjoignable · critique · firing · api · L'API catalogue ne répond plus, and the header counts 1 alerte(s) en mémoire · 1 notification(s) reçue(s) (1 alert in memory · 1 notification received). The raw format, http://localhost:8090/alertes.json, shows what Alertmanager sent:

{"recu_a":"2026-09-15T19:41:26+00:00","etat":"firing","nom":"APIInjoignable","severite":"critique","service":"api","resume":"L'API catalogue ne répond plus","description":"Prometheus n'arrive plus à lire http://api:8000/metrics depuis 30 secondes (cible api:8000).","debut":"2026-09-15T19:41:11.496Z","fin":"0001-01-01T00:00:00Z","labels":{"alertname":"APIInjoignable","instance":"api:8000","job":"api","labo":"observabilite","service":"api","severite":"critique"}}

Read debut (19:41:11, the time of the firing in Prometheus) and recu_a (19:41:26): fifteen seconds apart, including Alertmanager's 10 seconds of group_wait. fin at year 0001 means "not finished yet". Note recu_a: second line of your deliverable.

What the load sees, meanwhile:

.\labo.ps1 journal charge

labo-charge  | {"horodatage": "2026-09-15T19:40:36.600+00:00", "niveau": "WARNING", "message": "API injoignable : ConnectionError"}
labo-charge  | {"horodatage": "2026-09-15T19:40:40.925+00:00", "niveau": "WARNING", "message": "API injoignable : ConnectionError"}
labo-charge  | {"horodatage": "2026-09-15T19:40:45.393+00:00", "niveau": "WARNING", "message": "API injoignable : ConnectionError"}

And what you see if you call the API yourself:

Invoke-RestMethod http://localhost:8000/sante -TimeoutSec 5

Invoke-RestMethod : Le délai de l'opération a expiré.

(Without -TimeoutSec, PowerShell waits longer before giving up; curl.exe -sS http://localhost:8000/sante answers faster: curl: (7) Failed to connect to localhost:8000 after 2237 ms: Could not connect to server.)

The essential difference between a 404 and no answer. In step A.8, /cours/C9999 will answer 404: the API is running and politely tells you this course does not exist; it is counted in http_requetes_total{code="404"}, it is written in a WARNING log, and up stays at 1. Here, Le délai de l'opération a expiré (the operation timed out): nobody answers, there is neither code nor log on the API side, and it is up that drops to 0. Two situations, two signals, two places to look.

A.7 — Fix

.\labo.ps1 reparer

== Réparation ==
  ✔ API redémarrée
  api             .. prêt (6 s)
  ✔ taux d'erreurs remis à 0.01, lenteur à 0 ms

Les alertes passent en « resolved » dans les minutes qui suivent (voir http://localhost:8090).

The script did docker compose start api, waited for /sante to answer, then called /admin/reparer (useful for the other two outage scenarios). Check that the API is working:

.\labo.ps1 journal api

labo-api  | {"horodatage": "2026-09-15T19:41:48.720+00:00", "niveau": "INFO", "id_requete": "6a4228a46a56", "methode": "POST", "route": "/inscriptions", "code": 201, "duree_ms": 15.6, "message": "POST /inscriptions -> 201"}
labo-api  | {"horodatage": "2026-09-15T19:41:48.914+00:00", "niveau": "WARNING", "id_requete": "b4ebe9fa32d8", "methode": "GET", "route": "inconnue", "code": 404, "duree_ms": 0.4, "message": "GET /inexistant -> 404"}
labo-api  | {"horodatage": "2026-09-15T19:41:48.937+00:00", "niveau": "INFO", "id_requete": "732adf61ee17", "methode": "GET", "route": "/cours/{id}", "code": 200, "duree_ms": 25.7, "message": "GET /cours/C0043 -> 200"}

Then watch the alert go out, in the same order it came on. Timed on the course machine after reparer:

t+15 s  : APIInjoignable firing   (Prometheus)   · webhook : firing
t+40 s  : APIInjoignable inactive (Prometheus)   · webhook : firing
t+55 s  : APIInjoignable inactive (Prometheus)   · webhook : resolved

At the first successful scrape, up{job="api"} returns to 1 and the rule goes back to inactive; Alertmanager then sends a resolved notification to the webhook. Reload http://localhost:8090: two lines now for APIInjoignable, one firing and one resolved, and in /alertes.json the second has a filled-in fin field:

{"recu_a":"2026-09-15T19:42:26+00:00","etat":"resolved","nom":"APIInjoignable",…,"debut":"2026-09-15T19:41:11.496Z","fin":"2026-09-15T19:41:56.496Z",…}

fin minus debut: the outage lasted 45 seconds in Prometheus's eyes. Note recu_a of the resolved line: third line of your deliverable.

A.8 — Trigger a 404, for comparison

The API is running. Ask it for a course that does not exist:

Invoke-RestMethod http://localhost:8000/cours/C9999

Invoke-RestMethod : {"detail":"cours C9999 introuvable"}

This is an HTTP 404 error: the API answered. To see the code itself:

try { Invoke-WebRequest http://localhost:8000/cours/C9999 -UseBasicParsing } catch { $_.Exception.Response.StatusCode.value__ }

404

Retype P4 in Prometheus replacing 500 with 404: the route="/cours/{id}" series has increased by 1. Retype G1 adding |= "C9999": your request is there, level WARNING, with its id_requete. None of this exists for the outage of A.6: a stopped API counts nothing and writes nothing.

A.9 — Final check

.\labo.ps1 etat

== Conteneurs ==
NAME                 SERVICE         STATUS
labo-alertmanager    alertmanager    Up 23 minutes (healthy)
labo-alloy           alloy           Up 23 minutes (healthy)
labo-api             api             Up 53 seconds (healthy)
labo-cadvisor        cadvisor        Up 23 minutes (healthy)
labo-charge          charge          Up 23 minutes (healthy)
labo-grafana         grafana         Up 23 minutes (healthy)
labo-loki            loki            Up 23 minutes (healthy)
labo-node-exporter   node-exporter   Up 23 minutes (healthy)
labo-prometheus      prometheus      Up 23 minutes (healthy)
labo-webhook         webhook         Up 23 minutes (healthy)

== Supervision ==
  ✔ Prometheus répond — cibles up : 8/8
     séries en mémoire : 15395
     alertes : 0 active(s), 0 en attente (pending)
  ✔ Alertmanager répond (http://localhost:9093)
  ✔ Grafana répond (http://localhost:3000)
  ✔ Loki répond (http://localhost:3100)
  ✔ API catalogue répond — version 1.0.0, 64 cours
  ✔ Webhook répond — 2 alerte(s) reçue(s) (http://localhost:8090)

Labo : 10/10 services, 8/8 cibles up, 0 alertes actives.

What you must have: labo-api back to Up … (healthy) with a shorter time than the others (it has just restarted), 8/8, 0 active(s), 64 cours, 2 alerte(s) reçue(s) (the firing and the resolved), and the last line Labo : 10/10 services, 8/8 cibles up, 0 alertes actives.. On the course machine, this output still carried 1 alertes actives and 3 alerte(s) reçue(s) because of the residual TauxErreursEleve mentioned in A.6; on your machine, with only the api outage, the values are those above. This output and your three noted times are your deliverable. You can leave the lab running for workshops 06 and 07, or stop it with .\labo.ps1 arreter: the data is kept and demarrer resumes where you were.

All commands are typed in a bash terminal, from the lab3 folder. On macOS and Windows (WSL 2 or Git Bash), Docker Desktop must be running; on native Linux, docker info must answer without sudo (otherwise sudo usermod -aG docker $USER, then open a new session). The bash script needs curl.

B.0 — Clone the kit

cd ~
git clone https://github.com/hrhouma2/aiopsatlas-observabilite-labo-fr.git lab3
cd lab3
ls

You must see docker-compose.yml, labo.sh, labo.ps1, README.md, and the folders alertmanager, alloy, api, charge, grafana, loki, modules, outils, prometheus, webhook. On Linux and macOS, make the script executable once: chmod +x labo.sh (without it, ./labo.sh answers bash: ./labo.sh: Permission denied). If you already cloned the kit in lesson 03, skip this step and just do cd lab3.

B.1 — Check the prerequisites

./labo.sh prerequis

== Prérequis ==
  ✔ docker : Docker version 29.3.1, build c2be9cc
  ✔ le démon Docker répond
  ✔ docker compose : 5.1.1
  ✔ curl : présent
  ✔ mémoire disponible pour Docker : 31 Go
  ✔ processeurs : 20
  ✔ port 9090 libre
  ✔ port 9093 libre
  ✔ port 3000 libre
  ✔ port 3100 libre
  ✔ port 12345 libre
  ✔ port 9100 libre
  ✔ port 8080 libre
  ✔ port 8000 libre
  ✔ port 8090 libre

Tout est prêt. Lancez : ./labo.sh demarrer

Checkpoint: the last line is Tout est prêt.. The bash script checks one more line than PowerShell, curl : présent. The versions and memory are those of the course machine (Git Bash on Windows). If a port is occupied, lesson 03 explains what to do; for 3000, GRAFANA_PORT=3001 ./labo.sh demarrer.

B.2 — Start

./labo.sh demarrer

Expected end of output, once the images are downloaded:

== Attente que chaque service soit prêt ==
  prometheus       prêt (0 s)
  alertmanager     prêt (0 s)
  loki             prêt (0 s)
  alloy           .. prêt (6 s)
  node-exporter    prêt (0 s)
  cadvisor         prêt (0 s)
  api              prêt (0 s)
  webhook          prêt (0 s)
  charge           prêt (0 s)
  grafana         ... prêt (9 s)

Le labo est prêt.
  Grafana        http://localhost:3000   (utilisateur admin · mot de passe aiopsatlas2026)
  Prometheus     http://localhost:9090   (Status → Target health, puis onglet Graph)
  Alertmanager   http://localhost:9093
  API catalogue  http://localhost:8000/cours   ·   http://localhost:8000/metrics
  Webhook        http://localhost:8090   (les alertes reçues)
  Loki           http://localhost:3100/ready   ·   Alloy   http://localhost:12345
  node-exporter  http://localhost:9100/metrics   ·   cAdvisor   http://localhost:8080

Étape suivante : ./labo.sh etat   (laissez tourner 2 minutes pour avoir des courbes)

Checkpoint: ten prêt, then Le labo est prêt..

B.3 — Read etat

./labo.sh etat

== Conteneurs ==
NAME                 SERVICE         STATUS
labo-alertmanager    alertmanager    Up 11 minutes (healthy)
labo-alloy           alloy           Up 10 minutes (healthy)
labo-api             api             Up 4 minutes (healthy)
labo-cadvisor        cadvisor        Up 11 minutes (healthy)
labo-charge          charge          Up 10 minutes (healthy)
labo-grafana         grafana         Up 10 minutes (healthy)
labo-loki            loki            Up 11 minutes (healthy)
labo-node-exporter   node-exporter   Up 11 minutes (healthy)
labo-prometheus      prometheus      Up 11 minutes (healthy)
labo-webhook         webhook         Up 11 minutes (healthy)

== Supervision ==
  ✔ Prometheus répond — cibles up : 8/8
     séries en mémoire : 10602
     alertes : 0 active(s), 0 en attente (pending)
  ✔ Alertmanager répond (http://localhost:9093)
  ✔ Grafana répond (http://localhost:3000)
  ✔ Loki répond (http://localhost:3100)
  ✔ API catalogue répond — version 1.0.0, 64 cours
  ✔ Webhook répond — 0 alerte(s) reçue(s) (http://localhost:8090)

Labo : 10/10 services, 8/8 cibles up, 0 alertes actives.

Checkpoint: ten (healthy), 8/8, 64 cours, and the last line Labo : 10/10 services, 8/8 cibles up, 0 alertes actives.. The Conteneurs block comes from docker compose ps; you can type it yourself. Third path, in curl:

curl -s http://localhost:8000/sante
curl -s http://localhost:8090/sante

{"etat":"ok","version":"1.0.0","cours":64}
{"etat":"ok","alertes_en_memoire":0,"notifications":0,"alertes":0}

B.4 — The eight targets in Prometheus

In the browser, http://localhost:9090 → Status → Target health: eight 1 / 1 up blocks, state UP, Last scrape column always under 15 seconds. In curl, with python3 to read the JSON (or jq if you have it):

curl -s http://localhost:9090/api/v1/targets | python3 -c 'import json,sys; [print(t["labels"]["job"].ljust(14), t["health"], t["scrapeUrl"]) for t in sorted(json.load(sys.stdin)["data"]["activeTargets"], key=lambda t: t["labels"]["job"])]'

alertmanager   up http://alertmanager:9093/metrics
alloy          up http://alloy:12345/metrics
api            up http://api:8000/metrics
cadvisor       up http://cadvisor:8080/metrics
grafana        up http://grafana:3000/metrics
loki           up http://loki:3100/metrics
node-exporter  up http://node-exporter:9100/metrics
prometheus     up http://localhost:9090/metrics

B.5 — Queries P1 to P12 and G1 to G8

Wait two minutes after demarrer, then follow the Prometheus, in the Graph tab and Grafana, in Explore sections above; they are done in the browser, identically on all systems. The queries are ready to copy:

cat modules/01-le-labo/requetes.txt
cat modules/01-le-labo/requetes-logql.txt

Third path, a PromQL query in curl:

curl -s 'http://localhost:9090/api/v1/query?query=up' | python3 -m json.tool | head -n 20

You recognize in the JSON the same series as in the Table tab: "metric": {"__name__": "up", "instance": "localhost:9090", "job": "prometheus"} and "value": [1789500809.696, "1"].

Checkpoint: P1 returns 8 series at 1, P12 returns Empty query result, G1 returns JSON lines, G8 returns three curves.

B.6 — Break: stop the API

Note the time (date +%T), then:

./labo.sh casser api

== Panne : arrêt de l'API ==
 Container labo-api Stopping
 Container labo-api Stopped
  ✔ API arrêtée. La charge continue de frapper dans le vide.
  À observer : ./labo.sh etat  ·  http://localhost:9090/targets (api → down)
               http://localhost:9090/alerts (APIInjoignable : pending puis firing après 30 s)
               http://localhost:9093 et http://localhost:8090 (l'alerte arrive ~10 s après firing)

Pour tout remettre en ordre : ./labo.sh reparer

The script did docker compose stop api. You have about 70 seconds before the alert reaches the webhook. Observe through five paths.

Path 1, etat:

./labo.sh etat

== Conteneurs ==
NAME                 SERVICE         STATUS
labo-alertmanager    alertmanager    Up 22 minutes (healthy)
labo-alloy           alloy           Up 22 minutes (healthy)
labo-api             api             Exited (0) About a minute ago
labo-cadvisor        cadvisor        Up 22 minutes (healthy)
labo-charge          charge          Up 22 minutes (healthy)
labo-grafana         grafana         Up 22 minutes (healthy)
labo-loki            loki            Up 22 minutes (healthy)
labo-node-exporter   node-exporter   Up 22 minutes (healthy)
labo-prometheus      prometheus      Up 22 minutes (healthy)
labo-webhook         webhook         Up 22 minutes (healthy)

== Supervision ==
  ✔ Prometheus répond — cibles up : 7/8
  ✘ cible api (http://api:8000/metrics) : down — Get "http://api:8000/metrics": dial tcp: lookup api on 127.0.0.11:53: no such host
     séries en mémoire : 15116
     alertes : 2 active(s), 0 en attente (pending)
  ✘ APIInjoignable [critique] — L'API catalogue ne répond plus
  ✘ TauxErreursEleve [critique] — Plus de 5 % des requêtes de l'API échouent
  ✔ Alertmanager répond (http://localhost:9093)
  ✔ Grafana répond (http://localhost:3000)
  ✔ Loki répond (http://localhost:3100)
  ✘ API catalogue ne répond pas (http://localhost:8000)
  ✔ Webhook répond — 2 alerte(s) reçue(s) (http://localhost:8090)

Labo : 9/10 services, 7/8 cibles up, 2 alertes actives.

What to read: labo-api is Exited (0) (deliberate stop); Prometheus now reads only 7/8 targets and says why (lookup api … no such host: the name api has disappeared from the Docker network); APIInjoignable is active; the API does not answer; the webhook received the alert. This output was captured one minute after casser api, on a machine where casser erreurs had just been played: hence the second alert TauxErreursEleve. On your machine: 1 alertes actives, 1 alerte(s) reçue(s). If you run etat within the first 30 seconds, the alert is still pending and the webhook at 0: retry a minute later. Third path, in curl:

curl -s http://localhost:8000/sante

curl: (7) Failed to connect to localhost:8000 after 2237 ms: Could not connect to server

Path 2, the targets. Reload http://localhost:9090 → Status → Target health: the api block is at 0 / 1 up, state DOWN, Error column: Get "http://api:8000/metrics": dial tcp: lookup api on 127.0.0.11:53: no such host. Retype up in Query: up{instance="api:8000", job="api", service="api"} is at 0. Then up == 0: a single line.

Path 3, the alerts in Prometheus. Alerts menu. APIInjoignable changes state in three stages, timed on the course machine:

t+0 s   : APIInjoignable inactive      (Prometheus n'a pas encore rescrappé l'API)
t+40 s  : APIInjoignable pending       (up{job="api"} == 0 est vrai, le compte à rebours « for: 30s » tourne)
t+70 s  : APIInjoignable firing        (vrai depuis 30 s : Prometheus envoie à Alertmanager)
t+70 s  : APIInjoignable reçue par le webhook (firing)

Why 40 seconds before pending: up to 15 seconds for a scrape to fail (scrape_interval: 15s), then up to 15 seconds for the next rule evaluation (evaluation_interval: 15s). Why 30 more: for: 30s in alertes.yml. Retype ALERTS in Query:

ALERTS{alertname="APIInjoignable", alertstate="pending", instance="api:8000", job="api", service="api", severite="critique"}    1

then alertstate="firing". Note the time of the firing: first line of your deliverable. In curl, the same thing:

curl -s http://localhost:9090/api/v1/alerts | python3 -m json.tool

Path 4, Alertmanager. Open http://localhost:9093: the Alerts page shows a group alertname="APIInjoignable" service="api" (the group_by of alertmanager.yml) with the alert, its labels (instance="api:8000", job="api", labo="observabilite", severite="critique"), the summary L'API catalogue ne répond plus. The labo="observabilite" label comes from the external_labels of prometheus.yml. In curl:

curl -s http://localhost:9093/api/v2/alerts | python3 -c 'import json,sys; [print(a["labels"]["alertname"], a["status"]["state"], a["startsAt"]) for a in json.load(sys.stdin)]'

APIInjoignable active 2026-09-15T19:41:11.496Z

Path 5, the webhook. Open http://localhost:8090: a line APIInjoignable · critique · firing · api · L'API catalogue ne répond plus, header 1 alerte(s) en mémoire · 1 notification(s) reçue(s). The raw format:

curl -s http://localhost:8090/alertes.json | python3 -m json.tool

[
    {
        "recu_a": "2026-09-15T19:41:26+00:00",
        "etat": "firing",
        "nom": "APIInjoignable",
        "severite": "critique",
        "service": "api",
        "resume": "L'API catalogue ne répond plus",
        "description": "Prometheus n'arrive plus à lire http://api:8000/metrics depuis 30 secondes (cible api:8000).",
        "debut": "2026-09-15T19:41:11.496Z",
        "fin": "0001-01-01T00:00:00Z",
        "labels": {
            "alertname": "APIInjoignable",
            "instance": "api:8000",
            "job": "api",
            "labo": "observabilite",
            "service": "api",
            "severite": "critique"
        }
    }
]

debut (19:41:11) is the time of the firing in Prometheus; recu_a (19:41:26) arrives fifteen seconds later, including the 10 seconds of group_wait. fin at year 0001: not finished yet. Note recu_a: second line of your deliverable.

What the load sees:

./labo.sh journal charge

labo-charge  | {"horodatage": "2026-09-15T19:40:36.600+00:00", "niveau": "WARNING", "message": "API injoignable : ConnectionError"}
labo-charge  | {"horodatage": "2026-09-15T19:40:40.925+00:00", "niveau": "WARNING", "message": "API injoignable : ConnectionError"}
labo-charge  | {"horodatage": "2026-09-15T19:40:45.393+00:00", "niveau": "WARNING", "message": "API injoignable : ConnectionError"}

The essential difference between a 404 and no answer. In step B.8, /cours/C9999 will answer 404: the API is running and tells you this course does not exist; it is counted in http_requetes_total{code="404"}, it is written in a WARNING log, and up stays at 1. Here, curl: (7) Failed to connect: nobody answers, there is neither code nor log on the API side, and it is up that drops to 0.

B.7 — Fix

./labo.sh reparer

== Réparation ==
  ✔ API redémarrée
  api             .. prêt (6 s)
  ✔ taux d'erreurs remis à 0.01, lenteur à 0 ms

Les alertes passent en « resolved » dans les minutes qui suivent (voir http://localhost:8090).

The script did docker compose start api, waited for /sante, then called /admin/reparer. Check that the API is working:

./labo.sh journal api

labo-api  | {"horodatage": "2026-09-15T19:41:48.720+00:00", "niveau": "INFO", "id_requete": "6a4228a46a56", "methode": "POST", "route": "/inscriptions", "code": 201, "duree_ms": 15.6, "message": "POST /inscriptions -> 201"}
labo-api  | {"horodatage": "2026-09-15T19:41:48.914+00:00", "niveau": "WARNING", "id_requete": "b4ebe9fa32d8", "methode": "GET", "route": "inconnue", "code": 404, "duree_ms": 0.4, "message": "GET /inexistant -> 404"}
labo-api  | {"horodatage": "2026-09-15T19:41:48.937+00:00", "niveau": "INFO", "id_requete": "732adf61ee17", "methode": "GET", "route": "/cours/{id}", "code": 200, "duree_ms": 25.7, "message": "GET /cours/C0043 -> 200"}

Then the alert goes out, timed on the course machine after reparer:

t+15 s  : APIInjoignable firing   (Prometheus)   · webhook : firing
t+40 s  : APIInjoignable inactive (Prometheus)   · webhook : firing
t+55 s  : APIInjoignable inactive (Prometheus)   · webhook : resolved

Third path, watch in a loop:

watch -n 5 'curl -s http://localhost:8090/alertes.json | python3 -c "import json,sys; [print(a[\"recu_a\"], a[\"nom\"], a[\"etat\"]) for a in json.load(sys.stdin)]"'

2026-09-15T19:42:26+00:00 APIInjoignable resolved
2026-09-15T19:41:26+00:00 APIInjoignable firing

The resolved line carries a filled-in fin field (2026-09-15T19:41:56.496Z): 45 seconds of outage in Prometheus's eyes. Note its recu_a: third line of your deliverable. Ctrl+C to exit watch.

B.8 — Trigger a 404, for comparison

curl -s -i http://localhost:8000/cours/C9999

HTTP/1.1 404 Not Found
date: Tue, 15 Sep 2026 20:50:01 GMT
server: uvicorn
content-length: 36
content-type: application/json
x-id-requete: bbc7be667f1d

{"detail":"cours C9999 introuvable"}

The API answered: a 404 code, an x-id-requete identifier, a JSON body. Retype P4 in Prometheus with 404 instead of 500: the route="/cours/{id}" series has increased by 1. Retype G1 adding |= "bbc7be667f1d" (your own identifier, read in the header): your request is there, level WARNING. None of this exists for the outage of B.6: a stopped API counts nothing and writes nothing.

B.9 — Final check

./labo.sh etat

== Conteneurs ==
NAME                 SERVICE         STATUS
labo-alertmanager    alertmanager    Up 23 minutes (healthy)
labo-alloy           alloy           Up 23 minutes (healthy)
labo-api             api             Up 53 seconds (healthy)
labo-cadvisor        cadvisor        Up 23 minutes (healthy)
labo-charge          charge          Up 23 minutes (healthy)
labo-grafana         grafana         Up 23 minutes (healthy)
labo-loki            loki            Up 23 minutes (healthy)
labo-node-exporter   node-exporter   Up 23 minutes (healthy)
labo-prometheus      prometheus      Up 23 minutes (healthy)
labo-webhook         webhook         Up 23 minutes (healthy)

== Supervision ==
  ✔ Prometheus répond — cibles up : 8/8
     séries en mémoire : 15395
     alertes : 0 active(s), 0 en attente (pending)
  ✔ Alertmanager répond (http://localhost:9093)
  ✔ Grafana répond (http://localhost:3000)
  ✔ Loki répond (http://localhost:3100)
  ✔ API catalogue répond — version 1.0.0, 64 cours
  ✔ Webhook répond — 2 alerte(s) reçue(s) (http://localhost:8090)

Labo : 10/10 services, 8/8 cibles up, 0 alertes actives.

What you must have: labo-api back to Up … (healthy), 8/8, 0 active(s), 64 cours, 2 alerte(s) reçue(s) (the firing and the resolved), and Labo : 10/10 services, 8/8 cibles up, 0 alertes actives.. On the course machine, this output still carried 1 alertes actives and 3 alerte(s) reçue(s) because of the residual TauxErreursEleve mentioned in B.6; on your machine, with only the api outage, the values are those above. This output and your three noted times are your deliverable. You can leave it running for workshops 06 and 07, or ./labo.sh arreter: the data is kept.

etat says 9/10 services while you broke nothing. Look at which container is not (healthy). If it is grafana or alloy right after demarrer with (health: starting), wait thirty seconds. If it is labo-api in Exited, someone (maybe you, in a previous attempt) ran casser api: reparer. If a container is Restarting, read its log: .\labo.ps1 journal <service> or ./labo.sh journal <service>.

cibles up : 7/8 and APIInjoignable active, but the API answers on http://localhost:8000. Prometheus reads the API through the Docker network (http://api:8000/metrics), you through the published port (localhost:8000). If the API has just restarted, Prometheus may be one scrape (15 s) behind and the alert stays firing until the next evaluation, then a few more seconds for the webhook to receive the resolved. Wait a minute and rerun etat.

The alerts are slow: pending does not switch to firing. APIInjoignable has for: 30s; so it takes up to 15 s (scrape) + 15 s (evaluation) + 30 s (for) = 60 to 70 s for firing, then 10 s of group_wait for the webhook. It is not slow, it is set that way to avoid false alerts on an isolated miss. If after two minutes nothing moves, check that labo-api is indeed Exited (docker compose ps).

The webhook stays at 0 alerte(s) reçue(s) while Alertmanager shows the alert. Open http://localhost:9093 → Status: the Config section must show receiver: webhook and url: http://webhook:8090/alertes. Then journal webhook: you must see a POST /alertes line at every notification. If the labo-webhook container is not healthy, docker compose restart webhook.

Empty query result on P3, P5 or G1, right after demarrer. Prometheus needs at least one scrape (15 s) for P3, two for P5 (rate wants two points in [1m]), and Alloy takes a few seconds to send the first line to Loki. Wait two minutes after Le labo est prêt.. If {service="api"} stays empty after five minutes, check http://localhost:12345 (Alloy must be ready and its components Healthy) and journal alloy.

P10 returns NaN. histogram_quantile returns NaN when the [5m] window does not yet contain enough points. Wait five minutes after startup, or replace [5m] with [1m] to see a value sooner (less stable).

G1 returns zero lines while the API is running. First check the period (top right, Last 1 hour) and the label name (service, lowercase). Then http://localhost:12345: Alloy must answer Alloy is ready. and journal alloy must not show a repeated error. As a last resort, docker compose restart alloy.

parse error in Prometheus. The three most frequent, all seen on this page: unexpected identifier "api" in label matching, expected string (forgotten quotes: {job=api}); unexpected character inside braces: '5' ({code=500} instead of {code="500"}); expected type range vector in call to function "rate", got instant vector (forgotten [1m] window).

parse error in Loki. syntax error: unexpected IDENTIFIER: you forgot the braces (service="api" instead of {service="api"}). unexpected $end, expecting } or ,: missing closing brace. Zero lines without error: label name or case ({service="API"}, {app="api"}).

Windows only — Invoke-RestMethod displays odd characters (é) in course titles. It is the console display, not the API. [Console]::OutputEncoding = [Text.Encoding]::UTF8 before the command, or read in the browser.

Windows only — .\labo.ps1: "l'exécution de scripts est désactivée sur ce système" (script execution is disabled on this system). Set-ExecutionPolicy -Scope CurrentUser RemoteSigned, answer O, retry.

Windows only — demarrer fails with port is already allocated on 3000. Another program is already listening (often a Node application). $env:GRAFANA_PORT = '3001' then rerun demarrer; Grafana is then on http://localhost:3001 and etat displays it that way.

Native Linux only — permission denied while trying to connect to the Docker daemon socket. sudo usermod -aG docker $USER, close the session, reopen it, docker info must answer.

Native Linux and macOS — bash: ./labo.sh: Permission denied. The file is saved without the execute bit in the repository: chmod +x labo.sh once, or run bash labo.sh prerequis. Under Git Bash (Windows), the question does not arise.

You want to start from scratch. .\labo.ps1 reinitialiser or ./labo.sh reinitialiser deletes the containers and the volumes: Prometheus, Loki and Grafana start empty again. Then demarrer. Not to be done on a lab shared with other people.