Why observe: metrics, logs and alerts

Audience

beginner, no prior knowledge required

Duration

30 to 40 min

Module

1/7

Target skill

distinguish monitoring from observability, know the professional vocabulary (metric, log, trace, alert, SLI, SLO, SLA), recognize what a metric, a log line and an alert from the lab really look like, and place the ten services of the kit in the story of the course

In one image

Picture a car's dashboard. The gauges (speed, engine RPM, fuel level) give a numeric value at every instant: those are the metrics. The black box, the event recorder of a modern car, notes what happened, line by line, with the exact time: those are the logs. The warning lights (oil, battery, engine) come on by themselves when a monitored value leaves its normal range, without you having to stare at every gauge all the time: those are the alerts. The GPS, which retraces the complete route you followed with the time spent on each segment, is a trace. This image comes back throughout the module: every service in the lab plays the role of one of these four parts.

How it works

Three public outages that show why we observe

Nobody observes a system for fun. We observe it because one day something breaks, and we need to understand what, where, and since when, before it gets expensive. The three outages below are public: each company wrote and published its own post-mortem. They show what observability makes possible, and what it does not always make possible on the first try.

Cloudflare, November 18, 2025: when metrics alone are not enough. At 11:20 UTC, Cloudflare's network stops routing a large part of the world's traffic correctly: visitors to its customers' sites receive an error page. The cause: a permissions change on a ClickHouse database cluster doubles the size of an internal configuration file (the "feature" file of the anti-bot system), which exceeds a limit set in the code and crashes the core proxy. The HTTP 5xx error metrics are visible from the first minute, as a sharp spike on a graph. But the file was only generated incorrectly intermittently, on part of the cluster only: traffic recovered then fell over again every five minutes. This fluctuation first made the team believe it was a denial-of-service attack, not an internal failure. Metrics had to be cross-checked with the logs of the anti-bot module to identify the real cause. Main traffic is restored at 14:30, all systems return to normal at 17:06: nearly 5 hours and 46 minutes between the beginning and the complete end of the incident. Official source: Cloudflare outage on November 18, 2025.

AWS S3, February 28, 2017: when the monitoring tool depends on the system it monitors. At 9:37 (Pacific time), an Amazon S3 engineer runs a maintenance command meant to remove a few servers from a billing subsystem, in the Northern Virginia region (us-east-1). A typo removes a far larger number of servers than intended and brings down two critical S3 subsystems: the index (metadata and object location) and placement (allocation of new storage). S3 becomes unable to process GET, LIST, PUT and DELETE requests, and with it fall new EC2 instance launches, EBS, Lambda, and AWS's own official status dashboard, which depended on S3 to update itself. AWS had to communicate the state of the outage on its Twitter feed. The index is fully restored at 13:18, placement at 13:54: about 4 hours and 17 minutes between the start of the incident and the return to normal. Official source: Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region.

GitHub, August 14, 2024: when the monitoring itself triggers the outage. At 22:59 UTC, GitHub deploys a configuration change on its databases. This change breaks the ability of those databases to answer correctly the health checks sent by the routing layer. No longer receiving a valid answer, the routing layer declares these databases "unhealthy" and removes read access: GitHub.com becomes unreachable for all users from 23:02 to 23:38 UTC, that is 36 minutes. The team fixes it by rolling back the configuration change, then confirms through continuous monitoring that connectivity is restored before closing the incident at 00:30 the next day. The point to remember: it is not the absence of monitoring that caused the outage; it is one of the monitoring mechanisms itself, the health check, which, misconfigured, triggered it. Official source: GitHub Availability Report: August 2024.

The three outages overlap on one point: in all three cases, the company knew very quickly that something was wrong (error metrics move within seconds). What takes time is knowing why. That is exactly what the lab of this course makes you build, on a small scale, on your workstation.

Monitoring and observability, the professional vocabulary

Monitoring watches indicators chosen in advance, with thresholds already known: "the error rate exceeds 5%". It is effective for an outage we have already seen before. Observability is the ability to understand the internal state of a system from the signals it already produces (its metrics, its logs, its traces), including for a question we had not anticipated, without redeploying code to go and fetch it. Monitoring says "something is wrong"; observability helps answer "why, where, and since when".

Reference for the SLI, SLO and SLA definitions: Google SRE Book — Service Level Objectives. For observability as seen by the tools of the course: Prometheus — Overview and Grafana — Observability.

The essential difference between monitoring and observability: monitoring answers questions written in advance ("does the error rate exceed 5%?"). Observability lets you ask a question you had not planned ("which requests took more than 500 ms between 19:33 and 19:34?") and get the answer from what the system has already recorded.

What a grep on the server or an SQL query do not do

Before Prometheus and Grafana, the platform team connected to the server over SSH and typed grep ERROR /var/log/api.log. It works, for a single machine, a single file, a single moment. But:

• A grep only sees what is written on that server, in that file. With several API containers running in parallel, you have to connect to each one and cross-check by hand.
• A grep keeps no history beyond the current file: when the file rotates (log rotation) or the container restarts, whatever was not read is lost.
• A grep computes no trend: it shows lines, not a curve of "percentage of errors over the last five minutes".
• An SQL query on the catalog database (SELECT count(*) FROM inscriptions) says how many enrollments exist right now. It says nothing about HTTP request latency, about the error rate, nor about the state of the containers: that information is not in that database.
• Neither the grep nor the SQL query notifies anyone automatically. You have to remember to run them, again and again, or wait for a user to complain.

Observability fills these gaps: it centralizes the metrics and logs of every instance, it keeps a queryable history, it computes trends, and it notifies automatically, without waiting for someone to ask the question.

The story of the course: the platform, the catalog API and the team lead

You join the team running the online course platform used in the other courses of the catalog. The heart of the system is an API, the catalog API, written in Python with FastAPI, which exposes 64 courses (C0001 to C0064) on the routes /cours, /cours/{id}, /inscriptions, /lent, /sante and /metrics. A second program, the load generator (the charge service), calls this API continuously, as hundreds of students would while browsing the catalog, enrolling in a course, or landing on a page that no longer exists.

Your team lead asks a simple question, but one no grep and no SQL query answers in one go: "Is it working? For whom? Since when? And why does it break?" She does not want to log into the server. She wants a dashboard she can open herself, and an alert that warns her before a student writes in to complain.

That is the thread of the course. Each module adds a piece: metrics and PromQL (module 2), instrumentation of the API itself (module 3), Grafana dashboards (module 4), logs with Loki (module 5), alerts that warn before the complaint (module 6). Module 7 brings it all together. And at every step, the lab lets you deliberately break a piece of the platform (.\labo.ps1 casser api, casser erreurs, casser lenteur, casser disque) so that you learn to read the outage in the tools before fixing it.

The ten services of the lab

Step by step

You have not installed the lab yet: that is the work of lessons 03 and 04. This step-by-step makes you read four real outputs, captured on the course machine, so that you recognize a metric, a log and an alert when you produce them yourself. Keep this page open during lesson 04: you will find each of these four outputs on screen again.

1. A metric, as Prometheus reads it. Every 15 seconds, Prometheus calls http://api:8000/metrics. Here are four lines of that page, on the course machine:

   text

   # HELP http_requetes_total Nombre de requêtes HTTP reçues, par méthode, route normalisée et code de réponse.
   # TYPE http_requetes_total counter
   http_requetes_total{code="200",methode="GET",route="/cours"} 3247.0
   http_requetes_total{code="500",methode="GET",route="/cours"} 32.0

   What to see: a metric has a name (http_requetes_total), labels between braces (code, methode, route) and a value (3247.0). The two lines have the same name but different labels: they are two distinct series. The line # TYPE … counter says that this number only ever increases. Lesson 02 details the four types.

2. A log, as the API writes it. The API writes one JSON line per request on its standard output. .\labo.ps1 journal api displays them; here are two, on the course machine:

   text

   labo-api  | {"horodatage": "2026-09-15T19:33:25.839+00:00", "niveau": "INFO", "id_requete": "46bb533b33e9", "methode": "GET", "route": "/cours/{id}", "code": 200, "duree_ms": 13.9, "message": "GET /cours/C0038 -> 200"}
   labo-api  | {"horodatage": "2026-09-15T19:33:14.781+00:00", "niveau": "ERROR", "id_requete": "dc1c2ff189a6", "methode": "GET", "route": "/cours/{id}", "code": 500, "duree_ms": 0.1, "message": "GET /cours/C0019 -> 500"}

   What to see: a log is a precise event (that particular request, at that particular instant, for course C0038), whereas the metric of step 1 is a cumulative total (3247 successful /cours requests since startup). The id_requete field identifies a unique request; the same value is returned to the client in the x-id-requete HTTP header. The second line is a 500 error: the lab deliberately produces 1% of them in normal operation.

3. An alert, as the on-call team receives it. When the API is stopped (casser api), Prometheus can no longer read /metrics, the APIInjoignable rule switches to firing after 30 seconds, Alertmanager forwards it to the webhook. Here is what http://localhost:8090/alertes.json then contains, on the course machine:

   json

   {"recu_a":"2026-09-15T19:41:26+00:00","etat":"firing","nom":"APIInjoignable","severite":"critique","service":"api","resume":"L'API catalogue ne répond plus","description":"Prometheus n'arrive plus à lire http://api:8000/metrics depuis 30 secondes (cible api:8000).","debut":"2026-09-15T19:41:11.496Z","fin":"0001-01-01T00:00:00Z"}

   What to see: the alert has a name, a severity, a human-readable summary, and a firing state (it is ringing). The fin (end) field holds a null date as long as the alert is ongoing; after reparer, a second notification arrives with "etat":"resolved" and a real end date. Nobody had to refresh a page: the warning light came on by itself.

4. The overview, in one command. .\labo.ps1 etat summarizes the state of the ten services and of the monitoring. On the course machine, in normal operation:

   text

   == Supervision ==
     ✔ Prometheus répond — cibles up : 8/8
        séries en mémoire : 12350
        alertes : 0 active(s), 0 en attente (pending)
     ✔ Alertmanager répond (http://localhost:9093)
     ✔ Grafana répond (http://localhost:3000)
     ✔ Loki répond (http://localhost:3100)
     ✔ API catalogue répond — version 1.0.0, 64 cours
     ✔ Webhook répond — 0 alerte(s) reçue(s) (http://localhost:8090)
   
   Labo : 10/10 services, 8/8 cibles up, 0 alertes actives.

   What to see: 8/8 cibles up (Prometheus reads eight /metrics pages: the ten services minus charge and webhook, which expose none), 12350 series in memory (this number varies from one machine to another), 0 alertes actives (0 active alerts). This is the line you must find again at the end of every practice in the course.

5. The rule that links the metric to the alert. The alert of step 3 did not come out of nowhere: it is written in a file of the kit, prometheus/regles/alertes.yml, each rule of which Prometheus evaluates every 15 seconds (evaluation_interval: 15s in prometheus/prometheus.yml). Here is the first of the ten rules, as it is in the kit:

   yaml

   groups:
     - name: api-catalogue
       rules:
         - alert: APIInjoignable
           expr: up{job="api"} == 0
           for: 30s
           labels:
             severite: critique
             service: api
           annotations:
             resume: "L'API catalogue ne répond plus"
             description: "Prometheus n'arrive plus à lire http://api:8000/metrics depuis 30 secondes (cible {{ $labels.instance }})."

   What to see: expr is a metric (up, the one Prometheus makes itself at every read: 1 if the target answered, 0 otherwise) compared to a value; for: 30s is the duration during which the condition must stay true before the alert rings; resume and description are exactly the text the webhook displayed in step 3, with {{ $labels.instance }} replaced by api:8000. An alert is a metric, a threshold and a duration: nothing more. Module 6 will have you write your own.

If it breaks

This lesson has you install nothing, but three confusions come back as early as lesson 04. The messages below were genuinely triggered on the course machine.

• "The API is down, it answers 404": no. A 404 is a healthy response from a running service saying "this resource does not exist". On the lab, http://localhost:8000/cours/C9999 answers:

  text

  HTTP 404 {"detail":"cours C9999 introuvable"}

  An API that is really stopped answers nothing at all. During casser api, curl http://localhost:8000/sante returns:

  text

  curl: (7) Failed to connect to localhost:8000 after 2237 ms: Could not connect to server

  And Invoke-RestMethod http://localhost:8000/sante under PowerShell: Le délai de l'opération a expiré. (the operation timed out). A 404 is a WARNING-level log and a code="404" metric; a stopped API is up{job="api"} equal to 0.

• "There is no alert, the monitoring is not working": in Prometheus, the ALERTS metric returns an empty result when everything is fine. On the course machine, in normal operation, the query API answers:

  json

  {"status":"success","data":{"resultType":"vector","result":[]}}

  "status":"success" with "result":[]: the query is correct, there is simply nothing to show. This is the normal situation, not an outage. ALERTS only contains a series for a pending or firing alert.

• "A metric and a log are the same information": no. The metric http_requetes_total{code="500",methode="GET",route="/cours/{id}"} 27.0 says there have been 27 errors on this route since startup, without saying which ones. The log "GET /cours/C0019 -> 500" with "id_requete": "dc1c2ff189a6" says precisely which one. The metric is cheap and gets counted; the log is detailed and gets read. Module 5 links them through the id_requete field.

To remember

Monitoring watches thresholds known in advance; observability lets you understand an outage you had not anticipated, from the metrics, logs and traces a system already produces. A metric is a number measured over time, with a name, labels and a value (http_requetes_total{code="200",methode="GET",route="/cours"} 3247.0). A log is a timestamped event, one JSON line per request in the lab, with a unique id_requete. An alert notifies automatically when a condition stays true long enough (APIInjoignable after 30 seconds of unreachable API) and reaches the webhook with a firing then resolved state. A car's dashboard sums it all up: gauges = metrics, black box = logs, warning lights = alerts, GPS = traces, windshield = Grafana. Neither a grep on a single server nor an SQL query on the business database centralizes history, computes a trend, or notifies automatically. A 404 is the healthy response of a running service; a stopped API answers nothing and up equals 0. The thread of the course is the online course platform, its catalog API of 64 courses, its load generator and its ten services.

To go further

• The three post-mortems quoted, to read in full: they are short, written by the teams themselves, and each contains a minute-by-minute timeline resembling the one you will build in module 6 with casser and reparer.
• Prometheus — Overview: the official introduction page, which explains the pull model and the role of each component.
• Google SRE Book — Monitoring Distributed Systems: the chapter that fixed the vocabulary (the four golden signals: latency, traffic, errors, saturation). The lab's « API catalogue — signaux dorés » (catalog API — golden signals) dashboard, which you will open in lesson 04, follows exactly that breakdown.
• Grafana Loki — Overview: why Loki indexes labels and not the text of the logs, the difference with a full-text search engine.
• Lesson 02 revisits every term of this lesson with the real lines of the lab's /metrics: the four metric types, labels, cardinality, pull, and the exact structure of a JSON log line.