Read the Status and Fix It in 5 Minutes

10 min
Audience
beginner, lab started at least once
Duration
30 min
Module
1/7
Learning goal
diagnose a lab failure in three moves, recognize the twelve classic error messages, and apply the right remedy without breaking your data

In One Picture

When a car refuses to start, a good mechanic does not replace the engine: they look at the dashboard (which warning light is on?), they listen (what noise, at what moment?), and only as a last resort do they reset everything. The lab is troubleshot the same way. Move 1, etat: the dashboard — what is running, what answers. Move 2, journal <service>: the noise — the last hundred lines of what the service says, where the sentence that explains everything is almost always found. Move 3, reinitialiser: the reset, which erases containers and data and gives you back a fresh lab in two minutes. Between move 2 and move 3, this lesson gives you a catalog of twelve failures seen over and over, each with its exact message, its cause, and its remedy. Nine times out of ten, you will not go as far as move 3.

How It Works

What each move tells you:

MoveCommandWhat you readHow long
1./labo.sh etat · .\labo.ps1 etatSTATUS column: Up … (healthy), Up … (unhealthy), Exited (137), or missing container; then ✔/✘ per service and the counters10 s
2./labo.sh journal elasticsearch (or kibana, neo4j, opensearch)last 100 lines; look for ERROR, FATAL, bootstrap check, Exception1 min
3./labo.sh reinitialiser then demarrer, importer, charger-grapheasks for confirmation (oui); removes containers and volumes3 min

Move 3 only touches the labo-* containers and the three volumes labo-recherche-graphes_es-data, _neo4j-data, _os-data. You lose what you created yourself (sandbox indexes, dashboards, test nodes); the course data is reloaded with importer and charger-graphe.

A healthy log, so that you can later recognize a sick one (three real lines captured with journal):

text
labo-elasticsearch  | {"@timestamp":"2026-09-09T13:39:45.630Z","log.level": "INFO", "current.health":"GREEN","message":"Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[cours][0]]])." …}
labo-kibana         | [2026-09-09T13:37:16.402+00:00][INFO ][status] Kibana is now available
labo-neo4j          | 2026-09-09 13:38:32.401+0000 INFO  Started.

Elasticsearch speaks in JSON (one line per event, message field), Kibana in brackets [date][LEVEL][module], Neo4j in plain text. The three sentences above are the ones we want to see. Kibana also emits harmless WARN lines (Error initializing AI assistant resources: Platinum, Enterprise or trial license needed, Error while trying to load prerelease flag): ignore them.

Step by Step

The twelve failures, in order of frequency in the classroom. The messages marked "captured" were reproduced on the course lab; the others are quoted as the engines emit them.

  1. Port already in use (captured). demarrer fails immediately with:

    text
    Error response from daemon: failed to set up container networking: driver failed programming external connectivity on endpoint labo-elasticsearch (…): Bind for 0.0.0.0:9200 failed: port is already allocated

    Cause: another program is already listening on 9200 (a "bare-metal" Elasticsearch install, another Docker project), or on 5601, 7474, 7687. Remedy: prerequis tells you which port; find the culprit (Get-NetTCPConnection -LocalPort 9200 in PowerShell, sudo ss -ltnp | grep 9200 on Linux/macOS), stop it or, if you must keep it, change the host-side port in docker-compose.yml ("9202:9200") and adapt the addresses in the course.

  2. Container Exited (137). etat shows labo-elasticsearch Exited (137) 2 minutes ago, and the log stops dead without an error message. Cause: code 137 = killed by signal 9, almost always the OOM killer: Docker does not have enough memory for three JVMs (Elasticsearch and Neo4j each claim up to 1 GB of heap, plus Kibana). docker inspect labo-elasticsearch --format '{{.State.OOMKilled}}' answers true. Remedy: lesson 02, step 1 or 2: raise Docker's memory to 6 GB (.wslconfig then wsl --shutdown on Windows), then demarrer. Do not enable the OpenSearch profile under 6 GB.

  3. Linux: vm.max_map_count too low. Elasticsearch (or OpenSearch) exits within a few seconds and its log ends with:

    text
    bootstrap check failure [1] of [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
    ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/labo.log

    Cause: a limit of the host Linux kernel, which the container cannot change by itself. Remedy: sudo sysctl -w vm.max_map_count=262144, make it permanent in /etc/sysctl.conf (lesson 02, step 3), then demarrer. Under Docker Desktop (Windows, macOS), the value is already 262144.

  4. "Kibana server is not ready yet". The page http://localhost:5601 shows only this sentence. Cause: Kibana is waiting for Elasticsearch. Either it is too early (the first 40 seconds), or Elasticsearch has gone down (failure 2 or 3), and journal kibana repeats Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 172.x.x.x:9200. Remedy: etat. If Elasticsearch is healthy, wait a minute and reload; otherwise, fix Elasticsearch first, Kibana will follow on its own (restart: unless-stopped).

  5. Disk almost full: read-only indexes. At 95% disk usage, a standard Elasticsearch logs flood stage disk watermark [95%] exceeded on […] all indices on this node will be marked read-only and every write answers 429: cluster_block_exception … blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block]. In the lab, you will not see this message: the compose file sets cluster.routing.allocation.disk.threshold_enabled=false (verifiable with GET _nodes/settings?filter_path=nodes.*.settings.cluster.routing). The remaining risk is no space left on device in the log during importer. Remedy: docker system df to measure, docker image prune to free space, then rerun importer.

  6. Neo4j refuses the password (captured). In Neo4j Browser: Connection to instance failed — The client is unauthorized due to authentication failure. (details: Neo.ClientError.Security.Unauthorized). In etat: ✘ Neo4j ne répond pas, and on the command line cypher-shell says The client is unauthorized due to authentication failure. Cause A: typo — the password is aiopsatlas2026. Cause B, sneakier: the neo4j-data volume was created during a previous startup with another password; NEO4J_AUTH only applies at the very first initialization, and the log says so in black and white (captured line): Changed password for user 'neo4j'. IMPORTANT: this change will only take effect if performed before the database is started for the first time. Remedy A: retype it. Remedy B: reinitialiser then demarrer and charger-graphe.

  7. Container name conflict (captured). demarrer fails with:

    text
    Error response from daemon: Conflict. The container name "/labo-elasticsearch" is already in use by container "a26fe0358d63…". You have to remove (or rename) that container to be able to reuse that name.

    Cause: a labo-elasticsearch container already exists but does not belong to this Compose project, typically because you cloned the kit into a second folder, or ran a docker run --name labo-elasticsearch by hand. Remedy: docker rm -f labo-elasticsearch (the container only; the volume data remains), then demarrer. If the conflict affects all five names, docker rm -f $(docker ps -aq --filter name=labo-).

  8. Git Bash rewrites /labo paths (captured). On Windows, in Git Bash, a hand-typed command such as docker compose exec -T neo4j ls /labo/cypher answers:

    text
    ls: cannot access 'C:/Program Files/Git/labo/cypher': No such file or directory

    Cause: Git Bash's MSYS emulation converts every argument that starts with / into a Windows path before passing it to docker. Remedy: labo.sh exports MSYS_NO_PATHCONV=1 and MSYS2_ARG_CONV_EXCL='*' on its second line, so every command in the script is protected. For your own commands in Git Bash, prefix them: MSYS_NO_PATHCONV=1 docker compose exec -T neo4j ls /labo/cypher (captured: the list of .cypher files appears).

  9. PowerShell 5.1 and Invoke-RestMethod break accented characters (captured). You send a query with an accented word from PowerShell:

    powershell
    $corps = '{"analyzer":"french","text":"déployés en production"}'
    Invoke-RestMethod -Uri http://localhost:9200/cours/_analyze -Method Post -ContentType 'application/json' -Body $corps

    PowerShell 5.1 answers Le serveur distant a retourné une erreur : (400) Demande incorrecte.; on the Elasticsearch side the reason is x_content_parse_exception … Invalid UTF-8 middle byte 0x70. Cause: PowerShell 5.1 encodes the body in Latin-1 while Elasticsearch expects UTF-8; the same command under PowerShell 7 works. Remedy: use Kibana Dev Tools for every query in the course (UTF-8 guaranteed); labo.ps1 itself uses curl inside the containers. To script anyway: PowerShell 7, or -Body ([System.Text.Encoding]::UTF8.GetBytes($corps)).

  10. Docker Desktop is not running. Every docker command (hence prerequis, etat, demarrer) fails with:

    text
    error during connect: Get "http://%2F%2F.%2Fpipe%2FdockerDesktopLinuxEngine/v1.51/info": open //./pipe/dockerDesktopLinuxEngine: The system cannot find the file specified.

    (Cannot connect to the Docker daemon at unix:///var/run/docker.sock on macOS/Linux.) prerequis translates: ✘ le démon Docker ne répond pas — lancez Docker Desktop et attendez l'icône verte. Cause: Docker Desktop closed, or still starting after a Windows session. Remedy: launch Docker Desktop, wait until the icon stops animating, rerun the command. Containers set to restart: unless-stopped restart by themselves.

  11. Very slow download or TLS handshake timeout. During == Téléchargement des images ==:

    text
    Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: TLS handshake timeout

    or failed to copy: read tcp … connection reset by peer. Cause: saturated network, school Wi-Fi, corporate proxy; the images weigh 2.5 to 2.8 GB each. Remedy: nothing to repair, rerun demarrer: Docker resumes the layers already downloaded (Already exists). Behind a proxy, configure it in Docker Desktop → Settings → Resources → Proxies.

  12. Full volume or corrupted data: the reset. Varied symptoms: etat shows the cluster in red, the Elasticsearch log mentions CorruptIndexException or no space left on device, Neo4j stays stuck in recovery after a brutal PC shutdown, or more simply nothing makes sense anymore after a risky manipulation. Remedy, move 3:

    bash
    ./labo.sh reinitialiser      # répondre : oui
    ./labo.sh demarrer
    ./labo.sh importer
    ./labo.sh charger-graphe
    powershell
    .\labo.ps1 reinitialiser
    .\labo.ps1 demarrer
    .\labo.ps1 importer
    .\labo.ps1 charger-graphe

    What you should see: reinitialiser runs docker compose --profile opensearch down -v --remove-orphans and confirms ✔ labo remis à zéro. The images stay in cache: no new download, and the following trio takes two to three minutes. What you need to know: arreter (without -v) keeps the data; reinitialiser erases it — that is the only difference.

If Something Goes Wrong

  • etat lists no container and demarrer recreates everything every time → You are running the script from a second copy of the repository. Keep a single copy of the kit and always run the commands from its root.

  • journal shows thousands of unreadable JSON lines → Normal for Elasticsearch. Filter on keywords: in PowerShell .\labo.ps1 journal elasticsearch | Select-String 'ERROR|WARN|bootstrap', in bash ./labo.sh journal elasticsearch | grep -E 'ERROR|WARN|bootstrap'.

  • After reinitialiser, Kibana again shows « Vos données ne sont pas sécurisées » (Your data is not secure) → Normal banner of a Kibana without security; it comes back because Kibana's configuration was erased along with the volume. Click « Rejeter » (Dismiss).

Key Takeaways

  • Three moves, always in this order: etat (what is wrong), journal <service> (why), reinitialiser (last resort, erases the lab data but not the images).
  • The codes to recognize: Exited (137) = memory; port is already allocated = port taken; vm.max_map_count [65530] is too low = Linux to configure; Neo.ClientError.Security.Unauthorized = password; container name … already in use = docker rm.
  • The kit defuses two traps in advance: Elasticsearch's disk threshold is disabled, and labo.sh neutralizes Git Bash's path conversion.
  • On Windows, do not send accented characters through Invoke-RestMethod in PowerShell 5.1: Dev Tools and labo.ps1 handle it cleanly.
  • arreter keeps the data, reinitialiser erases it. The course data can always be reloaded with two commands.

Going Further

docker inspect --format '{{.State.Health.Status}}' labo-elasticsearch returns starting, healthy, or unhealthy: this is exactly what demarrer reads every three seconds. {{json .State.Health.Log}} shows the last five healthcheck results.