Why Elasticsearch, OpenSearch, Kibana, and Neo4j

Audience

Beginner, no prior knowledge required

Duration

20 to 30 min

Module

1/7

Learning Goal

Understand each lab tool's purpose, choose the right data model (table, document, graph) based on the question asked

Who Uses This, and Why

When you type three words into Google and get the answer in a fraction of a second, with your typos corrected, it's not a SQL database parsing billions of pages: it's an inverted index, a dictionary mapping "word → pages containing it". Elasticsearch and OpenSearch apply exactly this principle to your own data. And when Netflix suggests "people who liked this also watched…", or Facebook suggests a friend-of-a-friend, those are relationships being traversed, not rows being filtered: that's the job of a graph database like Neo4j.

What these companies have in common: they also have SQL databases for payments, orders, invoices. They add a search engine alongside for searching, and a graph database alongside for linking. That's exactly what you're going to build in this lab, in miniature.

The Definitions

• Elasticsearch and OpenSearch store JSON documents and perform fast full-text searches, even with typos.
• Kibana and OpenSearch Dashboards don't store data: they let you explore it and create charts and dashboards.
• Neo4j is a graph database, suited for complex relationships like prerequisites, recommendations, and links between users.
• SQL remains preferable for transactional data requiring strong consistency, like payments and invoices.
• The essential principle is to choose the tool based on the question: word search with Elasticsearch/OpenSearch, visualization with Kibana, and relationship exploration with Neo4j.

In One Picture

Imagine a large online school. At the front desk, a librarian finds in one second all courses talking about "deploying containers", even if you wrote "deployment" without an accent: that's Elasticsearch. On the walls, screens display the site's live traffic, error pages, visitor countries: that's Kibana, which stores nothing but draws what Elasticsearch contains. In the academic advising office, a counselor knows the links between people: who took what, which course is a prerequisite for which, which students are similar: that's Neo4j. And OpenSearch? It's the twin librarian, trained at the same school as Elasticsearch, hired by those who want a 100% open-source contract. Four tools, two ways to organize information: by words (search engine) or by links (graph).

How It Works

The Four Tools in the Lab

Elasticsearch and OpenSearch share the same internal engine, Apache Lucene, and the same principle: the inverted index. Instead of reading every document on each search, the engine builds once and for all a dictionary "word → list of documents containing it". Searching "docker kubernetes" then amounts to intersecting two lists of IDs, which takes milliseconds on millions of documents. Neo4j makes the opposite bet: it stores relationships as physical pointers between nodes, so "friends of my friends" is calculated by following arrows, without joins.

Vocabulary Translated Across Databases

Each engine invents its own terminology, but they often refer to the same idea as in relational databases. This table translates basic terms between classic SQL, search engines, and graphs.

Beware the word "index". In SQL, an index is an acceleration structure (B-tree) placed on a table. In Elasticsearch, an index is the equivalent of the table itself. And the inverted index is yet another thing: the internal mechanism "word → documents" that makes search instant. Three meanings for one word.

The Same Example in Three Models

Let's take three concepts from our school: courses, students, reviews left by students on courses.

1. In SQL (tables and joins) — three normalized tables, information is never duplicated, links go through foreign keys:

SELECT c.titre, AVG(a.note)
FROM cours c
JOIN avis a ON a.cours_id = c.id
JOIN etudiant e ON e.id = a.etudiant_id
WHERE e.ville = 'Montréal'
GROUP BY c.titre;

Perfect for consistency (a changed review is changed everywhere), less good for "all courses whose description contains deploy": LIKE '%deploy%' scans the entire table.

2. In Documents (Elasticsearch / OpenSearch) — a course is a single JSON document that includes what search needs, including the teacher's name and average rating already calculated. Here, exactly as is, the first document from the cours index in the lab:

{
  "id": "C0001",
  "titre": "Docker expliqué simplement",
  "description": "Dans ce cours accessible sans prérequis, vous apprenez à automatiser vos applications avec Docker. …",
  "categorie": "DevOps", "sujet": "Docker", "niveau": "debutant", "langue": "en",
  "prix": 129, "gratuit": false, "duree_heures": 5,
  "tags": ["docker", "linux", "helm", "devops"],
  "date_publication": "2024-08-14",
  "note_moyenne": 4.4, "nb_avis": 327,
  "professeur": { "id": "P001", "nom": "Karim Caron", "ville": "Gatineau" },
  "competences": ["Conteneurisation", "Intégration continue"]
}

The teacher's name is denormalized (copied into each course). If he changes his name, you'd need to reindex his courses: that's the price of instant search. Reviews live in a second index, avis, with cours_id as a simple field, no join possible at query time.

3. In Graph (Neo4j) — the same concepts become nodes, and links become first-class relationships that carry their own properties. Here are three real relationships from the lab graph around course C0001:

(:Etudiant {prenom: "Hugo"})-[:INSCRIT_A {progression: 100, note: 4}]->(:Cours {id: "C0001", titre: "Docker expliqué simplement"})
(:Cours {id: "C0001"})-[:PREREQUIS_DE]->(:Cours {id: "C0003", titre: "Docker : le guide complet"})
(:Professeur {prenom: "Karim", nom: "Caron"})-[:ENSEIGNE]->(:Cours {id: "C0001"})

The question "which courses have been taken by students who liked the same course as me?" is written in one line of Cypher and calculated by following three relationships, where SQL would chain three joins and Elasticsearch simply wouldn't know how to answer.

Architecture of a Typical Pipeline

In real life, logs arrive continuously (Filebeat, Logstash, or the app itself) and the graph database is fed by the business system. In the lab, two commands replace all that: importer sends NDJSON files to Elasticsearch's _bulk API, charger-graphe runs a Cypher script that reads CSVs. The two worlds don't communicate: you, or your app, choose who to ask each question.

Step by Step

Nothing to type in this lesson: the lab starts in lesson 03. The excerpts below were taken from the lab itself, so you know what you'll find in it.

1. The Dataset: An Online Course Platform. A single universe feeds all three engines, to compare what each does best.

What to notice: the 504 courses in the graph are the same as those in the courses index (same IDs C0001…C0504). You can search for a course in Elasticsearch then explore its prerequisites in Neo4j.

2. Categories are perfectly balanced. A terms aggregation on categorie returns:

Cloud 84 · DevOps 84 · Data 84 · Web Development 84 · AI 84 · Security 84

And on other fields: level → beginner 252, advanced 158, intermediate 94 ; language → fr 375, en 129 ; free → 75 free courses ; 72 distinct subjects (Docker, Kubernetes, Elasticsearch, Neo4j, React, AWS…) ; 30 teachers. What to notice: round numbers known in advance, convenient for checking your queries in modules 2 and 3.

3. A review, exactly as stored. First document from the reviews index:

{ "id": "A00001", "cours_id": "C0028", "etudiant": "Nathan", "ville": "Sherbrooke", "pays": "Canada",
  "note": 3, "texte": "Les vidéos sont bonnes, la partie théorique est dense.", "date": "2024-06-21", "utile": 33 }

What to notice: cours_id is a plain string. No join links this review to its course in Elasticsearch; your app makes the connection.

4. A Log Line. The most recent from the access index:

{ "id": "L012000", "@timestamp": "2026-09-08T23:39:29.000Z", "methode": "GET", "chemin": "/cours/C0430",
  "cours_id": "C0430", "categorie": "Cloud", "statut": 200, "octets": 168403, "duree_ms": 351,
  "ip": "169.199.97.7", "pays": "CA", "appareil": "desktop", "navigateur": "Chrome", "referent": "direct" }

Of the 12,000 lines: 10,829 in 200, 403 in 404, 355 in 301, 235 in 304, 122 in 500, 56 in 503. What to notice: the @timestamp field, essential to Kibana Discover for the time axis (module 4).

5. The Graph, counted by label and relationship type. Real result of two Cypher queries:

Courses 504 · Students 300 · Teachers 30 · Skills 22 · Cities 16
ENROLLS_IN 1654 · COVERS 994 · TEACHES 504 · LIVES_IN 330 · PREREQUISITE_FOR 230

What to notice: 1,654 enrollments for 300 students, or about 5.5 courses per student. It's this mesh that will make the recommendations in module 7 interesting.

If It Goes Wrong

• "I know SQL, why not do everything with PostgreSQL?" → You can, up to a point. PostgreSQL can do full-text search and recursive queries. But as soon as you want typo tolerance, relevance ranking, aggregations on millions of rows in real time, or variable-depth graph traversal, each specialized engine does the job in milliseconds where SQL takes exotic indexes and unreadable queries. The right reflex: SQL for transactional truth, Elasticsearch alongside for searching, Neo4j alongside for linking.

• "Elasticsearch or OpenSearch, do I have to choose now?" → No. This course has you work with Elasticsearch; module 5 replays the same queries on OpenSearch and lists the differences (a handful). What you learn applies to both.

• "Kibana, is it a database?" → No, and it's a common confusion. Kibana only stores your configuration (dashboards, views) in a hidden Elasticsearch index. All data you see comes from Elasticsearch; if Elasticsearch is down, Kibana displays "Kibana server is not ready yet" (lesson 04).

Key Takeaways

• Elasticsearch and OpenSearch organize information by words (inverted index); Neo4j organizes it by links (physical relationships). Kibana and OpenSearch Dashboards store nothing: they display.
• A JSON document is denormalized: it carries what's needed for search, at the cost of duplications. A SQL table normalizes; a graph links.
• Ask the question before choosing the tool: "which documents talk about…" → search engine ; "how is X linked to Y" → graph.
• The course dataset: 504 courses, 609 reviews, 12,000 accesses, 872 nodes and 3,712 relationships, with same course IDs everywhere.
• Lab versions are frozen (9.5.3 / 3.8.0 / 5.26): every result shown in the course was obtained with exactly these versions.

Going Further

The word "index" means two different things: in SQL, a B-tree acceleration structure attached to a table; in Elasticsearch, the equivalent of the table itself, split into shards (autonomous Lucene indexes). Module 2, lesson 01, covers this vocabulary in practice on the lab.