Exercise 2 — Evaluate the RAG on 10 golden questions

Guided practice5 min
Time
30 min
You need
the kit, the venv, Ollama running, the Chroma store of the manuals
Deliverable
week13/golden.json, week13/eval_results.csv, and your two rates

The lab kit of the course: https://github.com/hrhouma2/aiopsatlas-ml-data-diagnostics-labs-en

Goal

Your RAG answers questions about the manuals. How often is it right? Today you write ten questions that the six manuals can answer, with the manual that holds the answer and a keyword the answer must contain. Then you score two things: did the retriever bring the right manual in its top 3, and did the model's answer contain the keyword. You will find a failure and say where it happened.

Preview — the rest of the lesson is for enrolled readers.

Already enrolled with a code?

Your access is tied to your account, not to this link. Sign in with the same email you used in class: your course is waiting, no need to enter the code again.

Sign inNo account yet? Create one
This lesson is part of the “Week 13 — Observability and model reliability” module

The first modules of the course are open to everyone. For the rest you have three options: buy this course once and for all, subscribe, or enter the code handed out in class.

Are you a student on this course?

The code is tied to your account: sign in or create an account and it will be applied automatically when you come back.

No account yet? Create one