Quiz#
RAGAS Evaluation Metrics#
Question 1: Scenario: The LLM gives a brilliant, factually correct answer based on its pre-trained knowledge, but the retrieved context from your database contained entirely unrelated text. What is the Faithfulness score?
A. 1.0, because the answer is factually true.
B. 0.8, because it ignored the prompt.
C. 0.5, because the context was ignored.
D. 0.0, because none of the statements can be inferred from the retrieved context.
Answer: D
Question 2: If a user asks ‘What is the capital of Japan?’ and the LLM responds ‘Tokyo is a city in Japan with a large population, famous for cherry blossoms, and it serves as the capital.’, which metric might flag this answer as suboptimal?
A. Faithfulness (due to hallucination).
B. Context Recall (due to missing info).
C. Answer Relevancy (due to redundant/extra information not directly addressing only the prompt).
D. Context Precision (due to bad ranking).
Answer: C
Question 3: How does Context Precision handle irrelevant chunks that appear high up in the retrieved results (e.g., Position 1)?
A. It significantly penalizes the score because it calculates the ratio of relevant contexts at each top-k position.
B. It ignores them as long as a relevant chunk is at Position 5.
C. It boosts the score to encourage diversity.
D. It forces the LLM to rewrite the context.
Answer: A
Question 4: Scenario: You have an expert reference answer containing 4 key claims. Your retrieval system pulls contexts that only support 1 of those claims. What is the Context Recall score?
A. 1
B. 0.25 (1/4)
C. 0.5
D. 0
Answer: B
Question 5: Why does Ragas use an LLM (like GPT-4) as a ‘Judge’ for its metrics?
A. Because humans are incapable of reading RAG outputs.
B. To automate the evaluation process, minimizing the high costs and time associated with human ground-truth annotation.
C. To generate hypothetical vectors.
D. Because it is required by the Neo4j database.
Answer: B
Question 6: In the calculation of Answer Relevancy, why are ‘reverse-engineered’ questions generated?
A. To compare their embedding similarity against the original user question; high similarity means the answer directly addressed the prompt.
B. To train a new embedding model.
C. To populate the Graph database.
D. To ask the user for clarification.
Answer: A
Question 7: Which two Ragas metrics are specifically focused on evaluating the ‘Retrieval’ performance of a RAG system?
A. Faithfulness and Answer Relevancy
B. Answer Correctness and Faithfulness
C. Context Precision and Context Recall
D. Latency and Cost
Answer: C
Question 8: Which two Ragas metrics are specifically focused on evaluating the ‘Generation’ performance of a RAG system?
A. Faithfulness and Answer Relevancy
B. Context Precision and Context Recall
C. Retrieval Latency and Token Cost
D. Context Recall and Faithfulness
Answer: A
Question 9: If a RAG system has High Context Recall but Low Context Precision, what does this indicate about the retrieved chunks?
A. It found no useful information.
B. It found all the necessary information, but buried it among a lot of irrelevant noise (poor ranking).
C. It hallucinated the answer.
D. It ranked the exact right answer at position 1, but missed everything else.
Answer: B
Question 10: What is the first step in the calculation process for Context Recall?
A. Reverse-engineering questions.
B. Calculating cosine similarity.
C. Splitting the ‘reference answer’ (ground truth) into individual sentences/claims.
D. Generating an answer using GPT-4.
Answer: C
Question 11: In the ‘Green Tea’ Context Precision example, why does an irrelevant context at position 2 lower the final score?
A. Because Precision@2 drops to 0.5, pulling down the weighted average for subsequent relevant chunks.
B. Because the LLM deletes the irrelevant chunk.
C. Because it triggers a Faithfulness penalty.
D. Because it changes the user’s original question.
Answer: A
Question 12: What does it mean if Faithfulness evaluates to exactly 1.0?
A. The answer is 100% factually accurate to the real world.
B. The answer contains exactly 100 words.
C. Every single statement made in the generated answer can be directly supported by the retrieved context.
D. The retrieval process took exactly 1 second.
Answer: C
Question 13: Why is Answer Relevancy NOT considered a measure of ‘factuality’?
A. Because it uses BM25 instead of vectors.
B. Because it only checks if the answer conceptually aligns with what was asked, not whether the facts stated are true.
C. Because GPT-4 cannot evaluate facts.
D. Because it only measures the speed of the response.
Answer: B
Question 14: If your RAG system suffers from ‘hallucinations’, which metric will most directly drop?
A. Context Precision
B. Context Recall
C. Answer Relevancy
D. Faithfulness
Answer: D
Question 15: In the calculation process for Faithfulness, what happens after the answer is decomposed into claims?
A. The claims are translated.
B. The LLM verifies each statement to see if it can be inferred from the context.
C. The claims are stored in Neo4j.
D. The context is deleted.
Answer: B
Question 16: If a generated answer lacks necessary details requested in the prompt (e.g., asking for location and capital, but only giving location), what happens to Answer Relevancy?
A. It increases because the answer is shorter.
B. It stays the same.
C. It decreases because the reverse-engineered questions will not match the full scope of the original prompt.
D. It forces a re-retrieval.
Answer: C