How To Evaluate LLM Hallucinations and Faithfulness

Large language models are now embedded in real products, not just demos, and that’s where accuracy stops being a “nice to have” and starts becoming critical. I’ve seen firsthand how confident-sounding but incorrect outputs can quietly break trust, especially in domains like healthcare, law, and education, where users assume answers are grounded in facts.

The real challenge isn’t just avoiding wrong answers, but understanding why a model drifts from the source in the first place. That’s why I’m focusing this guide on faithfulness and hallucinations, not as abstract concepts, but as practical evaluation signals you can measure and improve. A basic understanding of tokenization also helps here, because once you see how text is broken down internally, the source of many hallucination patterns becomes easier to detect and evaluate.

What is Faithfulness in LLM?

Faithfulness means the model stays strictly aligned with the information it was given. In practice, I think of a faithful response as one that neither invents details nor subtly reshapes the original meaning. The output should reflect the source exactly, without adding, removing, or reinterpreting key facts.

For example, a user asking “Who led the Salt March and why?” and in the source doc says, “Mahatma Gandhi led the Salt March in 1930 to protest against British rule in India.”, but the model says “Jawaharlal Nehru led the Salt March in 1942 to protest high taxes” is not correct, the model changes the fact. 

The LLM model returns the unfaithful answer. The model was given the correct information but provided an incorrect response. This is unfaithful; the model did not stick to the source.

A faithful response = No made-up information + directly backed by the source.

What are LLM Hallucinations?

Hallucinations occur when a language model produces responses that sound correct but are not grounded in facts or the provided source. From my experience, these are the most dangerous failures because they don’t look wrong at first glance; they often read as confident, well-structured answers that quietly introduce false information. For example, if you are asking something like “Explain about LangChain”,  and LLM (gpt-3.5-turbo) replied with the name “LangChain is a company that provides pre-trained translation models for multilingual chatbots.” 

The model’s answer sounds believable, but it is not true. Since GPT-3.5-turbo was trained before the release of LangChain, it cannot know about the tool accurately. LangChain is not a company that builds translation models; it’s an open-source framework for developing LLM-powered apps. This is a hallucinated and unfaithful response because it changes the meaning and adds incorrect details that were never in the source.

The hallucination is a big problem, especially when people use the LLM model for important tasks like medical advice or legal help. That's why it is important to detect and reduce hallucinations. For instance, when working with multimodal systems that also generate speech, exploring tools for text-to-speech TTS can help evaluate how outputs are communicated beyond text.

2 Types of Hallucinations in LLMs

1. Factuality Hallucinations
2. Faithfulness Hallucinations

What are Factuality Hallucinations in LLM?

Factuality hallucination happens when the LLM model gives information that sounds true but is factually wrong. For example, you ask questions like “Who is the president of India in 2024?”, the model replies “Amit Shah is the president of India.” 

The sentence sounds correct, but it's factually wrong. The real president is Droupadi Murmu in 2024. This is a factual hallucination. These kinds of mistakes are risky, especially in areas like news, education, and healthcare. That’s why checking for factual accuracy is very important when using AI.

Two kinds of factuality hallucinations

Factual Inconsistency : 

Factual Inconsistency means the model gives answers that don't match the facts in the original information. It may change, add, or remove important details. For example, you ask the model, “When was the COVID-19 vaccine first rolled out?”, and if  LLM replies, “The COVID-19 vaccine was first rolled out in 2021.”This is a factual inconsistency because the actual year is 2020, but it changed the year. 

Factual Fabrication : 

Factual Inconsistency means the model makes up information that is not in the original source and is not true. It creates facts that were never mentioned. 

For example, if the source says, “Unicorns are mythical creatures often described as white horses with a single horn on their forehead" and you ask the model, "Where do unicorns live" but it replies, "Unicorns live in the forests of Scotland and are often seen by travelers" this is factual fabrication. 

The model added a fake detail (about unicorn lives but its methodology character) that was not in the original text. This kind of error is risky because the answer sounds real but is actually made up.

What are Faithfulness Hallucinations in LLM?

Faithfulness Hallucinations happen when the model doesn’t stick to the input or instruction. 

Three kinds of faithfulness hallucinations

1. Instruction Inconsistency → The model ignores the user's instructions.

Example: Instruction: “Translate this question to Spanish.” The model gives the answer in English instead.

2. Context Inconsistency → The model says something that doesn’t match the provided information.

Example:If the context says, “Ananya Sharma was born in Chennai, India,” and the user asks, “Where was Ananya Sharma born?” but the model replies, “Ananya Sharma was born in Mumbai,” this is context inconsistency. 

The model had the correct information, but gave an answer that contradicts it. Even though the question is simple and the answer sounds real, it doesn’t match the provided details, which makes it incorrect.

3. Logical Inconsistency → The model starts right, but makes a logical or calculation mistake. Example: In a math problem, it begins solving correctly, but messes up the final step.

Why Do We Need To Evaluate Hallucinations And Faithfulness? 

Evaluating hallucinations and faithfulness matters because confident-looking answers are often mistaken for correct ones. I’ve seen teams trust LLM outputs simply because they read well, only to discover later that subtle factual drift had already entered production workflows. Sometimes, LLMs can sound confident but give wrong or made-up information (hallucinations), or they may change the original meaning (lack of faithfulness). Since these systems rely on machine learning techniques trained on vast datasets, errors can easily propagate if left unchecked.

This can cause problems, especially in areas like healthcare, law, or education, where accurate information is very important. By checking how often LLMs make these mistakes, researchers and developers can improve their models to be more reliable, helpful, and safe to use in real-life situations.

How To Evaluate Hallucinations And Faithfulness In 7 Steps?

Human Evaluation – A person reads the LLM answer and checks if it’s correct and matches the source. This is the most accurate method.

Automatic Evaluation – Tools or models check if the LLM answer is supported by the source using techniques like: 

• Similarity checks (comparing answer and source)
• Fact-checking models (detecting unsupported claims)
• NLI (Natural Language Inference) (seeing if the answer logically follows the source). Many teams host these checks in UI for LLM with Gradio.

Code

This code creates a simple web app where you upload a PDF and ask a question. It reads the PDF, finds the most relevant sentence, and gives a sample answer. Then, it uses DeepEval to check how good the answer is. 

DeepEval tells if the answer has any made-up information (hallucination) or if it changes the meaning of the original text (faithfulness). It gives scores and reasons so you can understand how correct and reliable the answer is.

 Step 1: Import Dependencies

import gradio as gr
from sentence_transformers import SentenceTransformer
import torch
import fitz
import os
from openai import OpenAI
from deepeval.metrics import HallucinationMetric, FaithfulnessMetric
from deepeval.test_case import LLMTestCase
from google.colab import userdata	

These libraries help us:

• These are evaluation metrics from the DeepEval library. They help check if a model’s output is hallucinated (made-up) or faithful (true to the source).
• Wraps the model’s input, output, and context into a test case that can be passed into evaluation metrics.
•  Fitz from PyMuPDF helps read PDF files and extract text from them.

Step 2: Environment setup and loading the Sentence Embedding Model

# Set OpenAI key from Colab secrets
os.environ["OPENAI_API_KEY"] = userdata.get("OPEN_AI_API_KEY")


# Init OpenAI client
client = OpenAI()

model = SentenceTransformer('all-MiniLM-L6-v2')

It converts text (like questions or sentences) into numerical vectors for comparison.

Step 3: Initialize Evaluation Metrics

hallucination_metric = HallucinationMetric(threshold=0.5)
faithfulness_metric = FaithfulnessMetric()

We create two metrics:

• hallucination_metric: Checks if the response contains any made-up facts.
• faithfulness_metric: Checks if the response follows the context properly.

Step 4: Extract Text from PDF Function

    text = ""
    for page in doc:
        text += page.get_text()def extract_text_from_pdf(pdf_file):
    doc = fitz.open(pdf_file.name)  # FIX: open using file path


    return text

Open the uploaded PDF file using its filename. Loop through all the pages in the PDF and collect all the text into one string. Then return the complete text.

Step 5: Get GPT response

def get_llm_response(context, question):
    prompt = f"Context:\n{context}\n\nQuestion: {question}\nAnswer:"
    response = client.chat.completions.create(
        model="gpt-4o-mini", 
        messages=[{"role": "user", "content": prompt}],
        temperature=0.3
    )
    return response.choices[0].message.content.strip()

Suggested Reads- What are Temperature, Top_p, and Top_k in AI?

This function, get_llm_response, is used to ask a question to the LLM (like GPT-4o-mini) using a given context. It first creates a prompt by combining the context (a short part of the PDF related to the question) and the question itself. 

Then it sends this prompt to the LLM using the new chat.completions.create method. The model reads the context, understands the question, and generates an answer. Finally, the function returns that answer as a clean string. 

This function is helpful when you want the LLM to answer based on specific information, like from a PDF file.

 Step 6: Main Logic for PDF QA and Evaluation

def process_pdf_and_question(pdf_file, question):
    try:
        text = extract_text_from_pdf(pdf_file)
        reference_texts = [sent.strip() for sent in text.split(".") if sent.strip()]


        if not reference_texts:
            return "", "", "", "No text extracted from PDF.", ""


        question_embedding = model.encode(question, convert_to_tensor=True)


        best_score = -1.0
        best_context = ""
        for ref_text in reference_texts:
            ref_embedding = model.encode(ref_text, convert_to_tensor=True)
            similarity = torch.nn.functional.cosine_similarity(
                question_embedding.unsqueeze(0), ref_embedding.unsqueeze(0)
            ).item()
            if similarity > best_score:
                best_score = similarity
                best_context = ref_text


        model_output = get_llm_response(best_context, question)


        test_case = LLMTestCase(
            input=question,
            actual_output=model_output,
            context=[best_context],
            retrieval_context=[best_context]
        )
        hallucination_metric.measure(test_case)
        faithfulness_metric.measure(test_case)


        return (
            question,
            best_context,
            model_output,
            f"Hallucination Score: {hallucination_metric.score:.2f}\nReason: {hallucination_metric.reason}",
            f"Faithfulness Score: {faithfulness_metric.score:.2f}\nReason: {faithfulness_metric.reason}"
        )
    except Exception as e:
        return "", "", "", f"❌ Error: {str(e)}", ""

This function runs when a user uploads a PDF and asks a question. Extract all text from the PDF and then split it into sentences. This kind of preprocessing often uses chunking strategies in rag to ensure long documents are broken into manageable and retrievable pieces. If the PDF is not present, then return the error. Query_embedding is used to convert the user query into embeddings.

best_score = -1
best_context = ""

Initialise variables to track the best match.

for ref_text in reference_texts:
            ref_embedding = model.encode(ref_text, convert_to_tensor=True)
            similarity = torch.nn.functional.cosine_similarity(
                question_embedding.unsqueeze(0), ref_embedding.unsqueeze(0)
            ).item()

For each sentence from the PDF :

• Convert it into an embedding
• Compare it with the question using cosine similarity
• Higher similarity = better match

 if similarity > best_score:
                best_score = similarity
                best_context = ref_text

If this sentence is the best match so far, update the best_score and best_context.

model_output = get_llm_response(best_context, question)

This line calls the LLM to get an answer. It sends the best context (the most relevant sentence from the PDF) and the question to the get_llm_response function, which returns the model's response based on that context.

Suggested Reads- Unlocking LLM Potential Through Function Calling

test_case = LLMTestCase(
            input=question,
            actual_output=model_output,
            context=[best_context],
            retrieval_context=[best_context]
        )

 Wrap the input question, response, and matched context into a test case object.

 hallucination_metric.measure(test_case)
 faithfulness_metric.measure(test_case)

Run the evaluation to generate:

• Hallucination score (is anything made up?)
• Faithfulness score (does it follow the context?)

return (
            question,
            best_context,
            model_output,
            f"Hallucination Score: {hallucination_metric.score:.2f}\nReason: {hallucination_metric.reason}",
            f"Faithfulness Score: {faithfulness_metric.score:.2f}\nReason: {faithfulness_metric.reason}"
        )

Send back everything:

• Original question
• Best-matched context
• Fake response
• Similarity score
• Hallucination & Faithfulness result

 Step 7: Gradio Interface Setup

demo = gr.Interface(
    fn=process_pdf_and_question,
    inputs=[
        gr.File(label="Upload PDF", file_types=[".pdf"]),
        gr.Textbox(label="Ask a Question")
    ],
    outputs=[
        gr.Textbox(label="Your Question"),
        gr.Textbox(label="Best Matched Context"),
        gr.Textbox(label="LLM Output (OpenAI GPT)"),
        gr.Textbox(label="Hallucination Evaluation"),
        gr.Textbox(label="Faithfulness Evaluation")
    ],
    title="PDF QA + Hallucination & Faithfulness Checker (OpenAI GPT)",
    description="Upload a PDF and ask a question. The system finds the best context, gets a GPT answer, and evaluates hallucination & faithfulness using DeepEval."
)
demo.launch(share=True)

FAQ

What is the difference between hallucination and faithfulness in LLMs?

Hallucination refers to generating information that is factually incorrect or made up, while faithfulness measures whether the model stays aligned with the provided source without altering its meaning.

Why do LLMs hallucinate even when context is provided?

From my experience, hallucinations often happen due to token-level prediction, incomplete context retrieval, or weak alignment between the prompt and source. The model fills gaps with statistically likely text rather than verified facts.

How can hallucinations be detected automatically?

Hallucinations can be evaluated using techniques like Natural Language Inference (NLI), similarity checks, and specialized tools such as DeepEval that compare model outputs directly against the source context.

Is human evaluation still necessary?

Yes. Automated metrics are valuable, but I’ve found human review essential for catching subtle logical or contextual errors that automated systems may miss.

Conclusion 

DeepEval gives a practical way to measure something that’s otherwise easy to miss: whether an LLM is actually being truthful to its source. From working with document-based systems, I’ve found this kind of evaluation essential for catching hallucinations early and building AI systems that users can trust in real-world scenarios. With the help of DeepEval, it becomes easier to spot when the model makes up facts (hallucinations) or changes the original meaning (faithfulness issues). 

This kind of evaluation is important to build more accurate, safe, and reliable AI systems, especially when dealing with real-world documents and critical information. 

Happy learning!