Scrolls
5 min read
On this page

Transparency & Explainability

Overview

Users deserve to know when they are interacting with AI. Stakeholders deserve to understand why the AI made a particular decision. Regulators increasingly require both. Transparency is about disclosure: telling people that AI is involved. Explainability is about understanding: answering "why did the model produce this output?"

These are not nice-to-have features. They are rapidly becoming legal requirements. The EU AI Act mandates transparency for high-risk AI systems. GDPR Article 22 gives individuals the right not to be subject to decisions based solely on automated processing. The practical question is not whether to build explainable AI, but how.

Transparency: Telling Users What They Need to Know

AI Disclosure

When AI generates content, users should know:

def format_ai_response(response, source="llm", confidence=None):
    """Always disclose AI involvement in the response.
    
    Don't hide it in fine print. Make it visible.
    """
    disclosure = {
        "content": response,
        "metadata": {
            "generated_by": "ai",
            "model_type": source,
            "timestamp": datetime.now().isoformat(),
        },
    }
    
    if confidence is not None:
        disclosure["metadata"]["confidence"] = confidence
        
        if confidence < 0.7:
            disclosure["metadata"]["warning"] = (
                "This response has lower confidence. "
                "Please verify the information independently."
            )
    
    return disclosure

What to Disclose

Minimum disclosure:
  - That AI is involved in the interaction
  - What role AI plays (generating, recommending, filtering)
  - How to reach a human if needed

Better disclosure:
  - What data was used (e.g., "based on your purchase history")
  - How confident the system is
  - Known limitations ("this system may not handle X well")
  - How to opt out or override the AI decision

Best disclosure:
  - All of the above, plus
  - What the AI cannot do (explicit scope boundaries)
  - How the model was trained (at a high level)
  - How to provide feedback or dispute a decision
  - Regular transparency reports with aggregate metrics

Confidence Communication

Don't present uncertain answers as certain:

def communicate_confidence(response, confidence_score):
    """Adjust how you present information based on confidence.
    
    High confidence: present directly
    Medium confidence: present with caveats
    Low confidence: flag for human review or decline to answer
    """
    if confidence_score > 0.9:
        return {
            "response": response,
            "framing": "direct",
        }
    elif confidence_score > 0.7:
        return {
            "response": response,
            "framing": "qualified",
            "caveat": "Based on available information, though you "
                      "may want to verify this.",
        }
    elif confidence_score > 0.5:
        return {
            "response": response,
            "framing": "uncertain",
            "caveat": "I'm not fully confident in this answer. "
                      "Consider consulting a specialist.",
        }
    else:
        return {
            "response": None,
            "framing": "declined",
            "message": "I don't have enough information to answer "
                       "this reliably. Here's how to find help: ...",
        }

Explainability Methods

Feature Importance with SHAP

SHAP (SHapley Additive exPlanations) explains which features drove a specific prediction:

import shap

def explain_prediction_with_shap(model, input_data, feature_names):
    """Explain why the model made a specific prediction.
    
    SHAP values show how much each feature pushed the prediction
    up or down from the average.
    
    Example output:
      Base prediction: 0.35 (average probability of approval)
      Income > $80k:   +0.25 (increases approval probability)
      Credit score 720: +0.15
      Debt-to-income 0.4: -0.10
      Employment < 1yr: -0.08
      Final prediction: 0.57 (approved)
    """
    explainer = shap.TreeExplainer(model)
    shap_values = explainer.shap_values(input_data)
    
    # Create explanation for a single prediction
    explanation = []
    base_value = explainer.expected_value
    
    for i, (feature, value, shap_val) in enumerate(
        zip(feature_names, input_data[0], shap_values[0])
    ):
        if abs(shap_val) > 0.01:  # Only show significant features
            direction = "increases" if shap_val > 0 else "decreases"
            explanation.append({
                "feature": feature,
                "value": value,
                "impact": shap_val,
                "direction": direction,
            })
    
    # Sort by absolute impact
    explanation.sort(key=lambda x: abs(x["impact"]), reverse=True)
    
    return {
        "base_prediction": float(base_value),
        "final_prediction": float(base_value + sum(s["impact"] for s in explanation)),
        "top_factors": explanation[:5],
    }

Feature Importance with LIME

LIME (Local Interpretable Model-agnostic Explanations) works with any model:

from lime.lime_text import LimeTextExplainer

def explain_text_classification(model, text, class_names):
    """Explain why a text classifier made its prediction.
    
    LIME perturbs the input (removes words) and observes
    how the prediction changes. Words whose removal changes
    the prediction most are the most important.
    """
    explainer = LimeTextExplainer(class_names=class_names)
    
    explanation = explainer.explain_instance(
        text,
        model.predict_proba,
        num_features=10,
        num_samples=1000,
    )
    
    # Get the top contributing words
    word_importance = explanation.as_list()
    
    result = {
        "input": text,
        "predicted_class": class_names[model.predict([text])[0]],
        "important_words": [
            {
                "word": word,
                "weight": weight,
                "direction": "supports" if weight > 0 else "opposes",
            }
            for word, weight in word_importance
        ],
    }
    
    return result

Explaining LLM Decisions

LLMs are harder to explain than traditional ML models. Current approaches:

def explain_llm_with_rag_attribution(query, response, retrieved_docs):
    """For RAG systems: show which source documents informed the answer.
    
    This is the most practical form of LLM explainability:
    'Here is the answer, and here are the sources it came from.'
    """
    attributions = []
    
    for doc in retrieved_docs:
        # Check if the response contains information from this doc
        overlap = compute_information_overlap(response, doc["content"])
        
        if overlap > 0.3:  # Threshold for meaningful attribution
            attributions.append({
                "source": doc["title"],
                "url": doc.get("url"),
                "relevance_score": doc["search_score"],
                "information_overlap": overlap,
                "excerpt": doc["content"][:200],
            })
    
    return {
        "response": response,
        "sources": attributions,
        "source_coverage": len(attributions) / len(retrieved_docs),
    }

def explain_with_chain_of_thought(query):
    """Ask the LLM to explain its reasoning step by step.
    
    Note: this shows the model's stated reasoning,
    which may not reflect its actual computation.
    It is useful for transparency, not for debugging.
    """
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": "Before answering, explain your reasoning "
                           "step by step. Then provide your final answer. "
                           "If you are unsure about anything, say so.",
            },
            {"role": "user", "content": query},
        ],
    )
    
    return response.choices[0].message.content

Teaching the Model to Say "I Don't Know"

Most LLMs will confidently generate an answer even when they have no basis for one. Explicitly instruct the model to decline when appropriate. Key system prompt rules: only answer based on provided context, say "I don't know" when information is insufficient, explicitly state uncertainty on partial knowledge, never fabricate facts or citations, and recommend professionals for medical, legal, or financial questions.

The Right to Explanation

GDPR Article 22

GDPR Article 22 key points:

1. Individuals have the right not to be subject to decisions
   based solely on automated processing that significantly
   affect them.

2. When automated decisions are made, individuals have the right to:
   - Obtain meaningful information about the logic involved
   - The significance and envisaged consequences
   - Contest the decision
   - Obtain human intervention

What this means for AI systems:
   - You need a human review pathway
   - You need to be able to explain decisions in plain language
   - "The algorithm decided" is not a sufficient explanation
   - You must be able to describe the key factors

Building Compliant Explanations

A compliant explanation should include: the decision itself, a plain-language summary of the factors that drove it (with specific values and thresholds), and clear instructions for how to request a human review. For example: "Your loan application was denied. Your debt-to-income ratio (0.45) is above our threshold of 0.40. To request a human review, contact support@example.com."

Explainability for Different Audiences

  • End users: Plain language, actionable information ("We denied your loan because X. Here's how to improve.")
  • Business stakeholders: Feature importance rankings, confidence ranges
  • Regulators: Methodology, fairness metrics, audit trails, decision logs
  • Data scientists: SHAP/LIME per prediction, error analysis by slice
  • Legal/compliance: Risk assessments, decision logs with timestamps

Real-World Example: Explainable Credit Scoring

A fintech company builds an ML-based credit scoring system. Regulators require explanations for every denial.

Step 1: Model selection. They choose a gradient-boosted tree model instead of a neural network, partly because tree models are easier to explain with SHAP.

Step 2: Feature transparency. Every feature used by the model is documented with plain-language descriptions. "DTI_ratio" becomes "your total monthly debt payments divided by your monthly income."

Step 3: Per-decision explanations. When an application is denied, the system generates a letter listing the top 3 factors that contributed to the denial, with specific values and thresholds.

Step 4: Human review pathway. Any applicant can request a human review. A trained analyst reviews the model's explanation, the raw data, and the applicant's context before making a final decision.

Step 5: Regular auditing. Monthly fairness audits check that denial explanations are consistent across demographic groups. Quarterly reports are submitted to regulators.

Common Pitfalls

  • Treating explanation as an afterthought: If you build a black-box model first and try to explain it later, the explanations will be poor. Consider explainability from the start.
  • Confusing model confidence with explanation: Saying "the model is 87% confident" is not an explanation. Users need to know which factors drove the decision, not just how sure the model is.
  • Over-relying on post-hoc explanations: SHAP and LIME explain what the model did, not why the model is right. A biased model will have "explanations" that reflect its bias.
  • Providing explanations that nobody reads: A 10-page technical report is not transparency. Match the explanation depth and format to the audience.
  • Assuming chain-of-thought equals reasoning: When an LLM shows its "reasoning," it is generating plausible-looking text, not revealing its actual computation. Use it for user-facing transparency, not for debugging.
  • Not providing a path to human review: Explanations without recourse are theater. Users must be able to challenge automated decisions.

Key Takeaways

  • Transparency means disclosing AI involvement. Explainability means explaining why the AI made a particular decision. Both are increasingly required by law.
  • SHAP and LIME provide feature-level explanations for traditional ML models. For LLMs, RAG attribution and chain-of-thought are the most practical approaches.
  • Explanations should be tailored to the audience: plain language for users, feature importance for business stakeholders, detailed methodology for regulators.
  • "I don't know" is a valid and important model output. Systems that always produce confident answers are less trustworthy, not more.
  • GDPR Article 22 requires meaningful explanations, the ability to contest decisions, and access to human review. Build these capabilities from the start.
  • The best time to design for explainability is before you build the model. Choosing interpretable architectures and documenting features early is far cheaper than retrofitting explanations later.