RAG-Based Testing Series — Part 6: Automating RAG Quality Checks in CI/CD

.github/
└── workflows/
    └── rag_quality_checks.yml   ← GitHub Actions workflow

rag_test_framework/
├── config/
│   └── settings.py
├── core/
│   ├── retriever.py
│   ├── evaluator.py
│   └── rag_pipeline.py
├── tests/
│   ├── conftest.py
│   ├── test_retrieval.py
│   ├── test_faithfulness.py
│   └── test_edge_cases.py
├── data/
│   └── test_cases.json
├── reports/
│   └── (auto-generated, uploaded as CI artifacts)
├── run_tests.py
└── requirements.txt

# config/settings.py

import os

OPENAI_API_KEY = os.environ.get("OPENAI_API_KEY")

if not OPENAI_API_KEY:
    raise EnvironmentError(
        "OPENAI_API_KEY environment variable is not set.\n"
        "Set it locally with: export OPENAI_API_KEY=your-key\n"
        "In CI, add it as a GitHub Actions secret."
    )

name: RAG Quality Checks

on:
  push:
    paths:
      # Run when test cases or knowledge base changes
      - 'rag_test_framework/data/**'
      # Run when any core framework code changes
      - 'rag_test_framework/core/**'
      # Run when configuration (thresholds, models) changes
      - 'rag_test_framework/config/**'
      # Run when tests themselves change
      - 'rag_test_framework/tests/**'
      # Run when dependencies change
      - 'rag_test_framework/requirements.txt'

  pull_request:
    paths:
      - 'rag_test_framework/data/**'
      - 'rag_test_framework/core/**'
      - 'rag_test_framework/config/**'
      - 'rag_test_framework/tests/**'
      - 'rag_test_framework/requirements.txt'

  # Allow manual trigger from the GitHub Actions UI
  workflow_dispatch:

jobs:
  rag-quality-checks:
    name: RAG Quality Checks
    runs-on: ubuntu-latest

    steps:
      # ── 1. Check out the repository ──────────────────────────
      - name: Checkout repository
        uses: actions/checkout@v4

      # ── 2. Set up Python ─────────────────────────────────────
      - name: Set up Python 3.11
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: 'pip'   # cache pip installs between runs to speed up the workflow

      # ── 3. Install dependencies ───────────────────────────────
      - name: Install dependencies
        working-directory: rag_test_framework
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

      # ── 4. Run the RAG test suite ─────────────────────────────
      - name: Run RAG quality checks
        working-directory: rag_test_framework
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          mkdir -p reports
          pytest tests/ \
            -v \
            --tb=short \
            --json-report \
            --json-report-file=reports/rag_test_report.json \
            --json-report-summary

      # ── 5. Upload report as a downloadable artifact ───────────
      - name: Upload test report
        if: always()   # upload even if tests failed — you want the report either way
        uses: actions/upload-artifact@v4
        with:
          name: rag-test-report-${{ github.run_number }}
          path: rag_test_framework/reports/rag_test_report.json
          retention-days: 30

      # ── 6. Post summary to GitHub Actions summary page ────────
      - name: Post test summary
        if: always()
        working-directory: rag_test_framework
        run: python ci/post_summary.py reports/rag_test_report.json

# ci/post_summary.py

import json
import os
import sys


def post_summary(report_path: str):
    """
    Read the pytest JSON report and write a markdown summary
    to the GitHub Actions step summary page (GITHUB_STEP_SUMMARY).
    """
    if not os.path.exists(report_path):
        print(f"Report not found at {report_path}")
        sys.exit(1)

    with open(report_path) as f:
        report = json.load(f)

    summary  = report.get("summary", {})
    passed   = summary.get("passed", 0)
    failed   = summary.get("failed", 0)
    error    = summary.get("error", 0)
    total    = summary.get("total", 0)
    duration = round(report.get("duration", 0), 2)

    # Determine overall status
    if failed > 0 or error > 0:
        status_icon  = "❌"
        status_label = "FAILED"
    else:
        status_icon  = "✅"
        status_label = "PASSED"

    # Build the markdown summary
    lines = [
        f"## {status_icon} RAG Quality Checks — {status_label}",
        "",
        "| Metric | Value |",
        "|--------|-------|",
        f"| Total tests | {total} |",
        f"| Passed      | {passed} |",
        f"| Failed      | {failed} |",
        f"| Duration    | {duration}s |",
        "",
    ]

    if failed > 0 or error > 0:
        lines.append("### ❌ Failed Tests")
        lines.append("")
        for test in report.get("tests", []):
            if test["outcome"] in ("failed", "error"):
                lines.append(f"- `{test['nodeid']}`")
                # Include the failure message if available
                if "call" in test and "longrepr" in test["call"]:
                    # Truncate long failure output for readability
                    longrepr = test["call"]["longrepr"]
                    preview  = longrepr[:500] + "..." if len(longrepr) > 500 else longrepr
                    lines.append(f"  ```
{% endraw %}
\n  {preview}\n
{% raw %}
  ```")
        lines.append("")

    lines += [
        "### Test Breakdown",
        "",
        "| Test File | Tests | Status |",
        "|-----------|-------|--------|",
    ]

    # Group tests by file for the breakdown table
    file_results: dict = {}
    for test in report.get("tests", []):
        file_name = test["nodeid"].split("::")[0]
        if file_name not in file_results:
            file_results[file_name] = {"total": 0, "failed": 0}
        file_results[file_name]["total"] += 1
        if test["outcome"] in ("failed", "error"):
            file_results[file_name]["failed"] += 1

    for file_name, counts in file_results.items():
        icon = "✅" if counts["failed"] == 0 else "❌"
        lines.append(f"| `{file_name}` | {counts['total']} | {icon} |")

    summary_text = "\n".join(lines)

    # Write to GitHub step summary if running in CI
    github_summary_path = os.environ.get("GITHUB_STEP_SUMMARY")
    if github_summary_path:
        with open(github_summary_path, "a") as f:
            f.write(summary_text)
        print("✅ Summary written to GitHub Actions step summary.")
    else:
        # Running locally — just print it
        print(summary_text)


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: python ci/post_summary.py <path-to-report.json>")
        sys.exit(1)
    post_summary(sys.argv[1])

on:
  push:
    paths:
      - 'rag_test_framework/data/**'      # knowledge base changed
      - 'rag_test_framework/core/**'      # retrieval or pipeline logic changed
      - 'rag_test_framework/config/**'    # thresholds or models changed
      - 'rag_test_framework/tests/**'     # tests themselves changed
      - 'rag_test_framework/requirements.txt'

// data/test_cases_ci.json
{
  "knowledge_base": [ ... ],

  "retrieval_test_cases": [
    // Keep your 5 highest-signal retrieval cases
    // These should represent the most common and most critical query types
  ],

  "faithfulness_test_cases": [
    // 3-4 cases that cover your main faithfulness scenarios
  ],

  "edge_case_queries": {
    "out_of_scope":    ["What is the capital of France?"],
    "empty_retrieval": ["What is the pricing for the enterprise plan?"],
    "leading_questions": [ ... ]
  }
}

# tests/conftest.py

import json
import os
import pytest
from core.retriever import build_collection
from core.evaluator import build_evaluator

@pytest.fixture(scope="session")
def test_data():
    # In CI, use the smaller dataset. Locally, use the full one.
    ci_mode       = os.environ.get("CI", "false").lower() == "true"
    dataset_path  = "data/test_cases_ci.json" if ci_mode else "data/test_cases.json"

    with open(dataset_path) as f:
        return json.load(f)

@pytest.fixture(scope="session")
def collection(test_data):
    kb = test_data["knowledge_base"]
    return build_collection(
        collection_name="rag_test_kb",
        documents=[doc["text"] for doc in kb],
        doc_ids=[doc["id"] for doc in kb]
    )

@pytest.fixture(scope="session")
def evaluator():
    llm, embeddings = build_evaluator()
    return llm, embeddings

# Add this to the `on:` section of your workflow

  schedule:
    # Run every day at 2 AM UTC
    # This uses the full dataset, not the CI subset
    - cron: '0 2 * * *'

      - name: Run RAG quality checks
        working-directory: rag_test_framework
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          # Override CI mode for scheduled runs — use full dataset
          CI: ${{ github.event_name != 'schedule' && 'true' || 'false' }}
        run: |
          mkdir -p reports
          pytest tests/ -v --tb=short --json-report --json-report-file=reports/rag_test_report.json

rag_test_framework/
│
├── .github/workflows/
│   └── rag_quality_checks.yml     ← CI/CD trigger + orchestration
│
├── ci/
│   └── post_summary.py            ← GitHub Actions summary writer
│
├── config/
│   └── settings.py                ← all thresholds, model names, API keys
│
├── core/
│   ├── retriever.py               ← retrieval + Precision@K, Recall@K, MRR
│   ├── evaluator.py               ← RAGAS faithfulness + answer_relevancy
│   └── rag_pipeline.py            ← end-to-end RAG call
│
├── tests/
│   ├── conftest.py                ← shared session-scoped fixtures
│   ├── test_retrieval.py          ← Part 2 tests
│   ├── test_faithfulness.py       ← Part 3 tests
│   └── test_edge_cases.py         ← Part 4 tests
│
├── data/
│   ├── test_cases.json            ← full ground truth dataset
│   └── test_cases_ci.json         ← smaller CI subset
│
├── reports/
│   └── (timestamped JSON reports)
│
├── run_tests.py                   ← local single-command runner
└── requirements.txt

Developer pushes a commit that updates data/test_cases.json
    │
    ▼
GitHub detects the push matches a path filter
    │
    ▼
GitHub Actions spins up ubuntu-latest runner
    │
    ▼
Python 3.11 installed, pip cache restored
    │
    ▼
pip install -r requirements.txt
    │
    ▼
pytest tests/ runs with CI=true (uses test_cases_ci.json)
    │
    ├── test_retrieval.py   — Precision@K, Recall@K, MRR asserted
    ├── test_faithfulness.py — Faithfulness, no critical hallucinations
    └── test_edge_cases.py  — Empty retrieval, out-of-scope, leading questions
    │
    ▼
rag_test_report.json written to reports/
    │
    ▼
post_summary.py writes markdown table to GitHub step summary
    │
    ▼
Report uploaded as downloadable artifact (kept 30 days)
    │
    ├── All tests pass → ✅ Pipeline green, PR can merge
    └── Any test fails → ❌ Pipeline blocked, team notified

Part 1 ✅ — Understood what RAG is and why traditional testing breaks down
Part 2 ✅ — Gave retrieval quality a number: Precision@K, Recall@K, MRR
Part 3 ✅ — Gave hallucination detection a number: faithfulness scoring with RAGAS
Part 4 ✅ — Tested the edge cases that break RAG systems in production
Part 5 ✅ — Assembled everything into a structured, reusable framework
Part 6 ✅ — Automated the framework in CI/CD with GitHub Actions