A/B Testing Prompts in Production: Measuring the Impact of Prompt Changes

The Case for Prompt Experimentation

You rewrote your support agent's system prompt to be more concise. The team agrees it reads better. But does it actually perform better? Without measurement, prompt changes are gut-feel decisions. A/B testing brings the same rigor to prompt engineering that product teams apply to UI changes.

Prompt A/B testing means running two or more prompt variants simultaneously, splitting traffic between them, and measuring which variant produces better outcomes against defined metrics.

Experiment Design

Define clear hypotheses and metrics before writing any code.

flowchart LR
    PR(["PR opened"])
    UNIT["Unit tests"]
    EVAL["Eval harness<br/>PromptFoo or Braintrust"]
    GOLD[("Golden set<br/>200 tagged cases")]
    JUDGE["LLM as judge<br/>plus regex graders"]
    SCORE["Aggregate score<br/>and per slice"]
    GATE{"Score regress<br/>more than 2 percent?"}
    BLOCK(["Block merge"])
    MERGE(["Merge to main"])
    PR --> UNIT --> EVAL --> GOLD --> JUDGE --> SCORE --> GATE
    GATE -->|Yes| BLOCK
    GATE -->|No| MERGE
    style EVAL fill:#4f46e5,stroke:#4338ca,color:#fff
    style GATE fill:#f59e0b,stroke:#d97706,color:#1f2937
    style BLOCK fill:#dc2626,stroke:#b91c1c,color:#fff
    style MERGE fill:#059669,stroke:#047857,color:#fff

from dataclasses import dataclass, field
from datetime import datetime, timezone
from enum import Enum

class ExperimentStatus(str, Enum):
    DRAFT = "draft"
    RUNNING = "running"
    PAUSED = "paused"
    COMPLETED = "completed"

@dataclass
class PromptVariant:
    name: str
    prompt_content: str
    traffic_weight: float  # 0.0 to 1.0
    description: str = ""

@dataclass
class Experiment:
    id: str
    name: str
    hypothesis: str
    primary_metric: str
    secondary_metrics: list[str]
    variants: list[PromptVariant]
    min_sample_size: int = 1000
    status: ExperimentStatus = ExperimentStatus.DRAFT
    started_at: datetime = None
    results: dict = field(default_factory=dict)

    def validate(self):
        total_weight = sum(v.traffic_weight for v in self.variants)
        assert abs(total_weight - 1.0) < 0.01, (
            f"Variant weights must sum to 1.0, got {total_weight}"
        )
        assert len(self.variants) >= 2, "Need at least 2 variants"

Deterministic Traffic Splitting

Users must see the same variant consistently across sessions. Use hash-based assignment.

import hashlib

class TrafficSplitter:
    """Deterministic traffic assignment using consistent hashing."""

    def assign_variant(
        self, experiment_id: str, user_id: str,
        variants: list[PromptVariant]
    ) -> PromptVariant:
        """Assign a user to a variant deterministically."""
        hash_input = f"{experiment_id}:{user_id}"
        hash_value = int(
            hashlib.sha256(hash_input.encode()).hexdigest(), 16
        )
        # Normalize to 0.0 - 1.0 range
        bucket = (hash_value % 10000) / 10000.0

        cumulative = 0.0
        for variant in variants:
            cumulative += variant.traffic_weight
            if bucket < cumulative:
                return variant

        return variants[-1]  # Fallback to last variant

This approach ensures the same user always gets the same variant (deterministic) without storing assignments in a database. The hash function distributes users uniformly across buckets.

Metric Collection

Collect structured metrics for every interaction so you can compare variants fairly.

from datetime import datetime, timezone
import json
from pathlib import Path

@dataclass
class InteractionMetric:
    experiment_id: str
    variant_name: str
    user_id: str
    timestamp: datetime
    response_time_ms: float
    token_count: int
    user_rating: int = None          # 1-5 scale
    task_completed: bool = None
    escalated: bool = False
    error_occurred: bool = False
    custom_metrics: dict = field(default_factory=dict)

class MetricCollector:
    """Collect and store experiment metrics."""

    def __init__(self, storage_path: str = "experiment_metrics"):
        self.storage = Path(storage_path)
        self.storage.mkdir(exist_ok=True)

    def record(self, metric: InteractionMetric):
        """Record a single interaction metric."""
        filepath = (
            self.storage
            / f"{metric.experiment_id}_{metric.variant_name}.jsonl"
        )
        with open(filepath, "a") as f:
            f.write(json.dumps({
                "variant": metric.variant_name,
                "user_id": metric.user_id,
                "timestamp": metric.timestamp.isoformat(),
                "response_time_ms": metric.response_time_ms,
                "token_count": metric.token_count,
                "user_rating": metric.user_rating,
                "task_completed": metric.task_completed,
                "escalated": metric.escalated,
                "error_occurred": metric.error_occurred,
                **metric.custom_metrics,
            }) + "\n")

    def load_metrics(
        self, experiment_id: str, variant_name: str
    ) -> list[dict]:
        """Load all metrics for a specific variant."""
        filepath = (
            self.storage / f"{experiment_id}_{variant_name}.jsonl"
        )
        if not filepath.exists():
            return []
        metrics = []
        for line in filepath.read_text().strip().split("\n"):
            if line:
                metrics.append(json.loads(line))
        return metrics

Statistical Analysis

Do not just compare averages. Use proper statistical tests to determine whether differences are significant.

import math

class ExperimentAnalyzer:
    """Analyze A/B test results with statistical rigor."""

    def analyze_conversion(
        self, control_successes: int, control_total: int,
        treatment_successes: int, treatment_total: int,
        confidence_level: float = 0.95
    ) -> dict:
        """Compare conversion rates using a z-test."""
        p_control = control_successes / control_total
        p_treatment = treatment_successes / treatment_total
        p_pooled = (
            (control_successes + treatment_successes)
            / (control_total + treatment_total)
        )

        se = math.sqrt(
            p_pooled * (1 - p_pooled)
            * (1/control_total + 1/treatment_total)
        )

        if se == 0:
            return {"significant": False, "reason": "No variance"}

        z_score = (p_treatment - p_control) / se
        # Two-tailed z critical value for 95% confidence
        z_critical = 1.96 if confidence_level == 0.95 else 2.576

        return {
            "control_rate": round(p_control, 4),
            "treatment_rate": round(p_treatment, 4),
            "relative_lift": round(
                (p_treatment - p_control) / p_control * 100, 2
            ) if p_control > 0 else None,
            "z_score": round(z_score, 4),
            "significant": abs(z_score) > z_critical,
            "confidence_level": confidence_level,
            "recommendation": (
                "treatment" if z_score > z_critical
                else "control" if z_score < -z_critical
                else "no_difference"
            ),
        }

# Usage
analyzer = ExperimentAnalyzer()
result = analyzer.analyze_conversion(
    control_successes=340, control_total=1000,
    treatment_successes=385, treatment_total=1000,
)
# result["significant"] tells you if the difference is real

FAQ

How long should I run a prompt A/B test?

Until you reach statistical significance with your minimum sample size. Calculate the required sample size before starting based on your expected effect size. For most prompt changes, plan for at least 1,000 interactions per variant. Ending tests early based on preliminary results leads to false conclusions.

What metrics should I track for prompt experiments?

Track both quality metrics (task completion rate, user satisfaction, factual accuracy) and cost metrics (token usage, response time, escalation rate). The best primary metric depends on your use case — for a support agent, resolution rate matters most; for a coding assistant, code correctness is more important.

How do I handle experiments when prompts affect downstream agents?

In multi-agent systems, isolate the experiment to a single agent and hold all other agents constant. Measure the end-to-end outcome, not just the individual agent's output. If you change the triage agent's prompt, measure whether the downstream support agent still resolves issues successfully.

---

#ABTesting #PromptOptimization #StatisticalAnalysis #AIOps #ProductionAI #AgenticAI #LearnAI #AIEngineering