A/B Testing Agent Prompts and Models: Statistical Framework for Experiments

Why Standard A/B Testing Falls Short for Agents

Traditional A/B testing assumes each observation is independent and outcomes are binary (click or no click, convert or not). AI agent interactions are neither. A single conversation spans multiple turns, outcomes are multi-dimensional (accuracy, helpfulness, latency, cost), and the same prompt can produce different outputs due to model stochasticity. You need a statistical framework that accounts for these realities.

Experiment Design

Every experiment starts with a hypothesis, a primary metric, and a sample size calculation. Without these, you are just guessing with extra steps.

flowchart LR
    PR(["PR opened"])
    UNIT["Unit tests"]
    EVAL["Eval harness<br/>PromptFoo or Braintrust"]
    GOLD[("Golden set<br/>200 tagged cases")]
    JUDGE["LLM as judge<br/>plus regex graders"]
    SCORE["Aggregate score<br/>and per slice"]
    GATE{"Score regress<br/>more than 2 percent?"}
    BLOCK(["Block merge"])
    MERGE(["Merge to main"])
    PR --> UNIT --> EVAL --> GOLD --> JUDGE --> SCORE --> GATE
    GATE -->|Yes| BLOCK
    GATE -->|No| MERGE
    style EVAL fill:#4f46e5,stroke:#4338ca,color:#fff
    style GATE fill:#f59e0b,stroke:#d97706,color:#1f2937
    style BLOCK fill:#dc2626,stroke:#b91c1c,color:#fff
    style MERGE fill:#059669,stroke:#047857,color:#fff

from dataclasses import dataclass, field
from typing import Optional
from enum import Enum
import uuid
import math

class ExperimentStatus(Enum):
    DRAFT = "draft"
    RUNNING = "running"
    PAUSED = "paused"
    COMPLETED = "completed"

@dataclass
class Variant:
    name: str
    weight: float
    config: dict
    # config holds the actual differences: prompt, model, temperature, etc.

@dataclass
class Experiment:
    id: str = field(default_factory=lambda: str(uuid.uuid4()))
    name: str = ""
    hypothesis: str = ""
    primary_metric: str = "task_completion_rate"
    variants: list[Variant] = field(default_factory=list)
    status: ExperimentStatus = ExperimentStatus.DRAFT
    min_sample_size: int = 1000
    significance_level: float = 0.05
    minimum_detectable_effect: float = 0.05

    def required_sample_per_variant(
        self, baseline_rate: float = 0.7, power: float = 0.8
    ) -> int:
        p1 = baseline_rate
        p2 = baseline_rate + self.minimum_detectable_effect
        z_alpha = 1.96  # two-tailed, alpha=0.05
        z_beta = 0.84   # power=0.8
        pooled = (p1 + p2) / 2
        numerator = (
            z_alpha * math.sqrt(2 * pooled * (1 - pooled))
            + z_beta * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))
        ) ** 2
        denominator = (p2 - p1) ** 2
        return math.ceil(numerator / denominator)

Randomization and Assignment

Users must be consistently assigned to the same variant for the duration of the experiment. Use deterministic hashing, not random assignment per request.

import hashlib

class ExperimentAssigner:
    def assign(self, experiment: Experiment, user_id: str) -> Variant:
        hash_input = f"{experiment.id}:{user_id}"
        hash_val = int(
            hashlib.sha256(hash_input.encode()).hexdigest()[:8], 16
        )
        normalized = hash_val / 0xFFFFFFFF

        cumulative = 0.0
        for variant in experiment.variants:
            cumulative += variant.weight
            if normalized < cumulative:
                return variant

        return experiment.variants[-1]

Metrics Collection

Track every interaction with its experiment context. The metrics pipeline collects raw events that the analysis layer aggregates later.

from dataclasses import dataclass
import time

@dataclass
class ExperimentEvent:
    experiment_id: str
    variant_name: str
    user_id: str
    session_id: str
    metric_name: str
    metric_value: float
    timestamp: float = field(default_factory=time.time)

class MetricsCollector:
    def __init__(self):
        self._events: list[ExperimentEvent] = []

    def record(
        self,
        experiment: Experiment,
        variant: Variant,
        user_id: str,
        session_id: str,
        metrics: dict[str, float],
    ):
        for name, value in metrics.items():
            self._events.append(
                ExperimentEvent(
                    experiment_id=experiment.id,
                    variant_name=variant.name,
                    user_id=user_id,
                    session_id=session_id,
                    metric_name=name,
                    metric_value=value,
                )
            )

    def get_metric_values(
        self, experiment_id: str, variant_name: str, metric_name: str
    ) -> list[float]:
        return [
            e.metric_value
            for e in self._events
            if e.experiment_id == experiment_id
            and e.variant_name == variant_name
            and e.metric_name == metric_name
        ]

Statistical Significance Testing

For proportions like task completion rate, use a two-proportion z-test. For continuous metrics like response latency, use Welch's t-test.

import math
from typing import NamedTuple

class TestResult(NamedTuple):
    z_score: float
    p_value: float
    significant: bool
    control_rate: float
    treatment_rate: float
    relative_lift: float

def two_proportion_z_test(
    control_successes: int,
    control_total: int,
    treatment_successes: int,
    treatment_total: int,
    alpha: float = 0.05,
) -> TestResult:
    p1 = control_successes / control_total
    p2 = treatment_successes / treatment_total
    pooled = (control_successes + treatment_successes) / (
        control_total + treatment_total
    )
    se = math.sqrt(pooled * (1 - pooled) * (1 / control_total + 1 / treatment_total))

    if se == 0:
        return TestResult(0, 1.0, False, p1, p2, 0.0)

    z = (p2 - p1) / se
    # Approximate two-tailed p-value using normal CDF
    p_value = 2 * (1 - _normal_cdf(abs(z)))
    lift = (p2 - p1) / p1 if p1 > 0 else 0.0

    return TestResult(
        z_score=z,
        p_value=p_value,
        significant=p_value < alpha,
        control_rate=p1,
        treatment_rate=p2,
        relative_lift=lift,
    )

def _normal_cdf(x: float) -> float:
    return 0.5 * (1 + math.erf(x / math.sqrt(2)))

Running an Experiment End-to-End

Here is how you wire the pieces together in practice.

experiment = Experiment(
    name="reasoning_prompt_test",
    hypothesis="Adding chain-of-thought instructions improves task completion",
    primary_metric="task_completion_rate",
    variants=[
        Variant("control", 0.5, {"prompt": "You are a helpful assistant."}),
        Variant("treatment", 0.5, {
            "prompt": "You are a helpful assistant. Think step by step."
        }),
    ],
)

assigner = ExperimentAssigner()
collector = MetricsCollector()

# During agent execution
user_id = "user_42"
variant = assigner.assign(experiment, user_id)
agent_config = variant.config

# After task completes
collector.record(
    experiment, variant, user_id, "session_1",
    {"task_completion_rate": 1.0, "latency_ms": 1200.0},
)

Avoiding Common Pitfalls

One of the biggest mistakes is peeking at results too early. Every time you check significance, you increase the chance of a false positive. Decide the sample size upfront and only analyze after reaching it. If you must monitor results during the experiment, use sequential testing methods that adjust for multiple comparisons.

Another pitfall is ignoring user-level clustering. If a single user has 50 conversations, those 50 data points are not independent. Aggregate metrics at the user level first, then run the statistical test on user-level averages.

FAQ

How many samples do I need per variant?

It depends on your baseline rate and the minimum effect you want to detect. For a baseline task completion rate of 70% and a 5% minimum detectable effect, you need roughly 780 users per variant at 80% power. Use the required_sample_per_variant method to calculate this for your specific scenario.

Should I test prompt changes and model changes in the same experiment?

No. Changing multiple variables in one experiment makes it impossible to attribute results to a specific change. Test one variable at a time. If you need to test combinations, use a factorial experiment design with enough sample size to detect interaction effects.

How do I handle non-binary metrics like response quality scores?

Use Welch's t-test instead of the two-proportion z-test. Collect quality scores (for example from LLM-as-judge evaluations) as continuous values and compare the means between variants. The same sample size principles apply, though the calculation uses standard deviation instead of proportions.

---

#ABTesting #AIAgents #StatisticalTesting #ExperimentDesign #Python #AgenticAI #LearnAI #AIEngineering