Agent Conversation Mining: Discovering Patterns and Insights from Chat Logs

What Is Conversation Mining

Conversation mining is the process of analyzing large volumes of chat logs to discover patterns, recurring issues, user intents, and improvement opportunities that are invisible when reading individual conversations. It is the difference between reading 50 conversations and understanding 50,000.

For AI agents, conversation mining reveals which topics the agent handles well, where it struggles, what users actually ask for versus what you designed for, and how conversation patterns evolve over time.

Extracting and Structuring Conversations

Raw conversation data needs to be structured before analysis. Extract messages, pair them into exchanges, and compute basic features.

flowchart LR
    INPUT(["User intent"])
    PARSE["Parse plus<br/>classify"]
    PLAN["Plan and tool<br/>selection"]
    AGENT["Agent loop<br/>LLM plus tools"]
    GUARD{"Guardrails<br/>and policy"}
    EXEC["Execute and<br/>verify result"]
    OBS[("Trace and metrics")]
    OUT(["Outcome plus<br/>next action"])
    INPUT --> PARSE --> PLAN --> AGENT --> GUARD
    GUARD -->|Pass| EXEC --> OUT
    GUARD -->|Fail| AGENT
    AGENT --> OBS
    style AGENT fill:#4f46e5,stroke:#4338ca,color:#fff
    style GUARD fill:#f59e0b,stroke:#d97706,color:#1f2937
    style OBS fill:#ede9fe,stroke:#7c3aed,color:#1e1b4b
    style OUT fill:#059669,stroke:#047857,color:#fff

from dataclasses import dataclass, field
from datetime import datetime

@dataclass
class ConversationExchange:
    user_message: str
    agent_response: str
    timestamp: str
    turn_number: int
    response_length: int = 0
    user_message_length: int = 0

    def __post_init__(self):
        self.response_length = len(self.agent_response.split())
        self.user_message_length = len(self.user_message.split())

@dataclass
class StructuredConversation:
    conversation_id: str
    exchanges: list[ConversationExchange] = field(default_factory=list)
    total_turns: int = 0
    total_user_words: int = 0
    total_agent_words: int = 0

def structure_conversations(
    raw_messages: list[dict],
) -> list[StructuredConversation]:
    from collections import defaultdict
    grouped: dict[str, list] = defaultdict(list)
    for msg in raw_messages:
        grouped[msg["conversation_id"]].append(msg)

    conversations = []
    for conv_id, messages in grouped.items():
        messages.sort(key=lambda m: m["timestamp"])
        exchanges = []
        turn = 0
        i = 0
        while i < len(messages) - 1:
            if messages[i]["role"] == "user" and messages[i + 1]["role"] == "assistant":
                turn += 1
                exchanges.append(ConversationExchange(
                    user_message=messages[i]["content"],
                    agent_response=messages[i + 1]["content"],
                    timestamp=messages[i]["timestamp"],
                    turn_number=turn,
                ))
                i += 2
            else:
                i += 1

        conv = StructuredConversation(
            conversation_id=conv_id,
            exchanges=exchanges,
            total_turns=len(exchanges),
            total_user_words=sum(e.user_message_length for e in exchanges),
            total_agent_words=sum(e.response_length for e in exchanges),
        )
        conversations.append(conv)
    return conversations

Topic Extraction with LLM Batch Processing

For topic extraction at scale, batch-process conversations through a lightweight LLM to assign topics and intents.

from openai import OpenAI
import json

client = OpenAI()

TOPIC_PROMPT = """Analyze this conversation and extract:
1. primary_topic: the main subject (1-3 words)
2. user_intent: what the user wanted to accomplish
3. sub_topics: list of secondary topics discussed
4. sentiment: positive, neutral, negative, or frustrated

Return JSON with these fields."""

def extract_topics_batch(
    conversations: list[StructuredConversation],
    batch_size: int = 20,
) -> list[dict]:
    results = []
    for i in range(0, len(conversations), batch_size):
        batch = conversations[i:i + batch_size]
        for conv in batch:
            text = "\n".join(
                f"User: {e.user_message}\nAgent: {e.agent_response}"
                for e in conv.exchanges[:5]  # limit for cost
            )
            response = client.chat.completions.create(
                model="gpt-4o-mini",
                messages=[
                    {"role": "system", "content": TOPIC_PROMPT},
                    {"role": "user", "content": text},
                ],
                response_format={"type": "json_object"},
            )
            parsed = json.loads(response.choices[0].message.content)
            parsed["conversation_id"] = conv.conversation_id
            results.append(parsed)
    return results

Pattern Discovery

With topics assigned, aggregate them to find the most common topics, emerging trends, and correlations between topics and outcomes.

from collections import Counter

def discover_patterns(topic_results: list[dict]) -> dict:
    topic_counts = Counter(r["primary_topic"] for r in topic_results)
    intent_counts = Counter(r["user_intent"] for r in topic_results)
    sentiment_counts = Counter(r["sentiment"] for r in topic_results)

    # Find topics correlated with negative sentiment
    negative_topics = Counter()
    for r in topic_results:
        if r["sentiment"] in ("negative", "frustrated"):
            negative_topics[r["primary_topic"]] += 1

    # Calculate frustration rate per topic
    frustration_rates = {}
    for topic, neg_count in negative_topics.items():
        total = topic_counts[topic]
        frustration_rates[topic] = {
            "negative_count": neg_count,
            "total_count": total,
            "frustration_rate": round(neg_count / total * 100, 1),
        }

    return {
        "top_topics": topic_counts.most_common(20),
        "top_intents": intent_counts.most_common(15),
        "sentiment_distribution": dict(sentiment_counts),
        "high_frustration_topics": {
            k: v for k, v in sorted(
                frustration_rates.items(),
                key=lambda x: -x[1]["frustration_rate"],
            )
            if v["frustration_rate"] > 20 and v["total_count"] >= 10
        },
    }

Recurring Issue Detection

Beyond topics, conversation mining can detect recurring specific issues — questions that keep coming back, indicating a gap in documentation or product design.

def find_recurring_questions(
    conversations: list[StructuredConversation],
    similarity_threshold: float = 0.85,
) -> list[dict]:
    from difflib import SequenceMatcher

    first_messages = []
    for conv in conversations:
        if conv.exchanges:
            first_messages.append({
                "conversation_id": conv.conversation_id,
                "message": conv.exchanges[0].user_message.lower().strip(),
            })

    clusters: list[list[dict]] = []
    assigned = set()

    for i, msg_a in enumerate(first_messages):
        if i in assigned:
            continue
        cluster = [msg_a]
        assigned.add(i)
        for j, msg_b in enumerate(first_messages[i + 1:], start=i + 1):
            if j in assigned:
                continue
            ratio = SequenceMatcher(
                None, msg_a["message"], msg_b["message"]
            ).ratio()
            if ratio >= similarity_threshold:
                cluster.append(msg_b)
                assigned.add(j)
        if len(cluster) >= 3:
            clusters.append(cluster)

    return [
        {
            "representative": cluster[0]["message"],
            "count": len(cluster),
            "conversation_ids": [c["conversation_id"] for c in cluster],
        }
        for cluster in sorted(clusters, key=len, reverse=True)
    ]

FAQ

How do I handle conversations in multiple languages?

Translate all conversations to a common language before topic extraction. LLMs handle translation well, so you can add a translation step to your pipeline. Alternatively, use a multilingual embedding model and cluster on embeddings rather than text — this groups similar conversations regardless of language without explicit translation.

How often should I run conversation mining?

Run topic extraction daily on new conversations and a full pattern analysis weekly. Daily extraction keeps your topic distribution current and enables trend detection. The weekly full analysis includes pattern discovery, recurring issue detection, and cross-referencing with outcome data, which requires more context and is computationally heavier.

What should I do with the mining results?

Create an actionable feedback loop. For high-frustration topics, improve the agent's knowledge base or prompt instructions for those specific areas. For recurring questions, consider adding them to a FAQ or proactive messaging flow. For emerging topics, evaluate whether the agent needs new capabilities or tool access to handle them.

---

#ConversationMining #NLP #TopicModeling #TextMining #AIAgents #AgenticAI #LearnAI #AIEngineering