Building a Clinical Documentation Agent: AI-Assisted Medical Note Generation

The Documentation Burden

Physicians spend roughly two hours on documentation for every one hour of patient care. This documentation burden is a leading cause of clinician burnout. A clinical documentation agent listens to the encounter (via transcription), extracts structured medical information, generates a SOAP note draft, and presents it for physician review — cutting documentation time by 50 to 70 percent.

The agent does not replace the physician's clinical judgment. It handles the mechanical work of structuring information, allowing the physician to focus on accuracy and completeness during review.

The SOAP Note Structure

SOAP (Subjective, Objective, Assessment, Plan) is the standard format for clinical documentation. Each section has distinct content requirements:

flowchart LR
    CALLER(["Patient or Caregiver"])
    subgraph TEL["Telephony"]
        SIP["Twilio SIP and PSTN"]
    end
    subgraph BRAIN["Healthcare AI Agent"]
        STT["Streaming STT<br/>Deepgram or Whisper"]
        NLU{"Intent and<br/>Entity Extraction"}
        TOOLS["Tool Calls"]
        TTS["Streaming TTS<br/>ElevenLabs or Rime"]
    end
    subgraph DATA["Live Data Plane"]
        CRM[("CRM and Notes")]
        CAL[("Calendar and<br/>Schedule")]
        KB[("Knowledge Base<br/>and Policies")]
    end
    subgraph OUT["Outcomes"]
        O1(["Appointment booked"])
        O2(["Prescription refill request"])
        O3(["Triage to clinician"])
    end
    CALLER --> SIP --> STT --> NLU
    NLU -->|Lookup| TOOLS
    TOOLS <--> CRM
    TOOLS <--> CAL
    TOOLS <--> KB
    NLU --> TTS --> SIP --> CALLER
    NLU -->|Resolved| O1
    NLU -->|Schedule| O2
    NLU -->|Escalate| O3
    style CALLER fill:#f1f5f9,stroke:#64748b,color:#0f172a
    style NLU fill:#4f46e5,stroke:#4338ca,color:#fff
    style O1 fill:#059669,stroke:#047857,color:#fff
    style O2 fill:#0ea5e9,stroke:#0369a1,color:#fff
    style O3 fill:#f59e0b,stroke:#d97706,color:#1f2937

from dataclasses import dataclass, field
from typing import Optional
from datetime import datetime

@dataclass
class SOAPNote:
    patient_id: str
    encounter_date: datetime
    provider_id: str
    subjective: str = ""
    objective: str = ""
    assessment: str = ""
    plan: str = ""
    icd_codes: list[str] = field(default_factory=list)
    cpt_codes: list[str] = field(default_factory=list)
    status: str = "draft"  # draft, pending_review, signed
    review_comments: Optional[str] = None

    def to_formatted_note(self) -> str:
        return (
            f"ENCOUNTER NOTE - {self.encounter_date.strftime('%Y-%m-%d')}
"
            f"{'=' * 50}

"
            f"SUBJECTIVE:
{self.subjective}

"
            f"OBJECTIVE:
{self.objective}

"
            f"ASSESSMENT:
{self.assessment}

"
            f"PLAN:
{self.plan}

"
            f"ICD-10: {', '.join(self.icd_codes)}
"
            f"CPT: {', '.join(self.cpt_codes)}
"
        )

Transcript Processing Pipeline

The documentation agent takes raw encounter transcription and extracts structured information in stages:

from enum import Enum

class SpeakerRole(Enum):
    PROVIDER = "provider"
    PATIENT = "patient"
    NURSE = "nurse"

@dataclass
class TranscriptSegment:
    speaker: SpeakerRole
    text: str
    timestamp: float

class TranscriptProcessor:
    """Extracts structured clinical data from encounter transcripts."""

    SYMPTOM_KEYWORDS = [
        "pain", "ache", "fever", "cough", "nausea", "fatigue",
        "dizziness", "swelling", "rash", "bleeding", "shortness of breath",
    ]

    MEDICATION_PATTERNS = [
        "taking", "prescribed", "started", "stopped", "increased", "decreased",
    ]

    def extract_chief_complaint(self, segments: list[TranscriptSegment]) -> str:
        for segment in segments:
            if segment.speaker == SpeakerRole.PROVIDER:
                if "what brings you in" in segment.text.lower() or "how can i help" in segment.text.lower():
                    idx = segments.index(segment)
                    if idx + 1 < len(segments) and segments[idx + 1].speaker == SpeakerRole.PATIENT:
                        return segments[idx + 1].text
        # Fallback: first patient statement
        for segment in segments:
            if segment.speaker == SpeakerRole.PATIENT:
                return segment.text
        return ""

    def extract_symptoms(self, segments: list[TranscriptSegment]) -> list[dict]:
        symptoms = []
        for segment in segments:
            if segment.speaker != SpeakerRole.PATIENT:
                continue
            text_lower = segment.text.lower()
            for keyword in self.SYMPTOM_KEYWORDS:
                if keyword in text_lower:
                    symptoms.append({
                        "symptom": keyword,
                        "context": segment.text,
                        "timestamp": segment.timestamp,
                    })
        return symptoms

    def extract_medication_mentions(self, segments: list[TranscriptSegment]) -> list[dict]:
        mentions = []
        for segment in segments:
            text_lower = segment.text.lower()
            for pattern in self.MEDICATION_PATTERNS:
                if pattern in text_lower:
                    mentions.append({
                        "speaker": segment.speaker.value,
                        "context": segment.text,
                        "action": pattern,
                    })
                    break
        return mentions

SOAP Note Generator

The generator assembles extracted data into a structured note:

class SOAPNoteGenerator:
    def __init__(self, processor: TranscriptProcessor):
        self.processor = processor

    def generate(
        self,
        segments: list[TranscriptSegment],
        patient_id: str,
        provider_id: str,
        vitals: Optional[dict] = None,
    ) -> SOAPNote:
        chief_complaint = self.processor.extract_chief_complaint(segments)
        symptoms = self.processor.extract_symptoms(segments)
        medications = self.processor.extract_medication_mentions(segments)

        subjective = self._build_subjective(chief_complaint, symptoms, medications)
        objective = self._build_objective(vitals)

        return SOAPNote(
            patient_id=patient_id,
            encounter_date=datetime.utcnow(),
            provider_id=provider_id,
            subjective=subjective,
            objective=objective,
            assessment="[PENDING PROVIDER REVIEW]",
            plan="[PENDING PROVIDER REVIEW]",
            status="draft",
        )

    def _build_subjective(
        self, chief_complaint: str, symptoms: list[dict], medications: list[dict]
    ) -> str:
        lines = [f"Chief Complaint: {chief_complaint}"]
        if symptoms:
            symptom_list = list({s['symptom'] for s in symptoms})
            lines.append(f"Associated Symptoms: {', '.join(symptom_list)}")
        if medications:
            lines.append("Medication Discussion:")
            for med in medications:
                lines.append(f"  - {med['context'][:100]}")
        return "
".join(lines)

    def _build_objective(self, vitals: Optional[dict]) -> str:
        if not vitals:
            return "[Vitals not yet recorded]"
        parts = []
        if "bp" in vitals:
            parts.append(f"BP: {vitals['bp']}")
        if "hr" in vitals:
            parts.append(f"HR: {vitals['hr']}")
        if "temp" in vitals:
            parts.append(f"Temp: {vitals['temp']}")
        if "spo2" in vitals:
            parts.append(f"SpO2: {vitals['spo2']}")
        if "weight" in vitals:
            parts.append(f"Weight: {vitals['weight']}")
        return "Vitals: " + ", ".join(parts)

Review Workflow

The generated note is always a draft. The physician must review and sign:

@dataclass
class ReviewAction:
    action: str  # "approve", "edit", "reject"
    provider_id: str
    timestamp: datetime
    edits: Optional[dict] = None
    comments: Optional[str] = None

class NoteReviewWorkflow:
    def __init__(self):
        self.audit_trail: list[ReviewAction] = []

    def submit_for_review(self, note: SOAPNote) -> SOAPNote:
        note.status = "pending_review"
        return note

    def process_review(self, note: SOAPNote, action: ReviewAction) -> SOAPNote:
        self.audit_trail.append(action)

        if action.action == "approve":
            note.status = "signed"
        elif action.action == "edit":
            if action.edits:
                for field_name, new_value in action.edits.items():
                    if hasattr(note, field_name):
                        setattr(note, field_name, new_value)
            note.status = "pending_review"
        elif action.action == "reject":
            note.status = "draft"
            note.review_comments = action.comments

        return note

FAQ

Can the documentation agent auto-populate the Assessment and Plan sections?

The agent can suggest assessment and plan content based on the symptoms, history, and provider's documented treatment patterns. However, these sections require the most clinical judgment and should always be clearly marked as AI-suggested drafts. Many practices configure the agent to leave these sections blank for the physician to complete, generating only the Subjective and Objective sections automatically.

How does the agent handle multiple conditions discussed in one visit?

The agent identifies distinct clinical topics in the transcript and structures them as separate problem entries within the SOAP note. For example, if a patient discusses both knee pain and a medication refill for hypertension, the note will contain organized sections for each condition with their respective symptoms, findings, and plan items.

What happens if the transcription quality is poor?

The agent includes a confidence score for each extracted data point. Low-confidence extractions are flagged with brackets like "[unclear: possible mention of metformin]" so the reviewing physician knows to verify against their recollection. The agent never guesses — it surfaces uncertainty explicitly.

---

#HealthcareAI #ClinicalDocumentation #SOAPNotes #MedicalTranscription #Python #AgenticAI #LearnAI #AIEngineering