GPT Vision for CAPTCHA and Challenge Detection: Identifying Blocking Elements

The Problem of Blocking Elements

Browser automation agents frequently encounter elements that block their progress: CAPTCHAs, cookie consent banners, newsletter popups, login walls, age verification dialogs, and rate-limit notices. Traditional DOM-based detection fails because these elements vary enormously across sites in their HTML structure, but they all share recognizable visual patterns.

GPT Vision can identify these blockers instantly from a screenshot, classify their type, and help the agent decide how to proceed — without attempting to solve challenges, which raises ethical and legal concerns.

Detecting Blocking Elements

from pydantic import BaseModel
from openai import OpenAI

class BlockingElement(BaseModel):
    element_type: str  # captcha, cookie_banner, paywall, popup, etc.
    description: str
    severity: str  # blocking, dismissible, informational
    dismiss_strategy: str  # close_button, accept, scroll_past, none
    dismiss_button_x: int  # 0 if not dismissible
    dismiss_button_y: int
    blocks_main_content: bool

class PageBlockerAnalysis(BaseModel):
    has_blockers: bool
    blockers: list[BlockingElement]
    main_content_visible: bool
    recommended_action: str  # proceed, dismiss, wait, escalate

client = OpenAI()

def detect_blockers(screenshot_b64: str) -> PageBlockerAnalysis:
    """Detect blocking elements in a screenshot."""
    response = client.beta.chat.completions.parse(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a web page blocker detector. Identify any "
                    "elements that obstruct or block normal page "
                    "interaction. These include:\n"
                    "- CAPTCHAs (reCAPTCHA, hCaptcha, image challenges)\n"
                    "- Cookie consent banners\n"
                    "- Newsletter/subscription popups\n"
                    "- Login/paywall overlays\n"
                    "- Age verification dialogs\n"
                    "- Rate limiting or access denied notices\n"
                    "- Browser compatibility warnings\n\n"
                    "For each blocker, determine if it can be dismissed "
                    "with a simple button click and locate that button. "
                    "Do NOT suggest solving CAPTCHAs."
                ),
            },
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "Analyze this page for blocking elements.",
                    },
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": f"data:image/png;base64,{screenshot_b64}",
                            "detail": "high",
                        },
                    },
                ],
            },
        ],
        response_format=PageBlockerAnalysis,
    )
    return response.choices[0].message.parsed

Handling Dismissible Blockers

Cookie banners and newsletter popups can usually be dismissed with a button click. Build an automated dismissal handler.

flowchart LR
    GOAL(["High level goal"])
    PLAN["Planner LLM"]
    SCREEN["Screen capture<br/>every step"]
    VLM["Vision LLM<br/>reads UI state"]
    ACT{"Action type"}
    CLICK["Click coordinate"]
    TYPE["Type text"]
    KEY["Keyboard shortcut"]
    GUARD["Safety filter<br/>allow lists"]
    OS[("OS sandbox<br/>ephemeral VM")]
    DONE(["Goal verified"])
    GOAL --> PLAN --> SCREEN --> VLM --> ACT
    ACT --> CLICK --> GUARD
    ACT --> TYPE --> GUARD
    ACT --> KEY --> GUARD
    GUARD --> OS --> SCREEN
    OS --> DONE
    style PLAN fill:#4f46e5,stroke:#4338ca,color:#fff
    style GUARD fill:#f59e0b,stroke:#d97706,color:#1f2937
    style OS fill:#ede9fe,stroke:#7c3aed,color:#1e1b4b
    style DONE fill:#059669,stroke:#047857,color:#fff

from playwright.async_api import Page
import asyncio
import base64

class BlockerHandler:
    def __init__(self):
        self.dismissed_count = 0
        self.escalated_count = 0

    async def handle_blockers(
        self, page: Page, max_attempts: int = 3
    ) -> bool:
        """Detect and handle blocking elements. Returns True if
        the page is now clear for interaction."""
        for attempt in range(max_attempts):
            screenshot = await page.screenshot(type="png")
            b64 = base64.b64encode(screenshot).decode()

            analysis = detect_blockers(b64)

            if not analysis.has_blockers:
                return True

            handled_any = False
            for blocker in analysis.blockers:
                if blocker.severity == "dismissible":
                    if (blocker.dismiss_button_x > 0
                            and blocker.dismiss_button_y > 0):
                        await page.mouse.click(
                            blocker.dismiss_button_x,
                            blocker.dismiss_button_y,
                        )
                        self.dismissed_count += 1
                        handled_any = True
                        await asyncio.sleep(0.5)

                elif blocker.severity == "blocking":
                    if blocker.element_type == "captcha":
                        return await self._handle_captcha(
                            page, blocker
                        )
                    elif blocker.element_type == "paywall":
                        return False  # cannot bypass

            if not handled_any:
                break

            await asyncio.sleep(1)

        return analysis.main_content_visible

    async def _handle_captcha(
        self, page: Page, blocker: BlockingElement
    ) -> bool:
        """Handle CAPTCHA by escalating to human operator."""
        self.escalated_count += 1
        print(
            f"CAPTCHA detected: {blocker.description}. "
            "Escalating to human operator."
        )
        # In production, send a notification or queue for manual review
        return False

Pre-Navigation Blocker Check

Integrate blocker detection into your navigation workflow so every page visit is guarded.

class GuardedNavigator:
    def __init__(self):
        self.handler = BlockerHandler()

    async def safe_goto(self, page: Page, url: str) -> bool:
        """Navigate to a URL and handle any blockers."""
        await page.goto(url, wait_until="networkidle")

        # Wait a moment for popups to appear
        await asyncio.sleep(1.5)

        is_clear = await self.handler.handle_blockers(page)

        if not is_clear:
            print(f"Page blocked at {url}, cannot proceed")

        return is_clear

    async def wait_for_manual_resolution(
        self, page: Page, timeout: int = 300
    ) -> bool:
        """Wait for a human to resolve a blocker manually."""
        print(f"Waiting up to {timeout}s for manual resolution...")
        start = asyncio.get_event_loop().time()

        while asyncio.get_event_loop().time() - start < timeout:
            screenshot = await page.screenshot(type="png")
            b64 = base64.b64encode(screenshot).decode()
            analysis = detect_blockers(b64)

            if not analysis.has_blockers:
                print("Blocker resolved, continuing automation")
                return True

            await asyncio.sleep(10)  # check every 10 seconds

        print("Manual resolution timeout")
        return False

Classifying Challenge Types for Logging

Track what types of challenges your automation encounters across runs for monitoring.

from collections import Counter
from datetime import datetime

class ChallengeTracker:
    def __init__(self):
        self.encounters: list[dict] = []

    def record(
        self, url: str, blocker_type: str, resolved: bool
    ):
        self.encounters.append({
            "url": url,
            "type": blocker_type,
            "resolved": resolved,
            "timestamp": datetime.now().isoformat(),
        })

    def summary(self) -> dict:
        types = Counter(e["type"] for e in self.encounters)
        resolved = sum(1 for e in self.encounters if e["resolved"])
        return {
            "total_encounters": len(self.encounters),
            "resolved": resolved,
            "unresolved": len(self.encounters) - resolved,
            "by_type": dict(types),
        }

Ethical Considerations

This system detects and classifies challenges — it does not solve them. CAPTCHAs exist to prevent automated abuse. Solving them programmatically may violate terms of service and potentially laws like the CFAA. The proper response to a CAPTCHA is to either use the site's official API, escalate to a human operator, or respect the site's intent to block automation.

FAQ

Should GPT Vision be used to solve CAPTCHAs?

No. Using GPT Vision to solve CAPTCHAs raises ethical and legal concerns. CAPTCHAs are access control mechanisms, and bypassing them may violate the website's terms of service. Instead, use GPT Vision to detect CAPTCHAs, then either switch to an official API, queue the task for human completion, or skip that particular site.

How does the agent distinguish between a cookie banner and a CAPTCHA?

GPT-4V recognizes visual patterns effectively: cookie banners typically have "Accept" / "Reject" buttons with privacy-related text, while CAPTCHAs show image grids, text challenges, or checkbox widgets with "I'm not a robot" text. The model identifies these with high accuracy because these patterns are visually distinctive and well-represented in its training data.

Can blockers appear after initial page load?

Yes. Many sites trigger popups after a delay, after scrolling, or after a certain number of page views. Run blocker detection not just at page load but also before each interaction step in multi-step workflows. Some newsletter popups only appear 30-60 seconds into a session.

---

#CAPTCHADetection #GPTVision #BrowserAutomation #ChallengeHandling #WebScraping #EthicalAI #BlockerDetection #AgenticAI