UFO vs Browser Automation: Desktop Apps That Can't Be Automated with Playwright

The Automation Gap

Modern web automation is a solved problem. Playwright, Selenium, and Puppeteer can interact with any web application through well-defined DOM APIs. But a large portion of enterprise computing still happens in Windows desktop applications — ERP systems, medical records software, CAD tools, legacy accounting packages, and internal tools built with WinForms, WPF, or even MFC.

These applications have no DOM, no CSS selectors, and no REST APIs. They exist only as native Windows processes with graphical interfaces. This is the gap UFO fills.

Where Playwright and Selenium Fall Short

Browser automation tools operate on the DOM — the structured tree of HTML elements that represents a web page. Their core capabilities include:

flowchart LR
    GOAL(["High level goal"])
    PLAN["Planner LLM"]
    SCREEN["Screen capture<br/>every step"]
    VLM["Vision LLM<br/>reads UI state"]
    ACT{"Action type"}
    CLICK["Click coordinate"]
    TYPE["Type text"]
    KEY["Keyboard shortcut"]
    GUARD["Safety filter<br/>allow lists"]
    OS[("OS sandbox<br/>ephemeral VM")]
    DONE(["Goal verified"])
    GOAL --> PLAN --> SCREEN --> VLM --> ACT
    ACT --> CLICK --> GUARD
    ACT --> TYPE --> GUARD
    ACT --> KEY --> GUARD
    GUARD --> OS --> SCREEN
    OS --> DONE
    style PLAN fill:#4f46e5,stroke:#4338ca,color:#fff
    style GUARD fill:#f59e0b,stroke:#d97706,color:#1f2937
    style OS fill:#ede9fe,stroke:#7c3aed,color:#1e1b4b
    style DONE fill:#059669,stroke:#047857,color:#fff

# Playwright: Easy and reliable for web apps
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://app.example.com")

    # CSS selectors, text selectors, role selectors
    page.click("button.submit")
    page.fill("input[name='email']", "[email protected]")
    page.wait_for_selector(".success-message")

This works beautifully for web applications. But consider these scenarios where it fails completely:

Scenario 1: Legacy ERP System — A company runs SAP GUI, a native Windows application. There is no browser version available. Playwright cannot see or interact with SAP GUI windows.

Scenario 2: Desktop Accounting Software — QuickBooks Desktop stores data locally and has a native Windows interface. The web version exists but lacks features the accounting team depends on.

Scenario 3: CAD/Engineering Tools — AutoCAD, SolidWorks, and MATLAB are desktop applications with complex custom-rendered UIs.

Scenario 4: File System Operations with GUI — Renaming, moving, and organizing files through File Explorer with specific right-click operations and property modifications.

UFO's Approach to Desktop Automation

UFO uses the Windows UI Automation (UIA) framework combined with visual understanding:

# UFO approach: Works with any Windows application
# No DOM, no CSS selectors — just visual understanding

# Task: Automate a legacy inventory management application
task = """
In the Inventory Manager application:
1. Click the 'New Item' button in the toolbar
2. In the Item Name field, type 'Widget Pro X200'
3. Set the category dropdown to 'Electronics'
4. Enter 150 in the Quantity field
5. Enter 29.99 in the Unit Price field
6. Click Save
"""

# UFO handles this by:
# 1. Taking a screenshot of the application
# 2. Identifying labeled UI controls
# 3. Asking GPT-4V which control matches "New Item button"
# 4. Executing the click
# 5. Repeating for each step

Key Differences at a Glance

Dimension	Playwright	UFO
Target	Web browsers	Windows desktop apps
Element ID	DOM selectors	Vision + UIA tree
Reliability	Very high (deterministic)	Moderate (model-dependent)
Speed	Fast (direct API)	Slower (LLM per step)
Cost	Free	$0.01-0.03 per action
UI resilience	Breaks on selector changes	Adapts visually

When to Choose UFO Over Browser Tools

Use UFO when:

• The application is a native Windows desktop app with no web equivalent
• The application's UI changes frequently and maintaining selectors is costly
• You need to automate a one-off or infrequent task that does not justify writing a full script
• The application has no API and no scripting interface (no COM, no CLI)
• You need to work with file dialogs, print dialogs, and other OS-level UI elements

Use Playwright/Selenium when:

• The application is web-based or has a web interface
• You need high-speed execution (hundreds of actions per second)
• Reliability and determinism are critical (test suites, CI/CD)
• You want to avoid per-action API costs
• Cross-platform execution (Linux, macOS) is required

The Hybrid Approach

Many real-world workflows span both web and desktop applications. A common pattern is using Playwright for web portions and UFO for desktop portions:

from playwright.sync_api import sync_playwright
import subprocess

def hybrid_workflow():
    """Download report from web app, process in desktop Excel."""

    # Phase 1: Web automation with Playwright
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=False)
        page = browser.new_page()
        page.goto("https://analytics.company.com")
        page.fill("#username", "[email protected]")
        page.fill("#password", "secure-password")
        page.click("button[type='submit']")

        # Download the report
        with page.expect_download() as download_info:
            page.click("text=Export to Excel")
        download = download_info.value
        file_path = download.path()
        browser.close()

    # Phase 2: Desktop automation with UFO
    # Open the downloaded file in Excel and process it
    subprocess.run([
        "python", "-m", "ufo",
        "--task",
        f"Open {file_path} in Excel. "
        "Create a pivot table from the data. "
        "Add a chart showing monthly trends. "
        "Save the workbook."
    ])

    print("Hybrid workflow completed")

Enterprise Software That Needs UFO

Common enterprise applications that lack web interfaces or APIs:

• SAP GUI — the classic SAP client used by thousands of enterprises
• Oracle Forms — legacy Oracle application interfaces
• AS/400 terminal emulators — mainframe access through desktop clients
• Medical records systems — many healthcare applications are desktop-only
• Industrial control panels — SCADA and HMI interfaces
• Government systems — tax filing, licensing, and regulatory applications

For these applications, UFO provides an automation path that simply did not exist before vision-capable LLMs.

FAQ

Can I use UFO to automate Electron apps like VS Code or Slack Desktop?

Yes. Electron apps are rendered by Chromium but run as desktop applications. They expose UIA elements, so UFO can interact with them. However, since Electron apps are essentially web apps in a wrapper, you might also consider using Playwright with the Electron-specific API for better performance and reliability.

Is UFO fast enough for automated testing?

UFO is not designed for test automation. Each step requires an LLM API call (200-2000ms latency) plus screenshot processing. A 10-step task takes 20-60 seconds. For automated testing, use dedicated testing frameworks. UFO is best for workflow automation, data entry, and one-off tasks.

---

#DesktopVsWeb #PlaywrightAlternative #LegacyAutomation #EnterpriseRPA #MicrosoftUFO #HybridAutomation #WindowsApps