AI Agent SaaS Architecture: Designing a Multi-Tenant Agent Platform from Scratch

Why Agent SaaS Architecture Is Different

Building a SaaS platform that hosts AI agents for multiple customers is fundamentally different from building a single-purpose agent application. You are not just running one agent — you are running thousands of independently configured agents with different tools, different prompts, different data sources, and different usage patterns, all on shared infrastructure. The architectural decisions you make in the first month determine whether you can scale to your thousandth customer or need a painful rewrite at customer fifty.

The core tension in agent SaaS architecture is between isolation and efficiency. Each tenant wants their agents to feel like a dedicated system — private data, custom behavior, reliable performance. But you need shared infrastructure to keep costs manageable. Getting this balance right is the central design problem.

Service Decomposition

A well-decomposed agent platform separates concerns into services that can scale independently. Here is a proven decomposition:

flowchart LR
    CORPUS[("Pre-training corpus<br/>trillions of tokens")]
    FILTER["Quality filter and<br/>dedupe"]
    TOK["BPE tokenizer"]
    SHARD["Shard plus<br/>data parallel"]
    GPU{"GPU cluster<br/>FSDP or DeepSpeed"}
    CKPT[("Checkpoints<br/>every N steps")]
    LOSS["Loss curve plus<br/>eval gates"]
    SFT["SFT phase"]
    DPO["DPO or RLHF"]
    BASE([Base model])
    INSTR([Instruct model])
    CORPUS --> FILTER --> TOK --> SHARD --> GPU
    GPU --> CKPT --> LOSS
    LOSS --> BASE --> SFT --> DPO --> INSTR
    style GPU fill:#4f46e5,stroke:#4338ca,color:#fff
    style LOSS fill:#f59e0b,stroke:#d97706,color:#1f2937
    style INSTR fill:#059669,stroke:#047857,color:#fff

# service_registry.py — Core services in an agent SaaS platform

SERVICES = {
    "gateway": {
        "responsibility": "Authentication, rate limiting, tenant routing",
        "scales_with": "total_request_volume",
        "stateless": True,
    },
    "agent_runtime": {
        "responsibility": "Execute agent loops, manage tool calls, handle streaming",
        "scales_with": "concurrent_conversations",
        "stateless": True,
    },
    "agent_config": {
        "responsibility": "Store and serve agent definitions, prompts, tool configs",
        "scales_with": "total_agents_defined",
        "stateless": False,
    },
    "conversation_store": {
        "responsibility": "Persist conversation history, context windows",
        "scales_with": "message_volume",
        "stateless": False,
    },
    "tool_executor": {
        "responsibility": "Run tool calls in sandboxed environments",
        "scales_with": "tool_call_volume",
        "stateless": True,
    },
    "billing_meter": {
        "responsibility": "Track usage events, calculate costs, emit invoices",
        "scales_with": "event_volume",
        "stateless": False,
    },
}

The key insight is that agent_runtime and tool_executor are stateless and CPU/memory-intensive, so they scale horizontally. The config and conversation services are stateful but receive less traffic, so they scale vertically with database optimization.

Multi-Tenant Data Model

The data model must enforce tenant isolation at every layer. Here is the core schema:

# models.py — SQLAlchemy models for multi-tenant agent platform
from sqlalchemy import Column, String, ForeignKey, JSON, DateTime, Index
from sqlalchemy.dialects.postgresql import UUID
from sqlalchemy.orm import declarative_base
import uuid

Base = declarative_base()

class Tenant(Base):
    __tablename__ = "tenants"
    id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
    name = Column(String(255), nullable=False)
    plan = Column(String(50), nullable=False, default="free")
    api_key_hash = Column(String(128), nullable=False, unique=True)
    settings = Column(JSON, default=dict)

class Agent(Base):
    __tablename__ = "agents"
    id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
    tenant_id = Column(UUID(as_uuid=True), ForeignKey("tenants.id"), nullable=False)
    name = Column(String(255), nullable=False)
    system_prompt = Column(String, nullable=False)
    model = Column(String(100), default="gpt-4o")
    tools_config = Column(JSON, default=list)
    temperature = Column(String, default="0.7")

    __table_args__ = (
        Index("idx_agents_tenant", "tenant_id"),
    )

class Conversation(Base):
    __tablename__ = "conversations"
    id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
    tenant_id = Column(UUID(as_uuid=True), ForeignKey("tenants.id"), nullable=False)
    agent_id = Column(UUID(as_uuid=True), ForeignKey("agents.id"), nullable=False)
    external_user_id = Column(String(255))
    created_at = Column(DateTime, nullable=False)

    __table_args__ = (
        Index("idx_conversations_tenant_agent", "tenant_id", "agent_id"),
    )

Every table includes tenant_id, and every query filters on it. This is non-negotiable. A single missing tenant filter is a data breach.

API Gateway Design

The gateway authenticates requests, resolves the tenant, and routes to the correct service:

# gateway.py — Tenant-aware API gateway
from fastapi import FastAPI, Header, HTTPException, Depends
from functools import lru_cache
import hashlib

app = FastAPI()

async def resolve_tenant(x_api_key: str = Header(...)):
    key_hash = hashlib.sha256(x_api_key.encode()).hexdigest()
    tenant = await db.fetch_one(
        "SELECT id, plan, settings FROM tenants WHERE api_key_hash = :h",
        {"h": key_hash},
    )
    if not tenant:
        raise HTTPException(status_code=401, detail="Invalid API key")
    return tenant

@app.post("/v1/agents/{agent_id}/chat")
async def chat(agent_id: str, body: ChatRequest, tenant=Depends(resolve_tenant)):
    agent = await db.fetch_one(
        "SELECT * FROM agents WHERE id = :aid AND tenant_id = :tid",
        {"aid": agent_id, "tid": tenant["id"]},
    )
    if not agent:
        raise HTTPException(status_code=404, detail="Agent not found")
    return await runtime_service.execute(agent, body.messages, tenant)

Notice how the agent lookup includes tenant_id in the WHERE clause. Even if an attacker guesses another tenant's agent ID, the query returns nothing because the tenant filter excludes it.

Isolation Strategies

There are three isolation levels to choose from, each with different tradeoffs:

Shared database, shared schema — All tenants in one database, one set of tables. Cheapest to operate, hardest to ensure isolation. Use row-level security in PostgreSQL to add a safety net.

Shared database, separate schemas — Each tenant gets their own PostgreSQL schema. Better isolation, slightly higher operational overhead. Good for mid-market SaaS.

Separate databases — Maximum isolation, highest cost. Reserved for enterprise customers with compliance requirements.

Most agent platforms start with shared schema and graduate large customers to separate schemas when needed.

FAQ

How do I prevent one tenant's heavy usage from affecting other tenants?

Implement per-tenant rate limiting at the gateway layer and use separate queue partitions for the agent runtime. For the LLM calls specifically, maintain per-tenant token budgets and use a priority queue that deprioritizes tenants who are consuming above their plan limits. At the database level, use connection pooling with per-tenant connection limits.

Should I use a single LLM API key for all tenants or let each tenant bring their own?

Support both. Use your platform key as the default so customers can get started immediately, and offer a "bring your own key" option for customers who want to use their own OpenAI or Anthropic accounts. This reduces your LLM costs and gives enterprise customers more control. Store their keys encrypted at rest and never log them.

When should I split from a monolith to microservices?

Start with a modular monolith — all the services in one deployable unit but with clean module boundaries. Extract the agent runtime first when you need to scale it independently, then the tool executor when sandboxing becomes critical. Most platforms do not need full microservices until they pass 500 active tenants.

---

#SaaSArchitecture #MultiTenancy #AIAgents #APIDesign #SystemDesign #AgenticAI #LearnAI #AIEngineering