Kubernetes Jobs and CronJobs for Batch AI Agent Workloads

When to Use Jobs Instead of Deployments

Not every AI agent runs continuously. Many agent workloads are batch operations: processing a backlog of documents, generating weekly reports, reindexing a vector database, or evaluating model performance. These tasks run to completion and should not restart indefinitely. Kubernetes Jobs are designed for exactly this — they run Pods until successful completion rather than keeping them alive forever.

Basic Job: Single AI Agent Task

A Job creates one or more Pods and ensures they run to completion:

flowchart LR
    GIT(["Git push"])
    CI["GitHub Actions<br/>build plus test"]
    REG[("Container registry<br/>GHCR or ECR")]
    HELM["Helm chart<br/>values per env"]
    K8S{"Kubernetes cluster"}
    DEP["Deployment<br/>rolling update"]
    SVC["Service plus Ingress"]
    HPA["HPA<br/>CPU and queue depth"]
    POD[("Inference pods<br/>GPU node pool")]
    USERS(["Production traffic"])
    GIT --> CI --> REG --> HELM --> K8S
    K8S --> DEP --> POD
    K8S --> SVC --> POD
    K8S --> HPA --> POD
    SVC --> USERS
    style CI fill:#4f46e5,stroke:#4338ca,color:#fff
    style POD fill:#ede9fe,stroke:#7c3aed,color:#1e1b4b
    style USERS fill:#059669,stroke:#047857,color:#fff

# document-processing-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: document-processor
  namespace: ai-agents
spec:
  backoffLimit: 3
  activeDeadlineSeconds: 3600
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: processor
          image: myregistry/doc-processor:1.0.0
          resources:
            requests:
              memory: "1Gi"
              cpu: "500m"
            limits:
              memory: "4Gi"
              cpu: "2000m"
          env:
            - name: BATCH_ID
              value: "2026-03-17-intake"
            - name: OPENAI_API_KEY
              valueFrom:
                secretKeyRef:
                  name: ai-secrets
                  key: openai-api-key
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: document-storage

Key settings: backoffLimit: 3 retries the Job three times on failure. activeDeadlineSeconds: 3600 kills the Job if it runs longer than one hour. restartPolicy: Never prevents the container from restarting within the same Pod — failures create new Pods instead.

Parallel Jobs: Processing Large Batches

For large document batches, run multiple agent Pods in parallel:

# parallel-processing-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: batch-summarizer
  namespace: ai-agents
spec:
  completions: 100
  parallelism: 10
  completionMode: Indexed
  backoffLimit: 10
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: summarizer
          image: myregistry/summarizer:1.0.0
          env:
            - name: JOB_COMPLETION_INDEX
              valueFrom:
                fieldRef:
                  fieldPath: metadata.annotations['batch.kubernetes.io/job-completion-index']

This creates 100 indexed tasks, running 10 at a time. Each Pod receives its index through the JOB_COMPLETION_INDEX environment variable, which it uses to determine which chunk of data to process.

The Python agent uses the index to partition work:

import os

def get_work_partition():
    index = int(os.environ["JOB_COMPLETION_INDEX"])
    total_completions = 100
    # Fetch documents assigned to this partition
    offset = index * 50  # 50 documents per partition
    return fetch_documents(offset=offset, limit=50)

async def main():
    documents = get_work_partition()
    for doc in documents:
        summary = await summarize_document(doc)
        await store_summary(doc.id, summary)
    print(f"Partition {os.environ['JOB_COMPLETION_INDEX']} complete")

if __name__ == "__main__":
    import asyncio
    asyncio.run(main())

CronJobs: Scheduled Agent Tasks

CronJobs create Jobs on a schedule. This is ideal for recurring AI agent tasks:

# weekly-report-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: weekly-report-agent
  namespace: ai-agents
spec:
  schedule: "0 8 * * 1"  # Every Monday at 8:00 AM
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  startingDeadlineSeconds: 600
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: report-agent
              image: myregistry/report-agent:1.0.0
              envFrom:
                - secretRef:
                    name: ai-secrets
                - configMapRef:
                    name: report-config

concurrencyPolicy: Forbid prevents overlapping runs — if the previous report is still generating, the new run is skipped. startingDeadlineSeconds: 600 gives the scheduler a 10-minute window to start the Job if the cluster is under heavy load.

Monitoring Job Completion

Track Job progress programmatically:

# Watch Job status
kubectl get jobs -n ai-agents -w

# Check completion status
kubectl get job batch-summarizer -n ai-agents -o jsonpath='{.status.succeeded}/{.spec.completions}'

# View logs from a specific indexed Pod
kubectl logs job/batch-summarizer -n ai-agents --container=summarizer

Cleanup and TTL

Automatically clean up completed Jobs:

spec:
  ttlSecondsAfterFinished: 86400  # Delete 24 hours after completion

FAQ

How do I handle partial failures in parallel AI agent Jobs?

Set backoffLimit high enough to allow retries for transient failures like API rate limits. Use idempotent processing — each Pod should be able to re-process its partition safely. Store progress checkpoints in a database so failed Pods can resume from where they stopped rather than starting over.

What happens if a CronJob misses its schedule?

If startingDeadlineSeconds is set, Kubernetes counts missed schedules. If more than 100 consecutive schedules are missed, the CronJob stops creating new Jobs and logs a warning. Set a reasonable deadline window and monitor for MissSchedule events in your cluster.

Should I use Jobs or a message queue for batch AI processing?

Jobs are simpler for fixed-size batches where you know the total work upfront. Message queues with KEDA-scaled workers are better for continuous streaming workloads or when new items arrive unpredictably. For many AI agent use cases, a hybrid approach works well — a CronJob that enqueues items, combined with KEDA-scaled workers that process them.

---

#Kubernetes #BatchProcessing #CronJobs #AIAgents #Scheduling #AgenticAI #LearnAI #AIEngineering