CloudOpsBench is a new open benchmark for AI agents on cloud operations, led by Ang Chen’s group at the University of Michigan. Version 1.0 has 45 DevOps tasks on AWS — repairing failures, securing access, migrating services, rotating keys, rolling out updates, and decommissioning projects — and runs the same way in real AWS and in a high-fidelity emulator. The project is Apache-2.0 on GitHub under the CloudOpsBench organization.

Coffee summary

  • FACT: 45 real-style AWS DevOps tasks; identical runs in cloud and emulator; GCP/Azure/OCI extensions planned per secondary coverage.
  • FACT (CloudOpsBench blog): Claude Fable 5.1 completed 29 of 45 tasks; GPT-6 Astra completed 26. The public leaderboard shows Fable (max) near 60.6% resolution.
  • FACT (authors): Knowledge gaps were rare; about 80% of failures were reasoning/scope mistakes, about 20% were wrong actions — some irreversible.
  • CONTEXT (The Decoder): Digests round top results around 60% / 53%; cite the primary leaderboard or task counts when precision matters.

What happened

Unlike software-engineering benchmarks graded by unit tests, DevOps work is safety-critical and hard to score. CloudOpsBench tasks span decommissioning, access control, migration/deploy, configure/monitor, and repair. Authors break success into knowing services, reasoning about scope and environment, then acting safely.

Their failure taxonomy is blunt: models generally knew the AWS services involved. They still stopped surveys early, missed in-scope resources in other services (the MediaConvert queue example is vivid), or took destructive actions such as aborting an in-progress multipart upload and losing data.

Why it matters

Cloud agents are being sold as junior DevOps help. CloudOpsBench says the bottleneck is not missing docs — it is judgment about what is in scope and when to stop. A 20% wrong-action share is especially serious when deletes, key rotations, and network changes cannot be undone with Ctrl-Z.

That split also changes how teams should train and evaluate agents. More retrieval over AWS docs will not fix early stop decisions. Better environment surveys, explicit checklists of in-scope services, and forced confirmation before destructive calls matter more than another knowledge plugin.

Who should care

  • Platform and SRE teams piloting coding agents with cloud credentials.
  • Security and compliance owners who must bound blast radius.
  • Agent builders who need a frontier benchmark beyond SWE-bench-style coding.

Limitations

Tasks are AWS-only in v1.0. Leaderboard scores depend on harness and reasoning effort (Claude Code vs Codex vs Terminus, max vs high). Secondary digests sometimes round percentages differently from the live table — prefer cloudopsbench.ai for citations. Emulator fidelity is strong per the authors, but production IAM and org policies still differ. Forty-five tasks is a serious start, not a map of every cloud failure mode.

What to do next

  1. Do not give agents broad AdministratorAccess on production; use break-glass roles and dry-run modes.
  2. Require a human approval step before deletes, key rotations, and security-group changes.
  3. If you evaluate vendors, run CloudOpsBench (or its emulator) instead of trusting chat demos alone.
  4. Log every agent cloud mutation with who approved it and which resources changed.

AIImpish Take

CloudOpsBench is a useful cold shower: agents can recite AWS trivia and still leave queues behind or destroy uploads. Until reasoning about scope improves, treat cloud agents as draft operators behind a ticket queue — not autopilot for irreversible infra.

Related on AIImpish

Sources: CloudOpsBench official site and leaderboard; CloudOpsBench, How well do agents perform on DevOps tasks?; The Decoder summary; CloudOpsBench GitHub organization.