Brunel

Independent testing for software bots and AI agents.

An agent that says "done" is not the same as a task that is done. We test bots and AI agents on the business systems they actually run on, many times over, and check the result in the system itself.

First results

Two AI agents, 17 everyday tasks in an open-source ERP and helpdesk (Odoo and osTicket), five runs each, every result checked in the database.

Claude Sonnet 5.5
17 of 17
tasks right in all 5 runs
Claude Haiku 5.5
15 of 17
tasks right in all 5 runs
Claude Haiku 5.5
7 runs
reported a task as done that was still open

The failures were all one task: marking a project task as done. The cheaper agent moved it into the Done column instead of setting its status, then reported success. On screen it looked finished; in the database it was still open.

Read the full report

How we test

Cases, method and results are open: github.com/Brunel001/bench.

Work with us

If you build AI agents for businesses, or run bots in production, and want to know how reliable they really are, write to founder@brunelconsulting.com.