AI agents on Odoo and osTicket: first results
17 tasks · 2 agents · 170 runs · each task run from the same starting copy of the system · 2026-11-17
Claude Sonnet 5.5
Claude Haiku 5.5
Task by task
| Task | Claude Sonnet 5.5 passed | actions | Claude Haiku 5.5 passed | actions | Person actions |
|---|---|---|---|---|---|
| odoo-crm-update-001 | 5/5 | 7.2 | 5/5 | 7.4 | 7 |
| odoo-date-001 | 5/5 | 11.4 | 5/5 | 17.2 | 10 |
| odoo-hidden-001 | 5/5 | 7.0 | 5/5 | 8.0 | 6 |
| odoo-inventory-adjust-001 | 5/5 | 12.0 | 5/5 | 12.4 | 10 |
| odoo-inventory-receive-001 | 5/5 | 5.0 | 5/5 | 6.8 | 5 |
| odoo-lookalike-001 | 5/5 | 11.0 | 5/5 | 13.0 | 10 |
| odoo-nochange-001 | 5/5 | 7.8 | 5/5 | 7.0 | 7 |
| odoo-partial-001 | 5/5 | 9.6 | 5/5 | 14.0 | 9 |
| odoo-project-task-001 | 5/5 | 11.0 | 1/5 | 11.4 | 10 |
| odoo-project-task-002 | 5/5 | 11.0 | 2/5 | 11.0 | - |
| odoo-sales-quote-001 | 5/5 | 21.0 | 5/5 | 20.6 | 16 |
| ost-assign-001 | 5/5 | 6.0 | 5/5 | 6.0 | 6 |
| ost-create-001 | 5/5 | 27.6 | 5/5 | 29.0 | 16 |
| ost-create-002 | 5/5 | 27.8 | 5/5 | 27.8 | 16 |
| ost-note-001 | 5/5 | 5.2 | 5/5 | 5.0 | 8 |
| ost-priority-001 | 5/5 | 5.0 | 5/5 | 6.6 | 5 |
| ost-reply-close-001 | 5/5 | 8.0 | 5/5 | 8.2 | 6 |
Passed = runs out of k where the agent finished, stayed off forbidden screens, and the database showed the correct result. Actions = clicks, typing and key presses (waits not counted), averaged over runs.
Where the failures happened
Claude Haiku 5.5: 7 failed runs
odoo-project-task-001, run 2: wrong_end_state
Action 5: red = where this run clicked (left_click), green = where a passing run clicked on the same screen.
The screen right after.
odoo-project-task-001, run 3: wrong_end_state
Action 5: red = where this run clicked (left_click), green = where a passing run clicked on the same screen.
The screen right after.
odoo-project-task-002, run 1: wrong_end_state
Action 5: red = where this run clicked (left_click), green = where a passing run clicked on the same screen.
The screen right after.
odoo-project-task-002, run 2: wrong_end_state
Action 5: red = where this run clicked (left_click), green = where a passing run clicked on the same screen.
The screen right after.
Method
- Each task runs against a disposable copy of an open-source system (Odoo 17, osTicket), reset to the same starting data before every run.
- Every task runs k times. pass@1 is the share of all runs that passed; pass^k is the share of tasks that passed every time. The 95% range is a Wilson interval on pass^k.
- Results are checked in the database, not on screen; forbidden screens (settings, admin) fail a run even if the result is right.
- A person did every task by hand once; their action count is shown for comparison.
- Native dropdown lists are drawn inside the page so the agent can see them in screenshots.
- Costs are estimated from token counts and list prices; small tasks, so ranges are wide.