The world's most realistic benchmark for database AI-friendliness. A 100-entity enterprise schema across 10 domains. 100 files, ~10K LOC. SWE-bench F2P+P2P verification. Real CLI agents. Live databases. No claims, no hypotheses, no verdicts. Just raw data from real agents on real databases.
We asked two independent AI coding agents (Codex gpt-5.6-terra and Claude Sonnet) to add a field to an existing codebase with a 100-entity schema (4,000+ lines of inline schema code) across 100 files (~10K LOC). SWE-bench F2P+P2P verification: all new tests must pass AND no existing tests may break. Each agent was compared only to itself — never cross-CLI.
No hypotheses. No verdicts. Just the raw numbers — you decide what they mean.
Each CLI counts tokens differently. Claude Code includes the system prompt, tool definitions, and cache read/creation in every token count — so its numbers are inherently higher. Codex counts usage events differently. These numbers are never compared cross-agent. Each agent is compared only to itself across the three database lanes. The percentage differences (e.g. +177%) are within-agent — they measure how much MORE the same agent spends on Postgres vs MongoDB, not how one agent compares to the other.
Every benchmark claims to be fair. This one proves it with 10 independent defenses — any one of which would catch bias if it existed.
No mocks. No simulated agents. No estimated tokens. Every number comes from a real AI agent writing real code against a real database. vanilla mode, anti-cheat, within-agent comparison — zero bias.
If your team runs 1,000 AI database-tasks per month, here's the monthly token bill per database design — based on the measured medians. MongoDB saves significant money on every agent.
Assumptions: 1,000 tasks/month. Token cost estimated at $3/M input tokens (Claude) and $2/M input tokens (Codex). within-agent comparison only — absolute costs are not cross-comparable between agents because CLIs count tokens differently. vanilla mode: no skills, no code reviews, no subagents. anti-cheat: source-pattern scan, protected files, cheat-signal detection.
Based on research into real enterprise database complexity (SAP, Salesforce, Workday, GitLab). 100 entities across 10 business domains, written as inline schema code the agent must read. MongoDB stores them as 55 collections with embedded arrays. Postgres stores them as 100 normalized tables with foreign keys and CHECK constraints.
The agent receives a change ticket — exactly what a developer gets in an enterprise. It must understand the existing schema, find where the field goes, write the migration, update the query, and not break anything.
"Add a preferredPaymentMethod field (string, optional,
enum: credit_card / wire_transfer / ach / paypal) to the
accounts entity. Update the
getOrderSummary query to include the account's
preferredPaymentMethod in the result."
src/schema.mjs (4,000+ lines of inline schema)
to understand the 100-entity schema
src/accounts.mjs, src/orders.mjs,
src/invoices.mjs, src/products.mjs,
src/support.mjs — 27 model files with working CRUD
code
src/queries.mjs for the order summary query
npm test until it passes — tests check regression
AND new behavior