Safebots · the matchup
A fair comparison against the popular AI coding tools — Cursor, Copilot, Claude Code, Devin, Cody. Where we're ahead, where they are, and the one thing we haven't proven yet. Each claim is tagged so you can see how we know it.
measured we ran it and have numbers · architectural a structural fact, not a benchmark · open not yet proven
01 · How it works
Every popular AI coding tool works the same way underneath. You ask a question, it skims through your code looking for relevant bits, answers, and then forgets everything it just learned. Ask again and it does all that work over. Ours reads your whole project one time and keeps a map of it — every function, setting, and label — so answering a question is a quick lookup on the map. It updates only the parts that changed.
The skim-and-forget approach is good at fuzzy questions like "find code that looks like this." The map is good at exact ones like "what uses this, and what will I break if I change it." That's the whole idea: the AI model is a rented engine you can swap out anytime, and the map is the part that actually knows your project and keeps getting more valuable. Almost everything below follows from keeping the map.
02 · The cost
The other tools work by opening file after file, and each new file gets re-read alongside everything already opened. The bigger the project and the longer the session, the more that piles up — the cost grows faster than the work does. Our map is built by fast, plain scanning of the code (no AI needed for that part), and the AI is called only for the few spots that genuinely need judgment. So we never get on that treadmill. In our tests the map answered questions 3.4 to 9.2 times cheaper than the open-every-file approach, and the gap grows with bigger projects and repeat questions. A team of these tools re-reads the same code all day; the map reads it once and is done. Full breakdown: grokonomics.
↑ scorecard03 · Cheaper AI works
This is the core idea, and we measured it on real jobs. On three real coding tasks, a small, cheap AI given the map matched a much pricier one and beat the most expensive tier — same tasks, real dollar costs, and the results actually run and checked line by line. In one case the cheap AI got a detail right that the expensive one got wrong, because the map handed it the fact instead of making it guess. The point is simple: once the project is already mapped, most everyday code changes just don't need the priciest AI. The other tools can't say that, because they figure the project out from scratch each time.
The full three-model, three-task grid — plans, diffs, the Fable refusals, and the cost compounded layer on layer (cheaper model → index removes re-reading → warm cache → amortization) — is laid out and grounded in real API runs in The Substrate Ledger. The short version: the cheap AI matched the pricey one, beat the most expensive tier, and the map answered 3 to 9 times cheaper than the open-every-file approach.
04 · Precision
| Question | Embedding / agentic SOTA | The graph |
|---|---|---|
| "What calls this?" | Similar-looking code, maybe misses callers | The exact caller set — it's an edge |
| "What breaks if I change it?" | Inferred, often incomplete | The transitive blast-radius, across files |
| "Do we already do X?" | Can confidently hallucinate a yes | A real yes/no from the feature graph |
| "Where is this string / style set?" | Grep-ish | The exact key + every read-site + current value |
| "Who's authorized to call this?" | Not modeled | The resolved guard (role/modifier/gate) |
05 · Reach
No SOTA coding tool connects your app's PHP/JS to the platform it calls, to the Solidity a web3 call reaches, to the config that gates it, to what customers said in support tickets. Because it's all one graph, a single question spans them: "a JS getContract(…).addToCirculation() call → the on-chain function → its onlyRole guard → who holds that role → the user whose wallet it resolves to." That traversal — proven across four artifact kinds in our tests — has no equivalent in a tool that only sees the files in front of it.
06 · Trust
A synthesized conclusion carries derivedFrom links to every fact it rests on, so it's auditable back to a line of code, and it's automatically marked stale when any of those facts change on a re-grok. SOTA answers are confident and sourceless — you can't tell what a claim depends on, or whether it went out of date when you edited the file underneath it. Provenance + staleness is what lets this say "not built" or "this may be out of date" instead of bluffing.
07 · Execution
SOTA agents run your tests in your shell or your CI — on the live tree, with your credentials, after a commit. We run them in a ZFS clone inside a sealed sandbox before proposing anything: the change is applied to a snapshot, tested, and the snapshot is discarded — pass or fail, the live tree and git history are untouched until a human approves the diff. A hostile or broken generated test corrupts a clone that evaporates, not your repo. That's an integration SOTA coding agents structurally don't have, because they don't own the execution substrate — they borrow yours.
This safe-copy testing comes from the Safebox layer. On its own, the tool falls back to running tests in a container you already trust — we label that clearly, so you know it's a lighter guarantee.
08 · Safety
The agent runs behind host-filtered egress (it can reach only allowlisted hosts — a compromised dependency has nowhere to send your secrets), holds only capabilities it was granted, and proposes, never merges — enforced by a whitelist where "merge" simply isn't an available operation. SOTA safety is largely instruction-and-guardrail: "please don't do the bad thing." Ours is: the bad thing isn't reachable. Same thesis as the rest of the stack — trust the substrate, not a promise.
↑ scorecard09 · Compounding
SOTA writes fresh code per task, and duplicates accumulate into a maintenance swamp. Ours checks the content-addressed tool catalog first: exact match → reuse the hash, generate nothing; close → a grow-only successor that adds options without breaking anything pinned to the old one; genuinely different → a new lineage, never silently a "breaking upgrade." The collection compounds in value as it grows instead of sprawling — a discipline enforced by content-addressing and a superset-check, not by hoping.
↑ scorecard11 · Why you can trust it
The reason a cheaper model over the grok beats a frontier model exploring is that the grok hands it proven facts, not plausible ones. The Safebox toolchain makes this literal: on the U language it runs sixteen mechanistic analyses — every one an AST walk, a symbol-table query, or a hash, none involving an LLM — so each result is reproducible and checkable, and a consumer (human, tool, or model) can rely on it. A partial list of what is proven rather than assumed:
| Analysis | Proves (mechanically, zero LLM) |
|---|---|
| Effect & capability inference | purity/effects propagated bottom-up, then checked — a -E marker on an effectful fn is an error, not a lie |
| Type & effect checker | mutability, nullability, declared-throws, overload ambiguity, MVCC write paths — rejected at compile, not hoped |
| Compile-time folding | a pure z f over values is executed and replaced; anything unfoldable stays an honest runtime call |
| Relation graphs | the unique root store, and edges grouped by distance from it — the shape queried, not inferred |
| Supply-chain audit & lockfile | module fingerprints and dependency provenance by hash |
The map runs the same way on any project: do the plain, checkable work first, and only ask the AI for the parts that truly need judgment. That's why a fact marked certain means something — it came from a scan you can re-run and verify, not from an AI that can sound confident and still be wrong. (Analysis list contributed by the Safebox team.)
↑ scorecard12 · The whole line
Because support tickets, surveys, and calls are grokked into the same graph as the code, demand can be clustered, ranked by measured buildability (blast-radius, not a guess), proposed, built through review, and — since every request kept its provenance — the exact people who asked get told when it ships. SOTA coding tools sit inside a developer's editor; this runs the understand-to-ship loop for a business. Different job, not a faster version of the same one.
↑ scorecard13 · Where SOTA is ahead
A comparison that only lists our wins wouldn't be worth reading. The top coding tools are genuinely better than us at some things, and here they are:
| Axis | Frontier coding tools | Us |
|---|---|---|
| Fluent code generation in the moment | Excellent — years of tuning | We lean on models for this; it's not our edge |
| Fuzzy "find code like this" | Embeddings shine | A graph is the wrong tool for fuzzy recall |
| Zero setup | Install and go | A grok index + Safebox is heavier to stand up |
| Ecosystem / IDE integration | Deep, mature | Early |
| Language breadth proven at scale | Broad | ~10 languages, fidelity drops on dynamic dispatch |
So this isn't "we beat them." Use the coding tools for what they're great at, and use the map to run the whole job around them. Together they're stronger than either alone.
↑ scorecard14 · The open question
Straight talk, because the whole stack is built on not bluffing: the understanding and decision layers are done and tested — 150+ assertions across 17 suites across grok, gate, safety, identity, currency-gating, corpus-at-scale, synthesis, and reuse, every guarantee proven by a negative control. The hinge — does the generator produce a genuinely usable tool? — we decomposed rather than left as one big unknown, and shrank it hard:
| Sub-risk of "usable generated tool" | Status |
|---|---|
| (a) emitted code is syntactically valid | verified (with negative control) |
| (b) it conforms to the declared interface it was generated for | verified (missing input/output rejected) |
| (c) it's shaped to run in the sandbox clone (entry + inputs + a test) | verified (no-test rejected) |
| (d) it does the real task, executed end-to-end in an attested box | OPEN — needs a live Orchestrator.runTool |
So the open question is no longer "does any of this work" — it's the single, bounded experiment (d): run one generated tool through a live attested sandbox and confirm it produces the right answer. The harness for exactly that is written and turnkey (tests/hinge-live.test.js) — it fails honestly today (no Orchestrator present) and passes only in a real deployment. We name it, decompose it, and hand you the experiment rather than a claim — that's the difference between a matchup and a pitch deck.
It costs less, answers more exactly, sees more of your system, and follows the job all the way to a shipped change that people hear about — measured where we could measure it. Rent the AI; keep the map. The other tools are still better at writing fresh code in the moment and fuzzy search. And we're honest about the one thing left to prove — the whole idea is here.— the whole matchup, in three lines↑ scorecard