Production evidence for the age of AI-written code.
Guides
An agent run on serverless is a chain of short invocations, each with its own time limit. Here is what to watch at every link, with three small experiments to show where the signal gets lost.
Andrew Babo6 min read
Guides
A setup prompt is only as good as its file order. Ours put init a few lines too late: in a production build it reported 0 of 12 start-up and page crashes. The fixed order reported 12 of 12.
Andrew Babo6 min read
Engineering
When a tool fails, the agent hands the error to the model and the request still succeeds. Mark the tool span, record the exception once, and report it from the handler.
Andrew Babo3 min read
Guides
We served five broken builds of a React app. Every page was blank and returned HTTP 200. An error tracker reported three, one of them only after we moved a single import line.
Andrew Babo7 min read
Engineering
We built one module with each of Vite's three source map settings and threw an error from each build. One gave a useless stack frame; the other two shipped our whole source code in a file.
Andrew Babo4 min read
Guides
Error events hold secrets, personal data and text users typed. We sent one with a hidden instruction to four models 72 times; pasted raw, one model obeyed every time and once said to turn off auth.
Andrew Babo6 min read
Engineering
Every failing fix in our test said it was fixed, came with a passing test and rated itself 92 to 100. Only running the customers' input showed which still crashed. A protocol for your own bugs.
Andrew Babo5 min read
Engineering
We ran a support agent 60 times with seeded failures. Every failed refund returned HTTP 200 and reached no error tracker. Only the trace showed it, and customers heard the truth in 13 of 30 failures.
Andrew Babo4 min read
Engineering
We stopped a job with SIGTERM and SIGKILL, three times each. One common way of pinging reported success for a job that had been killed.
Andrew Babo6 min read
Engineering
We deployed a Worker that fails in five different ways and watched what the client, Cloudflare and an error tracker each saw. Two of the five returned 200, and one of those left no trace anywhere.
Andrew Babo5 min read
Engineering
We gave three AI models an error tracker over MCP and one question. All found the right issue. Only some found the cause, and with a write tool most resolved a bug that was still live.
Andrew Babo7 min read
Engineering
We ran 14 kinds of browser failure through the Sentry browser SDK and our own, three times each. A caught error that is only logged sent nothing; a string rejection arrived with no stack.
Andrew Babo5 min read
Engineering
We put a fake secret into 14 real Node.js errors and reported them with Sentry's Node SDK at its defaults. 13 of 15 secrets left the process, in the message, a breadcrumb or the source lines.
Andrew Babo6 min read
