Blog · 2026-08-22 · 6 min read
Our empty wallet is not your zero
We audited 160 of our own failed engine calls. 64% were us: credit and quota, not the engines. Here is what we built so a vendor's outage can never be sold to you as your invisibility.
There is a failure in this business worse than being wrong, and most of the category commits it daily: selling you an infrastructure problem as a market finding. A vendor's API call fails, the dashboard prints "your brand: 0% on this engine," and you just inherited a crisis that never happened. Budgets move on that zero. Agencies get fired on that zero. Nobody checks it, because a zero is silent, plausible and convenient.
An admission first: in August we ran the forensics on our own stored failures, 160 failed engine calls across two production runs, every one with its raw error payload kept. The distribution was humbling. 103 of the 160, 64%, traced to our own provider accounts: out of credit, over quota. Another roughly 21% were output-budget truncation, our own configuration cutting answers short. The engines were barely the story. We were the story, and we are telling it to you anyway, because a measurement company's failures are part of its methodology, and methodology you cannot inspect is just marketing.
What we built from it
Technical information first. A billing-class refusal is never stored as an answer. The classifier reads the actual refusal payload, and when the cause is our credit or our quota, the unit stays pending and the run holds. It does not complete around the hole. A report cannot misread a failure that was never allowed to become data.
Second, retryable failures get one bounded retry and then one bounded repair sweep, and the word that matters is bounded. Timeouts and rate limits retry once inside the run; before a run may finish, stored failures get a single claimed repair pass. An unbounded retry loop re-bills the engines forever, and we learned the expensive way that spend discipline and measurement discipline are the same discipline wearing different hats.
Third, a daily balance sentinel probes every provider we pay for a fraction of a cent, and a billing or auth refusal emails us before it can touch a customer's run. The failure we most need to catch is our own, so that is the one with the alarm wired to it.
And the disclosure rule that predates all of this still stands: any failed call that does get stored is excluded from every metric, an engine with zero successful calls is dropped rather than zeroed, and the report shows the failure counts in a banner, per engine, on every audit where they occur. Absence of data is disclosed as absence of data. The grounding rule is the other half of that promise.
The one question to ask any vendor
"When your API call fails, what does my dashboard show?" It is the sharpest methodology question in this category, and we would put money on most vendors not having an answer ready. Ask whether failures are stored, whether they are excluded from the metrics, and whether the report discloses them. If a vendor has never audited their own failure modes, every zero on that dashboard is unexamined by definition, and you are paying for the privilege of guessing.
Ours are on the report. Counts, per engine, in a banner. We would rather show you a smaller number we can defend than a bigger one we cannot, and we think that is exactly the instinct you want in the people measuring your business. Run the free scorecard and read the banner yourself; if it is empty, that is a measurement too.
— The AEOSearch team