43 → 0: Governing AI in production
Why ungoverned AI is a production incident waiting to happen
Most teams adopt LLMs the same way: a first feature calls the provider SDK directly, it
works, and the pattern copies itself. Six months later there are dozens of new Anthropic()
and fetch("https://api.openai.com/...") calls spread across the codebase. Each one is an
independent liability:
- No spending ceiling. A retry loop, a traffic spike, or a prompt-injection attack can run the token bill to five figures before anyone notices — the first signal is the invoice.
- No audit trail. "Which feature spent what, for which tenant, last month?" has no answer, so Finance can't forecast and Compliance can't attest.
- No PII boundary. Untrusted text flows into third-party models with no redaction and no record of what crossed the perimeter.
- No injection defense. Customer messages and product fields get interpolated straight into prompts.
- Vendor lock-in. Every call hardcodes one provider, so switching models — or surviving a provider outage — is a code change in every file.
We had exactly this: 43 call sites reaching the provider SDKs directly. Here's how we took that to zero, and the architecture that keeps it there.
The core principle: one gate, no bypass
The single most important decision is where the controls live. The naive design puts a
checkRateLimit() call before each callAI() — which means every new call site has to
remember to add it. That is the root cause of ungoverned sprawl: governance that depends on
discipline always decays.
Instead, the rate-limit and budget gates live inside the gateway. A caller cannot reach a model without passing them, because there is no code path to the provider that skips them:
if (/\b(408|429|5\d\d)\b/.test(msg)) return true;
return /overloaded|rate.?limit|timeout|timed out|aborted|ETIMEDOUT|ECONNRESET|ENOTFOUND|EAI_AGAIN|fetch failed|socket hang up|network error/i.test(msg);
}
async function withAiRetry(fn, { label = "ai call" } = {}) {
let attempt = 0;
for (;;) {
try {
return await fn();
} catch (err) {
if (attempt >= AI_MAX_RETRIES || !isRetriableAiError(err)) throw err;
attempt++;
const backoff = Math.min(AI_RETRY_CAP_MS, AI_RETRY_BASE_MS * 2 ** (attempt - 1));
const waitMs = backoff + Math.floor(backoff * 0.25 * Math.random()); // +0–25% jitter
console.warn(
`[ai-client] retriable error (attempt ${attempt}/${AI_MAX_RETRIES}) on ${label}: ` +
`${err?.message || err}; retrying in ${waitMs}ms`,
Two details that matter in production:
- Fail-closed on money paths. A
BUDGET_EXCEEDEDerror is terminal — it is never retried. Only genuinely transient signals (429/5xx/timeout) retry; an exhausted budget stops the call cold. - Fail-open on infra blips. If the limiter itself errors (a Redis hiccup), we log and continue rather than halt every AI call across every tenant. The gate degrades safely in both directions.
Every call is accounted — automatically
Cost attribution can't be a separate step either, or it drifts. The gateway writes a usage record on every successful call: model, input/output tokens, estimated USD cost, feature, and tenant — fire-and-forget, so a logging failure never blocks the response:
// `gateShop` (= shop || ANONYMOUS_SHOP), NOT raw `shop`, so a call without
// ctx.shop is metered against the shared anonymous budget bucket instead of
// bypassing the cap entirely (it's already rate-limited under that bucket).
if (!skipBudget) {
try {
const { checkAiBudget } = await import("./plan-gate.server.js");
const estimate = Number.isFinite(estimatedTokens) ? estimatedTokens : maxTokens * 4;
const bg = await checkAiBudget(gateShop, estimate);
if (!bg.allowed) {
const err = new Error(bg.reason || "AI monthly budget exceeded");
err.code = "BUDGET_EXCEEDED";
err.budget = bg;
throw err;
}
} catch (err) {
if (err?.code === "BUDGET_EXCEEDED") throw err;
// Budget is a MONEY control. Default posture is fail-open (availability),
// but operators can opt into fail-closed (AI_BUDGET_FAIL_CLOSED=true) so a
// budget-check outage can't allow unbounded spend across all shops.
if (process.env.AI_BUDGET_FAIL_CLOSED === "true") {
throw Object.assign(new Error("AI budget check unavailable — failing closed."), { code: "BUDGET_EXCEEDED" });
}
console.warn("[ai-client] budget check failed (fail-open):", err.message);
Cost math comes from a single pricing source of truth (ai-models.js) imported by both the
usage logger and the budget gate — so the cost dashboard and the budget enforcement can never
silently disagree. "Spend per feature, per model, per tenant" becomes a SQL query instead of a
project.
Built-in defenses, not bolt-ons
Prompt-injection sanitization
Untrusted text gets a defensive clamp before interpolation — control characters stripped, classic override preambles defanged to inert markers, length bounded. It's a mitigation paired with output validation, not a silver bullet, but it raises the floor everywhere it's applied:
}).catch(err => console.error("[ai-client] usage log failed:", err.message));
}
return json ? safeParseJson(text) : text;
}
// ── callAIFromMessages (Phase 2 migration helper) ────────────────────────────
// Accepts the SAME param shape the routes used to pass to
// `anthropic.messages.create({ model, max_tokens, system, messages })` and
// routes it through the governed callAI() — so migrating a direct-SDK call site
// is a wrapper swap, not a prompt rewrite. Returns the response TEXT directly
// (callAI's return), replacing the old `message.content[0].text` access.
//
// Flattens the user/assistant turns into a single prompt; for the common
// single-user-message generation routes that's just that one message. Multi-
// turn conversational endpoints (chatbot) should use a real messages path, not
Conservative error classification
Resilience is only safe if you retry the right errors. The classifier is deliberately conservative: gate errors (rate-limit, budget) and auth/validation 4xx are terminal, so a request that can never succeed never loops:
// Cohere retired the command-light / command (and the unversioned r/r-plus
// aliases) on 2025-09-15. Remap stored prefs to the closest live tier.
"command-light": "command-r7b-12-2024",
"command": "command-r-08-2024",
"command-r": "command-r-08-2024",
"command-r-plus": "command-r-plus-08-2024",
};
/** Map a retired model id to its live replacement; pass live ids through. */
export function normalizeModelId(model) {
return (model && LEGACY_MODEL_ALIASES[model]) || model;
}
Retries themselves are bounded, with exponential backoff plus jitter, owned in one place — the provider SDKs' own retry is disabled at the call site so behavior is uniform and observable across all five vendors.
Provider-portable by construction
Because every call goes through one entry point, the provider and model are configuration, not
code. The gateway dispatches to Anthropic, OpenAI, Gemini, Mistral, or Cohere behind an
identical request shape — switching models, or failing over during an outage, is a settings
change. Agentic tool-use runs through the same governed path (callAITools), so copilots get
the same gates, retry, and accounting as a one-shot generation.
Enforcement: making bypass impossible, not just discouraged
Architecture without enforcement decays back to sprawl. A custom ESLint rule
(no-direct-ai-sdk) bans importing the provider SDKs anywhere outside the gateway. Its
exception ledger is empty — there are no grandfathered files. A new direct-SDK call fails
CI before it can merge.
A second guard catches the subtler failure mode — a call that reaches the gateway but forgets
to pass tenant context. In strict mode (AI_CLIENT_STRICT_SHOP=true, on in dev/CI) that
throws; in production it emits a traced, deduplicated Sentry event so any unmetered path
surfaces as a single rising-count issue rather than silent drift.
How we got from 43 to 0
The migration was mechanical by design:
- Inventory. Grep every provider-SDK import and
fetchto a model endpoint. 43 sites. - Build the shims.
callAIfor one-shot prompts,callAIFromMessagesfor the existingmessages.create({...})shape (a wrapper swap, not a prompt rewrite), andcallAIToolsfor the agentic copilots that had been forced onto the raw SDK. - Migrate in waves, smallest blast radius first, each behind the same gates.
- Sanction the exceptions. Five governed peers (image analysis, a worker, the sales copilot) keep their own gates + usage logging for good reason — they're documented, not forgotten.
- Lock the door. Turn on the ESLint rule with an empty ledger so the count can't climb back up.
The whole program landed with the test suite green — 986 tests at completion.
What this buys you
| Before | After |
|---|---|
| 43 ungoverned call sites | 1 gateway, 0 bypasses |
| No spending ceiling | Hard per-tenant monthly budget cap |
| "What did AI cost us?" → unknown | Cost per feature/model/tenant is a query |
| PII flows raw into models | Sanitization at the interpolation boundary |
| One hardcoded provider | Five providers, switch by config |
| Governance by code review | Governance by CI — bypass fails the build |
This is the architecture behind Sumeru Systems' AI Governance Audit — we map every AI call site in a codebase, estimate worst-case cost exposure, and deliver a prioritized remediation plan against the nine dimensions above. Reach out at support@sumeru.systems.