LLM governance: a working framework

Most enterprise AI governance in 2026 is a document. Usually a good one: it lists the approved tools, forbids pasting confidential data into public chatbots, and asks everyone to watch their API spend. Then it goes up on the intranet, and it governs nothing, because no request on its way to a model provider ever passes through a PDF.

The fix is not a better document. It is to take each rule the document states, find the point in the request path where a system can check it, and put a mechanism there.

What is LLM governance?

LLM governance is the set of controls that decide who in an organization can use which models and tools, with what data, at what spend, and with what record of it. The controls that hold are enforced technically, at the gateway, the identity provider and the admin plane, not stated in a policy document.

It is the operational subset of AI governance. Model risk, evaluation and retention policy sit above it and belong to a different committee. Five layers cover the part that has to hold on the request path:

1. Access. Who reaches which models and tools. Enforced by your identity provider and the gateway’s virtual keys.
2. Data boundaries. What may leave the perimeter. Enforced by guardrails that screen every request before it goes out.
3. Spend. What it may cost, and what happens at the limit. Enforced by budgets on the key.
4. Visibility. One record of who used what, at what cost. Enforced by routing all traffic through one gateway that logs it.
5. Change discipline. How anything new gets added. Enforced by default-deny in the admin plane.

The policy lives on the intranet. The request goes to the provider's API. Nothing on that path reads the intranet.

A control answers two questions a policy cannot: where is the rule checked, and what happens when it trips. “Don’t paste customer data into chatbots” answers neither. “Outbound prompts pass a PII check before egress, and a hit blocks the request” answers both. Here is where each of the five checks runs:

The five layers, and where each one is enforced

1. Access: who reaches which models and tools

The failure is the key that outlives its reason: a contractor sets up a provider key for a pilot, leaves, and the key stays, because rotating it means finding out which nine scripts break.

The control is identity. People sign in through SSO, their team decides which models they can reach, and every person and application gets its own virtual key instead of a copy of the provider’s key. In SecondStack, directory groups from your identity provider map to teams. Models are granted to the organization, to named teams or to named people, separately for chat and for API access, and nothing is reachable until it is granted. Provider keys stay inside the gateway, encrypted, and no one gets a copy.

Deprovisioning is where this layer is tested. Sync jobs reconcile the directory, the admin plane and the gateway on a schedule, so when someone leaves, their AI access leaves with them. Nothing new on the offboarding checklist.

2. Data boundaries: what may leave the perimeter

This failure is quieter: a support engineer pastes a full customer record into a chat to draft a polite reply. The policy forbade it, and nothing was positioned to notice.

The control is a guardrail in the request path that screens prompts, attachments, tool results and replayed history before they reach the provider. Because it sits in front of the gateway, it covers chat, IDE plugins and cron jobs alike. In SecondStack this is SecondGuard: detectors for secrets and personal data run in parallel, the strictest verdict wins, and the verdict is block, mask in place, or log. Policies attach globally, to one model, or to a tag on a set of models, so external providers can carry a stricter posture than internal ones. What it catches, and when to mask rather than block, is its own post.

3. Spend: what it may cost, and what happens at the limit

The failure is the job nobody watched: a batch enrichment script pointed at the flagship model, still running when the office lights come back on.

Two different problems hide under one label. A runaway script is a technical fault, and a hard cap on its key is the only mechanism it respects. A team drifting past its monthly envelope is a management question, and freezing everyone because one estimate was wrong is the wrong answer to it. So: hard caps on the keys that scripts and agents use, and for teams a high safety-net cap with threshold alerts that reach the person who can tell a launch from a bug. In SecondStack, budgets exist at every level from a single key to the organization. Any figure you enter is a hard cap, and alerts fire on the way up to it. Which cap belongs at which level is its own post.

4. Visibility: one queryable record of everything

The failure: the CFO asks what the company spent on LLMs last quarter, by team, and the answer is four provider invoices.

The control is one point that all traffic crosses, with attribution. Every request through the LLM gateway carries a virtual key, so the record knows who sent it, to which model, how many tokens, at what cost, and ControlTower turns that into usage per user, team, key and model. Administrative changes get an append-only history that the database itself refuses to rewrite, exportable to whatever your security team already reads. That history is what an access review actually needs. All of it lands in your own PostgreSQL.

5. Change discipline: how anything new gets added

The failure is the tool nobody approved: a new provider or an MCP plugin in a team’s workflow, because one person had a card and a key.

The control is default-deny with an explicit enable step. In SecondStack a model or provider becomes reachable only after an admin configures it in ControlTower and grants it, and it takes effect on the next gateway deploy. Anyone can build an MCP connector for themselves, but only an admin can turn one on for the whole organization. What no software supplies is an owner and a cadence: one admin owns the catalog, and new models and tools land on a fixed schedule rather than on request, so “not yet” comes with a date.

Is a blocklist governance? Is a certificate?

Three things wear the governance label without earning it.

A blocklist moves the traffic rather than the risk, onto personal devices where none of the five layers can see anything. It has a place and a ceiling, both covered in detecting shadow AI.

A certificate attests that a process exists and is audited. ISO 27001 applies to SecondStack through T1A, our implementation partner, and procurement will rightly ask for it. It does not sit in the request path, so it belongs beside the five layers, not instead of them.

A committee is good at deciding what the rules should be. A board that also tries to enforce them, approving use cases one meeting at a time, teaches teams to ship first and ask later. Let people set policy, and let machines enforce it.

Start with the gateway, tune the guardrails second

Put a gateway you control in front of the traffic first. That one component carries spend, visibility and the key half of access on day one, and nobody’s workflow changes. Guardrails come second, and need time in log-only mode before anyone gets blocked. Change discipline arrives with the admin plane and survives only if it has an owner and a cadence.

Through all five layers, hold one test: the governed path has to be more convenient than the ungoverned one. If it is slower than a personal ChatGPT account, people use the personal account, and governance ends up covering the traffic that matters least. That is why the five layers here share one identity and one database you own. Hold every vendor to the convenience test. Ours included.