SecondStack vs. "just LiteLLM": what you actually get for an in-house AI Gateway

A common and fair question: “LiteLLM is open source and we could run it ourselves — what is the difference between using SecondStack and using LiteLLM directly?”

The short answer: SecondStack is not a replacement for LiteLLM — LiteLLM runs inside SecondStack as the routing and metering engine. We are active contributors to the LiteLLM open-source project and maintain our own production builds. What SecondStack adds is everything an enterprise needs around the gateway: SSO-governed self-service, hierarchical budget governance, organization-level access control, a hardened security perimeter, and ongoing maintenance of LiteLLM itself, so you never have to operate or license it separately.

Below is the difference, point by point.

1. LiteLLM — supported and maintained, included

Running LiteLLM “directly” is not free in practice:

• Operations: upgrades, schema migrations, performance tuning, incident response, and keeping pace with a project that releases very frequently.
• Licensing: the features an enterprise actually needs from day one — SSO/SAML, JWT auth, audit logs, SLA support — sit in LiteLLM Enterprise, a commercial tier with non-public, individually negotiated pricing. Public references put the premium tier at roughly $30,000/year.

With SecondStack, LiteLLM support and maintenance is part of the engagement: there is no separate LiteLLM license to buy. Our team are frequent contributors to LiteLLM upstream; we maintain our own builds that track upstream and cherry-pick high-priority bug and security fixes ahead of official releases.

When something breaks at 100 RPS on a Friday, it is our problem, not yours.

2. Self-service API keys under your enterprise SSO

End users sign in with your corporate identity provider (SAML/OIDC) and create and manage their own API keys in a clean, purpose-built portal, with their budgets, usage, and limits visible in the same place.

This is technically achievable in vanilla LiteLLM, but with caveats: the native LiteLLM admin UI is an operator console, not an end-user portal — it is dense, exposes far more than an end user should see, and SSO for it requires the Enterprise license. So “free self-serve keys in LiteLLM” is not actually free, and not actually end-user-friendly.

3. Budget governance that matches how organizations work

LiteLLM provides the low-level primitives: keys, users, teams, budgets. SecondStack overlays the organizational logic on top of them — and in the end provisions and enforces everything through LiteLLM configuration, deployed automatically by ControlTower. Alerts and notifications are SecondStack’s own.

What the overlay gives you:

• Top-down hierarchy: Organization → Department (upcoming release) → Team → User. Budgets are managed where the accountability sits, by the people who own them (department heads, team supervisors), not by a central gateway admin editing YAML.
• Per-capita budgets by design. We deliberately prioritize per-user budgets, managed hierarchically, over aggregate team-level hard caps. A shared departmental “bucket” means one misbehaving user or runaway script can hard-lock the entire team out of LLM access. In our view, aggregate departmental spend should be governed through visibility, alerts, and bounded per-capita limits, not horizontal hard caps that punish everyone at once.
• Self-serve budget allocation. Users distribute their personal budget across apps, agents, and API keys themselves, including an explicit “unallocated” reserve: available to the user on request, but not silently spendable until they explicitly allocate it. Vanilla LiteLLM has no such concept.
• One-time spend grants (upcoming release). When a user runs out of budget mid-period and has a legitimate need, a team supervisor or department head can issue a one-time top-up for the current period only — instead of permanently raising a monthly watermark that, realistically, no one ever rolls back.

4. Access control to models and MCP tools, the way organizations think about it

Same philosophy: SecondStack expresses access in terms your organization already uses — User, Group, Team, Department, Organization, default-deny — and compiles it down to LiteLLM configuration.

It is also channel-aware, which vanilla LiteLLM has no concept of. A real example from our deployments: give all users access to a premium model (e.g. Claude Opus) through the chat web UI, where spend is naturally self-limiting, while restricting API access to that same model to a small allow-list, to prevent unbounded agentic/coding workloads from draining budgets. In vanilla LiteLLM this is an awkward multi-entity construction to build and maintain; in ControlTower it is a natural, first-class policy.

5. A drastically smaller attack surface

This point matters more than it might seem. LiteLLM exposes a very broad management REST API — hundreds of endpoints — and that surface has a documented security track record: privilege escalations, authentication bypasses, and similar issues have repeatedly been found and fixed, traceable through the project’s public security advisories and fix history. We can walk you through representative examples on request.

To be clear: the LiteLLM team promptly fixes these issues, and we report and contribute fixes upstream ourselves. But the lesson stands: in a fast-moving project with a management API this broad, vulnerabilities will keep being discovered, and most of them live in the management surface.

SecondStack’s posture:

• Only the minimally necessary inference endpoints are exposed (/v1/chat/completions, /v1/messages, /models, /images/generations, …). The entire LiteLLM management API — where the bulk of the published vulnerabilities live — is unreachable from outside the platform. Administration goes through ControlTower: a curated, SSO-guarded surface.
• For vulnerabilities on the inference path itself, our maintained builds let us ship priority security fixes immediately, without waiting for the next upstream “stable” release train.

6. When you are ready for end-user chat, it is already there

Even if the AI Gateway is the immediate priority, the next question arrives on its own: once models are routed in-house, departments will want a web chat UI on top of them — with or without a central plan. Grassroots adoption of Open WebUI, LibreChat, etc. is the typical pattern, with all the shadow-IT issues that brings.

SecondStack includes an enterprise-grade Chat App on the same platform: same SSO, same budgets, same access policies, same audit trail. LiteLLM’s built-in chat UI is a rudimentary testing tool, not a mature product for end users. You do not have to roll out our Chat App on day one, but when the demand appears, it is already there — open it up and invite users, not a new procurement cycle.

When “just LiteLLM” is the right call

To be fair about it: vanilla LiteLLM is an excellent proxy, and sometimes it is all you need. If a small platform team is fronting one or two internal applications, nobody outside that team ever touches a key, and you have the engineering capacity to own upgrades and security patching yourself, then run LiteLLM directly and skip the platform. The open-source tier covers that scenario well.

The calculus changes when keys go to end users, budgets need owners other than the gateway admin, or the management API has to live inside a real corporate security perimeter. That is the point where “just LiteLLM” quietly becomes a platform project, and where SecondStack starts earning its keep.

SecondStack vs. LiteLLM: side by side

You keep everything that makes LiteLLM attractive — open source, OpenAI-compatible API, self-hosted in your own perimeter — and gain the governance, security, and support layer that turns a proxy into an enterprise platform.