By Illia Sokolov, Senior Technology Fellow at ABCloudz

Most days, our AI spend for development stayed below $100.

Then one day hit $490.

No big production launch. No planned load test. No obvious reason in the data.

It was clear, visible, and hard to ignore.

The problem was that the chart could only tell us one thing with confidence: the money was gone.

It could not tell us who invited it to leave.

A bit of context first

Like many engineering teams, we started using generative AI to speed up development. Developers were using it more often, and we were gradually bringing it into the way our teams worked day to day.

At first, we tracked costs in the most obvious way. The providers already had the data. Amazon Bedrock, OpenAI, and other vendors knew how much we were spending. So we pulled provider cost data through their APIs, connected it to Grafana, and built dashboards to watch AI spend over time.

This was a reasonable first step. We could see daily spend. We could compare providers. We could spot unusual days. We did not have to wait for a billing surprise at the end of the period.

For a while, that felt like enough.

Then that one day appeared on the chart and broke the pattern.

Who or what created that spike?

Okay, $500 in one day was not going to bankrupt us. But who could promise that tomorrow it would not be $1,000? And the day after that, $2,000?

“We do not really have a problem spending even $10,000 a day if we know the money is going into real work, something valuable, something that will pay off,” we said during an internal discussion about that $490 day.

And then the uncomfortable part became obvious.

We did not have a good enough way to take one daily spend number apart and see who spent what, where it went, why it happened, and whether it actually produced enough value.

A provider could show us tokens, usage, and cost. But it could not tell us what each request meant inside our work.

We needed a way to connect each LLM request to its source before it reached the provider.

Where should that attribution happen so every AI-enabled tool did not have to implement its own tracking logic?

Looking for the right place to solve it

We briefly considered adding tracking inside every tool that used an LLM.

Technically, it could work. Practically, it felt fragile. Every team would need to pass metadata the same way. Every new workflow would need the same discipline. One missed place, and the next spike would come with another mystery inside it.

So we looked for an AI gateway.

After comparing several options, we chose LiteLLM. It gave us an open-source proxy layer we could put directly into the request path, adapt to our workflow, connect to Grafana, and use for virtual keys, budgets, limits, and logs.

For organization-wide AI cost attribution, the common request path turned out to be a more reliable control point than instrumentation repeated independently across every AI-enabled application.

See what we compared before choosing LiteLLM
Tool Why it could fit Why it did not fit our case
Portkey Virtual keys, routing, fallbacks, rate limits, budget controls Broader gateway platform and a less direct fit for our self-hosted, Grafana-based workflow
Helicone Logs, cost tracking, request visibility Strong observability focus, with less emphasis on request-path control and budget guardrails
TrueFoundry Team, user, model, and project controls; budgets; request blocking; governance Heavier platform scope and more governance than we needed for a fast internal rollout
Cloudflare AI Gateway Managed logging, caching, rate limiting, retries, fallbacks More managed and ecosystem-specific than our internal attribution model required
LiteLLM Request-path proxy, multiple providers, virtual keys, budgets, limits, logs, self-hosted setup No major blocker for our case

The choice was specific to what we needed. We already had Grafana and provider-level cost collection. We were looking for a layer that could sit directly in the LLM request path and add attribution and control without requiring us to replace the rest of the monitoring workflow.

How LiteLLM gave each request a name

So what changed after we added LiteLLM?

Before that, the request went almost directly to the provider. Somewhere in the configuration, we had a real OpenAI, Amazon Bedrock, or other provider key. A developer, tool, or application sent an LLM request. The provider counted the tokens. Later, that usage came back to us as part of the overall spend.

The interesting part happened before we ever saw the money in Grafana.

LiteLLM changed that part of the path. The main mechanism was virtual keys.

A real provider key is the credential that actually gives access to OpenAI, Amazon Bedrock, or another provider. Instead of giving that credential directly to every application or workflow, we put LiteLLM in between.

On the outside, each developer, feature, or workflow can get its own virtual key. Inside LiteLLM, that key is connected to the real provider credentials behind the scenes.

Every requester still reaches the provider, but now the request arrives with an identity we control.

  • This virtual key belongs to this project.
  • This one belongs to this feature.
  • This one belongs to this team.
  • This one belongs to this developer or service account.

We could attribute AI spend to the units that actually mattered to us, such as projects, features, teams, developers, and service accounts, even when they shared the same underlying provider account.

We can split or group those keys according to what we need to measure.

And once spend had names, the next question became obvious.

How much freedom should each source have?

See how virtual keys separate usage without creating separate provider accounts

A virtual key does not replace the provider account. It adds another layer of identity in front of it.

For example, one application can have separate virtual keys for a configuration analyzer, planning feature, assistant, and test workflow. All of them may ultimately use the same provider credentials.

That means provider account structure no longer has to define the level at which we measure application usage.

The same mechanism also simplifies work across multiple providers because applications can access different models through the proxy instead of implementing separate provider-specific integrations throughout the application.

How budget limits turned tracking into guardrails

As soon as we started assigning virtual keys to real sources, we ran into the next setup question.

The sources were not equal.

How much can each source actually spend?

We still wanted to keep using AI to speed up development. But a developer workflow, an automated test, an experimental assistant, and a real product feature should not all have the same room to spend.

So we started putting budgets around the sources themselves.

In LiteLLM, budgets can be assigned to teams, users, features, virtual keys, and other units we want to control.

If a source is still within its limit, the request goes to the provider. If the budget is gone, LiteLLM stops the next request and returns a budget-limit response.

Budgets can reset daily, weekly, or monthly. We can also limit request behavior itself, including requests per minute, tokens per minute, and maximum parallel requests.

A virtual key became more than a name in our cost data. It became a controllable unit with an owner, a budget, and usage limits.

There is an important distinction here, though.

A budget limit works well as a safety boundary for predictable or potentially runaway AI usage. It should not become a substitute for investigating why a useful workflow is expensive.

For example, an automated test may only need a very small budget if its job is simply to verify that an LLM integration works. A development workflow doing real AI-assisted work may legitimately need much more.

If a useful workflow repeatedly reaches its limit, the answer may be a larger budget.

If it reaches the limit because of retries, unnecessary calls, or unexpected agent behavior, the workflow itself needs attention.

See where AI budget limits are useful and where they can become misleading

A hard limit is particularly useful when normal consumption is predictable.

Automated tests are one example. If a test should make only a small number of real LLM calls, a low budget can prevent a retry loop or other failure from continuing indefinitely.

The same principle can protect experimental or agentic workflows that may behave unpredictably.

The limit itself does not tell us whether the underlying usage is good or bad. It only defines a boundary.

That distinction matters because cost optimization should not automatically mean forcing every workflow toward the lowest possible spend.

How logs showed what was happening inside usage

Budget limits gave us boundaries.

But they still did not answer the most interesting question.

What actually happened?

Let’s say one virtual key spent more than usual. Now we can see its name. We can see the owner. We can see the budget. We can even stop the next request if the limit is gone.

But to understand the situation, we still need to see what was happening inside that usage.

That is where logs helped.

LiteLLM gave us a request-level picture. We could look at usage not only as cost, but as a sequence of specific actions.

  • What requests were sent?
  • Which model was used?
  • How many tokens did they consume?
  • Was it a normal working scenario, repeated retries, an expensive model, an oversized prompt, or a workflow that wandered farther than expected?

The next action depended on the answer:

  • If the usage was normal and produced a useful result, the source might simply need a bigger budget.
  • If the cost came from retries, we needed to check the retry logic.
  • If a workflow used a model that was too expensive for the task, we could consider a cheaper one.
  • If a prompt inflated token usage, we could optimize the prompt.
  • If a test flow made too many calls, it needed a stricter policy or limit.

The same cost spike can represent productive work or an implementation problem. Cost alone is not enough to decide what should be optimized.

Now that we had names, budgets, and logs, we could bring not only the financial picture into Grafana, but the operational picture as well.

See what request-level monitoring can reveal and how access to it can be controlled

Depending on the request and integration, logs can provide data such as:

  • request source;
  • user or service account;
  • model;
  • tokens;
  • cost;
  • tools involved;
  • request details;
  • response details.

That does not mean every stakeholder needs access to full prompts and responses.

For client environments, request-level visibility can be governed according to the client’s requirements using approaches such as role-based access, controlled access, aggregation, redaction, and retention rules.

Business stakeholders may only need cost summaries. Engineering leads may need request diagnostics. Security or compliance teams may need their own visibility and retention policies.

Grafana connected the financial and operational picture

This is where we came back to Grafana. Only now it was not the same monitoring picture we started with. In the first version, Grafana showed us the financial side: how much we spent, on which day, and with which provider.

LiteLLM added a second layer: virtual keys, users, teams, features, models, tokens, budgets, limits, and requests. Each layer was useful on its own, but incomplete. Provider data gave us the financial truth, but not the behavior behind it.

LiteLLM showed usage close to the time requests happened, but its dollar value was still an estimate based on usage and configured model pricing.

So we brought both views together in Grafana.

Operational estimates and provider billing serve different purposes. Near-real-time usage data is useful for control and investigation; provider billing remains the source to reconcile against for actual financial cost.

Now a strange day on the chart no longer looked like one lonely number.

We could start with actual spend and go lower:

  • Which project grew?
  • Which feature?
  • Which virtual key?
  • Which user or service account?
  • Which model?
  • How many tokens?
  • Which limits fired or did not fire?

The dashboard became an investigation path from actual provider spend down to the activity that generated it.

And to keep that picture from living only inside dashboards, we added regular reports.

The people who need this information can receive a summary daily, weekly, or monthly depending on how closely they need to watch it. Reports can show total spend, provider split, costs by project, feature, user, or service account, and comparisons between LiteLLM estimates and final provider billing.

Make AI spend visible before it becomes a surprise

The $490 spike was not important because $490 itself was a dangerous amount.

It was important because we could see the cost but could not explain it well enough.

We now have several layers of control working together:

  • provider data tells us what we actually paid;
  • virtual keys tell us where usage came from;
  • budgets and limits define boundaries;
  • logs help explain what happened;
  • Grafana and reports make that information usable by the people who need it.

The useful target is not minimum AI spend. It is explainable AI spend: knowing what consumed the budget, whether the usage was justified, and where intervention is actually needed.

At ABCloudz, we use this workflow in our own AI-assisted development and AI-enabled systems, and we can bring the same type of visibility and control into client projects.

This is particularly useful when a client owns and manages the AI resources used by our development team. The client can keep those resources under its own control while still getting visibility into how they are being consumed during development.

If your AI spend is becoming visible but still hard to explain, we can help connect usage to the people, teams, tools, features, budgets, and reports behind it, so AI can keep scaling without turning into another mystery on the bill.

Ready to start the conversation?