Skip to content

Cost lookup can't price a model reached directly from its vendor's endpoint #191

Description

@Alexsun1one

Cost is looked up with the model id the provider echoes back (completion.modelresponse.model), matched against the LiteLLM table by exact key, else the longest key that's a prefix of it.

Most of that table is route-qualified — 2421 of 2983 keys contain a /. So a model I reach directly from its vendor's own OpenAI-compatible endpoint echoes a bare id that isn't a key at all, and never gets a cost. Running lookup against a freshly pulled table, with the ids exactly as they appear in my usage records:

mimo-v2.5-pro            miss
k3                       miss
MiniMax-M3               miss
gpt-4o-mini-2024-07-18   exact
claude-sonnet-4-6        exact

mimo-v2.5-pro is in the table, as openrouter/xiaomi/mimo-v2.5-pro. But I don't go through OpenRouter — I hit Xiaomi's endpoint directly, so that rate isn't what I'm paying. k3 and MiniMax-M3 have no entry under any route.

I'm flagging that because I don't think a looser matcher fixes this. Matching a bare id onto a route-qualified key would attach a price from a route the operator isn't using, silently, which is worse than no price. The missing information isn't in the model id — it's which route the binding points at, and only the operator has that.

So the shape that looks right to me is the binding carrying its own pricing, a rate card key or explicit rates, which overlaps with what #175 is doing. That's a design call rather than a bug, though.

Separately: none of these states are visible today. cost_usd is simply omitted when no price comes back, so "not priced", "the table didn't load" and "free" are indistinguishable in the log. That's what made me assume for two weeks that cost tracking didn't exist.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions