inferrail

Integrations

How to send traffic through a self-hosted Inferrail gateway, attach attribution, and group requests into units of work. For the exact supported request surface, see PRODUCT.md.

Before you start

The gateway process calls the provider, so the gateway process needs the provider key. A key that exists only in your application’s process is not passed along. Set the key in the terminal where you run inferrail serve:

export OPENAI_API_KEY=...        # for /v1/chat/completions
export ANTHROPIC_API_KEY=...     # for /v1/messages
inferrail serve --quickstart

--quickstart registers both providers and passes any model id through to the matching one. Only the provider whose key is set will succeed. Real requests are billed by your provider as usual.

The model ids in the examples below are only examples. Inferrail doesn’t choose a model: send any model your account can use. inferrail models lists them and shows which ones have a price (needed under a dollar budget).

Clients then point at the gateway. Unless you set INFERRAIL_GATEWAY_TOKEN, the gateway ignores the client’s API key, so any placeholder works.

OpenAI SDK and compatible clients

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="not-needed")
client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Say hello in five words."}],
    extra_headers={"X-Inferrail-Attribute-Customer": "acme"},
)

Or, without code changes: export OPENAI_BASE_URL=http://127.0.0.1:8000/v1.

With curl:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "X-Inferrail-Attribute-Customer: acme" \
  -d '{"model": "gpt-4o-mini",
       "messages": [{"role": "user", "content": "Say hello in five words."}]}'

The response is the standard choices/usage shape plus a non-standard inferrail block (route, provider, latency, retries) that OpenAI clients ignore. Supported: text messages, streaming (stream: true), and tool/function calling, structured outputs (response_format), and the other provider-valid fields listed in PRODUCT.md. Rejected with an error rather than silently dropped: n != 1, non-text content parts, fields whose cost the gateway can’t account for, and any unknown field. See examples/basic_chat_request.py.

Anthropic SDK

POST /v1/messages is a separate Anthropic-compatible passthrough with streaming and tool use (ADR 0014). The Anthropic SDKs append /v1/messages themselves, so their base URL stops at the origin, without /v1:

import anthropic

client = anthropic.Anthropic(base_url="http://127.0.0.1:8000", api_key="not-needed")
client.messages.create(
    model="claude-haiku-4-5-20251001",
    max_tokens=256,
    messages=[{"role": "user", "content": "Say hello in five words."}],
)

Or export ANTHROPIC_BASE_URL=http://127.0.0.1:8000. See examples/anthropic_messages_request.py.

Claude Code

Point Claude Code at the gateway and give a run (or a whole loop of runs) an id and a dollar budget with ANTHROPIC_CUSTOM_HEADERS:

export ANTHROPIC_BASE_URL=http://127.0.0.1:8000
export ANTHROPIC_CUSTOM_HEADERS=$'X-Inferrail-Attribute-Work-Id: loop-42\nX-Inferrail-Budget-Usd: 20'
claude -p "..." --model claude-sonnet-5
inferrail work loop-42

Every invocation that sends the same work id draws from the same budget, so an orchestrator that runs Claude Code round after round (build, review, fix) can cap the whole loop, not just one session. When the budget can’t cover the next request, the gateway refuses it with HTTP 402 before the provider and Claude Code stops with API Error: 402 budget 'work_id:loop-42:per_work' ... would be exceeded.

To give every session its own budget instead, without choosing ids yourself, start the gateway with --work-id-header X-Claude-Code-Session-Id (Claude Code sends its session id on every request) and declare only the amount:

inferrail serve --quickstart --app-mode --work-id-header X-Claude-Code-Session-Id
export ANTHROPIC_CUSTOM_HEADERS='X-Inferrail-Budget-Usd: 5'
claude -p "..."
inferrail report --by work_id      # one row per session

An explicit X-Inferrail-Attribute-Work-Id still wins over the session header. The same flag works for other agents that identify their session, e.g. OpenCode’s x-opencode-session-id; repeat it to list several, first present wins (work_id_headers: in inferrail.yaml does the same).

Things to know:

Tested with Claude Code 2.1.289 (requests accepted and forwarded, refusals stop the session, two invocations sharing one budget) against a stand-in upstream; not yet confirmed end to end against the Anthropic API.

The inferrail serve --quickstart banner in release 0.4.3 prints the Anthropic base URL with a trailing /v1, which makes the SDK request /v1/v1/messages. Use the origin shown above. The banner is fixed on main.

Choosing a model

"model" first selects a named route from inferrail.yaml (for example default), which maps to a provider and model. If default_provider (or default_anthropic_provider) is set, a model that matches no route is forwarded unchanged to that provider. Quickstart turns this passthrough on; explicit configs leave it off unless you set it. Named routes always win. Design: ADR 0007.

Cost is computed only when the provider reports usage and a verified price is on file for that provider and model. Otherwise pricing and estimated_cost_usd are null, never a guessed 0. The built-in price catalog applies only to the providers’ own default endpoints; for a custom base_url, add prices under pricing: in inferrail.yaml.

Framework configuration

These are configuration examples. CI tests the OpenAI wire protocol these frameworks use (test_agent_e2e.py), not the frameworks themselves.

# LangChain
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="not-needed",  # or your INFERRAIL_GATEWAY_TOKEN if auth is enabled
    model="default",
)
# LlamaIndex
from llama_index.llms.openai_like import OpenAILike

llm = OpenAILike(
    model="default",
    api_base="http://127.0.0.1:8000/v1",
    api_key="not-needed",
    is_chat_model=True,
    context_window=8192,
)
# CrewAI
from crewai import LLM

llm = LLM(
    model="openai/default",  # "openai/" prefix required by CrewAI
    base_url="http://127.0.0.1:8000/v1",
    api_key="not-needed",
)

Agents and apps

Config-only setups for tools that make many model calls per task. Each gives one run (or session) an id and a dollar ceiling; `inferrail work

` and `inferrail report --by work_id` show what it cost. All were run against a stand-in OpenAI upstream with the versions noted (no provider spend); "on refusal" is what the tool does when the gateway returns 402. ### goose (per session) A custom provider can send goose's session id under any header name: ```json { "name": "inferrail", "engine": "openai", "display_name": "OpenAI via Inferrail", "api_key_env": "", "requires_auth": false, "base_url": "http://127.0.0.1:8000/v1", "models": [{"name": "gpt-4o-mini", "context_limit": 128000}], "headers": {"X-Inferrail-Budget-Usd": "2.00"}, "session_id_header_override": "X-Inferrail-Attribute-Work-Id" } ``` Save as `~/.config/goose/custom_providers/inferrail.json` and select the `inferrail` provider. goose v1.53.0. On refusal: stops with its generic "add more credits" message, no retry loop. ### ZeroClaw (one ceiling across all your instances, Inferrail Hosted) ZeroClaw's cost tracker is per process, so several ZeroClaw instances (on one machine or many) each spend up to their own limit. To give all of them one ceiling, point a `custom` provider at [Inferrail Hosted](../hosted/budget_gateway/README.md) and send a shared run id and ceiling with `extra_headers`: ```toml schema_version = 3 [providers.models.custom.inferrail] uri = "https://inferrail-cost-gateway.onrender.com/hosted/v1" model = "gpt-4o-mini" api_key = "irw_..." # Inferrail Hosted workspace key: POST /hosted/v1/workspaces [providers.models.custom.inferrail.extra_headers] X-Provider-Api-Key = "sk-..." # your OpenAI key, sent per request, never stored X-Inferrail-Attribute-Work-Id = "zeroclaw-2026-10" # the same id on every instance = one shared budget X-Inferrail-Budget-Usd = "5.00" # the ceiling for that id, across all instances [agents.assistant] model_provider = "custom.inferrail" risk_profile = "default" [risk_profiles.default] ``` Every call reserves its worst-case cost against the shared ceiling before it goes to OpenAI, so concurrent instances can't overshoot it together. Use a new work id (for example per month) to start a new budget; a work id keeps the first ceiling it was given. `GET /hosted/v1/work/zeroclaw-2026-10` (with the workspace key) shows the calls and their cost. All instances on one work id count as one governed run. ZeroClaw v0.8.5 (release binary), two separate instances with their own config directories, against Inferrail Hosted with a stand-in OpenAI upstream reporting usage sized from each request: with a $0.01 ceiling, 2 calls were answered ($0.004832; ZeroClaw's default prompt is about 16K tokens per turn), then 14 calls from both instances were refused before reaching the provider. On refusal: ZeroClaw prints "The selected model provider rejected the request" and doesn't retry; the reason is in `GET /hosted/v1/work/`. ZeroClaw's startup `GET /models` probe gets a 404, which it ignores. ### Hermes Agent (per conversation) A custom provider can send Hermes' conversation id under any header name (`session_affinity_header`): ```yaml model: default: gpt-4o-mini provider: capped providers: capped: base_url: http://127.0.0.1:8000/v1 key_env: INFERRAIL_PLACEHOLDER_KEY extra_headers: X-Inferrail-Budget-Usd: "2.00" session_affinity_header: X-Inferrail-Attribute-Work-Id ``` Hermes v0.21.5 (2026.9.24, Docker image). On refusal: `Billing or credits exhausted: HTTP 402: budget ...`, one request, no retry loop. ### Qwen Code (per session) ```json { "modelProviders": { "openai": [{ "id": "gpt-4o-mini", "envKey": "INFERRAIL_PLACEHOLDER_KEY", "baseUrl": "http://127.0.0.1:8000/v1", "generationConfig": { "customHeaders": { "X-Inferrail-Attribute-Work-Id": "qwen-${session_id}", "X-Inferrail-Budget-Usd": "2.00" }, "samplingParams": {"max_tokens": 4000} } }] }, "outboundCorrelation": {"allowDynamicHeaderValues": true} } ``` In `~/.qwen/settings.json`, with `INFERRAIL_PLACEHOLDER_KEY=unused` exported. Qwen Code 0.25.0. On refusal: `[API Error: 402 budget ...]`, exit 1, one request. ### Crush (per run) `extra_headers` expands environment variables and drops a header that expands to empty, so a run is capped only when you name it: ```json { "providers": { "capped": { "type": "openai-compat", "base_url": "http://127.0.0.1:8000/v1", "api_key": "unused", "extra_headers": { "X-Inferrail-Attribute-Work-Id": "${INFERRAIL_WORK_ID}", "X-Inferrail-Budget-Usd": "${INFERRAIL_WORK_ID:+${INFERRAIL_BUDGET_USD:-2.00}}" }, "models": [{"id": "gpt-4o-mini", "name": "gpt-4o-mini", "context_window": 128000, "default_max_tokens": 4000}] } } } ``` `INFERRAIL_WORK_ID=ticket-4411 crush run "..."`. crush v0.97.1. On refusal: `payment required: budget ...`, exit 1, no retries. Per interactive session: `--work-id-header x-session-id` on the gateway (crush sends a hashed session id) with a static `X-Inferrail-Budget-Usd`. ### OpenCode (per day) Provider headers are static, so cap a project per day: ```json { "provider": { "capped": { "npm": "@ai-sdk/openai-compatible", "options": { "baseURL": "http://127.0.0.1:8000/v1", "apiKey": "unused", "headers": {"X-Inferrail-Attribute-Project": "opencode"} }, "models": {"gpt-4o-mini": {"name": "gpt-4o-mini"}} } }, "model": "capped/gpt-4o-mini" } ``` ```bash inferrail budget set --scope project --scope-value opencode --window daily --mode block --limit-usd 5 ``` opencode 1.18.34. On refusal: prints the budget error and exits, two requests, no retry loop. Per session instead: start the gateway with `--work-id-header x-opencode-session-id` (opencode sends it on every request) and put only `"X-Inferrail-Budget-Usd": "3"` in `options.headers`. ### PR-Agent (per review run, GitHub Actions) Run the gateway and PR-Agent's CLI in the same job: ```yaml - run: pip install pr-agent inferrail - env: {OPENAI_API_KEY: "$"} run: | nohup inferrail serve --quickstart --app-mode --no-telemetry > gateway.log 2>&1 & curl -s --retry 20 --retry-connrefused --retry-delay 1 http://127.0.0.1:8000/health - env: GITHUB__USER_TOKEN: $ OPENAI__KEY: unused OPENAI__API_BASE: http://127.0.0.1:8000/v1 CONFIG__MODEL: gpt-5 LITELLM__EXTRA_HEADERS: '{"X-Inferrail-Attribute-Work-Id": "pr-$", "X-Inferrail-Budget-Usd": "0.50"}' run: python -m pr_agent.cli --pr_url "$" review ``` pr-agent 0.47.0. On refusal: two attempts, then `Failed to review PR`. ### crawl4ai, ScrapeGraphAI, Prefect, Atomic Agents (per job) ```python # crawl4ai: one id per crawl job LLMExtractionStrategy( llm_config=LLMConfig(provider="openai/gpt-4o-mini", api_token="unused", base_url="http://127.0.0.1:8000/v1"), extra_args={"max_tokens": 800, "extra_headers": { "X-Inferrail-Attribute-Work-Id": job_id, "X-Inferrail-Budget-Usd": "2.00"}}, instruction="...", ) # ScrapeGraphAI: pass a ChatOpenAI as model_instance config = {"llm": {"model_instance": ChatOpenAI( model="gpt-4o-mini", base_url="http://127.0.0.1:8000/v1", api_key="unused", max_tokens=800, default_headers={"X-Inferrail-Attribute-Work-Id": job_id, "X-Inferrail-Budget-Usd": "1.50"}), "model_tokens": 128000}} # Prefect: the flow run id is the work id OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="unused", default_headers={ "X-Inferrail-Attribute-Work-Id": f"prefect-{prefect.runtime.flow_run.id}", "X-Inferrail-Budget-Usd": "5.00"}) # Atomic Agents: one instructor client per request, shared by every agent instructor.from_openai(OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="unused", default_headers={"X-Inferrail-Attribute-Work-Id": request_id, "X-Inferrail-Budget-Usd": "0.25"})) ``` crawl4ai 0.9.4, scrapegraphai 2.3.0, prefect 3.8.7, atomic-agents 2.10.3. **Prefect tasks in several processes or on several workers (Inferrail Hosted).** A local gateway on each worker would give each one its own ledger, so a flow run fanned out across processes could spend the ceiling once per process. Point every task at [Inferrail Hosted](../hosted/budget_gateway/README.md) instead, so the flow run has one ceiling wherever its tasks run: ```python client = OpenAI( base_url="https://inferrail-cost-gateway.onrender.com/hosted/v1", api_key=workspace_key, # POST https://inferrail-cost-gateway.onrender.com/hosted/v1/workspaces default_headers={ "X-Provider-Api-Key": openai_key, # sent per request, never stored "X-Inferrail-Attribute-Work-Id": f"prefect-{prefect.runtime.flow_run.id}", "X-Inferrail-Budget-Usd": "5.00", }, ) ``` Prefect 3.8.8, one flow run with `ProcessPoolTaskRunner(max_workers=4)`, four tasks in four processes making 10 calls each, a $0.002 ceiling, and a stand-in for OpenAI that reports usage sized from each request: with a separate ceiling per task, the run spent $0.007164 (3.6x); with the one shared work id, 9 calls were answered and 31 refused before the provider, and the run spent $0.001791. **Retries:** a refused call can't pass on retry, but some retry layers resend it anyway (free, since refusals never reach the provider, but wasted). Prefect `@task(retries=...)` and instructor's default `max_retries` both do; skip 402 with a Prefect `retry_condition_fn`, and pass instructor `max_retries=Retrying(stop=stop_after_attempt(3), retry=retry_if_exception_type(ValidationError))`. ## Voice agents Inferrail has no native voice support. It does not handle audio, speech-to-text, text-to-speech, the OpenAI Realtime API, or WebSocket sessions, and it does not account for full call cost. A voice stack can still route its **text LLM stage** through Inferrail if that stage lets you set an OpenAI- or Anthropic-compatible base URL and sends a supported request shape (text messages, optionally streaming or tool calls). Only that stage's tokens and cost appear in receipts. No voice framework integration (LiveKit, Pipecat, Vapi, Retell, or others) has been tested by this project. Treat compatibility as something to verify in your own stack: send one request, then check that a receipt appears in `inferrail report`. ## Attribution Three ways to attach business context, all landing in the same `attributes: dict[str, str]` on the receipt: - **HTTP header** (gateway): `X-Inferrail-Attribute-: `, for example `X-Inferrail-Attribute-Task-Id: bug_9281`. These headers are never forwarded to the provider ([attribution.py](../src/inferrail/gateway/attribution.py)). - **CLI flag** (`inferrail try`): `--customer`, `--workflow`, or generic `-a =`. - **Ambient, for nested agent calls**: `inferrail.track_task` attaches `X-Inferrail-Attribute-Task-Id` to every request inside a `with` block or decorated function. ```python import inferrail from openai import OpenAI client = OpenAI( base_url="http://127.0.0.1:8000/v1", api_key="not-needed", # base_url must match the client's own base_url: the header is only # attached to requests going to that destination. http_client=inferrail.attributed_http_client(base_url="http://127.0.0.1:8000/v1"), ) @inferrail.track_task(task_id="bug_9281") def fix_bug(): client.chat.completions.create(...) # tagged automatically run_subagent() # nested calls too ``` Sync and async are both supported (`attributed_async_http_client`), and concurrent tasks never cross-contaminate. See [ADR 0009](/adr/0009-ambient-task-tracking.html). **Attribute values are stored exactly as sent.** Use identifiers, not names, emails, secrets, or message text. Then slice spend by any dimension: ```bash inferrail report --by customer inferrail report --by workflow inferrail report --by provider # also: model, route, or any attribute name ``` ## Work and outcomes Give related requests the same `work_id`, then declare an outcome when your application knows one: ```bash inferrail try "Review this contract clause" --model -a work_id=contract_review_42 inferrail try "Identify remaining risks" --model -a work_id=contract_review_42 inferrail work outcome contract_review_42 --status completed inferrail work contract_review_42 inferrail work --all ``` `inferrail try` sends one real, billed request with `OPENAI_API_KEY` to the model you name (`inferrail models` lists yours). Over HTTP, use `X-Inferrail-Attribute-Work-Id: contract_review_42`. Work Economics reports the **known** attributed inference cost for that work. If any receipt in the work has unknown cost, the report counts it separately rather than folding it into the total, so a known subtotal is not a complete bill. It is not COGS, margin, or business value, and Inferrail does not interpret your outcome labels. `inferrail transaction ` gives the older receipt-only grouping by `task_id` ([ADR 0008](/adr/0008-task-transactions.html)). ## MCP `inferrail mcp` is a stdio MCP server with two read-only tools over your local receipts file: | Tool | What it answers | |---|---| | `get_spend` | Known cost, tokens, and request counts grouped by `provider`, `model`, `route`, or any attribute you tag requests with (`customer`, `workflow`, `work_id`), optionally within a time window. Requests with unknown pricing are counted separately, not as `$0`. | | `get_health` | Whether the gateway answers `GET /health`, plus the most recent receipt. | Neither runs inference, spends provider budget, changes configuration, or writes files. Receipts store usage and cost metadata without persisting prompt or response bodies, so the tools have none to return. Grouping by `customer`, `workflow`, or `work_id` only covers requests that were sent with that tag ([Attribution](#attribution)). Claude Code: ```bash claude mcp add inferrail -e INFERRAIL_RECEIPTS_PATH=/absolute/path/to/inferrail-receipts.jsonl -- uvx --with "mcp>=2.0" inferrail mcp ``` Claude Desktop, Cursor, and other clients that use `mcpServers` (VS Code uses the same entry under `servers`): ```json { "mcpServers": { "inferrail": { "command": "uvx", "args": ["inferrail", "mcp"], "env": { "INFERRAIL_RECEIPTS_PATH": "/absolute/path/to/inferrail-receipts.jsonl" } } } } ``` Without `uvx`: `pip install inferrail`, then use `inferrail mcp` as the command. Set `INFERRAIL_RECEIPTS_PATH` to your receipts file. Clients start the server from their own working directory, so the default `./inferrail-receipts.jsonl` is rarely the right place. For `serve --app-mode`, point it at `receipts.db` in Inferrail's data directory (`~/.local/share/inferrail` on Linux, `~/Library/Application Support/inferrail` on macOS, `%APPDATA%\inferrail` on Windows). Then ask, for example: *"How much did work contract-review-42 cost?"* If your requests carried `work_id=contract-review-42`, the agent calls `get_spend` with `by: "work_id"` and reads that group. Full tool contract: [inferrail-mcp/README.md](../inferrail-mcp/README.md).