GPU routing demo
Send every request to theright GPU, automatically.
Cut inference spend without trading away latency or uptime. Run the router below, change its policy, break things on purpose and watch it recover.
Starting the simulation…
Savings estimator
Three numbers about your setup give a cautious range. The method and every assumption are shown below the result.
Estimated monthly saving
$3,400 to $8,500
About 8% to 21% of spend, or $40,800 to $102,000 a year.
Estimate, not a guarantee. Real savings depend on your traffic and contracts.
How this is worked out
saving = spend × (batch share × batch rate + interactive share × interactive rate) × setup factor
- Batch rate 15% to 35%: batch work can move to spot and cheaper GPUs and wait for spare capacity.
- Interactive rate 4% to 12%: chat must stay fast, so there is less room to move it.
- One cloud or API provider: factor 0.8. Fewer pools to choose between, mostly regions and GPU types.
- Two or more providers: factor 1. The most routing room: prices and capacity differ between providers.
- Self-hosted GPUs: factor 0.6. Capacity is already paid for; savings come mainly from better utilisation and spill-over.
The low rates give the bottom of the range and the high rates the top, each rounded to two significant figures. These are cautious assumptions, not measured results.
How it is built
- 01
Ingress
Requests arrive through one endpoint, are authenticated and tagged with their tenant and request class.
- 02
Classifier
Each request is classed (interactive, long-context, batch) and its size estimated from the prompt.
- 03
Scorer
Every pool is scored on live price, queue depth, measured latency, health and model fit, as in the demo.
- 04
Policy
Per-class rules apply on top: latency targets, budget caps, spot allowed or not, data residency.
- 05
Dispatcher
The request goes to the best pool, with a fallback chain if that pool errors or times out.
- Scoring inputs
- Price per GPU-hour, queue depth, measured p50/p95 per pool, health-check state, GPU memory and model fit, and region round-trip time.
- Fallback
- Health checks take a pool out of rotation; in-flight work is retried elsewhere; recovering pools ramp back in gradually.
- Observability
- Every decision is logged with its inputs and a plain-language reason, with metrics and alerts for spend, latency and errors.
- Deployment
- Runs in your cloud account, on your own hardware, or across both. Your keys and contracts stay yours.
Who it is for
AI product teams
Chat and agent products where latency matters and traffic is spiky.
Inference-heavy startups
Growing spend across more than one provider, GPU type or region.
Data and ML platforms
Shared GPU capacity serving many internal teams with different needs.
Proof goes here: a named deployment, a measured before and after, or a quote used with permission. Added in content/placeholders.ts.
See it with your own traffic.
We will walk through how routing would work on your workloads, and what it would take to build.