Product Requirements Document: Inference API
Status: Draft v0.1
Owner: Product
Last updated: 2026-04-10
1. Summary
An OpenAI-compatible, multi-tenant inference API serving open-weight LLMs and embedding models on our owned-and-operated GPU fleet. We sell inference-as-a-service to teams that want OpenAI-grade DX without sending data to a hyperscaler, without managing vLLM clusters, and with predictable pricing on dedicated capacity when they want it.
2. Target customers and positioning
Who buys this
- Regulated and data-sensitive teams (fintech, healthtech, legal, government contractors, EU companies with GDPR exposure) who can’t or won’t ship prompts to OpenAI or to US hyperscaler regions. We offer a contractual data-residency story because we own the racks and know which building each request lands in.
- Latency-sensitive product teams in our geographic region (voice agents, IDE autocomplete, real-time translation, gaming NPCs) where 50-100ms of network RTT matters and a regional POP beats us-east-1.
- Cost-conscious scale-ups burning $50k-$500k/month on OpenAI/Anthropic who want to migrate the cheap-and-cheerful 80% of their traffic to open weights without hiring a platform team to run vLLM themselves.
- Teams that need dedicated capacity guarantees — agencies, batch pipelines, eval harnesses — that get rate-limited or priced unpredictably on shared serverless inference platforms.
Why us over Together / Fireworks / Anyscale / DIY vLLM
| Competitor |
Their edge |
Our wedge |
| Together / Fireworks |
Huge model menu, serverless |
We offer contractual dedicated capacity on owned hardware with regional data residency. They’re hyperscale-on-rented-cloud; we’re regional-on-owned-metal. |
| Anyscale / Modal |
Ray ergonomics, serverless |
We’re a product, not a platform — drop-in OpenAI SDK, no Ray to learn. |
| Self-managed vLLM on cloud GPUs |
Full control |
We own the GPUs so our $/token is structurally lower than anyone renting H100s by the hour. Customer doesn’t run kubectl. |
| OpenAI / Anthropic |
Frontier quality |
Open weights are good enough for ~70% of production workloads, at ~10-20% of the cost, and the data never leaves our region. |
Our durable edge
- We own the metal. Our marginal cost of a token is electricity + amortization, not an AWS markup. We can profitably price 30-50% under Together/Fireworks on the models we choose to host, especially for reserved capacity.
- We are local/regional. A POP in our region is a hard moat against us-east-1 latency for any customer in that region.
- We can sell dedicated nodes. Hyperscalers won’t carve out a single H100 node for a $20k/month customer. We will.
3. API surface
v1 endpoints (OpenAI-compatible)
POST /v1/chat/completions — streaming and non-streaming, tool/function calling, JSON mode (via grammar-constrained decoding)POST /v1/completions — legacy text completion (kept because batch/eval pipelines still use it)POST /v1/embeddingsGET /v1/models — lists what this tenant can callGET /v1/models/{id} — model metadata (context length, modality, pricing tier, dedicated/shared)
v1 native extensions
POST /v1/batch — submit a JSONL file of requests, get an async job back. Priced ~50% of sync. Mirrors OpenAI Batch API shape so SDKs work.GET /v1/usage — per-API-key token counts, request counts, $ spend, broken down by model and day.- Response metadata: every response includes
x-provocapi-node-id, x-provocapi-model-version, x-provocapi-queue-ms, x-provocapi-ttft-ms headers for customer-side observability.
- LoRA adapter routing: customer passes
model: "llama-3.1-70b-instruct:my-adapter-v3" and we route to a vLLM instance with that adapter loaded.
Explicit non-goals for v1
- No fine-tuning API (deferred to v2; customers BYO weights instead)
- No image generation, audio (TTS/STT), or vision models (deferred; we’ll add Llama 3.2 Vision and Whisper in v1.5)
- No Assistants API, threads, file storage, or vector store (we are inference, not a RAG product)
- No moderation endpoint
- No realtime/WebRTC API
4. Model catalog strategy
Curation principle
Start tight: 8-12 hand-picked models at launch, not 50. Every model on the menu must justify the VRAM it occupies. Add models when (a) a customer commits revenue or (b) a new release demonstrably beats an incumbent on a benchmark we care about.
Launch catalog (proposed)
| Model |
Size |
GPU class |
Use case |
| Llama 3.1 8B Instruct |
8B |
RTX 5090 / RTX PRO 6000 |
Cheap chat, classification, routing |
| Llama 3.1 70B Instruct |
70B |
2x H100 80GB or 2x RTX PRO 6000 96GB |
Flagship general-purpose |
| Llama 3.1 405B Instruct |
405B |
8x H100 80GB |
Premium tier, reserved only |
| Qwen 2.5 72B Instruct |
72B |
2x H100 / 2x RTX PRO 6000 |
Strong multilingual, coding |
| Qwen 2.5 Coder 32B |
32B |
1x H100 / 1x RTX PRO 6000 |
Code completion, IDE plugins |
| Mistral Small 3 24B |
24B |
1x RTX PRO 6000 / 1x A100 80GB |
Mid-tier chat |
| DeepSeek-V3 (or successor) |
671B MoE |
8x H100 |
Reasoning, premium |
| BGE-M3 |
560M |
RTX 5090 |
Multilingual embeddings |
| E5-Mistral-7B-Instruct |
7B |
RTX 5090 |
High-quality embeddings |
| Nomic Embed v1.5 |
137M |
RTX 5090 |
Cheap embeddings |
ASSUMPTION TO VALIDATE: Llama 3.1 70B in FP16 is ~140GB VRAM; fits on 2x H100 80GB or 2x RTX PRO 6000 96GB with tensor parallelism. If your H100 nodes are 4x or 8x configurations, we get better throughput per node by stacking multiple TP=2 replicas. Confirm node SKUs.
How customers request models
- Public catalog: any tenant can call any listed model, billed per-token at posted rates.
- Private models: tenants on dedicated tier can request a model not in the public catalog; we evaluate and load it on their dedicated capacity.
BYOM (bring-your-own-model)
- v1: limited BYOM. Dedicated-tier customers can ship us HuggingFace repo URLs for permissively-licensed models in supported architectures (Llama, Mistral, Qwen, Gemma families). We pull, validate, and deploy on their dedicated nodes. SLA: 1 business day.
- v1: LoRA adapters. Any customer (shared or dedicated) on a supported base model can upload LoRA adapters via
POST /v1/adapters (HF repo or direct upload, max 500MB). Adapter is hot-loaded into the vLLM instance serving that base model.
- Not in v1: custom architectures, quantization-on-the-fly, S3-pull from customer buckets. Deferred to v1.5.
5. Tenant model
Auth
- API keys, prefix
pk-prov- (live) and pk-prov-test- (sandbox).
- Keys are hashed at rest (argon2id), prefix shown in dashboard for identification.
- Rotation: customers can issue/revoke unlimited keys per workspace.
- No OAuth in v1 (deferred). No per-user keys; workspace-level only.
Tiers
| Tier |
Capacity |
Rate limits |
Pricing |
Isolation |
| Shared |
Pooled vLLM instances |
200 RPM / 500k TPM default, raisable |
Per-token, public rates |
Logical only (per-key queues, fair scheduling) |
| Reserved |
Customer reserves N replicas of model X for M months |
None (your replicas) |
Flat $/replica/month + electricity passthrough |
Dedicated vLLM processes; customer can have noisy LoRA load patterns without affecting others |
| Dedicated nodes |
Customer leases entire physical nodes |
None |
Flat $/node/month |
Physical isolation; customer namespace, customer-only k8s taints |
Usage metering
- Every request emits a usage event (input tokens, output tokens, model, tenant, latency, node) into a Kafka/Redpanda topic → ClickHouse.
- Dashboard shows near-realtime (60s lag) usage; billing is end-of-day reconciled.
6. SLA and reliability
Targets (v1, conservative)
| Metric |
Shared tier |
Reserved tier |
| Monthly uptime |
99.5% |
99.9% |
| TTFT p50 (chat, 70B model, ~500 input tokens) |
< 400ms |
< 250ms |
| TTFT p95 |
< 1200ms |
< 600ms |
| TTFT p99 |
< 2500ms |
< 1500ms |
| Output TPS p50 (70B, single stream) |
> 35 tok/s |
> 45 tok/s |
| Embeddings p95 latency (batch of 32) |
< 300ms |
< 200ms |
ASSUMPTION TO VALIDATE: TPS targets depend on H100 vs. RTX PRO 6000 (FP8 perf differs significantly). Validate against benchmark runs on actual hardware before publishing.
Failure handling
- Node failure: in-flight requests fail; customer SDK retries; load balancer ejects node within 10s. Affected models reload on healthy nodes within 90s (warm pool keeps weights cached on local NVMe).
- Model reload: zero-downtime for any model with replica count ≥ 2. Single-replica models (rare, premium-tier only) get a maintenance window.
- Capacity: we maintain 25% headroom on shared pool; alert at 70% sustained utilization, scale at 80%.
7. Pricing
Shared (per-token, USD per 1M tokens)
| Model |
Input |
Output |
| Llama 3.1 8B |
$0.10 |
$0.15 |
| Mistral Small 3 24B |
$0.20 |
$0.30 |
| Qwen 2.5 Coder 32B |
$0.30 |
$0.45 |
| Llama 3.1 70B / Qwen 2.5 72B |
$0.55 |
$0.75 |
| DeepSeek V3 |
$0.50 |
$1.20 |
| Llama 3.1 405B |
$2.50 |
$3.50 |
| BGE-M3 / E5-Mistral / Nomic |
$0.02 - $0.05 per 1M tokens |
— |
Reference: Together charges ~$0.88 in / $0.88 out for Llama 70B; Fireworks ~$0.90. We undercut by 30-40% on the models we choose to host. Hyperscaler equivalents (Bedrock Llama 70B) are ~$2.65/$3.50.
Reserved
- $/replica/month, contracted in 1/3/12 month terms with 20%/35%/50% commit discounts.
- Example: 1x Llama 70B reserved replica on 2x H100 = $4,500/month flat, no per-token charge. Breakeven vs. shared at ~6B input + 6B output tokens/month.
Batch
- 50% of sync per-token rates. SLA: 24h completion.
8. Developer experience
- Drop-in OpenAI SDK compatibility.
client = OpenAI(base_url="https://api.provocapi.com/v1", api_key=...) works in Python, JS, Go SDKs without modification.
- Docs site (Mintlify or Docusaurus): quickstart, migration-from-OpenAI guide, model cards, API reference auto-generated from OpenAPI spec.
- Dashboard (Next.js): API key management, usage charts, billing, request log explorer (last 7 days, redacted prompts unless customer opts in), model catalog, LoRA adapter upload.
- Playground: chat UI with model picker, parameter sliders, copy-as-curl/Python.
- Status page (statuspage.io or Vercel-hosted): per-model uptime + latency.
9. v1 scope
Ships in v1
- 10-12 model catalog (above)
- Chat completions, completions, embeddings, models, batch, usage endpoints
- Streaming (SSE)
- API key auth, shared + reserved tiers
- Per-tenant rate limits and metering
- LoRA adapter upload + serving
- Dashboard with usage, keys, billing
- Playground
- Docs + status page
- Single region (our primary DC)
Deferred (v1.5 / v2)
- Vision models (Llama 3.2 Vision, Qwen2-VL)
- Whisper / TTS
- Fine-tuning API
- Multi-region routing
- Full BYOM with custom architectures
- Customer S3-pull for model weights
- OAuth / SSO / per-user keys
- Assistants-style stateful APIs
- Speculative decoding configurability per request
- SOC 2 (start the audit at v1 launch, achieve at v1.5)
Explicit non-goals (probably never)
- Image generation (commodity, race-to-zero, not our market)
- Frontier model training (we are not in the training business)
- A vector database (partner instead)