provocapi

Technical Requirements Document: Inference API

Status: Draft v0.1 Owner: Engineering Last updated: 2026-04-10 Companion to: PRD.md

1. Inference engine: vLLM

Decision

vLLM is the v1 engine. We standardize on it for every model in the launch catalog.

Why vLLM over the alternatives

What we give up vs. TensorRT-LLM

We accept this for v1. Operational simplicity and LoRA support outweigh the throughput delta at our scale.

Engine abstraction layer

We wrap vLLM behind an internal InferenceWorker interface so we can swap engines per-model later (e.g., move Llama 405B to TensorRT-LLM if its throughput justifies the operational tax). The interface:

InferenceWorker:
  - load(model_id, dtype, tp_size, lora_modules) -> WorkerHandle
  - generate(request) -> AsyncIterator[Token]
  - embed(request) -> Vector
  - unload()
  - health() -> Status
  - metrics() -> {gpu_util, kv_cache_used, queue_depth, ...}

The router (§2) only talks to this interface. v1 has one implementation: VLLMWorker. v1.5 may add TRTLLMWorker for high-volume models.

2. Architecture

Diagram

flowchart TB
    Client[Customer App<br/>OpenAI SDK] -->|HTTPS| LB[Envoy<br/>Edge LB + TLS]
    LB --> GW[API Gateway<br/>FastAPI<br/>auth · rate limit · validation]
    GW --> Router[Model Router<br/>Go service<br/>tenant queue · model dispatch]
    GW --> Meter[(Usage Events<br/>Redpanda)]
    Router --> Registry[(Model Registry<br/>Postgres)]
    Router --> WP1[vLLM Worker Pool<br/>Llama 70B<br/>2x H100 · TP=2]
    Router --> WP2[vLLM Worker Pool<br/>Llama 8B<br/>1x RTX 5090]
    Router --> WP3[vLLM Worker Pool<br/>BGE-M3 Embeddings<br/>1x RTX 5090]
    Router --> WPN[vLLM Worker Pool<br/>...]
    Meter --> CH[(ClickHouse<br/>usage + billing)]
    WP1 -.metrics.-> Prom[Prometheus]
    WP2 -.metrics.-> Prom
    WP3 -.metrics.-> Prom
    Prom --> Graf[Grafana + Alertmanager]
    GW -.auth lookup.-> KV[(Redis<br/>API key cache<br/>rate limit counters)]

    subgraph K8s[Kubernetes Cluster - bare metal]
      LB
      GW
      Router
      WP1
      WP2
      WP3
      WPN
    end

Layer-by-layer

2.1 Edge: Envoy

2.2 API Gateway: FastAPI (Python)

A stateless Python service that:

  1. Validates request schema (Pydantic models matching OpenAI’s OpenAPI spec).
  2. Resolves API key → tenant via Redis lookup (Postgres on miss).
  3. Enforces per-tenant rate limits (token bucket in Redis, redis-cell module).
  4. Tags request with tenant_id, model_id, request_id, priority, and forwards to the Model Router over HTTP/2 keep-alive.
  5. Streams the response back as SSE without buffering (FastAPI StreamingResponse).
  6. Emits a usage event to Redpanda on stream completion (or failure).

Why Python here despite the latency tax: the gateway is I/O-bound, and Python lets us share Pydantic schemas with the public OpenAPI spec and SDK validation. Per-request overhead is ~3-5ms which is rounding error vs. TTFT.

2.3 Model Router: Go service

The router is the only component that knows the live state of the worker pool. It:

  1. Maintains an in-memory map of model_id → [worker_endpoints] updated from Kubernetes API watch on labeled vLLM pods.
  2. For each incoming request, picks a worker using least-outstanding-requests with vLLM’s num_running_requests metric scraped every 1s via the /metrics endpoint. Falls back to round-robin if metrics are stale.
  3. Maintains per-tenant queues in front of each worker pool to enforce fair scheduling under contention (token-bucket-weighted by tier — reserved tenants get priority lanes).
  4. Handles SSE proxying with backpressure-aware buffering.
  5. Handles LoRA adapter routing: requests for model:adapter are pinned to workers that have that adapter loaded; if none do, the router calls vLLM’s /v1/load_lora_adapter to hot-load.

Why Go here: the router is the hot path for every token streamed. Go’s goroutine model handles thousands of concurrent SSE connections per pod with low memory overhead, and avoids Python GIL contention.

2.4 Worker pool

Each vLLM replica runs in its own pod with vllm serve exposing the OpenAI-compatible HTTP server on a ClusterIP. Pods are labeled model=<id>, version=<sha>, tier=shared|reserved|dedicated, tenant=<id|shared>. The router watches pod events and updates routing tables in real time.

Multi-GPU (tensor parallel) models: a single pod requests N GPUs from the device plugin and runs vllm serve --tensor-parallel-size N. Pods requesting >1 GPU must land on a node with N matching GPUs (enforced by node affinity rules — see §3).

Pipeline parallelism across nodes: not in v1. We avoid multi-node TP because (a) it requires NVLink/InfiniBand we may not have everywhere and (b) it’s an operational nightmare. The largest model that fits on one node sets our ceiling. For Llama 405B in FP8 (~410GB), we need an 8x H100 node.

ASSUMPTION TO VALIDATE: Do you have any 8x H100 nodes? If not, 405B serving requires either FP4 quantization (quality risk) or multi-node TP over InfiniBand. Confirm node configurations and interconnect fabric.

3. Node management and fleet partitioning

Cluster

Fleet partitioning

The cluster spans nodes used for this product and nodes used for other workloads. We isolate via:

  1. Node labels:
    • provocapi.io/pool=inference-api — opt-in to this product.
    • provocapi.io/gpu-class=h100|a100-80|a100-40|rtx-pro-6000|rtx-5090
    • provocapi.io/gpu-count=1|2|4|8
    • provocapi.io/tenant=shared or provocapi.io/tenant=<tenant_id> for dedicated nodes
  2. Taints: every inference-api node carries provocapi.io/pool=inference-api:NoSchedule. Only our workloads tolerate it. This prevents accidental scheduling of other workloads onto our nodes — and prevents our workloads from drifting onto nodes outside the pool.
  3. Namespace: all inference-api workloads live in the inference-api namespace with a ResourceQuota and LimitRange. NetworkPolicies restrict ingress to Envoy and the management plane only.
  4. Dedicated tenants get an additional taint provocapi.io/tenant=<id>:NoSchedule and node affinity rules on their pods.

Node lifecycle

ASSUMPTION TO VALIDATE: I’m assuming each node has ≥2TB local NVMe for model weight caching. Llama 405B FP8 is ~410GB on disk, FP16 is ~810GB. Confirm NVMe per node and whether we need a network filesystem (Ceph, JuiceFS) for the weight cache instead.

4. Model loading and lifecycle

Storage hierarchy

  1. Origin: internal model registry — an S3-compatible object store (MinIO deployed in-cluster on dedicated storage nodes, or a managed S3 if we have one). We mirror HuggingFace Hub there on model intake to (a) avoid HF rate limits, (b) pin exact revisions, (c) survive HF outages.
  2. Node-local cache: /var/lib/provocapi/models/<model_id>/<revision>/ on local NVMe. Managed by a model-warmer DaemonSet that pulls weights when a vLLM pod requests them (init container) and runs an LRU eviction when free space drops below 20%.
  3. GPU VRAM: vLLM loads from the node-local cache at pod start.

Model intake pipeline

When we add a model to the catalog:

  1. Engineer creates a Model CR (custom resource) with repo, revision, dtype, tp_size, gpu_class_allowed, serving_args.
  2. A controller pulls weights from HuggingFace → MinIO (with checksumming).
  3. Smoke test job runs vLLM against the weights on a test node (correctness benchmark + latency baseline).
  4. On pass, a Helm release deploys the production worker pool.
  5. Router picks up the new pods via watch.

Warm vs. cold loading

Versioning

5. Multi-tenancy and isolation

Noisy-neighbor controls

  1. Per-tenant queues at the router (§2.3) with weighted-fair scheduling. Tier weights: dedicated=∞ (own pool), reserved=10x, shared=1x.
  2. vLLM admission control: each worker has --max-num-seqs capped to keep KV cache utilization <85%. The router stops sending requests when a worker hits its admission ceiling rather than queueing inside vLLM (which gives us less observability).
  3. Per-request token limits enforced at the gateway based on tenant tier (shared: 8k output max default; reserved: model max).
  4. Output token rate limits per key to prevent runaway agents from one customer eating shared capacity.

Isolation guarantees by tier

Tier Process GPU Node Network
Shared Shared vLLM workers Shared GPUs Shared nodes Shared namespace
Reserved Dedicated vLLM workers (own pods) Shared GPUs OK Shared nodes OK Shared namespace
Dedicated Dedicated vLLM workers Dedicated GPUs Dedicated nodes (taints) Dedicated namespace + NetworkPolicy

We do not offer cryptographic memory isolation (confidential computing) in v1. If a customer needs that, they need dedicated nodes. Document this clearly.

6. LoRA adapter serving

7. Observability

Metrics (Prometheus)

The metrics that actually matter:

Logging

Tracing

Alerts (Alertmanager)

Pageable:

Non-pageable (Slack):

8. Security

9. Deployment and operations

10. Capacity planning

Model → GPU class fitment table

All numbers FP16 unless noted. KV cache budget assumes 8k context and target concurrency. Validate against benchmarks on your actual hardware.

Model Weights KV cache budget Min config Preferred config
BGE-M3 / Nomic Embed <2GB <2GB 1x RTX 5090 32GB 1x RTX 5090 (many replicas)
E5-Mistral 7B ~14GB ~6GB 1x RTX 5090 32GB 1x RTX 5090
Llama 3.1 8B ~16GB ~8GB 1x RTX 5090 32GB 1x RTX 5090
Mistral Small 3 24B ~48GB ~16GB 1x A100 80GB or 1x RTX PRO 6000 96GB 1x RTX PRO 6000
Qwen 2.5 Coder 32B ~64GB ~16GB 1x A100 80GB or 1x RTX PRO 6000 96GB 1x H100 80GB
Llama 3.1 70B / Qwen 2.5 72B FP16 ~140GB ~30GB 2x H100 80GB TP=2 OR 2x RTX PRO 6000 96GB TP=2 2x H100
Llama 3.1 70B FP8 ~70GB ~30GB 1x H100 80GB 1x H100 (much better $/token)
DeepSeek V3 (671B MoE, FP8) ~370GB active ~50GB 8x H100 80GB or 4x H100 with offload 8x H100
Llama 3.1 405B FP8 ~410GB ~80GB 8x H100 80GB TP=8 8x H100
Llama 3.1 405B FP16 ~810GB ~150GB Multi-node (not in v1) —

ASSUMPTION TO VALIDATE: RTX PRO 6000 96GB. The Blackwell-based RTX PRO 6000 has 96GB; older Ada RTX 6000 has 48GB. Confirm which SKU you have — it changes the fitment table significantly. Same for whether your A100s are 40GB or 80GB.

Heterogeneous routing rules

The Model CRD declares gpu_class_allowed: [h100, rtx-pro-6000]. Node affinity translates this to scheduling constraints. Rules of thumb baked into operator defaults:

Replicas per node

A node hosts as many replicas as fit in VRAM minus a 10% safety margin, and whose combined --max-num-seqs doesn’t saturate the node’s PCIe bandwidth or CPU (the API server side of vLLM is non-trivial). Empirical guidance:

Forecasting

Adding capacity

A new node joins the pool via:

  1. PXE boot → Talos or Ubuntu → join k3s
  2. NVIDIA GPU Operator detects GPUs, installs drivers
  3. Apply labels + taints
  4. model-warmer DaemonSet pre-pulls catalog weights to local NVMe (~30min for the full catalog)
  5. Operator schedules worker pods per the per-model replica targets in the Model CRDs
  6. Router picks up new endpoints via k8s watch
  7. Node enters rotation

End-to-end target: <2 hours from rack-and-stack to serving traffic, assuming weights are already mirrored in the internal registry.

Open questions for validation

  1. Node SKUs: exact GPU counts per node, NVMe capacity, NIC speed, NUMA topology. The fitment and routing tables assume canonical configs.
  2. Interconnect: do we have NVLink within nodes and InfiniBand between nodes? Affects whether multi-node TP is on the table for 405B.
  3. k8s baseline: is k3s acceptable, or do you already run RKE2/kubeadm? Switching costs are real.
  4. Object storage: do we have S3-compatible storage in-house, or do we need to deploy MinIO?
  5. Secret store: Vault assumed; alternative is sealed-secrets if Vault is too heavy.
  6. Region count for v1: confirmed single region? Multi-region routing changes the gateway design.
  7. Metering accuracy guarantees: is “near-realtime, end-of-day reconciled” acceptable for billing, or do customers need real-time hard cutoffs?