Inference Engines: vLLM, TensorRT-LLM & SGLang

what a serving engine actually adds

A raw model + framework (e.g. a Hugging Face transformers model in a Python loop) can generate correct output, but does none of the following by default: continuous batching across concurrent requests, PagedAttention-style KV cache management, request scheduling/queuing, or kernel-level optimization of the attention and matmul operations for the specific GPU it's running on. A serving engine's whole job is closing that gap between "produces correct tokens" and "serves production traffic efficiently." See KV Cache & Continuous Batching for the mechanisms these engines implement.

the major engines, compared

Engine	Strength	Trade-off
vLLM	PagedAttention origin, broad model support, straightforward to deploy, strong community/ecosystem	Not always the single fastest for every model/hardware combination
NVIDIA TensorRT-LLM	Deep kernel-level optimization specifically for NVIDIA GPUs, often the fastest on that hardware	NVIDIA-only; more involved build/compile step per model
SGLang	Strong at structured generation and complex multi-call prompting patterns (agents, constrained decoding), competitive raw throughput	Smaller ecosystem than vLLM, though growing quickly
Hugging Face TGI	Tight integration with the Hugging Face model hub/ecosystem	Historically behind vLLM/TensorRT-LLM on raw throughput benchmarks

a minimal vLLM server

pip install vllm

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --quantization awq \
    --max-model-len 8192

The server exposes an OpenAI-compatible API by default — existing client code written against OpenAI's SDK typically works against it by just changing the base URL, which is a big part of why vLLM became a common default for self-hosting.

choosing between them

If...	Lean toward
You want the fastest path to a working self-hosted server on common GPUs	vLLM
You're squeezing maximum throughput out of NVIDIA hardware you already control end to end	TensorRT-LLM
Your workload is agentic/structured-generation-heavy with many chained calls per user request	SGLang

Benchmark all serious candidates against your own model and traffic pattern before committing — published benchmarks vary by model size, hardware, and request pattern enough that a generic "X is fastest" claim rarely transfers cleanly to a specific deployment.

where to go from here

GPU Memory Planning for LLMs — sizing the hardware these engines run on.
KV Cache & Continuous Batching — the mechanisms these engines are built around.
GPU Optimization — the lower-level kernel work these engines rely on.