zettelkasten

V Llm

Last updated: 8/9/2025

vLLM

High-throughput, memory-efficient LLM serving with PagedAttention vLLM is an open-source inference and serving engine designed to maximize LLM throughput by eliminating KV-cache memory waste via PagedAttention, continuous batching, and optimized CUDA kernels, exposing both a Python API and an OpenAI-compatible server for easy integration.[4][3][1]

Core Idea

  • The bottleneck in LLM serving is the growing, dynamic KV-cache; naive allocation leads to 60%–80% memory waste due to fragmentation and over-reservation.[2][3]
  • PagedAttention treats KV-cache like virtual memory: splits it into fixed-size blocks so keys/values can be stored non-contiguously, enabling flexible allocation, sharing, and near-zero waste.[3][2]
  • With a more efficient KV-cache, vLLM increases batch sizes and supports continuous batching, driving higher tokens/s at lower latency and cost.[1][3]

Key Concepts

  • PagedAttention: KV-cache partitioned into blocks; attention kernels fetch blocks via a block table, allowing non-contiguous physical layout while preserving logical order.[5][2][3]
  • KV-cache manager: virtual-memory-like mapping, supports dynamic growth, sharing across sequences/requests, and reduced duplication.[2][3]
  • Continuous batching: accepts new requests between decoding steps to keep GPUs saturated and reduce p50 latency while boosting throughput.[1]
  • Execution optimizations: CUDA/HIP graphs, FlashAttention/FlashInfer kernels, speculative decoding, chunked prefill, quantization (GPTQ/AWQ/INT4/INT8/FP8).[1]
  • Distributed support: tensor/pipeline/data/expert parallelism across accelerators to serve larger models and higher loads.[6][1]
  • Interfaces: Python LLM class for offline inference; OpenAI-compatible API via vllm serve for easy drop-in with existing clients.[4]

Why It Matters

  • Higher throughput (2–4× in the paper’s evaluations) and lower memory footprint translate to better utilization and lower serving cost, especially for long prompts, large models, and complex decoding like beam search.[7][2]
  • Efficient KV-cache sharing and pre-allocation let operators push more concurrent sequences without OOMs, improving QoS under bursty traffic.[2][1]

How To Use (mental model)

  • Offline: from vllm import LLM, SamplingParams; run llm.generate() for batch prompts.[4]
  • Serving: vllm serve or python -m vllm.entrypoints.openai.api_server --model to expose OpenAI-compatible endpoints (chat/completions).[4]
  • Tuning: gpu_memory_utilization determines fraction of GPU reserved for cache; adjust max_num_seqs or max_num_batched_tokens to control concurrency and KV-cache size.[8][9]

Operational Notes

  • Version behavior: gpu_memory_utilization changed semantics in vLLM 0.6.4 to act as a global GPU memory limit including non-Torch allocations, complicating multi-model sharing relative to 0.6.3; plan deployment accordingly.[10]
  • Capacity pressure: vLLM can swap KV to CPU RAM or recompute under load to sustain service, trading latency vs memory.[6]
  • Hardware coverage: supports NVIDIA/AMD/Intel CPUs/GPUs, Gaudi, TPU, AWS Trainium/Inferentia, and more.[1]

Comparisons/Context

  • Compared to traditional attention memory layouts, PagedAttention avoids contiguous allocation requirements and reduces fragmentation, enabling larger dynamic batches.[3][2]
  • Alternative approaches (e.g., vAttention) explore dynamic KV management without paging, retaining contiguous virtual cache; it’s an adjacent design trade-off space worth tracking.[11]

Pitfalls

  • Misconfigured gpu_memory_utilization can lead to underfilled cache (low throughput) or oversubscription (OOM/preemption).[9][8]
  • Multi-tenant GPUs require careful coordination after 0.6.4 due to global limit behavior, especially when launching multiple servers/processes.[10]

Links/Entry Points

  • Docs and feature list, including quantization and batching strategies.[1]
  • Architecture overview with LLM API and OpenAI server usage examples.[4]
  • Original PagedAttention paper for algorithmic details and throughput results.[12][2]
  • Introductory blog post explaining KV-cache waste and PagedAttention’s design.[3]

Related

  • FlashAttention/FlashInfer for kernel-level attention speedups integrated in vLLM.[1]
  • Speculative decoding for latency/throughput gains in generation pipelines.[1]
  • Serving frameworks and schedulers that benefit from continuous batching and KV reuse.[1]