A developer has published a free learning project that takes viewers from a blank file to a working large language model inference engine. The project, called samir-vllm, pairs more than six hours of video with more than 80 pages of handwritten notes and diagrams. Its goal is to show what actually happens between a prompt and the generated tokens that come back.
What does the samir-vllm course cover?
The series is organized as a build. It starts with an overview of AI inference and moves through loading model weights, embedding lookups, normalization, rotary position embeddings, and the query, key and value projections. Later videos cover self-attention, the feed-forward network, the decode step and the KV cache, and then batching and paged KV caches. A finale covers paged attention. Companion notes for each video are available in Markdown and PDF form.
The engine is written in C++17 and CUDA and built with CMake. It targets the Llama 3.2 1B Instruct model and loads weights directly from a model.safetensors file, copying the raw BF16 tensors to GPU memory. A memory-mapped loader is also included to avoid an extra copy on the CPU side.
Much of the code is aimed at the parts of inference that most tutorials skip. The project includes a local byte-pair encoding tokenizer that handles Unicode and special tokens, with a dedicated test executable. Its CUDA kernels cover embedding lookups and RMSNorm, which reduces in FP32 while applying the learned BF16 weights. Prefill runs all 16 layers of the model, including grouped-query attention, the SwiGLU multilayer perceptron and the residual connections. A GPU argmax step selects each next token.
How does the engine manage KV cache memory?
The memory system is where the project does most of its teaching. A shared pool of BF16 KV pages holds 16 tokens of keys and values per physical page, with a block table for each sequence and layer. The default pool is 128 MiB and can be changed with a command-line flag. Decode attention reads each sequence's cached pages directly, using an FP32 online softmax, rather than gathering a contiguous prefix.
The engine supports both static batching, which waits for a group to finish before admitting more requests, and continuous batching, which admits queued prompts when enough page budget is free. Requests that do not fit wait in first-in, first-out order. A request whose budget exceeds the entire pool is rejected with an error. The author describes this approach as conservative. It avoids preemption and recomputation, but a large requested output budget can limit how many requests run at once, even if the model stops early.
The README also flags a limit that often surprises people. BF16 math can round differently at different batch sizes, so when two vocabulary logits are nearly tied, greedy output can differ between single and batched runs. Bitwise identical output across batch sizes is not guaranteed.
What hardware and licenses do you need to run it?
Running the engine requires a CUDA-capable NVIDIA GPU, the CUDA Toolkit, CMake 3.24 or newer, a C++17 compiler, ICU development libraries and the nlohmann/json header. Users also need the Llama 3.2 weights and tokenizer, which are governed by Meta's Llama 3.2 Community License and Acceptable Use Policy and are not covered by the project's MIT license.
The licensing is split. The engine source code in src, include, tests and python directories is MIT-licensed. The notes, scans, illustrations and PDFs are rights reserved. Personal reading and study are allowed, but redistributing or republishing the materials, free or paid, requires prior written permission.
This article was written with the assistance of AI.
News Factory APP - agentic news to boost your SEO & AEO.