What is DeepInfer and what does it do?

DeepSeek has released DeepInfer, an open-source inference framework that the company says powers its large-scale production services. The code is hosted on GitHub and licensed under the Apache License 2.0. The current release is built for offline inference, and it targets the job of running large models across many accelerators at once.

The first supported model is DeepSeek-V4.1-Flash. The default checkpoint, deepseek-ai/DeepSeek-V4.1-Flash-1017, downloads into the Hugging Face cache on the first run. Developers who already have a local copy can point the framework to it with an environment variable instead.

DeepInfer ships with a set of optimized kernels, including FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and custom TileLang kernels. It also supports several parallelism strategies, among them attention data parallelism, balanced context parallelism and mixture-of-experts expert parallelism with EPLB. An asynchronous scheduler overlaps batch scheduling on the CPU with execution on the device, and it is turned on by default.

Decoding and output controls

The framework uses a speculative decoding method the README calls DSpark. It relies on draft tokens and confidence estimates to decide when to verify them. Developers can set the maximum draft length, which defaults to 5.

A second feature, which the project calls decoder SWA bounded replay, cuts prefill work by replaying only a limited suffix of the input. For the default V4.1 settings, the retained suffix is 140 tokens. The kept portion must cover the sliding window plus the maximum backtracking length.

DeepInfer also offers batch-invariant inference, which keeps model results identical no matter how requests are grouped into batches. That feature works only on NVIDIA hardware, as does deterministic sampling, which produces the same output for the same input tokens and sampling parameters. Structured outputs let developers constrain generation to a regular expression, a JSON schema or an EBNF grammar.

Hardware, setup and what is missing

Which hardware does DeepInfer support?

The framework supports NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Users need Linux, GCC 13.0 or later and Python 3.12. NVIDIA users also need CUDA 13, and Ascend users need CANN 9.2. The build process starts with creating an empty Python 3.12 environment, setting the DEEPINFER_ACCELERATOR variable to nvidia or ascend, and running the build script.

What is missing from the DeepInfer release?

The release is not a complete serving stack. According to the README, a full inference service also requires a KV cache system, separation of prefill and decode workloads, and load balancing across data-parallel replicas for both stages. The project says most of the engine functionality needed for that separation is already implemented, but the components themselves are not part of this release.

The project's documentation describes its performance benchmarks as run on four SM100 GPUs for prefill and 64 for decode, with each set inside a single scale-up domain. A sweep script is included so others can reproduce the tests.

DeepInfer was originally built primarily on vLLM, the README says, with additional inspiration from SGLang and TensorRT-LLM. DeepSeek credits those projects' developers in the release notes.

Questo articolo è stato scritto con l'assistenza dell'IA.
News Factory APP - notizie agentiche per potenziare il tuo SEO e AEO.