Computer scientists have long been aware of the limitations of storing Ternary Large Language Models (LLM) in a conventional format. The prevailing deployment format, which packs five ternary weights into one byte, rounds up to 1.625 bits per weight due to the power-of-two group sizes used in practice. However, researchers have now introduced a new storage method called BITCOS that breaks the 1.58-bit barrier, potentially leading to more efficient AI processing.

The BITCOS method is based on the finding that zeros account for up to 51.5% of all weights in 29 ternary LLM models. By introducing a dense presence bitmap plus a compacted sign vector, BITCOS costs 2 - z bits per weight element given a zero density z in the model's weights. This makes it more compact than the five-trit packing in 26 of the 29 tested models, reaching 1.485 bits per weight on the sparsest of them.

BITCOS is also amenable to efficient unpacking on modern processors and GPUs. Researchers have presented optimized unpacking sequences for AVX-512, AVX2, and Intel Xe2 GPUs. Measured against production state-of-the-art ternary matrix-vector multiplication kernels, the realized gain with the proposed layout is up to 1.28 times at the zero densities real-world ternary models exhibit.

The end-to-end LLM inference results on five different platforms, including client and server CPUs, integrated and discrete Xe2 GPUs, show that decode throughput improves by up to 1.18 times on CPUs and 1.27 times on GPUs. This breakthrough has significant implications for the development of more efficient AI models, particularly in the context of LLM optimization and AI search optimization.

The BITCOS Method

The BITCOS method is a simple yet effective way to store ternary LLM models. By taking into account the actual symbol distribution of the models, it is able to store weights more compactly than existing methods. This is particularly significant in the context of AI models, where storage efficiency can have a major impact on processing speed and overall performance.

Este artigo foi escrito com a assistência de IA.
News Factory APP - notícias agênticas para impulsionar seu SEO e AEO.