Posts

All the articles I've posted.

• Techniques for KV Cache Optimization in Large Language Models

  Posted on:February 25, 2024 |  at 08:00 AM (12 min read)

  This post explores techniques for optimizing the Key-Value (KV) cache in large language models, from Grouped-query attention to PagedAttention and distributed cache management.

• Understanding how LLM inference works with llama.cpp

  Posted on:November 11, 2023 |  at 04:00 PM (34 min read)

  In this post we will understand how large language models (LLMs) answer user prompts by exploring the source code of llama.cpp, a C++ implementation of LLaMA, covering subjects such as tokenization, embedding, self-attention and sampling.

• IOPS, the silent killer of cloud databases

  Posted on:August 20, 2023 |  at 08:00 AM (9 min read)

  Despite advancements in cloud infrastructure and storage technology, IOPS is still a significant bottleneck for cloud databases. This post explains the source of this bottleneck and techniques to solve it.