"FlashAttention: Speeding Up Transformers with Modern Attention Kernels"
Transformers unlocked modern AI, but their attention mechanism remains one of the most stubborn performance bottlenecks at scale. This book is written for experienced ML engineers, systems practitioners, GPU programmers, and advanced researchers who want to understand not just that FlashAttention is faster, but exactly why it is faster, when it is faster, and how its design reshapes real-world transformer execution on modern hardware.
Across the book, readers move from the exact semantics of standard scaled dot-product attention into the GPU IO model that makes naive implementations collapse at long sequence lengths. From there, the text develops FlashAttention v1 in depth, then follows the major redesigns in FlashAttention-2 and FlashAttention-3, covering online softmax, tiling, kernel fusion, work partitioning, hardware specialization, framework integration, compatibility constraints, and rigorous benchmarking. By the end, readers will be able to reason about attention kernels as engineering artifacts, evaluate backend choices with evidence, and attribute performance gains correctly instead of relying on marketing claims or anecdotal speedups.
The treatment is architectural, implementation-aware, and deliberately exacting. Rather than presenting FlashAttention as a black-box package, the book frames it as a durable design pattern for building high-performance exact attention kernels across evolving GPU generations and software stacks.
