Introduction xix
Chapter 1: Introduction 1
Overview of GenAI Applications and Large Language Models 1
The Rise of Large Language Models 1
Neural Networks, Transformers, and Beyond 2
GenAI vs. LLMs: Whatâs the Difference? 5
The Three-Layer GenAI Application Stack 6
The Infrastructure Layer 6
The Model Layer 7
The Application Layer 8
Paths to Productionizing GenAI Applications 9
Sample LLM-Powered Chat Application 11
The Importance of Cost Optimization 12
Cost Assessment of the Model Inference Component 12
Cost Assessment of the Vector Database Component 19
Benchmarking Setup and Results 20
Other Factors to Consider 23
Cost Assessment of the Large Language Model Component 24
Summary 27
Chapter 2: Tuning Techniques for Cost Optimization 29
Fine-Tuning and Customizability 29
Basic Scaling Laws You Should Know 30
Parameter-Efficient Fine-Tuning Methods 32
Adapters Under the Hood 33
Prompt Tuning 34
Prefix Tuning 36
P-tuning 39
IA3 40
Low-Rank Adaptation 44
Cost and Performance Implications of PEFT Methods 46
Summary 48
Chapter 3: Inference Techniques for Cost Optimization 49
Introduction to Inference Techniques 49
Prompt Engineering 50
Impact of Prompt Engineering on Cost 50
Estimating Costs for Other Models 52
Clear and Direct Prompts 53
Adding Qualifying Words for Brief Responses 53
Breaking Down the Request 54
Example of Using Claude for PII Removal 55
Conclusion 59
Providing Context 59
Examples of Providing Context 60
RAG and Long Context Models 60
Recent Work Comparing RAG with Long Content Models 61
Conclusion 62
Context and Model Limitations 62
Indicating a Desired Format 63
Example of Formatted Extraction with Claude 63
Trade-Off Between Verbosity and Clarity 66
Caching with Vector Stores 66
What Is a Vector Store? 66
How to Implement Caching Using Vector Stores 66
Conclusion 69
Chains for Long Documents 69
What Is Chaining? 69
Implementing Chains 69
Example Use Case 70
Common Components 70
Tools That Implement Chains 72
Comparing Results 76
Conclusion 76
Summarization 77
Summarization in the Context of Cost and Performance 77
Efficiency in Data Processing 77
Cost-Effective Storage 77
Enhanced Downstream Applications 77
Improved Cache Utilization 77
Summarization as a Preprocessing Step 77
Enhanced User Experience 77
Conclusion 77
Batch Prompting for Efficient Inference 78
Batch Inference 78
Experimental Results 80
Using the accelerate Library 81
Using the DeepSpeed Library 81
Batch Prompting 82
Example of Using Batch Prompting 83
Model Optimization Methods 83
Quantization 83
Code Example 84
Recent Advancements: GPTQ 85
Parameter-Efficient Fine-Tuning Methods 85
Recap of PEFT Methods 85
Code Example 86
Cost and Performance Implications 87
Summary 88
References 88
Chapter 4: Model Selection and Alternatives 89
Introduction to Model Selection 89
Motivating Example: The Tale of Two Models 89
The Role of Compact and Nimble Models 90
Examples of Successful Smaller Models 91
Quantization for Powerful but Smaller Models 91
Text Generation with Mistral 7B 93
Zephyr 7B and Aligned Smaller Models 94
CogVLM for Language-Vision Multimodality 95
Prometheus for Fine-Grained Text Evaluation 96
Orca 2 and Teaching Smaller Models to Reason 98
Breaking Traditional Scaling Laws with Gemini and Phi 99
Phi 1, 1.5, and 2 B Models 100
Gemini Models 102
Domain-Specific Models 104
Step 1 - Training Your Own Tokenizer 105
Step 2 - Training Your Own Domain-Specific Model 107
More References for Fine-Tuning 114
Evaluating Domain-Specific Models vs. Generic Models 115
The Power of Prompting with General-Purpose Models 120
Summary 122
Chapter 5: Infrastructure and Deployment Tuning Strategies 123
Introduction to Tuning Strategies 123
Hardware Utilization and Batch Tuning 124
Memory Occupancy 126
Strategies to Fit Larger Models in Memory 128
KV Caching 130
PagedAttention 131
How Does PagedAttention Work? 131
Comparisons, Limitations, and Cost Considerations 131
AlphaServe 133
How Does AlphaServe Work? 133
Impact of Batching 134
Cost and Performance Considerations 134
S3: Scheduling Sequences with Speculation 134
How Does S3 Work? 135
Performance and Cost 135
Streaming LLMs with Attention Sinks 136
Fixed to Sliding Window Attention 137
Extending the Context Length 137
Working with Infinite Length Context 137
How Does StreamingLLM Work? 138
Performance and Results 139
Cost Considerations 139
Batch Size Tuning 140
Frameworks for Deployment Configuration Testing 141
Cloud-Native Inference Frameworks 142
Deep Dive into Serving Stack Choices 142
Batching Options 143
Options in DJL Serving 144
High-Level Guidance for Selecting Serving Parameters 146
Automatically Finding Good Inference Configurations 146
Creating a Generic Template 148
Defining a HPO Space 149
Searching the Space for Optimal Configurations 151
Results of Inference HPO 153
Inference Acceleration Tools 155
TensorRT and GPU Acceleration Tools 156
CPU Acceleration Tools 156
Monitoring and Observability 157
LLMOps and Monitoring 157
Why Is Monitoring Important for LLMs? 159
Monitoring and Updating Guardrails 160
Summary 161
Conclusion 163
Index 181
