"promptfoo in CI: Regression Testing Prompts, Tools, and RAG Pipelines"
Modern LLM systems fail in ways that traditional test suites rarely catch: prompts drift, tool calls silently change, retrieval quality degrades, and superficially good outputs conceal serious regressions. This book is for experienced engineers, platform teams, and technical leads who want to bring CI-grade discipline to prompt-driven systems. It treats promptfoo not as a convenience utility, but as an engineering framework for turning fragile LLM behavior into versioned, testable, reviewable contracts.
Across the book, readers learn how to design robust `promptfooconfig.yaml` specifications, build high-fidelity datasets, choose assertions that match real failure modes, and operationalize evaluations in CI/CD pipelines with artifacts, thresholds, and quality gates. It goes beyond simple prompt checks to cover tool and agent trajectory validation, RAG grounding and retrieval regressions, cost and flakiness control, and adversarial red-team workflows for prompt injection, tool abuse, and data leakage. The result is a practical blueprint for making LLM quality measurable, reproducible, and enforceable before release.
The treatment assumes fluency with software delivery, automated testing, and modern AI application architecture. Rather than simplifying the subject, it focuses on the operational details, design trade-offs, and maintenance patterns that matter in production environments, especially where models, prompts, and system boundaries evolve continuously.
