"Marquez Lineage Backend: Operating OpenLineage Metadata Services"
Lineage is only as trustworthy as the service that records it. This book targets experienced platform engineers, data infrastructure operators, and senior data engineers responsible for running Marquez as an OpenLineage backend in production. Rather than reintroducing lineage concepts, it focuses on what you are actually operating: a contract-driven metadata system whose correctness, freshness, and availability depend on how producers emit events and how the backend absorbs failure.
You'll learn to reason precisely about the OpenLineage-Marquez contract, including identity construction, lifecycle semantics (START/COMPLETE/FAIL), idempotency, duplication, and out-of-order delivery. The book goes deep on facets and schema evolution, offering operational strategies for mixed producer versions without outages. It then turns to ingestion transport and API semantics—timeouts, retries, backpressure, request sizing, and security at the boundary—before building a durable naming architecture that prevents graph fragmentation. Finally, it treats Marquez as a data system: PostgreSQL capacity planning, indexing and slow-query workflows, safe migrations, retention, and disaster recovery, followed by production deployment patterns and observability-driven day-2 operations.
Readers should be comfortable with HTTP services, distributed systems failure modes, and PostgreSQL operations. The emphasis is on decision criteria, anti-patterns, and runbook-grade workflows you can apply immediately in Kubernetes, local stacks, or
