Scalable Big Data Architecture

Chapter 1: I think I have a Big (data) Problem (20 pages) Chapter Goal: This chapter aims to introduce you to the topology of common existing limitations when it comes to dealing with large amounts of data, and what are the common solutions to those problems. The goal here is to lay down the foundation of a heterogeneous architecture that will be described in the following chapters. 1- Identifying Big Data symptoms 2- Understanding the Big Data projects ecosystem 3- Creating the foundation of a long term Big Data architecture Chapter 2: Early Big Data with No-SQL (30 pages) Chapter Goal: This chapter aims to describe how a No-SQL database can be a starting point for your Big Data project, how it can deal with large amounts of data, what are the limits of this model and how it can be scaled to a full-fledged Big Data project. 1- Choosing the right No-SQL database 2- Introduction to Couchbase 3- Introduction to Elasticsearch 4- Using No-SQL cache in a SQL based architecture Chapter 3: Big Data processing jobs topology (30 pages) Chapter Goal: The more data you get, the more important it is to split the processing into different jobs depending on the topology of the processing. 1- Big Data Job processing strategy 2- Smart data extraction from No-SQL database 3- Short term processing jobs. 4- Long term processing jobs. Chapter 4: Big Data Streaming Pattern (30 pages) Chapter Goal: This chapter helps the readers to understand what are their options when it comes to dealing with streaming high data throughput. 1- Identifying streaming data sources 2- Streaming with Big Data projects (Flume) versus Enterprise Service Bus 3- Processing architecture for stream data Chapter 5: Querying and Analysing Patterns (30 pages) Chapter Goal: In this chapter, the readers will understand how to leverage the processing work through long term & real time data querying. 1- "Process then Query" strategy versus real-time querying 2- Process, store and query data in Elasticsearch 3- Real-Time querying using Spark Chapter 6: How About Learning from your Data? (30 pages) Chapter Goal: This chapter will introduce the concept of machine learning at different level of the preceding described patterns and through different relative methodology. 1- Introduction to machine learning 2- Supervised and Unsupervised learning 3- A simple example of Machine learning 4- Using MLlib for machine learning Chapter 7: Governance Considerations (20 pages) Chapter Goal: Monitoring, and more generally governance is extremely important when dealing with architecture that involves all the previous patterns. This chapter is to safeguard the reader from major issues, and to gain visibility and control over the architecture. 1- Data Quality 2- Architecture Scalability 3- Security 4- Monitoring