| Edition |
2nd ed. |
| Description |
1 online resource (345 pages) |
|
text file |
| Contents |
Cover; Copyright; Credits; About the Author; About the Reviewer; www.PacktPub.com; Customer Feedback; Table of Contents; Preface; Chapter 1: A First Taste and What's New in Apache Spark V2; Spark machine learning; Spark Streaming; Spark SQL; Spark graph processing; Extended ecosystem; What's new in Apache Spark V2?; Cluster design; Cluster management; Local; Standalone; Apache YARN; Apache Mesos; Cloud-based deployments; Performance; The cluster structure; Hadoop Distributed File System; Data locality; Memory; Coding; Cloud; Summary; Chapter 2: Apache Spark SQL. |
|
The SparkSession -- your gateway to structured data processingImporting and saving data; Processing the text files; Processing JSON files; Processing the Parquet files; Understanding the DataSource API; Implicit schema discovery; Predicate push-down on smart data sources; DataFrames; Using SQL; Defining schemas manually; Using SQL subqueries; Applying SQL table joins; Using Datasets; The Dataset API in action; User-defined functions; RDDs versus DataFrames versus Datasets; Summary; Chapter 3: The Catalyst Optimizer; Understanding the workings of the Catalyst Optimizer. |
|
Managing temporary views with the catalog APIThe SQL abstract syntax tree; How to go from Unresolved Logical Execution Plan to Resolved Logical Execution Plan; Internal class and object representations of LEPs; How to optimize the Resolved Logical Execution Plan; Physical Execution Plan generation and selection; Code generation; Practical examples; Using the explain method to obtain the PEP; How smart data sources work internally; Summary; Chapter 4: Project Tungsten; Memory management beyond the Java Virtual Machine Garbage Collector; Understanding the UnsafeRow object. |
|
The null bit set regionThe fixed length values region; The variable length values region; Understanding the BytesToBytesMap; A practical example on memory usage and performance; Cache-friendly layout of data in memory; Cache eviction strategies and pre-fetching; Code generation; Understanding columnar storage; Understanding whole stage code generation; A practical example on whole stage code generation performance; Operator fusing versus the volcano iterator model; Summary; Chapter 5: Apache Spark Streaming; Overview; Errors and recovery; Checkpointing; Streaming sources; TCP stream. |
|
File streamsFlume; Kafka; Summary; Chapter 6: Structured Streaming; The concept of continuous applications; True unification -- same code, same engine; Windowing; How streaming engines use windowing; How Apache Spark improves windowing; Increased performance with good old friends; How transparent fault tolerance and exactly-once delivery guarantee is achieved; Replayable sources can replay streams from a given offset; Idempotent sinks prevent data duplication; State versioning guarantees consistent results after reruns; Example -- connection to a MQTT message broker. |
| Note |
Controlling continuous applications. |
| Summary |
Advanced analytics on your Big Data with latest Apache Spark 2.xAbout This Book* An advanced guide with a combination of instructions and practical examples to extend the most up-to date Spark functionalities.* Extend your data processing capabilities to process huge chunk of data in minimum time using advanced concepts in Spark.* Master the art of real-time processing with the help of Apache Spark 2.xWho This Book Is ForIf you are a developer with some experience with Spark and want to strengthen your knowledge of how to get around in the world of Spark, then this book is ideal for you. Basic knowledge of Linux, Hadoop and Spark is assumed. Reasonable knowledge of Scala is expected. What You Will Learn* Examine Advanced Machine Learning and DeepLearning with MLlib, SparkML, SystemML, H2O and DeepLearning4J* Study highly optimised unified batch and real-time data processing using SparkSQL and Structured Streaming* Evaluate large-scale Graph Processing and Analysis using GraphX and GraphFrames* Apply Apache Spark in Elastic deployments using Jupyter and Zeppelin Notebooks, Docker, Kubernetes and the IBM Cloud* Understand internal details of cost based optimizers used in Catalyst, SystemML and GraphFrames* Learn how specific parameter settings affect overall performance of an Apache Spark cluster* Leverage Scala, R and python for your data science projectsIn DetailApache Spark is an in-memory cluster-based parallel processing system that provides a wide range of functionalities such as graph processing, machine learning, stream processing, and SQL. This book aims to take your knowledge of Spark to the next level by teaching you how to expand Spark's functionality and implement your data flows and machine/deep learning programs on top of the platform. The book commences with an overview of the Spark ecosystem. It will introduce you to Project Tungsten and Catalyst, two of the major advancements of Apache Spark 2.x. You will understand how memory management and binary processing, cache-aware computation, and code generation are used to speed things up dramatically. The book extends to show how to incorporate H20, SystemML, and Deeplearning4j for machine learning, and Jupyter Notebooks and Kubernetes/Docker for cloud-based Spark. During the course of the book, you will learn about the latest enhancements to Apache Spark 2.x, such as interactive querying of live data and unifying DataFrames and Datasets. You will also learn about the updates on the APIs and how DataFrames and Datasets affect SQL, machine learning, graph processing, and streaming. You will learn to use Spark as a big data operating system, understand how to implement advanced analytics on the new APIs, and explore how easy it is to use Spark in day-to-day tasks. Style and approachThis book is an extensive guide to Apache Spark modules and tools and shows how Spark's functionality can be extended for real-time processing and storage with worked examples. |
| Local Note |
eBooks on EBSCOhost EBSCO eBook Subscription Academic Collection - North America |
| Subject |
Spark.
|
|
Data mining.
|
|
Data mining. |
|
COMPUTERS -- General. |
| Genre/Form |
Electronic books.
|
| Other Form: |
Print version: Kienzler, Romeo. Mastering Apache Spark 2.x - Second Edition. Birmingham : Packt Publishing, ©2017 |
| ISBN |
9781785285226 |
|
178528522X |
|
9781786462749 |
|
1786462745 |
| Standard No. |
9781786462749 |
|
