Frank Kane's Taming Big Data with Apache Spark and Python

coverpage

Title Page

Credits

About the Author

www.PacktPub.com

Why subscribe?

Customer Feedback

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Downloading the color images of this book

Errata

Piracy

Questions

Getting Started with Spark

Getting set up - installing Python a JDK and Spark and its dependencies

Installing Enthought Canopy

Installing the Java Development Kit

Installing Spark

Running Spark code

Installing the MovieLens movie rating dataset

Run your first Spark program - the ratings histogram example

Examining the ratings counter script

Running the ratings counter script

Summary

Spark Basics and Spark Examples

What is Spark?

Spark is scalable

Spark is fast

Spark is hot

Spark is not that hard

Components of Spark

Using Python with Spark

The Resilient Distributed Dataset (RDD)

What is the RDD?

The SparkContext object

Creating RDDs

Transforming RDDs

Map example

RDD actions

Ratings histogram walk-through

Understanding the code

Setting up the SparkContext object

Loading the data

Extract (MAP) the data we care about

Perform an action - count by value

Sort and display the results

Looking at the ratings-counter script in Canopy

Key/value RDDs and the average friends by age example

Key/value concepts - RDDs can hold key/value pairs

Creating a key/value RDD

What Spark can do with key/value data?

Mapping the values of a key/value RDD

The friends by age example

Parsing (mapping) the input data

Counting up the sum of friends and number of entries per age

Compute averages

Collect and display the results

Running the average friends by age example

Examining the script

Running the code

Filtering RDDs and the minimum temperature by location example

What is filter()

The source data for the minimum temperature by location example

Parse (map) the input data

Filter out all but the TMIN entries

Create (station ID temperature) key/value pairs

Find minimum temperature by station ID

Collect and print results

Running the minimum temperature example and modifying it for maximums

Examining the min-temperatures script

Running the script

Running the maximum temperature by location example

Counting word occurrences using flatmap()

Map versus flatmap

Map ()

Flatmap ()

Code sample - count the words in a book

Improving the word-count script with regular expressions

Text normalization

Examining the use of regular expressions in the word-count script

Running the code

Sorting the word count results

Step 1 - Implement countByValue() the hard way to create a new RDD

Step 2 - Sort the new RDD

Examining the script

Running the code

Find the total amount spent by customer

Introducing the problem

Strategy for solving the problem

Useful snippets of code

Check your results and sort them by the total amount spent

Check your sorted implementation and results against mine

Summary

Advanced Examples of Spark Programs