Udemy
    •  
    •  
    •  
    •  
    •  
    •  
    •  
    •  
Subscribe
Turn what you know into an opportunity and reach millions around the world.
Learn More
Your cart is empty.
Keep shopping
Big Data Processing with Apache Spark
Rating: 3.8 out of 5(7 ratings)
60 students

Big Data Processing with Apache Spark

Efficiently tackle large data sets and big data analysis challenges using Spark and Python
Created byPackt Publishing
Last updated 1/2019
English
English [Auto],

What you'll learn

  • Write your own Python programs that can interact with Spark
  • Implement data stream consumption using Apache Spark
  • Recognize common operations in Spark to process known data streams
  • Integrate Spark streaming with Amazon Web Services
  • Create a collaborative filtering model with Python and the movielens dataset
  • Apply processed data streams to Spark machine learning APIs

Course content

4 sections • 27 lectures • 3h 29m total length

  • Course Overview2:33

    Let us begin our learning journey on Data Processing with Apache Spark. In this course, we will learn to efficiently tackle large datasets and perform big data analysis with Spark and Python. The GitHub link for this course is - https://github.com/TrainingByPackt/Big-Data-Processing-with-Apache-Spark-eLearning

  • Installation and Setup4:50

    Before we start learning about Apache Spark, let us ensure that we have access to the ecosystem and all of the necessary tools. To begin with, let us download and install Spark and set up our computer environment by installing the following:

    • Spark 2.4

    • OS: Windows 7 SP1 64-bit, Windows 8.1 64-bit, or Windows 10 64-bit

    • Python 3.0 or above

    • Amazon Web Services (AWS) account

  • Lesson Overview3:35

    Let us begin the course by learning about data processing fundamentals using RDDs, datasets, and APIs. In this section, we will learn how to use Spark for data processing and get an introduction into Spark SQL and various Spark dataframes.   

  • Introduction to Spark and Resilient Distributed Datasets16:14

    Now that we are familiar with the lesson overview has a whole, let  us begin to learn the use of Spark for data processing and get an  introduction of Spark SQL and various Spark DataFrames. In this section,  we will learn about the programming languages supported by Spark, its  components and the deployment modes. 

  • Operations Supported by the RDD API15:41

    RDDs support two types of operations: transformations and actions. These operations along with some actions will be used to execute the dataset. Let us further learn into the concepts and use them practically in the demo.   

  • Map Reduce Operations7:28

    In continuation from the previous section, let us dive deep into the concepts of map reduced functions and their execution.   

  • Self-Contained Python Spark Programs10:47

    The interactive Python interface is a great tool for simple computations. Its functionality is limited even as the computing operations grow in complexity. In this section, let us learn to write Python programs that can interact with a Spark cluster outside of the interactive console. Further to this section, we will also learn about functional programming.   

  • Nested Functions and Standalone Python Programs10:10

    Nested function are functions inside other functions. The most important advantage of this paradigm is that the outer scope cannot see what is happening in the inner function. Nonetheless, the inner scope can access variables in the outer scope. Now, let us look at an example of a function using the syntax. Further to this section, we will also learn about Standalone Python Programs.   

  • Introduction to SQL, Datasets, and DataFrames14:16

    Before we understand how each of these function along with Spark, let us first know what they mean. A dataset is a distributed collection that provides additional metadata about the structure of the data that is stored. A DataFrame is a dataset that organizes information into named columns. DataFrames can be built from different sources, such as JSON, XML, and databases. In this section, let us cover each of them in detail. For further information on movielens datasets.   

  • Lesson Summary0:43

    Let us quickly recap our learning from this lesson.     

  • Test Your Knowledge

Requirements

  • Prior experience of working with Python is recommended.

Description

Processing big data in real  time is challenging due to scalability, information consistency, and  fault-tolerance. Big Data Processing with Apache Spark teaches you how  to use Spark to make your overall analytical workflow faster and more  efficient. You'll explore all core concepts and tools within the Spark  ecosystem, such as Spark Streaming, the Spark Streaming API, machine  learning extension, and structured streaming.

You'll begin by learning data processing fundamentals using Resilient  Distributed Datasets (RDDs), SQL, Datasets, and Dataframes APIs. After  grasping these fundamentals, you'll move on to using Spark Streaming  APIs to consume data in real time from TCP sockets, and integrate Amazon  Web Services (AWS) for stream consumption.

By the end of this course, you’ll not only have understood how to use  machine learning extensions and structured streams but you’ll also be  able to apply Spark in your own upcoming big data projects. 

About the Author

Manuel Ignacio Franco  Galeano is a computer scientist from Colombia. He works for Fender  Musical Instruments as a lead engineer in Dublin, Ireland. He holds a  master's degree in computer science from University College, Dublin UCD.  His areas of interest and research are music information retrieval,  data analytics, distributed systems, and blockchain technologies.

Nimish Narang has graduated from UBC with a degree in biology and  computer science in 2016. He has developed Mobile apps for Android and  iOS since 2015. He is focused on data analysis and machine learning from  the past two years and has previously published Keras and Professional  Scala with Packt. 

Who this course is for:

  • This course is for you if you are a software engineer, architect, or IT professional who wants to explore distributed systems and big data analytics

Report abuse