Big Data Processing with Apache Spark
What you'll learn
- Write your own Python programs that can interact with Spark
- Implement data stream consumption using Apache Spark
- Recognize common operations in Spark to process known data streams
- Integrate Spark streaming with Amazon Web Services
- Create a collaborative filtering model with Python and the movielens dataset
- Apply processed data streams to Spark machine learning APIs
Course content
- Preview02:33
- Preview04:50
- 03:35Lesson Overview
- 16:14Introduction to Spark and Resilient Distributed Datasets
- 15:41Operations Supported by the RDD API
- 07:28Map Reduce Operations
- 10:47Self-Contained Python Spark Programs
- 10:10Nested Functions and Standalone Python Programs
- 14:16Introduction to SQL, Datasets, and DataFrames
- 00:43Lesson Summary
- 3 questionsTest Your Knowledge
Requirements
- Prior experience of working with Python is recommended.
Description
Processing big data in real time is challenging due to scalability, information consistency, and fault-tolerance. Big Data Processing with Apache Spark teaches you how to use Spark to make your overall analytical workflow faster and more efficient. You'll explore all core concepts and tools within the Spark ecosystem, such as Spark Streaming, the Spark Streaming API, machine learning extension, and structured streaming.
You'll begin by learning data processing fundamentals using Resilient Distributed Datasets (RDDs), SQL, Datasets, and Dataframes APIs. After grasping these fundamentals, you'll move on to using Spark Streaming APIs to consume data in real time from TCP sockets, and integrate Amazon Web Services (AWS) for stream consumption.
By the end of this course, you’ll not only have understood how to use machine learning extensions and structured streams but you’ll also be able to apply Spark in your own upcoming big data projects.
About the Author
Manuel Ignacio Franco Galeano is a computer scientist from Colombia. He works for Fender Musical Instruments as a lead engineer in Dublin, Ireland. He holds a master's degree in computer science from University College, Dublin UCD. His areas of interest and research are music information retrieval, data analytics, distributed systems, and blockchain technologies.
Nimish Narang has graduated from UBC with a degree in biology and computer science in 2016. He has developed Mobile apps for Android and iOS since 2015. He is focused on data analysis and machine learning from the past two years and has previously published Keras and Professional Scala with Packt.
Who this course is for:
- This course is for you if you are a software engineer, architect, or IT professional who wants to explore distributed systems and big data analytics
Instructor
![Packt Publishing](data:image/svg+xml,%3Csvg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64"%3E%3C/svg%3E){kind=avatar}
- 3.9 Instructor Rating
- 59,001 Reviews
- 353,963 Students
- 1,418 Courses
Packt has been committed to developer learning since 2004. A lot has changed in software since then - but Packt has remained responsive to these changes, continuing to look forward at the trends and tools defining the way we work and live. And how to put them to work.
With an extensive library of content - more than 4000 books and video courses -Packt's mission is to help developers stay relevant in a rapidly changing world. From new web frameworks and programming languages, to cutting edge data analytics, and DevOps, Packt takes software professionals in every field to what's important to them now.
From skills that will help you to develop and future proof your career to immediate solutions to every day tech challenges, Packt is a go-to resource to make you a better, smarter developer.
Packt Udemy courses continue this tradition, bringing you comprehensive yet concise video courses straight from the experts.