Distributed Tasks Demystified with Celery, SQS & Python

Name: Distributed Tasks Demystified with Celery, SQS & Python
Rating: 4.0 (223 reviews)

Conjure up your first Python scalable background worker

Created byTimothy Mugayi

Last updated 5/2019

English

What you'll learn

Fundamentals of multithreading in python
How to implement distributed tasks with Python & Django
Implement message passing communication between processes to build parallel applications
How to scale on the cloud with AWS Simple Queue Service (SQS)
Learn how to build a distributed social media data ingestor

Course content

7 sections • 39 lectures • 4h 27m total length

Introduction5:11
Prepping up your environment7:23
In this lesson we cover all the packages that we need to get started with the course and prep up our development workspace.

Blocking vs non blocking (part 1)6:09
Blocking vs non blocking (part 2)5:00
Concurrency Consumer & Producer problem a deep dive6:37
Explore the producer–consumer concurrency problem, race conditions, and how mutual exclusion, semaphores, and message queues enable safe, distributed processing with first-in, first-out delivery.
Solving Consumer producers problem with Mutual Exlusion6:18
Controlling threads with conditions (Part 1)2:23
In this lesson we will discusses thread synchronization mechanisms in Python and how to synchronize access to shared resources, We take the consumer & producer problem from a different angle with conditions
Controlling threads with conditions (Part 2)8:15
Controlling threads with conditions (Part 3)3:50
Daemon threads by example (Part 4)2:05
Daemonize producer and consumer threads to run in the background, and keep the main thread alive with a loop so the program exits cleanly when interrupted.
Consumer producer a thread safe FIFO queue5:56
Explore how to implement a thread-safe, first-in-first-out producer-consumer pattern in Python by using the built-in queue data structure with put, get, and task_done to handle multiple producers efficiently.

Getting started with Celery5:52
Explore how to convert a Python function into a Celery task using the task decorator, configure a broker, and run producer and consumer processes to execute background work.
Celery backends & Asyncresult by example8:43
Python exception handling back to the basics13:41
Exception handling in Celery Explained9:22
Celery scheduled periodic tasks (Part 1)4:44
Celery scheduled periodic tasks (Part 2)4:42
Explore celery scheduled periodic tasks, including how to create a periodical task with a decorator, run every few seconds, and execute via celery beat, with interval and cron-like triggers.
Celery scheduled periodic tasks How to apply Mutex (Part 3)10:38
In this lesson, we will touch base on a key important aspect of incorporating locks in distributed workers. Distributed locks are a useful primitive in many environments where different processes must operate on shared resources in a mutually exclusive way.
Celery scheduled periodic tasks solar schedules1:20
Learn to schedule periodic tasks with Celery using solar schedules tied to sunset and coordinates, explore configurations, and apply UTC time references with practical samples.

Introduction to distributed tasks with AWS SQS13:59
Explore how to design decoupled, scalable distributed applications using AWS SQS as a message conduit between producers and workers, comparing standard and FIFO queues and introducing priority-based concepts.
Creating your first AWS SQS Queue with your AWS Console5:20
In this lesson we learn how to configure AWS simple queue service for distributed applications
How to create a AWS SQS background worker in python (Part 1)8:02
How to create a AWS SQS background worker in python (Part 2)9:41
PENDING LAUNCH 28/10/2017
Dead-letter Queues the theory7:11
In this lesson we take a look at what are dead letter queues and how they fit in distributed services with sqs
Dead-letter Queues illustrated10:16
In this lesson we see a live example of how to implement dead letter queues in AWS SQS
How to bypass AWS SQS (Simple Queue Service) 256kb payload limit10:32
In this special video we take a look at how to bypass AWS SQS 256kb hard limit when sending large data via SQS queues.

Introduction Project #11:04
With the advent of numerous Social media platforms, the need has become of real importance to have the ability to tap into these platforms for their data. Such data allows businesses to understand the wants and needs of customers. Data extraction poses a fantastic use case to explore how to build and orchestrate distributed processes to pull data from various social media platforms. In this series of data injestor project lessons, we explore how to build our own distributed data injestor that will allow us to aggregate raw data while taking all concepts learnt and implementing them into a real-world use case that has been implemented by large multi-million dollar cooperation’s.
Real world examples of data ingestors3:59
Explore real-world data ingestors that pull social media data for analytics and sentiment insights, building distributed back-end pipelines with connectors and tools like Twitter API and IBM Watson.
Creating a twitter developer application and Authentication Token6:16
Create a Twitter developer application, generate the access token and secret, and set the callback. Then use the Python Twitter library to authenticate with these credentials and perform read/write actions.
Building your first social ingestor twitter (Part 1)1:22
Walk through building a Twitter ingestor by installing the Python Twitter library, setting up a Django scrappy project, and creating a test file to connect to Twitter.
Building your first social ingestor twitter (Part 2)3:33
Authenticate with the Twitter API using your consumer key and secret, verify credentials, and fetch a user timeline while handling non ascii characters and encoding on Windows and Linux.
Building your first social ingestor twitter Rate Limits (Part 3)8:52
Most APIs are subject to a limit on how many calls can be made per second (or minute, or other short time period), This is in order to protect servers from being overloaded and maintain high quality of service for other clients wishing to use a service. In This lesson we explore twitter rate limits and how to overcome them
Building your first social ingestor twitter Handle (Part 4)12:10
Design a Twitter data ingestor using the user timeline REST API to fetch tweets with paging and rate-limit considerations. Clean data with regex and export to CSV using Python.
Building your first social ingestor twitter Handle (Part 5)7:51

Basic fundamentals of SMTP and transactional email Services4:20
In this lesson we dive into the core basics of how SMTP works and look at what are your options for transactional email services
Creating your first background email worker (Part 1)11:48
In this series of lessons we take a look at how we can craft out our email background worker.
Creating your first background email worker (Part 2)11:22
Creating your first background email worker (Part 3)3:20

Requirements

Have access to the internet
Elementary understanding of any programming language
Sublime Anaconda Python IDE or or any python IDE installed
Your enthusiasm to learn how to scale distributed applications

Description

This course teaches beginners to industry professionals the fundamental concepts of Distributed Programming in the context of python & Django. We look at how to build applications that increase throughput and reduce latency. In this course, we will take a dive intially in the irst part of the course and build a strong foundation of asynchronous parallel tasks using python-celery a distributed task queue framework. We will explore AWS SQS for scaling our parallel tasks on the cloud. These fundamentals will aid you in building scalable Python solutions for virtually any python project. By the end of this course, you will have learnt how to use popular distributed programming frameworks for python and Django. Through concepts learnt, you will discover the world of distributed computing with Python and how easy it is to build distributed components into your python or Django projects.

Why take this course?

As to be python developer or Django web application developer, Its important to learn how to build applications that are able to process long running jobs or tasks in a non blocking way e.g sending mass email, map reduce, running high computational functions. Building a chatting applications.
This course builds up your skills in distribute programming giving you the tools you need to scale your applications
This course consist of projects you can use to implement in your real world projects with the sole emphasis of allowing your applications to become distributed and have asynchronous components
During the course, you will have online access to the instructor and get individualized answers to your questions posted on forums.
As an enrolled student you will get lifetime access to over 30 lectures plus and counting including future series of lessons
This course comes with a 30 day money back guarantee! This means If you are not satisfied in any way, you'll get your money back. Though you may just miss out of the future existing lessons planned ahead and free source code for your own personal or business projects. So what are you waiting for?

"Come join me and lets build distributed Python apps in a way that will advance your career, enhance your knowledge and potiential to earn even more. The best part is this doesnt need to be hard it actually can be fun"

Who this course is for:

Complete beginners to python programming language
Non Python programmers looking to explore distributed programming in python
Aspiring Technical Founders looking to implement distributed applications
Anyone wishing to learn how distributed programming works in python
Anyone wishing to implement asynchronous programming in python web applications

Distributed Tasks Demystified with Celery, SQS & Python

What you'll learn

Explore related topics

Course content

Introduction2 lectures • 13min

Getting started with Multithreading in Python9 lectures • 47min

Core Celery Distributed Tasks8 lectures • 59min

Distributed tasks with AWS SQS7 lectures • 1hr 5min

Distributed data ingestor Project #18 lectures • 45min

Distributed Email Workers Project #24 lectures • 31min

Python Development Tools1 lecture • 8min

Requirements

Description

Who this course is for: