Python for webscraping, scheduling & automation (Selenium)

Name: Python for webscraping, scheduling & automation (Selenium)
Rating: 4.4 (36 reviews)

Scrap any website online, schedule and automate complex Python scripts and create web apps with Selenium and Flask

Created byRomain Baud

Last updated 7/2023

English

What you'll learn

Using Python to automate tasks
Scrapping content on websites like Linkedin, Instagram, Google, ...
Scheduling & automating Python scripts
Setting up a server to host a Python application
Connecting a Python app to databases
Using Python for sending emails and text messages

Course content

5 sections • 26 lectures • 6h 44m total length

Introduction1:55
Scrape data from any website, including Instagram and LinkedIn, and automate Python scripts with scheduling that runs while you sleep, plus database connections, Excel manipulation, and email or text delivery.
Python refresher course0:32
Using Python to modify Excel files15:02
Retrieving data from the web (APIs, exchange rates, weather forecast, ..)9:21
Sending emails with Python11:26
Learn to send emails with Python using the SMTP library to connect to a mail server and alert yourself about web scraping results.
Sending text messages with Python8:49
Learn to send text messages with Python using Twilio for alerting and daily notifications. Install the Twilio library, configure clients and numbers, and send messages to clients.

Introduction to Selenium & website structure9:31
Sending information through search bars18:13
Scraping information from any website28:15
Learn to scrape information from websites using Selenium, iterate through results, extract names, addresses, and links, handle pagination, and save data to an Excel file with pandas.
Scraping linkedin searches29:03
Learn to scrape linkedin searches using selenium, extracting names, titles, and profile links, and save results to Excel with Python while handling login and page navigation.
Scraping forever loading lists16:24
Master using selenium to scroll endlessly, executing JavaScript with execute_script and a while loop to load more content from feeds like Twitter, LinkedIn, and Facebook.

Introduction to schedulers (Blocking scheduler)18:25
Learn python scheduling for web scraping with a blocking scheduler using apscheduler, configuring interval, date, and cron triggers, and passing arguments for automated tasks.
Introduction to web apps with the Flask framework10:40
Build your first Flask web app to trigger web scraping and automation scripts, set up routes and forms, and prepare for production and scheduling with a secret key.
Sending information to your web app9:21
Learn to send user input from a web form to your Python web app, using Flask to handle requests, get parameters, and a results route to trigger scraping scripts.
Webscrapping from a web app22:48
Automate web scraping with Selenium by connecting a Flask app, logging into LinkedIn, and searching for people in Paris using location inputs and result scraping.
Advanced scheduling with Apscheduler13:42
Learn to schedule web scraping tasks in the background with APScheduler, enabling delay, date-based and interval runs, misfire grace time, and database integration.
Connecting a database to your scheduler14:28
Connect your Python scheduler to a PostgreSQL database to store and manage jobs beyond local memory, using job stores, configure misfire grace time, and persist job data for scalable automation.
Connecting your web app to a database20:36

Securing your application16:48
Preparing for production7:40
Prepare for production by moving from a development Flask server to a production setup on PythonAnywhere or DigitalOcean, enabling headless Selenium to scrape and print results with names and links.
Production on Python Anywhere (without scheduler)30:03
Deploy a Python web scraping app to PythonAnywhere or DigitalOcean, using virtualenv and a requirements file to install libraries and run selenium tasks in production without a scheduler.
Before starting Digital Ocean0:21
Production on Digital Ocean (with scheduler)1:28:24
Note: Be sure to use Ubuntu 20.XX instead of Ubuntu 22.XX as it seems that some commands on Ubuntu 22.XX might not work.
Production on AWS0:38

Requirements

Python language or familiar with other programming languages

Description

Even though low-code and no-code technologies are becoming more and more popular, scraping any website and automating any task can still be difficult. This course gives you the tools so that you can use Python to automate any task on a browser, like creating a Linkedin or Instagram bot.

This course offers practical solutions to the most challenging problems that you will meet while trying to retrieve data from websites during your growth hacking journey: scraping email addresses, creating an Instagram bot, ... With this course, you will understand how you can automate any Python script to perform actions online. You will learn how to connect your application to databases and how to put it to production. This course is broken into 23 chapters that address all the subjects that you need to understand to build any automation or web scraping tool. You will learn:

- How to modify files with Python

- How to scrap data from any website (Linkedin, Instagram, etc)

- How to send emails & text messages with Python

- How to connect your application to databases (MySQL, PostgreSQL, MongoDB)

- How to automate & schedule Python scripts

- How to put your application to production with different providers

I have intentionally designed this course so that you can come back any time and find what you need in separate videos and course chapters. This course is part on an ongoing Python’s series including:

- Python for web development

- Python for web scrapping, scheduling & automation (this course)

- Python for mobile app backends & APIs

Who this course is for:

Python and full-stack developers, engineering students, self-taught developers

Python for webscraping, scheduling & automation (Selenium)

What you'll learn

Explore related topics

Course content

Basics of automating with Python6 lectures • 47min

Advanced webscraping5 lectures • 1hr 41min

Advanced automating7 lectures • 1hr 50min

Production6 lectures • 2hr 24min

Wrapping up2 lectures • 3min

Requirements

Description

Who this course is for: