Teach on Udemy

Turn what you know into an opportunity and reach millions around the world.

Learn More

Your cart is empty.

Keep shopping

Web Scraping with Python, Ruby & import. io

Name: Web Scraping with Python, Ruby & import. io
Rating: 3.8 (71 reviews)

Learn exactly how to scrape web data from many popular websites even if you have very little experience as a programmer!

Created byCharlie Craine

Last updated 10/2016

English

English [Auto],

What you'll learn

Scrape a variety of websites
Scrape websites with Ruby
Scrape websites with Python
Scrape websites with Import. io

Course content

7 sections • 46 lectures • 4h 37m total length

Introduction: What you will learn1:30
YES you will learn to scrape websites! And no, you don't need to be a highly skilled developer!

Introduction to the new Import io Web Scraping tool1:38
Learn what is new in the brand new web-based version of Import io
Setting up the new Import io3:12
Create a new extractor in import.io, understand pricing limits on queries, and compare using the paid interface versus Python or Ruby scrapers for large-scale data.
Wikipedia - Learn to scrape a table from Wikipedia5:15
Wikipedia - Learn to grab content from a Wikipedia page4:53
CNN.com - Learn how to grab content from a CNN article page3:03
Learn to scrape CNN article content by building a content extractor, creating columns for headline, content, and author, and exporting the data as a csv for reuse.
IMDB - Scrape the most popular movies from IMDB2:34
NFL.com - Learn how to Scrape the top QB stats4:44
Learn to scrape NFL quarterback stats and draft data using python, ruby, and import. io, build clean data views, and enable weekly updates and API integrations.
NFL Draft - Learn how to Scrape all rounds of the NFL Draft7:01
You will learn to scrape the NFL draft picks and how to add URLs to your web scraper.

Download Import.io0:12
Import.io overview: learn how to install and get up and running0:33
Oh, it's kind of Magic: Grab products from IKEA2:12
Scrape Wikipedia Data and Map it5:29
Scrape Wikipedia and map the data. Added August 28, 2015
Learn how to Extract data from IMDB4:09
Learn to run an extractor on IMDB top 250, load pages, create columns for title, year, IMDB id, and rating, then export to Google Sheets.
Extract data from Macys.com4:51
Yelp: Learn how to scrape information from Yelp!4:50
Yelp Advanced: Scrape the Top 100+ Pizza places in New York City8:03
Learn to crawl the top 100 pizza places in New York City using a crawler, train Import.io to collect name, neighborhood, address, and phone, and download the data.
Yelp: Learn how to Search and scrape Yelp results with an automated process!8:11
Learn to create a Yelp connector to search, save settings, and scrape results, train fields like restaurant and phone, then query with pagination and export data to csv or excel.
Learn how to crawl an entire website and capture over 40k products!6:26
Learn to build and train a web crawler to crawl an entire website, extract data from about 42,000 products, and export results to CSV, Excel, or JSON with pagination.
From Import.io to Google Sheets. Get it into the cloud!2:19
BONUS: How to extract specific data - Zip Codes in Yelp3:08
Learn how to extract a zip code from Yelp
Learn how to Install and Setup Ruby1:48
Simple Script: Learn How to Scrape Macys.com products12:20
Next Step: Macys.com getting a bit more advanced as we build your skills4:11
Advance your web scraping skills on macys.com with ruby by extracting product name, original price, and sale price into an array, then prep to export to csv.
CSV: Now you have Macys.com data learn how to put it into a CSV file4:41
Learn how to export scraped Macy's data to a CSV file by creating headers, writing items from an array, and saving the file for further analysis, including handling Excel quirks.

Walmart: How to scrape all pages from a product category on Walmart.com7:55
Spoof a site and look like you are web browser!2:27
Google + Mechanize = How to scrape search results from Google with Mechanize4:32
Learn to crawl google search results with mechanize in ruby by submitting the google form. Extract results via the search input q and page elements.
BONUS: Scraping YouTube's Most Popular4:53
Learn to crawl YouTube's most popular videos, extracting image links, titles, channels, published dates, and view counts using a reusable scraping template.
BONUS: Scraping Zillow by Zip Code6:44
BONUS: Using JSON to scrape Walmart's website8:58

Installing and setting up Python8:02
Simple Script: Learn how to crawl IMDB15:07
Learn how to submit a web form6:15
Learn to submit web forms with Python by inspecting form actions, filling input fields, and posting data with requests; build a scraper that handles AJAX and JavaScript.
Install Selenium and PhantomJS - Trust me you'll want these!3:31
Install Selenium and PhantomJS to crawl Ajax sites that load content on the fly, using a headless browser to fetch dynamically updated data and compare methods.
Yes, you can crawl JavaScript! Learn to scrape Ajax with Selenium11:36
Simple: Crawl and export data from Nordstrom.com into a CSV6:00
Learn to export Nordstrom product titles to a csv using Python, by reusing existing code, writing rows to a file, and validating the simple data pipeline.
Next Step: Crawl NFL data - Grab all the QB data from 2014!9:20
Learn to crawl NFL quarterback data from 2014 by fetching the full table with requests and BeautifulSoup, extracting all rows and cells, and saving the results to a data file.
Advanced: Export NFL QB data to a CSV5:39
Export Data to MySQL16:27
Explore setting up and connecting Python to MySQL, create a database and products table, and insert headlines to store data in utf8 with practical code examples.

Requirements

How to install software on their computer
Be able to understand a reasonable level of programming -- much of it will be explained
Follow step-by-step instructions

Description

Latest Update:
March 9, 2017 - Please note that import io is no longer a free program. They just recently started to charge for use of the platform. I would love some feedback about whether to keep the tutorials up for this portion or to remove and add more lessons from Python, Scrapy and/or Ruby.

_____________________________________________________________

You'll be web scraping in less than an hour!!

This course teaches you three ways to scrape websites:

Import. io is a software that you download and is a really quick way to start web scraping quickly. It is not without its limitations but depending on what you need or plan to do it could do everything you need. No programming experience necessary!

If you prefer to learn a little programming along the way, and aren't afraid to invest a little time, you can learn how to do web scraping with Ruby and/or Python. Both have their pluses and minuses but both can do just about anything you could imagine.

Even if you have never used Python or Ruby you can learn how to still use these programs as a web scraper. But if you are willing to learn you can scrape just about any website there is online!

Want to do some web scraping of super popular websites? That's how you will learn!

This course comes complete with code for every lecture for Ruby and Python!

You will learn to web scrape popular websites like:

Zillow
Craigslist
Yelp
IMDB
Wikipedia
Google
Macys
YouTube
NFL
Nordstrom
Wal-Mart
Twitter

You will learn to install Python and Ruby. You'll learn how to download what you web scrape into CSV files and with Python you will learn how to put your data into a database.

Other lessons include how to get around obstacles such as websites with ajax that load content dynamically. You'll also learn how to crawl multiple pages of websites and get all the data you need!

With Import. io you will learn how to download a website that has over 40,000 products on it. Yes, 40,000 products and you will learn how to scrape all of those products.

Stop imagining you can grab data off websites and use them for your next web project, you an do it now.

Please note: Software installation can be done for both Mac OSX and Windows. I use a Mac so it is more focused on OSX however instructions are available for Windows.

Update: January 17, 2016 - Added three Scrapy videos. How to setup scrapy and scraping Craigslist
Update: October 17, 2015 - Added Bonus lesson on extracting zip codes from Yelp
Update: September 15, 2015 - Added How to find JSON and scrape it on Walmart
Updated: September 1, 2015 - Three new Twitter API scraper videos. How to scrape user tweets, users searching for keywords and how to follow those users through Ruby/Twitter API
Update: August 28, 2015 - Added Wikipedia scraper and map

Enroll now!

Who this course is for:

Anyone who has an interest or idea where they want to utilized data from other websites

Web Scraping with Python, Ruby & import. io

What you'll learn

Explore related topics

Course content

Course Introduction1 lecture • 2min

Import.io - Free, Easy to use software built for scraping websites8 lectures • 32min

Import io - Download tool16 lectures • 1hr 13min

Ruby - Don't know Ruby? It doesn't matter because you will soon!6 lectures • 35min

Python - Don't know Python? It doesn't matter because you will soon!9 lectures • 1hr 22min

Scrapy - Use Scrapy built for Python to create your web crawler3 lectures • 29min

Bonus: Ruby - How to Scrape via Twitter API3 lectures • 24min

Requirements

Description

Who this course is for: