Professional Web Scraping with Java
- 1.5 hours on-demand video
- 2 articles
- 13 downloadable resources
- Full lifetime access
- Access on mobile and TV
- Certificate of Completion
Get your team access to 4,000+ top Udemy courses anytime, anywhere.Try Udemy for Business
- Have a solid understanding of web scraping with Java
- Beeing able to scrape practically any web page (static AND dynamic / AJAX) though you learn the concepts behind web scraping
- Download, parse and extract data from websites with Jsoup
- Call web APIs in Java with Unirest
- Export your data as CSV or JSON
- Build web scrapers that stay undetected and do not get blocked or banned
We introduce the Jsoup library. It helps downloading, parsing and extracting elements from a page using CSS selectors. It has a lot of similarities of jQuery - so prior knowledge of jQuery is helpful but not necessary.
Also we develop a simple example program using this library...
In this lecture we discuss what a dynamic / AJAX web page is and how it differs from a static one.
You will learn how to become invisible and hide traces of beeing a web scraper. This will help you avoid getting blocked or banned.
Bonus: in the resource section you find an undercover web scraper that builds upon the google scraper from the previous lecture. You can use this as a foundation for creating your own scrapers....
In this lecture we use the same example from the previous lecture and scrape data from peoplefinders.com - except this time we do it using Proxycrawl so you can see and understand the difference between the two approaches and won't be distracted by using another website for demonstration.
Also included is the full source code with a ready to use class to integrate in your own scrapers if you like to use proxycrawl...
- You should already be familiar with Java and Maven at a basic to medium level (the course will not show you how to setup Java, Maven or an IDE)
- You should be familiar with HTML/CSS and know how to use your browser's developer tools
- You should know about CSS selectors, though we use them for scraping static web pages
- Prior knowledge of jQuery helps you getting started faster with Jsoup, though this is not required
- You should know what a web API and AJAX is (basic level is enough)
In this short and concise course you will learn everything to get started with web scraping using Java.
You will learn the concepts behind web scraping that you can apply to practically any web page (static AND dynamic / AJAX).
We start with an overview of what web scraping is and what you can do with it.
Then we explain the difference in scraping static pages vs dynamic / AJAX pages. You learn how to classify a website in one of the two categories and then apply the right concept in order to scrape the data you want.
Now you will learn how to export the scraped data either as CSV or JSON. These are some popular formats that can be used for further processing.
Unfortunately many websites try to block scrapers or sometimes you just do not want to be detected. In the section going undercover you will learn how to stay undetected and avoid getting blocked.
At the end of the course you can download the full source code of all the lectures and we discuss an outlook to some advanced topics (private proxies, cloud deployment, multi threading ...). Those advanced topics are covered in a follow up course I am going to teach.
Why you should take this course
Stop imagining you can scrape data from websites and use the skills for your next web project, you can do it now.
- Stay ahead of your competition
- Be more efficient and automate tedious, manual tasks
- Increase your value by adding web scraping to your skill set
- Anyone with an interest in learning web scraping and understanding the concepts
- Anyone who likes a short and concise course
- This course is NOT an introduction to Java
- This course will NOT show you how to setup your development environment
- This course is intended to get you started with web scraping. Very advanced topics (e.g. private proxies, cloud deployment, multi threading) are discussed but not implemented in this course. I will do an an advanced / enterprise level course on this separately...
- Windows, Mac, or Linux PC