Udemy
    •  
    •  
    •  
    •  
    •  
    •  
    •  
    •  
Subscribe
Turn what you know into an opportunity and reach millions around the world.
Learn More
Your cart is empty.
Keep shopping
How to Build a Web Scraper Bot Application with Golang
Rating: 3.6 out of 5(12 ratings)
120 students

How to Build a Web Scraper Bot Application with Golang

Build Go application using MongoDB, creating telegram channel, scraping jobs from popular sites, send jobs to telegram
Created byYEVHEN MAKSYMENKO
Last updated 12/2020
English

What you'll learn

  • Architect large, scalable apps using a golang language
  • Solve problems with scrapping data from jobs sites
  • Build a Golang app to send data from Mongo DB to Telegram channel
  • Develop a service app with Golang and MongoDB
  • Leverage your Golang skills to build a complex web app
  • Write nothing but production-level code. No cutting corners!

Course content

7 sections • 30 lectures • 1h 46m total length

  • What Application Does ?2:29

    Build a Golang web scraper bot that gathers job posts from Stack Overflow, stores them in MongoDB, and broadcasts updates to a Telegram channel with title, company, and position.

  • Create entry endpoint3:54

    Create an entry endpoint for a Golang web scraper, establish a project structure with feeds, and define a public feed interface to manage per-site routines and Telegram broadcast.

  • Create a public feed interface3:29
  • Model Post strucuture3:19
  • Create MongoDB connection3:36

    Create a MongoDB connection in Go by importing the mongo package, connecting to localhost, selecting the database and collection, and validating the session for posts.

  • Define base methods Model Post2:46
  • Implement base methods Model post3:56

    Implement base methods for the post model and support bulk posts, with created time tracking and status flags, while testing edge cases.

  • Base feed: create base methods3:58
  • Create parse page method4:44

    Implement a parse page method in a Go-based web scraper, building a clear document context and reliable text extraction to drive data collection.

Requirements

  • Basic knowledge of Golang and Mongo is required
  • You must be familiar and comfortable with the command line

Description

Scalable, production-ready code? Its here!


How This Course Works

This will expose you to challenging problems and clever solutions when handling subjects like parse - scrapping jobs from sites.


What Technology You'll Use

We are building a backend application, we'll use Goalng - Go. Data for each service is held in either a Mongo database.

This is a scary list of technologies! Not familiar with some of these? No problem! The course is built assuming that you only know the basics of Golang and Mongo. No other knowledge is needed - you will learn everything you need to know.

What You'll Be Able to Do

By the time you complete this course, you will be able to:

  • Architect feeds parsing application

  • Determine whether your app is a good fit for a select area of webcams to watch for objects

  • Organize and enhance the reusability of code in large projects


What You'll Learn

An absolute incredible number of topics are covered in this course. Here is a partial list of what you'll do:

  • Create a telegram channel

  • Posts messages to telegram channel

  • Store parsed data to mongo db

  • Used popular solution to parse data from sites

  • And much more!

Who this course is for:

  • Golang engineers looking to parse data from sites and sending to Telegram channels
  • This course is *not* designed for sysadmins focused on infrastructure deployment

Report abuse