Building a Search Engine

All algorithms and secrets reveled
5 reviews
TAUGHT BY
  • QScutter Tutorials a place to learn technology

    QScutter is a Indian based company that offers an ever growing range of high quality eLearning solutions that teach using studio quality narrated videos backed-up with practical hands-on examples. The emphasis is on teaching real life skills that are essential in today's commercial environment. We provide tutorials for almost all IT topics.

WHAT'S INSIDE
  • Lifetime access to 52 lectures and 5 quizzes
  • 5+ hours of high quality content
  • A community of 500+ students learning together!
SHARE

Building a Search Engine

All algorithms and secrets reveled
5 reviews

HOW UDEMY WORKS?

Discover courses made by experts from around the world.

Take your courses with you and learn anytime, anywhere.

Learn and practice real-world skills and achieve your goals.

COURSE DESCRIPTION

With "Developing a Search Engine", you will learn everything about Search Engines, even if you've never build one before!

The full course has several video lectures, divided into several chapters. Each chapter will give you a new level of knowledge in Search Engine development. We'll start from the basics of Search Engine development to more advanced and the most popular algorithms used now a days.

"Building a Search Engine" will give you a new perspective on how the Internet works and after you completed the course you will be able to create your own Search Engine with the latest technology and algorithms. Hope you enjoy!

NOTE: In order to keep you up to date in the world of Search Engine Development all the chapters will be updated regularly with new lectures, projects, quizzes and any changes in future versions of all the programming languages covered on the course.

Why Learn Search Engine Development?

The internet is the fastest and largest platform ever created for humans to learn, communicate, share, or create businesses of any kind, and all of this in just 15 years! It is estimated that in the next 2 or 3 years more than 80%%%% of the companies around the world will become internet dependent which will cause a huge demand for Search Engine developer in this market. As the World Wide Web grows Search Engines needs to upgraded proportionally.

Learning Search Engine Development will give you the opportunity to start ahead of other competitors by giving you the knowledge of the most recent web technologies and how to better apply them on your future projects. Knowing Search Engine Development will give you the ability to control and create anything on the web.

How this course will help you to get a Job?

At present the fastest growing technology in the Internet is Search Engines. Google makes thousands of changes every year and employees larger number of engineers who can make their Search Engine more efficient as the structure of the web is becoming larger and complicated. Other companies are employing Search Engine experts to optimize their websites to appear on top results in Search Engines.

I promise you would have never had such kind of learning experience.

Welcome to "Building a Search Engine"

    • Internet
    • OS X, Windows or Ubuntu
    • Over 52 lectures and 5.5 hours of content!
    • Cover all algorithms used in Search Engines
    • Introduce you to Big Data technologies and explain how to use them to build a Search Engine
    • Couse contents are updated regularly.
    • Reveles all secret spam fighting techniques used by GOOGLE
    • Needs to know basics of Networking
    • Needs to know basics of Web Development
    • Should be familier with atleast one programming language

THE UDEMY GUARANTEE

30 day money back guarantee
Lifetime access
Available on Desktop, iOs and Android
Certificate of completion

CURRICULUM

  • SECTION 1:
    Introduction to Building a Search Engine
  • 1
    What you will learn?
    06:48

    This lecture gives you an overview of the course. This lecture will tell you the things you are going to learn and also importance of this lecture in your life.

    Things you will learn:

    1. Search Engine architecture (crawler, indexer, query processor and parser).

    2. Web crawler algorithms and efficiency.

    3. Spider traps.

    4. Web scraping

    5. Spam fighting.

    6. Replication and sharding

    7. HTTP attacks

    8. Query understanding

    9. Spell checking algorithm and using apache solr.

    10. Auto complete

    11. Big data

    12. SEO

    Much more.

  • 2
    Prerequisites
    03:06

    This lecture tells you the what you need to know to understand this course:

    1. Basics of web development.

    2. Basics of networking

    3. Any one programming language

    4. Basics of data structures and algorithms.

    5. Basics of Database management systems.

  • SECTION 2:
    Getting started with Search Engine
  • 3
    What is a Search Engine?
    03:11

    Introduces you to Search Engine. Talks about difference between search engine and web search engine. Gives you an overview of World Wide Web. Provides definition and explanation to these topics briefly.

  • 4
    Features of a good search engines
    06:06

    Festures of a good Web Search Engine:

    1. Index large number of documents.

    2. Prevents spider traps.

    3. Ranking webpages using pagerank algorithm.

    4. Understanding user queries.

    5. Auto complete

    6. Query clustering.

    7. Better web scraping techniques.

    Much more

  • 5
    Brief history of Search Engines
    02:34

    This lectures gives you a brief history about search engines. It will help you to get motivated and get started with further lectures.

  • 6
    Difference between Search Engine and Web Directory
    02:20

    This lecture explains the difference between web search engine and web directory. Google, bing, yahoo are web search engines. But dmoz, ewd etc are web directories. Once upon a time Yahoo used to be a web directory.

  • 7
    What is a Metasearch Engine?
    01:58

    This lecture gives you a difference between metasearch engine and a web search engine. DuckDuckGo is a metasearch engine but google is a search engine. Its easy to create a metasearch engine. Metasearch engine requires less resources and can be rapidly created and deployed successfully.

  • 8
    What is Social Search?
    02:58

    This lectures explains one of the most important feature integrated into most of the search engines called as social search. This features helps you find more organic results. And makes the search more meaningful. Integrating social search requires and a lot of users using your search engine and must have put up their personal information into your search engine. Social feature can also be enabled using ip tracking and understanding user queries. Social search is a application of machine learning.

  • 9
    What is Filter Bubble?
    03:12

    Filter bubble is also a search engine feature. Social search looks for related documents and puts algorithms on the top of web graph. But filter bubble used clicks, location, bookmarks, favorites and many more things to rate and display documents.

  • 10
    Open source search engines
    02:52

    Instead of trying to build a search engine from scratch its a good choice to use a open source search engine. It will save time and you will have your search engine build up quickly. There are a large number of open source search engine. Most of them are well documentated.

  • 11
    How Search Engines work?
    04:42

    This lectures gives you a overview of common architectures used in modern search engines.

    Components of a search engine:

    1. Parser

    2. Crawler

    3. Indexer

    4. Query Processor

    Bad design of any one component will lead to a bad search engine. Every component needs to be designed carefully and tested for every situation before deploying.

  • 12
    Basic questions
    3 questions
  • SECTION 3:
    Web Crawler
  • 13
    Introduction to Web Crawler
    10:58

    A web crawler is a component of a search engine that downloads information from the World Wide Web. features of a good web crawler:

    1. Downloads large number of documents.

    2. Takes less CPU time

    3. Consumes less bandwidth.

  • 14
    HTTP 301 vs HTTP 302 Redirects
    02:44

    There are two types of redirects codes supported by HTTP protocol.

    1. 301 -> Web server responds to 301 redirect if the file is moved permanently.

    2. 302 -> Web server responds to 302 redirect of the file is moved temporary.

  • 15
    DNS Caching
    05:14

    DNS caching a crawler optimization feature. It helps helps to tackle this problems.

    1. Bandwidth

    2. Time

  • 16
    Multithreading vs Asynchronous Crawling
    03:36

    A good crawler always fetches large number of web pages in less time and consumes less bandwidth.

    1. Web pages are downloaded by multiple threads.

    2. Web pages are downloaded by asynchronous sockets.

  • 17
    Data Compression and Caching
    02:26
  • 18
    Webgraph
    05:27
  • 19
    robots.txt and sitemap.xml
    09:29

    robots.txt and sitemap.xml are two very important files every website must include in their root directory.

    1. robots.txt contains rules for crawler.

    2. sitemap.xml provides the crawler architecture of the website web directory.

  • 20
    Crawling policy
    10:00

    There are three different algorithms a web crawler must follow:

    1. Selection policy

    2. Re-visit policy

    3. Politeness policy

  • 21
    Crawler Identification
    04:19

    user-agent field in http protocol is used by the crawler to introduce itself to the web server. Web crawler identification helps the web servers to take many major decisions.

  • 22
    Crawling the deep web
    03:12
  • 23
    Spider traps
    03:58

    Spider traps are different techniques by which a web crawler can be put into an problem. A good web crawler should prevent all kinds of spider traps. Everyday hackers find new spider traps techniques and you should be intelligent enough to catch them and rectify your crawler code to escape from the traps.

  • 24
    Popular libraries
    05:01
  • 25
    Open source web crawlers
    01:37
  • 26
    Crawler questions
    5 questions
  • SECTION 4:
    Parser
  • 27
    What is a Parser?
    02:36

    Parser is a component of a search engine responsible for web scraping. A good parser should always parse different types of documents like:

    1. html

    2. pdf

    3. doc

    4. ppt

    5. many more

    And also prevent spam.

    Like:

    1. invisible text

    2. advertisement text.

  • 28
    What to Parse and What not to Parse?
    03:58

    Parse only what your users want. For example you are creating a mp3 search engine then no need to download and parse pdf files. You only need to download .mp3 url's. So this decision is very important.

  • 29
    Spam fighting
    06:47
  • 30
    Open source parsers
    01:51
  • SECTION 5:
    Indexing
  • 31
    What is Indexing?
    01:16

    Index is a data structure into which documents can put into quickly and also retrieved quickly. Index data structure is used in almost all types of application. A pdf reader indexer the whole document and finds the page number when your search for a word in the document. Similarly a search engine also indexes.

  • 32
    Index design factors
    04:41

    Index design factors:

    1. Merge factors.

    2. Storage techniques.

    3. Index size.

    4. Lookup speed.

    5. Fault tolerance.

  • 33
    Inverted indices
    05:22

    Inverted index is a index data structure most widely used into search application to search for matching documents according to text. Understanding inverted index is very important.

  • 34
    The forward index
    03:34
  • 35
    Sharding
    05:34

    Sharding is the best technique to split the inverted index into multiple computers for fast and efficient querying.

  • 36
    Index questions
    3 questions
  • SECTION 6:
    Text Processing
  • 37
    Text Analysis and Query Processing
    06:57
  • SECTION 7:
    Getting deep into Search Engines
  • 38
    PageRank
    09:18
  • 39
    Query Clustring
    04:41
  • 40
    Spell checking
    01:44
  • 41
    Spell checker
    88 pages
  • 42
    Questions
    3 questions
  • SECTION 8:
    Search Engine Storage
  • 43
    Parallel Computing vs Distributed Computing
    01:37
  • 44
    Memcached
    03:15
  • 45
    Google Big Table
    05:43
  • 46
    Google File System and MapReduce
    04:10
  • 47
    Apache Solr
    04:19
  • 48
    Storage questions
    4 questions
  • SECTION 9:
    Search Engine Optimization
  • 49
    Introduction to SEO
    01:50
  • 50
    White hat versus black hat techniques
    Text
  • 51
    On-Page SEO, TIPS and TRICKS
    13 pages
  • 52
    Off-Page SEO, TIPS and TRICKS
    12 pages
  • SECTION 10:
    Apache Solr
  • 53
    How does solr work?
    07:44
  • 54
    Configuring and launching Solr
    09:28
  • 55
    Solr Cloud and Multiple schema.xml
    03:36
  • 56
    Solr complete reference
    Upcoming

    This documents covers everything about apache solr in details. If you have any problem in understanding any topic please let us know we will make a video for that specific topic and explain it to you.

  • SECTION 11:
    Bye, Bye Lesson
  • 57
    Bye Bye
    01:02

    If you need any other tutorials regarding this topic please post them on questions section I will create and upload the videos as soon as I can.

    Don't forget to give a review.

    Thanks

UDEMY BY THE NUMBERS

5,200,000
Hours of video content
19,000,000
Course Enrollments
5,800,000
Students

RATING

  • 3
  • 0
  • 1
  • 0
  • 1
AVERAGE RATING
NUMBER OF RATINGS
5

REVIEWS

  • Noor Islam
    My dream course

    "very nice course. worth the money. lots of techniques explained well"

  • Sadaqat Ghafoor
    A complete reference to searach engine

    This course gives you a great knowledge about search engine technologies. I learned a lot of new things

  • Thomas Zhenhua Wang

    I only finished about 15% of the class and I no longer want to continue. When I sign up this class, I want to do something hands-on to actually BUILD a search engine. I think many others may think the same way, but after finishing 15%, all I learnt is some basic concepts and those concepts are so basic like this is my first time on the Internet. Also the lecturer is basically reading the slides or there is only a one-page slide with just a title on it.

  • Bader Alotaibi
    Unique and Awesome

    it's the only course on planet that teaches you how search engine works im so happy for taking this course the benefits of taking this class are: *understand how search engine works and that helps you rank your website on top of all search engines *learn how create your own search engine website *real live project that's coming very soon * the price is very very very cheap only $20 that can transform your online business 180 degrees ** Finally my rating for this course is 4.5 / 5