Building a Search Engine

Introduces you to Search Engine. Talks about difference between search engine and web search engine. Gives you an overview of World Wide Web. Provides definition and explanation to these topics briefly.

Festures of a good Web Search Engine:

1. Index large number of documents.

2. Prevents spider traps.

3. Ranking webpages using pagerank algorithm.

4. Understanding user queries.

5. Auto complete

6. Query clustering.

7. Better web scraping techniques.

Much more

This lectures gives you a brief history about search engines. It will help you to get motivated and get started with further lectures.

This lecture explains the difference between web search engine and web directory. Google, bing, yahoo are web search engines. But dmoz, ewd etc are web directories. Once upon a time Yahoo used to be a web directory.

This lecture gives you a difference between metasearch engine and a web search engine. DuckDuckGo is a metasearch engine but google is a search engine. Its easy to create a metasearch engine. Metasearch engine requires less resources and can be rapidly created and deployed successfully.

This lectures explains one of the most important feature integrated into most of the search engines called as social search. This features helps you find more organic results. And makes the search more meaningful. Integrating social search requires and a lot of users using your search engine and must have put up their personal information into your search engine. Social feature can also be enabled using ip tracking and understanding user queries. Social search is a application of machine learning.

Filter bubble is also a search engine feature. Social search looks for related documents and puts algorithms on the top of web graph. But filter bubble used clicks, location, bookmarks, favorites and many more things to rate and display documents.

Instead of trying to build a search engine from scratch its a good choice to use a open source search engine. It will save time and you will have your search engine build up quickly. There are a large number of open source search engine. Most of them are well documentated.

This lectures gives you a overview of common architectures used in modern search engines.

Components of a search engine:

1. Parser

2. Crawler

3. Indexer

4. Query Processor

Bad design of any one component will lead to a bad search engine. Every component needs to be designed carefully and tested for every situation before deploying.

A web crawler is a component of a search engine that downloads information from the World Wide Web. features of a good web crawler:

1. Downloads large number of documents.

2. Takes less CPU time

3. Consumes less bandwidth.

There are two types of redirects codes supported by HTTP protocol.

1. 301 -> Web server responds to 301 redirect if the file is moved permanently.

2. 302 -> Web server responds to 302 redirect of the file is moved temporary.

DNS caching a crawler optimization feature. It helps helps to tackle this problems.

1. Bandwidth

2. Time

A good crawler always fetches large number of web pages in less time and consumes less bandwidth.

1. Web pages are downloaded by multiple threads.

2. Web pages are downloaded by asynchronous sockets.

robots.txt and sitemap.xml are two very important files every website must include in their root directory.

1. robots.txt contains rules for crawler.

2. sitemap.xml provides the crawler architecture of the website web directory.

There are three different algorithms a web crawler must follow:

1. Selection policy

2. Re-visit policy

3. Politeness policy

user-agent field in http protocol is used by the crawler to introduce itself to the web server. Web crawler identification helps the web servers to take many major decisions.

Spider traps are different techniques by which a web crawler can be put into an problem. A good web crawler should prevent all kinds of spider traps. Everyday hackers find new spider traps techniques and you should be intelligent enough to catch them and rectify your crawler code to escape from the traps.

Parser is a component of a search engine responsible for web scraping. A good parser should always parse different types of documents like:

1. html

2. pdf

3. doc

4. ppt

5. many more

And also prevent spam.

Like:

1. invisible text

2. advertisement text.

Parse only what your users want. For example you are creating a mp3 search engine then no need to download and parse pdf files. You only need to download .mp3 url's. So this decision is very important.

Index is a data structure into which documents can put into quickly and also retrieved quickly. Index data structure is used in almost all types of application. A pdf reader indexer the whole document and finds the page number when your search for a word in the document. Similarly a search engine also indexes.

Index design factors:

1. Merge factors.

2. Storage techniques.

3. Index size.

4. Lookup speed.

5. Fault tolerance.

Inverted index is a index data structure most widely used into search application to search for matching documents according to text. Understanding inverted index is very important.

Sharding is the best technique to split the inverted index into multiple computers for fast and efficient querying.

This documents covers everything about apache solr in details. If you have any problem in understanding any topic please let us know we will make a video for that specific topic and explain it to you.