A search engine is made up of many complex parts, which vary from search engine to search engine. Although there are two main parts, which all search engine have to have to call them self’s a search engine. Firstly a Spider/Crawler which we covered earlier, is what goes out in to the web and gathers the Data. Now logically something has to happen to this data for it to provide accurate results against a search engine query (what you type in the search box), this is where the indexer comes into play. The indexer takes the data collected by the spiders then analyzes, stores and catalogues the information. This is done so there is a set structure, for retrieval against a given query. Think of the indexer as a Super Liberian, taking a set of books analyzes them for there topic then storing them into relevant place on the shelf’s. They also place them within a category in a computer so when a customer ask for a book on a certain topic the Liberian can type the information in to the computer and give the exact location of the said book or books based on the topic requested.

That was a very loose interpretation of what a indexer does. In fact what a search engine indexer does is a lot more complicated than that, especially when it comes to cataloging information. First because of the shear scale of the information it has to categorize, as this much information and many process has to be undertook to obtain relevance of the said information to the query.

The major issue when it comes to relevance is the current technology the web is based on which is its natural language. Unlike are real world Liberian, a computer can not differentiate between the mean of the same word. For example if your search for “Polo”, a search engine can’t determine whether your searching for the sweet, the car type or the game with out more information, hence if you enter that into the search engine you will get an array of result based on different topics. This is the next stage of web semantic index coming to us in 2013, where the indexer will be able to differentiate between the above example. Don’t ask me how because it is still in the development stage but it is coming.

Related posts:

1. Keyword Matrix and Search Engine Optimisation.
2. What is a search engine?
3. The role of the search engine
4. What does a search engine consist of?
5. Search engine Optimisation (SEO) 101 Series Meta Tags