This is such a broad topic that entire texts are dedicated to it, We will look at this from a perspective of information retrieval for search engine results. We will keep this as readable as possible, leaving out the PHYSICS of the matter and concentrating on the application and review.

For Search Engines and Information Retrieval Systems we look to rd's that make up a database with 75-200 million records, This is huge. The Shear numbers will lead to inaccurate results because the information generally is not fielded.

So to be accurate in Search Engine retrieval the algorithms being used must do one of two things

1. Build Fields based on a full text span
2. Validate fields with a full/partial text query

 #1 has been around for the longest, due to the nature of human speech/communication patterns this method becomes less and less reliable as the number of records in the database are increased.

internet With the growing number of web sites and documents on the internet today it is becoming harder to keep up with #1, to deal with this many search engines have instituted a partial grab of the document.

e.g. The first 5000 Characters. If you use a search engine like Netscape Compass Server you will see how easy it is to assign the value of characters reviewed / returned by the crawler.

So the record is not formed from the whole document but the patterns in the first few paragraphs.

internet In addition this formula removes noise words based on a noise word list. This is important because many new people to evaluating search will, with no training, recommend that you duplicate words meaninglessly, perhaps you repeat design 15 times in the first paragraph. This word will most likely be removed from the record formation as "Noise" adding no value to the description of the document.

#2 This option is not new to business and has been widely adopted as a way to handle the overwhelming number of documents on intranets and internets today. This option reads metadata, in HTML, XML and as simple XML or html records. This is simple enough in addition it then validates what the author or metadata entry person had entered about the document versus the logic in step one.

If one understand the above topics they are closer to higher search engine rankings and more importantly, the construction of good documents!