Crawlers (or bots) are used to assemble information available on the web. By using web site navigation menus, and reading inside and external hyperlinks, the bots begin to understand the context of a page. Of course, the words, images, and different knowledge on pages additionally assist search engines like google and yahoo perceive what they’re