BYTE MARKS
Web crawler programs seek out Net nuggets

By Burt Lum

The thought of spiders, bots and Web crawlers can get a little creepy. Just imagine if you were a Web site and you had all these cyber-critters crawling over you. That is what is going on with your favorite site.

These Web crawlers, as they are generically known, are programs that go out on the Net and constantly look for key words from Web sites. They then return to their owner and populate a database. This database is what you end up searching when you go to search engines such as Google or Alta Vista.

While reading the Q&A for the Googlebot (google.com/bot.html), I was surprised to learn that Web sites are checked on average once every few seconds. That explains why a search result on one day can be different from the next.

One example of this was my recent search on the phrase "Web crawler." On one day I got a top listing for a page promoting photos of Russia — a gateway page, as it clearly stated on the opening page.

A gateway page is a trick of the trade to optimize positioning by the search engines. It also functions as a gateway or entry point into the main Web site. Gateway pages are populated with keywords and internal tags designed to register with the search engines.

It's a controversial technique. Some Web designers (rankwrite.com/gatewaymyth.htm) see it as an excuse for weak content. On a search, the next day the page was nowhere to be found.

Getting a top ranking on a search result requires not only coding techniques but also constant registration — for example on Google's index list at google.com/addurl.html — all behind-the-scenes stuff we take for granted most of the time. ;-)

Reach Burt Lum at burt@brouhaha.net.