Search directories are a hugely important part of the web. It is truly impressive what you can turn up by searching for keywords on just about any subject. There are really two types of web index, the search spiders or worms which crawl over the web tracing and indexing URL's they find starting from those submitted, and the structured indices where each URL submitted is checked, and classified.
Despite constant updating, all of these directories index only a small fraction of the whole web which is growing at a phenomenal rate. The spiders seem to have a hard time just keeping up with the URL's which are submitted to them. I used to put a lot of effort into submitting the URL's of my pages to these directories and following how they were taken up. It always takes a few weeks for them to come round and even after a few months I had only managed to get about half my pages indexed on any of them. Then one weekend almost all my pages disappeared from some of the larger directories. I suppose that the CompuServe web page server must have been down when they visited so the URL's got wiped off.
Maintaining these directories must require a huge effort so those which work deserve applaud. When the web was new several of these services poped up. Now they have almost all either gone commercial or fallen into disuse. The best of what remains are listed here. Perhaps market forces will be sufficient to encourage the invention of a technique to make these directories more complete.
Run by Digital on an enviable array of hardware, this is certainly one of the best search indices with a huge fast database and excellent advanced search features. It also has an extensive archive of usenet news. The ability to search specifically for links to a certain page combined with phrases, wildcards and other specifiers seems to suggest that it was designed by someone intelligent.
![[ALTA VISTA]](alta6.gif)
This is the newest search robot and it claims to have the most complete database. It is produced by the successful e-zine HotWired togther with Inktomi. After a few teething problems it now looks pretty good and its web database is very extensive. I often use it to find links which Alta Vista may have missed. The ability to select a search for words, names, phrases etc is good for beginners but it is a pity these things cannot be combined in the way they can on Alta Vista (if you know how). HotBot have recently added a usenet search.
Another very good and well maintained searchable index which usually finds what you are looking for. This one can also search recent usenet articles but not for many groups. It lacks advanced search options but has a useful collection of short reviews. For some reason the search field clears each time you go back to it which can be a pain if you wanted to edit it. Excite seems to be exceptional quick at picking up URL's which are submitted.
Another large database which works well. The find similar pages feature is brilliant. It has a very good searchable usenet archive and a categorical directory
Another very large database but it sometimes comes back with too many junk links. it is a useful place to search and is kept up-to-date. They have now teamed up with Point who provide the "top 5%" directory.
This old spider is still very popular and well maintained. Not a bad alternative although it is falling behind the others on features. Recently it has been spruced up but I think the database has some way to go before they can catch up with the competition.
The original organised web directory is still one of the best ways to start looking. It can be searched too but is much more selective than the search engines. I find that good pages submitted here do now get taken up after just a few weeks, and pages in the index get plenty of visits as a result. Their recent success on the stockmarket will no doubt give them cash to expand and will also be an incentive to others to improve their act.