Site Overview:

Read me

Site Resources:


Digital Music

Digital Music

Digital Workforce

Search Engines

Web Browsers

MP3 Players

Scour Exchange




:::: :::
::From :::
the vault:::

MP3 Law Proposed

Napster & Democracy

Death of Web Design

Bagging on
E-Commerce Trends

The new napster


Are the Users


:: : : :

digital music :
file sharing : gnutella : napster
subscribe napster newsletter : more news More:
archives : submit story

Directories and Search Engines
Since the beginning, we've tried to catalog and organize the Web - hardly an easy task when you consider the phenomenal rate at which the Internet has grown.

And have we succeeded? Unfortunately for the most part the answer is probably a resounding no. With the exception of standouts like Google, Yahoo and the Open Directory Project using most search engines and directories is a frustrating experience, one which causes plenty of people to simply give up.

Happily this seems to be changing, as companies like Google bring more powerful search algorithms to bear on the problem.

Best of Class: Search Engine - Google

Developed by researchers Larry Page and Sergey Brin at Stanford University, Google has impressed with results far more accurate than first-generation search engines. A simple, straightforward interface is a welcome relief from the hectic, increasingly commercial services offered by competitors.

In early 2000 Yahoo dropped well-regarded Inktomi as its search engine provider in favor of Google, as good a sign as any that this search engine is coming into it's own.

If things like the theory behind search engines interest you, Page and Brin's 1998 outline for the project is archived in full at the Stanford web site.

pluses: relatively high accuracy
minuses: none that we can think of.

The San Francisco Chronicle ran a good story on the challenges Google faces in the future. Read it here.

Upgraded to Best of Class [Directory] - Yahoo

Its easy to forget that when Yahoo isn't offering you free e-mail, streaming stock quotes, point and click e-commerce and a thousand other services it is also, just as it was in the beginning, a web directory. And a very useful web directory at that.
Unrivaled in size and in general all around comprehensiveness, Yahoo operates a international network of directories, which taken together, are as close a thing as we have to an accurate map of cyberspace.

A dedicated team of editors does an excellent job of maintaining Yahoo's system of categories, a fact that becomes readily apparent if you compare the site's listings with competitors like Looksmart and For an opposing viewpoint read last year's review. Have we learned anything since then?

Er, maybe a little bit...

pluses: Bigness
minuses: Bigness. Annoying ez-click branding.

Underground: NewsSearch.Userland

Although it could easily be argued that this search tool isn't really a search engine at all, in the sense that most search engines index the entire web or large portions of it, NewsSearch.userland is still a valuable addition to your arsenal of web power tools. Described by its creators as a search engine with an editorial focus, Newssearch combs a tightly focused group of cool meta resources and weblogs including Scripting News , Tomalak's Realm and Hack-the-Planet for links to news stories at the major news sites and smaller web sites.

If you're interested in a particular story in the news, like say napster or the presidential election, a search here will probably turn up dozens of links to important stories, all pretty tightly controlled for quality - you'll almost never follow a link here to pr drivel or mush. You'll also be exposed to dozens if not hundreds of smaller sites it would have taken you a long time to find otherwise. Unfortunately, only links through 1999 are archived, although hopefully that will change.

pluses: the human factor filtered, editorially-focused search blows away many advertising driven news sites.
minuses: limited archives - expired links

Is bigger better? AltaVista
An early leader among the major search engines, AltaVista has clearly lost major ground in the past few years. If you have good eyes and plenty of time to kill you may want to try making the switch, because when it comes down to sheer numbers - AltaVista is the clear winner.

Until quite recently this search engine had more web pages indexed than any other, the reason AltaVista routinely returns more results than other search engines.
Unfortunately this vast ocean of data makes using AltaVista efficiently very difficult, if not impossible. In 2000 Altavista announced it would begin including paid placements in it's search results.


Now used primarily by professional researchers in business and academia - this search engine does a remarkably good job of turning up articles in magazines and other publications. Unfortunately, you'll have to pay to download most of the articles you find here.

The good news is much of the material is worth it. On the other hand, if you can afford to pay these rates, you can probably afford Lexis-Nexis.


Doing other things - Inktomi

Originally a pure search engine company, Inktomi's business has grown rapidly in other directions to the point where the search side represents only a fraction of the company's business. Inktomi's network management software, which allows web sites to ease bandwidth problems by caching frequently accessed content, has became the main focus here.

Makes you wonder how they'd do if they concentrated on that little search engine.

Downgraded in 2000 -

Relevance-based searching promised to deliver more meaningful search results by indexing web site popularity. And that's exactly what DirectHit was claiming last year, when this search engine first started getting attention.

Unfortunately, DirectHit's performance has not yet lived up to its creators claims. Under new management following the company's acquisition by AskJeeves, DirectHit has become much more visible but hardly more accurate.

The results DirectHit returns are supposed to be more useful because they are indexed by popularity. Unfortunately, it turns out popularity may have little to do with the quality of search engine results. Unrelated and low-quality results seem to be the norm here. Stick with Google.

Pretty web site though.

Natural Born Loser: Askjeeves

Everybody has to have a gimmick, the old saying goes. AskJeeves has two. The company's British butler mascot, may be the most recognized corporate mascot since Joe Camel. The natural language search queries AskJeeves offers are an obvious commercial draw, even if their effectiveness is sometimes open to question.

For many people natural language queries like "where can I buy seedless grapes in San Jose?" are more...well..*natural* than the complicated Boolean phrases other search engines require, one reason this search engine remains popular.

AskJeeves has had as much success as anybody forging alliances with content providers and web retailers, meaning you're likely to run into the little cartoon butler frequently in your web travels.

ClueTrain Manifesto author David Weinberg writes "I don't know what AskJeeves is doing, but I know I don't like it."

pluses: natural language queries.
minuses: paid placements. limited scope. jeeves.

The first search engine to go completely commercial, Goto has both won fans and provoked screams of outrage from purists. It has also inspired imitators like

Its worth noting, however, that most of the fans work for Goto's cash-strapped search engine rivals. The company charges advertisers a flat fee for ever surfer who clicks on a link. Thankfully, (for the time being at least) Goto makes no pretense of hiding this fact from visitors. Results clearly indicate the rate an advertiser will pay if you click on a link.

Useful, as more than one critic has commented, if you're interested in studying the state of advertising on the Net or if you're a site owner with deep enough pockets to pay for the traffic Goto seems to be able to come up with.

A little less useful if you're searching for meaningful content.
Same idea. Different advertisers.


It is unclear exactly what's going behind closed doors at looksmart. One minute they're a directory, the next they're not.

As directories go, Looksmart is a relatively useful resource, although nobody in their right mind would dare to compare the company's offerings with either Yahoo or the Open Directory Project.

There is a perception that this is a company that has lost both considerable momentum and direction, a company that has become preoccupied with finding new ways to generate revenue. Visit Looksmart and you'll find plenty of innovation - most of it aimed at finding ways to squeeze more money out of advertisers.

There was a time when Looksmart was an up-and comer.

That time has apparently passed.

Postcript: LookSmart recently changed it's slogan to "The Quality Web Directory." A surefire sign that the directory's editorial staff may finally have been overrun by the marketing department.

For a revealing look inside the company from the point of view of some of the company's disgruntled employees click here

Fairfax IT, an Australian web site, also ran a good piece on the company's problems.

The site formerly known as the Mining Company changed names last year, switching to the focus group-approved and infinitely hipper takes what seems to be a more human and personal approach to organizing and collating the web than many of the other directories. Each topic the site covers is assigned to an expert guide responsible for that section.

The guide pulls together relevant links and organizes the section using templates provided by the company. The results are fairly predictable. is uneven. Some categories are strong and others are weak, depending on the amount of effort the guide responsible feels inclined to put in. Yet, on the whole, there is something about the human quality here that makes more pleasurable to browse than the endless maze of hierarchies at Yahoo.

Character, more than anything else, is the trait that most directories and search engines lack - the missing element that makes surfing many sites about as stimulating as reading a phone book. You'll find it here - in varying degrees.

pluses and minuses:
even out.


| Add URL || subscribe | feedback | digital music | file sharing | gnutella | napster | archives
© the 2000, 2001