The Oxford Knowledge Company

This is the story (so far) of Dmoz, aka the Open Directory Project (ODP), how it has taken over from Yahoo as the largest Web directory and why we should all be using the Google-flavoured variety.

 

Everybody loves Yahoo, it seems - except people who can't get their sites listed in the world-famous directory. This apparent exclusivity has dogged webmasters for years, to whom a Yahoo listing can often be a major but frustratingly unattainable goal. In their defence, Yahoo editors have a nigh on impossible task keeping up with the submissions they receive. Yahoo never has been a directory of every site on the Web and it never will.

If you are keen for your own site to join the elite collection of sites listed by Yahoo, your best bet is to pay $199 for the Business Express Service and hope for the best. Read the small print - this service doesn't guarantee inclusion - just that an editor will consider your application within 7 business days.

But this is old news. In the past year, various developments of a rival directory, the Open Directory Project (ODP, also known as Dmoz or the Netscape Open Directory), have resulted in two major changes to the status of the Yahoo Directory. The first is that Yahoo is no longer the largest human-edited directory on the Web - ODP is. The second is that the ubiquity of ODP content is such that one could argue that it has replaced Yahoo in importance to web marketers.

This ubiquity stems from the development of the project under an "open source" agreement. Linux is the best known and arguably the most successful such project, wherein source code is freely available to software developers who can create their own versions of a product, perhaps enhancing it with new features, and then, if they feel like it, offering this new version back to the developer community. Internet veterans love this; for many, it embodies the free and generous spirit of the Internet which many feel is being eroded.

ODP started as a project named GnuHoo, (Hoo to create the parallel with Yahoo), then NewHoo, then the Netscape Open Directory. Volunteer editors would list and edit categorized web sites to produce a Yahoo-style searchable directory. Now a veritable army, the editorial staff (still all volunteers) stands at over 30,000 people.

The big break for ODP came when it offered its content free to any takers - and promptly appeared in Netscape's Netcenter portal. Lycos followed suit shortly, then Altavista. So, even though most web marketers still didn't know about it, a listing on ODP would now result in a top level appearance in search results at Netcenter, Altavista, and Lycos.

That was just the beginning. ODP claims that their data is now used on over 4,000 Web sites and numbers over 2 million pages. By comparison, estimates put the size of Yahoo's directory at around 1.8m (although Yahoo doesn't publish these figures, there are people out there working it out based on looking at the number and size of the categories…).

Netcenter, Lycos and Altavista weren't the only takers. Other search portals soon followed suit, most notably Google, to which we will return. And it wasn't just the portals. ODP content started to pop up all over the Web.

We at Oxford Knowledge first noticed this when performing one of those vanity tests where you search across many search engines to check that they all know about your Web site. Using our favorite and trusty search tool BullsEye, we found that aside from real results from our own Web site, we were also picking up references to our company on other sites, many of which on closer inspection turned out to be displaying some form of ODP results.

Our inclusion in ODP meant that we were being referred to left right and centre - on other sites whose ODP versions were being indexed by the major search engines.

Other companies with interesting Web technology began to use the freely available directory content to showcase their technology.

WebBrain (http://www.webbrain.com) and Oingo (http://www.oingo.com) are two of my favourite examples of this type of exercise. To search them is to search ODP - with slightly different flavours.

WebBrain allows users to explore a dynamic picture of related information, instead of searching through long lists of text. So for example, typing in "internet searching" quickly brings back some results, and a picture appears linking the phrase "internet searching" to related terms showing relationships between these terms in such a way as to suggest the next port of call - help and tutorials, perhaps, or newsletters and mailing lists?

WebBrain is a useful tool for people who search for information without a definite strategy, just allowing concepts to occur as potentially useful when prompted.

Oingo's approach uses the fact that different words often mean the same thing - and some words or phrases may be better search terms than others. It allows you to refine your original search, perhaps excluding some possible contexts and meanings.

My own impression is that like Web Brain, this is likely to work best for people who don't already know how to use online search terminology…which is probably most people.

The third example of a company using ODP content to showcase is Google, the upstart search engine company which has taken the world of search by storm. Google's PageRank technology means that it is the world's largest search engine to use the popularity of a Web site as one of the key factors in assessing its relevance.

If you haven't yet switched to Google for Web searching then you have treat in store. Google users may not have noticed it, but for the past six months or so, Google has included categorized results from the Open Directory.

Google's way of using ODP content is unlike any other major search portal's - firstly, it ranks results not just alphabetically (like Yahoo), but using the PageRank technology to list results in order of relevance to your search query. Secondly, Google's search index includes an index of the actual content of all pages listed in the ODP content. This means that when you search the Google version of ODP (known as the Google Web Directory), you search the titles, summaries and actual content of all pages listed (as is the case with the InPharm Knowledge Database). And since you can choose to restrict your search to a chosen category, this turns Google into a specialised search engine, focussing a precision instrument onto whichever portion of ODP that you choose.

Yahoo had better watch out. They already use Google technology to provide Web searching results. ODP content is larger than Yahoo's directory and rumour has it, getting as hard to be listed in as Yahoo. But for now, Web site marketers should take comfort from an ODP listing in the absence of a Yahoo listing!