The Oxford Knowledge Company

Quick history lesson. Back in 1996 when the Web was but two years old, a Reuters-commissioned survey claimed that 'information overload' was a significant contributor to increased stress in the workplace. The problem had been noticeably intensified with the advent of email and the Web. Do you remember the Web in 1996? It was a much smaller place. And email? Those were the days when people still wrote each other excitedly saying "I've just started using my email account!" Yet, information overload was a problem even then…

 

Two years later the same team set out to follow-up on the previous study. This time, some of the findings indicated hope for the future - improved knowledge management systems, search tools and email-filtering technology had helped to get the problem under control, in the eyes of some respondents.

In 2001 another survey (from Roper Starch, commissioned by WebTop, a search engine company) shows that even after the millions spent on improving search technology, people aren't satisfied, with 71% experiencing 'Web rage' - even though 78% of respondents admitted that they 'often' or 'always' find what they were looking for.

Another recent study (by Keen.com/Lewis, Mobillo and Associates) shows that people ask an average of four questions per day that require them to go beyond their friends, family or sales staff for an answer. When real-life contacts fail us, apparently, 31% of us turn to Web search engines.

Well, I know I do. Used properly, a good Web search can provide an answer, hopefully the right answer, within seconds, to a very large proportion of the trivial questions you ask.

This is potentially great news for consumers. The main changes we'll need to see to satisfy this group are the following: search engines that respond to voice and search engines that you don't need to be in front of a computer to use.

For example, you're in the kitchen, you're cooking, you need a recipe. With current technology you aren't likely to want to leave the kitchen, turn on your computer and search for a recipe, so fridges or microwave ovens that are Web-search-enabled would be terrific.

The major search engine companies are currently working on technologies in this vein; Google and BMW have a joint venture to develop search-enabled cars and WebTop are working on voice-activated search, to name but two.

In business, however, most questions aren't trivial and they aren't so easily answered either. The main reason for this continues to be that the information lies in the Invisible Web - pages of information in database-driven Web sites that cannot be indexed by Web search engines. To access the Invisible Web, typically you need to use a private database search. This could be something readily open to the public - like Amazon's shopping search, or it might be something you need an account and password to access.

Let's imagine I need to find the latest sales figures of Vancomycin. Could I find this information on the Web? It would be great if I could, but I shouldn't count on it and I shouldn't waste too much time if I don't - this is just the sort of information that is likely to be found on the Invisible Web.

In an ideal world, a search on the Web will take me straight to the result if it can be found on the Visible Web, but if not, the next best thing is an indication of which sites have this information in their Invisible Web pages. The latter requires the owners of that sort of information to make at least meta-information of this nature available to the Web search engines.

So, on Google, a search for vancomycin AND "sales figures" results in 21 results, two of which are the same news story that happens to quote a figure of $300 million as part of an analyst's report on a company selling Zyvox (indicated for vancomycin-resistant organisms). Can we trust this, the one and only reference? Maybe, since the analyst's company is a well-known one (Robertson Stephens).

The other 19 results are useless or misleading. Worst of all, though, is the fact that not one result comes from a site which does have that sort of information, like IMS Health or BioScorpio, both sites that sell information about prescription drug sales. The fault is not Google's, but of these companies who do not even make the fact that they hold this information available to Google or other Web search engines. Why they don't do this probably has something to do with the fact that it requires extra effort and thought.

The future, therefore, should be in XML. When XML is commonly used, it will be relatively straightforward to make certain elements of the information you hold generally available, and other elements private, or available at a price.

However, before XML can solve the search problems, a number of major industry-wide agreements must be reached about browser standards and information standards. For example, workers within different disciplines will have to agree on the appropriate structures for their various documents. For the life sciences industries, that could mean agreements on formats for disease information, drug information or any other type of specific information. The regulators will have a field day!

Moreover, search indexing and search interface technologies must be developed to interrogate XML pages.

The latter could happen quite quickly, given the speed of development at companies like Google, Northern Light and FAST. The industry-standard agreements are another story. It could therefore be some time, but in the meantime it is definitely worth developing an interim solution.

So, if you have information that you want to sell, it could be that the best way to promote it is to tell the search engines that you have it. Invest in making a few pages that describe all the types of facts and figures you have for sale. And remember to submit these pages to the search engines!

Postcript: Here's a sobering thought. In the not-too-distant future, this article itself will be a high ranking result for any search for 'vancomycin AND 'sales figures'. Yet, what do I know about that…?