Problem Description and Related Work

Examples of Database services on the Web included Yahoo [11], Lycos [9], and Infoseek [8]. These services provide Web users with a way of finding information on the Web. They can be distinguished by the way they collect data for their database.

• Robots - The most common and most effective way is by using "robot"s. "Robots" are programs that repeatedly request documents from the Web, and obtain data such as keywords or titles from these documents. Services such as WebCrawler, Lycos, and Infoseek use this method to collect data.

• User Entry [11] - Another method is via user entry. Services such as Yahoo allow users to add their own document information into the database.

• Selective Collection - This is a newer method of collecting data. Similar to the "robot"s, this service will collect indexes from subscribed servers or users. The indexes are constructed by the users or server managers, and are in a format recognised by the service. The indexes will be updated whenever it is necessary. An example of a service that uses selective collection is AliWeb. A disadvantage for selective collection is the these databases are limited to the sites that subscribe to it.

Problems associated with the methods above include updating databases and access to these services.

Updating the databases

The "robot"s have the biggest problem with updating. Documents are being put on the Web, taken out from the Web, or modified, every minute, but it can take several weeks, or months for the "robot"s to come back to a document for updating, and the data might have been changed long before the "robot" could update its information. As the Web increases in size, these robots will need to cover more ground, as a consequence the length of time needed to update data in their databases will increase dramatically.

Another problem with the "robot"s is that it is possible for these "robot"s to request a lot of documents from one server, overloading the server, although this problem can be solved by having conditions imposed on robots, so they do not request more than one document at any one time.

Updating in a user entry based services can also be a problem. Such a service has to keep track of the data entered, so that it can update the database when a document becomes invalid. This can take a long time.

Access databases

Another problem with these databases services is gaining access to them. Because these databases can be used by everybody on the Web, there are times when the number of people accessing these services reaches numbers beyond what they can handle. As the number of Web users increases, this problem will grow worse.

Sometimes a database service will "mirror" itself, thus allowing search processes to be done at other sites. But typically this solution will only last for a month or so before this "mirrored" database starts to have the same congestion problem.

Another problem with these databases is their location. Most of these databases are located overseas, so accessing them to look for something is like having to fly all the way to a far away place to use their database. This is a waste of Internet resources.

• The Cause and A Solution
  • Searching through a distributed system
  • Updating the local index
  • Updating the List