Don't Let the Web Bugs Bite: How to Stop Spidering.

Spiders and other

automated software agents are sent out by their owners to "search and retrieve"

useful information from the internet on behalf of their owners. They are

invaluable tools for ensuring search engines are kept up-to-date and for market

research purposes. But they are also used by companies to dig the dirt on their

rivals.

To what extent is spidering legal and

what can be done to prevent it? Solicitor Raffi Varoujian of City law firm

Field Fisher Waterhouse discusses the options.

Spiders can be assigned a specific set of tasks to ensure that they only

bring back information which is useful to the sender. This is done by providing

them with either a URL or directions to a server's file directory. Once there,

it collates data from the chosen location and via any hypertext links and

returns them to its owner.

Most businesses actually welcome visits from spiders which are being used

to modify search engines. Some even lure them there by using metatags which

direct them more easily to their websites in a bid to increase their chances of

being favourably ranked by them.

However, the intentions of spiders are not always so honourable. They can

be used by competitors to find out everything, from details of a rival's

products and pricing to their annual profits and identifying failing areas of

their business - a worrying prospect, when one considers that a competitor can

send out thousands of spiders a day.

Antidotes

The

issue of exactly what legal measures one can take to prevent spiders entering a

website have been discussed at great length in the US where guidance is

…