Information on the fly is heading for your desktop. David Tebbutt looks beyond the limitations of search engines to information as it happens, drilling down into the deep web, and introducing a feedmesh.

Searching is a fundamental part of your life, right? Whether it's searching third-party databases, your own intranet, the web or your desktop, you're probably never far from a search engine. But the problem with search engines is they look only at the past and they don't look at everything.

Someone at the Online Information conference said: "If it isn't in Google, it probably doesn't exist." While amusing, this is simply not true. The tragedy is that it's what a lot of internal clients believe.

On the desktop, indexing is pretty much real-time, and with third-party providers it is concurrent with the arrival of fresh information. With web search engines, it depends when the crawling and index updates take place. With systems such as Google, this can be weeks after new material appears, depending on the engine's algorithms for determining the relative importance of a website.

All the traditional search engines extract their information from the HTML of the web pages. They usually start from the root of the website; if a page isn't linked to, then it isn't crawled.

Blogs changed things because they started using RSS feeds to syndicate their contents. By reproducing part or all of the blog in XML and adding other contextual information, anyone could pick up the feeds and share them. A number of services sprung up around watching blog feeds. In search terms, most copied traditional search engines by indexing the content of the feeds.

One exception is PubSub Concepts (www.pubsub.com), which scans XML feeds and delivers hits to its customers in real-time, as they flow through the filter. Anyone can try it out, it's free. Put in a "when this happens" search argument and wait. It's similar ...