Google! Here's how to achieve 'really intelligent search'

Peter Cochrane examines how to bring AI to bear on search engines - but do Google et al really want to make search more efficient?

At a modest estimate, I spend about 80 per cent of my time trying to find data and information, with another two-to-five per cent invested in validating, fact checking and analysing what I have found to make sure it is reasonable and meets my needs.

This leaves only around 15-to-18 per cent of my time to do something useful and to add value in some way. But it doesn't have to be this way - it really doesn't.

Ten years ago we didn't have what it takes to create a solution, but now we do. Let me demonstrate, using three well-known and popular search engines. This is the number of publications I find when I search these general categories:

By any measure this is both quite remarkable and generally useless at the same time! The search is completed in less than one second, but how do I find what I want in a listing of 787,000 publications let alone 313,000,000? The best my browsers can offer is a handful of the most popular read or retrieved publications out a massive stack, and as a rule they are wholly irrelevant to the work I am pursuing at the time.

So now the real fun begins: refining the search terms and homing-in on the likely areas, one layer at a time as I to try and find the gems of information I need.

This is where my time evaporates: Poor titles, reference search terms, tagging, difficult to decode abstracts and so on all add to the complexity. This is so frustrating when under pressure to get the job done and to deliver and, more especially, when document and fact validation also has to be completed for every item located. For sure, I am not alone in my frustration. Most professionals suffer all this to one degree or another, and we all need help - AI help.

Start the AI engine

What is now feasible is no more than an engineering problem and does not require any scientific or big breakthroughs.

An artificial intelligence (AI) engine that collects and analyses data about what I am working on by observing what I type, talk about, read, and analyses my models and searches would have all the clues and necessary data to help search out all the ‘hard to find materials' I really need.

And my machine could be doing this 24/7 so that when I do search ‘general AI', for instance, I immediately get the five most pertinent publications, slide sets, graphics, animations and abstracts. This would be a great start, which at a later stage may well become more of a ‘work partner' than just a ‘search buddy'.

When will this happen? Not for some time I think. The problem is not the technology, but the commercial model. Search engine providers make money on advertising and, thus, the number of clicks a user makes. The last thing they need is more search efficiency and users spending less time on their service.

But I think they are wrong. My output would certainly improve and, I suspect, I would end up spending more time using their service.

Destroying a successful business model is hard, but it is what IT and the internet does and, if today's providers don't explore how they can use AI to provide better search, I think we can safely assume that some new AI ‘assisted engines' will emerge and quickly enjoy the kind of rapid adoption that Google did back in 1999 and 2000.

This is made all the more certain as AI analytics is deployed along with sensor networks and instrumentation. Probably the biggest opportunity here is the creation of intelligent desktops and work spaces powered by AI, including intelligent search and analytics.

Peter Cochrane OBE is an ex-CTO of BT who now works as a consultant focusing on solving problems and improving the world through the application of technology.