My Index is Better Than Your Index
While this may well be the case, as ‘my index’ is a parallel processing real-time powerhouse of IoT, It is completely irrelevant to this particular discussion.
Enterprise Search, and Search in general, is said to be reliant on a good index. So much so that enterprise search vendors charge hundreds of thousands of dollars each year to help you maintain the index. So it must be good!
This may or may not be the case. I am definitely not going to go into the merits of who has and who has not got a good index, because in the world of enterprise search, it should be irrelevant. But let’s take a logical look at it from a completely different viewpoint.
Up until recently enterprise search was designed around the index and tagging the data enclosed within it. The index was not completely self-created. It required external human intervention to be able to sort and categorize the information it holds. This means that from the moment the index is live, the content is skewed to be in favor of the creator of the tags. Their searches I am sure will be relevant, while all other users have to put up with almost correct. This is why repeated studies have found that 20% of a knowledge workers time is wasted looking for information: Not finding, looking!
The Search Algorithm
This has been a highly publicized area, but in reality most enterprise searches use a basic Key Word algorithm and place the emphasis on the index to bring back a result. Microsoft have even gone back to making Key Words the predominant search inside SharePoint 2016. A Key Word search is exactly that. Key word. Put in words, find them, and then sort the results based on a false notion of the more times that a word is found, the more relevant the result must be. This supposition is just plain wrong and the uptake by users of enterprise search applications would seem to me at least to be proof of this.
It is time that the algorithms changed. Users should be at the forefront of all searches. Not just ease of use, but results that matter, are correct and that mean that the wasted time searching is drastically cut back. The index is irrelevant, it is the search algorithm and the search query that make or break a result. The index is just there to enhance the speed of a search. If you query any index correctly, you will get back a good result. Search a great index badly, and you will get back a bad result.
It is time for the industry to take a long hard look at itself. Who are they working for? The answer should always be their user base.