Four years ago I wrote about a metasearch project called Grub. At the time, it was basically being hailed as an open-source way to index the interweb, in real-time.
And it pretty much went nowhere for a number of reasons, one of which was that the parent company was free-riding off of its install base.
Another reason is that end-users trying to search the results had a subpar experience when compared to the dominant search engines of the time, especially the bigger players like Yahoo, MSN, and Google.
And now it seems as if a bit of fresh air has been blown back into Grubs little, presumably, unsalted heart.
Last week, Jimmy Wales (co-creator of Wikipedia) purchased the spider/crawler from its beleaguered parent company and intends on “truly” open-sourcing it this time.
In Om Malik’s write-up of this event, he mentions that building a scalable infrastructure was perhaps the prime culprit in Grub’s original demise; I still think that is merely one piece of the overall service.
No matter how big your index is, providing relevant results and presenting them in a useful manner still seems like the most important attribute desired by consumers.
And while a number of factors have gone into success, arguably one of the reasons Google dominates the current industry is because the developers have figured out not just how to index the web in a timely manner, but also how to present it in a productive, meaningful way — this despite the fact that both Yahoo and MSN have indices of comparable sizes and scope.
Here’s to hoping that Wales & Co. will not fulfill Santayana’s maxim. Heck, if they really are on top of things this time, I might even utilize the fat pipe I have here to help with the project.
Note: be sure to check out the Daft Punk song, “Technologic“