Some of my thoughts on the state of web search (and a comparison with search in the enterprise) are posted here on my blog.

In the short term, I think keyword search + Wikipedia is working reasonably well for web search. In the long term, I’d like to see an approach to web search that overcomes the adversarial issue of spam without resorting to obscurity. But I don’t think anyone has cracked that nut yet.

Wikia is really painfully slow. Hopefully they will fix that, but the results were also not “stellar” especially if you tried whole phrases, rather than single terms. Mahalo suffers from this as well. Mahalo can never be your only search engine because if you take any phrase and add the word Competitor, or clone to it you get nothing. You can’t do a search for “Whole Foods Competitor” or “iPod Clone” or even “php foreach” and likely you never will be becasue you can’t hit the edge cases.

Instead focusing on creating results tuned for huge classes of topics is where I am putting my time. Today I added 24k “modifiers” to our search for the top prescription drugs of 2002. I’d have liked this to be of 2007, but the truth is I take open data where I can get it, and there was a government report that had this data and so I manipulated it, and built rules around it. Tomorrow who knows what the next 10k rules will be for, but my changes effect 24k pages I know about, and 240+k that I don’t about drug interactions, and disease results. Not one entry at a time 100’s of thousands at a time.

Some interesting points. Thanks.

I don’t agree with a lot of Google’s approach to relevance ranking (see my brief debate with Amit Singhal). But I think Google is right to avoid tweaking with individual queries. Search tuned on a per-query basis is not a scalable approach, and at best reduces to the Wikipedia we already have.