Comments on: The Future of Search http://aabs.wordpress.com/2007/09/04/the-future-of-search/ Closed weekends and holidays. Sat, 05 Jul 2008 04:33:50 +0000 http://wordpress.org/?v=MU By: Andrew Matthews http://aabs.wordpress.com/2007/09/04/the-future-of-search/#comment-14567 Andrew Matthews Thu, 06 Sep 2007 00:17:43 +0000 http://aabs.wordpress.com/2007/09/04/the-future-of-search/#comment-14567 <i>I just received this comment via private email from an architect on the search team within Microsoft, who has asked to remain anonymous, but who has some very interesting ideas:</i> It will take some time to digest this article. I will comment on one point right now, however: <blockquote>Search engines assume that if many others link to a page then it must be valuable and so rank it higher.</blockquote> This is way off base. Ranking is an extremely complex process, taking into account a wide range of variables, including inbound links, meta data, heuristics, the quality and nature of inbound links (just speculating, a link from Wikipedia might rank very highly compared to other links), and so on. There are also algorithms to detect bot-generated content. Query logs (what people search for and what they click (and whether they stay there or quickly come back)) are also bountiful sources of ranking algorithms. And these are just the obvious things that the big engines figured out years ago. The most important challenge -- and competitive playing field -- is capacity, and not just to hold the web, but to compute on it. We hold tens of billions of documents, so even URI-only graph algorithms have to be distributed. That's why Google and Microsoft (and all the other transnats) are buying land near power stations as fast as they can. The computational platform itself, is also a key strategic piece. Whoever can more elegantly express and efficiently execute computations over petabytes of data has a significant competitive edge. I just received this comment via private email from an architect on the search team within Microsoft, who has asked to remain anonymous, but who has some very interesting ideas:

It will take some time to digest this article. I will comment on one point right now, however:

Search engines assume that if many others link to a page then it must be valuable and so rank it higher.

This is way off base. Ranking is an extremely complex process, taking into account a wide range of variables, including inbound links, meta data, heuristics, the quality and nature of inbound links (just speculating, a link from Wikipedia might rank very highly compared to other links), and so on. There are also algorithms to detect bot-generated content. Query logs (what people search for and what they click (and whether they stay there or quickly come back)) are also bountiful sources of ranking algorithms. And these are just the obvious things that the big engines figured out years ago.

The most important challenge — and competitive playing field — is capacity, and not just to hold the web, but to compute on it. We hold tens of billions of documents, so even URI-only graph algorithms have to be distributed. That’s why Google and Microsoft (and all the other transnats) are buying land near power stations as fast as they can. The computational platform itself, is also a key strategic piece. Whoever can more elegantly express and efficiently execute computations over petabytes of data has a significant competitive edge.

]]> By: Andrew Matthews http://aabs.wordpress.com/2007/09/04/the-future-of-search/#comment-14563 Andrew Matthews Wed, 05 Sep 2007 23:31:36 +0000 http://aabs.wordpress.com/2007/09/04/the-future-of-search/#comment-14563 This article can also be found at http://blog.vortexdna.com/andrew-matthews-on-the-future-of-search/ This article can also be found at http://blog.vortexdna.com/andrew-matthews-on-the-future-of-search/

]]> By: Mills Davis http://aabs.wordpress.com/2007/09/04/the-future-of-search/#comment-14548 Mills Davis Wed, 05 Sep 2007 12:51:58 +0000 http://aabs.wordpress.com/2007/09/04/the-future-of-search/#comment-14548 Currently, there are a number of ventures and product development efforts focused on adding linguistics to search (moving beyond keywords), semantic search (combining knowledge models with linguistics in the search), vertical search (finding a particular kind of information or answer in a particular domain), media search (being able to search video, and other kinds of file formats) and so on. There is also a trend towards question answering and what I would call semantic research rather than search that involves training on questions, briefs or documents on the fly to then read and extract specific information of interest and to then write and report syntheses of findings. Currently, there are a number of ventures and product development efforts focused on adding linguistics to search (moving beyond keywords), semantic search (combining knowledge models with linguistics in the search), vertical search (finding a particular kind of information or answer in a particular domain), media search (being able to search video, and other kinds of file formats) and so on. There is also a trend towards question answering and what I would call semantic research rather than search that involves training on questions, briefs or documents on the fly to then read and extract specific information of interest and to then write and report syntheses of findings.

]]> By: Andrew Matthews http://aabs.wordpress.com/2007/09/04/the-future-of-search/#comment-14529 Andrew Matthews Wed, 05 Sep 2007 00:08:30 +0000 http://aabs.wordpress.com/2007/09/04/the-future-of-search/#comment-14529 Hi Sudha, I'm an optimist - I don't think that there is a problem that we can't overcome, given sufficient incentive. Hi Sudha,

I’m an optimist - I don’t think that there is a problem that we can’t overcome, given sufficient incentive.

]]> By: sudha http://aabs.wordpress.com/2007/09/04/the-future-of-search/#comment-14526 sudha Tue, 04 Sep 2007 23:50:32 +0000 http://aabs.wordpress.com/2007/09/04/the-future-of-search/#comment-14526 Wow, great insights into what lies ahead in future of search. Do we have more issues than we can solve? Wow, great insights into what lies ahead in future of search. Do we have more issues than we can solve?

]]> By: Andrew Matthews http://aabs.wordpress.com/2007/09/04/the-future-of-search/#comment-14496 Andrew Matthews Tue, 04 Sep 2007 13:42:46 +0000 http://aabs.wordpress.com/2007/09/04/the-future-of-search/#comment-14496 This article was invited by Kaila at http://blog.vortexdna.com as party of their 'Dream team of Search' series. And is posted there too. This article was invited by Kaila at http://blog.vortexdna.com as party of their ‘Dream team of Search’ series. And is posted there too.

]]>