Comments on: Wikia to launch new social search engine, more on Monday http://venturebeat.com/2008/01/04/wikia-to-launch-new-search-engine-jan-7/ News About Tech, Money and Innovation Wed, 25 Nov 2009 20:47:38 +0000 http://wordpress.org/?v=2.8.5 hourly 1 By: Overheard: Why do we need Search Wikia? - Overheard in the tech blogosphere http://venturebeat.com/2008/01/04/wikia-to-launch-new-search-engine-jan-7/comment-page-1/#comment-721156 Overheard: Why do we need Search Wikia? - Overheard in the tech blogosphere Tue, 08 Jan 2008 17:23:42 +0000 http://venturebeat.com/2008/01/04/wikia-to-launch-new-search-engine-jan-7/#comment-721156 [...] wrote earlier: We’ve tested Grub, the service’s way of crawling the Internet’s web sites to collect data. [...] [...] wrote earlier: We’ve tested Grub, the service’s way of crawling the Internet’s web sites to collect data. [...]

]]> By: VentureBeat » Search Wikia gets slammed, here’s our review http://venturebeat.com/2008/01/04/wikia-to-launch-new-search-engine-jan-7/comment-page-1/#comment-720330 VentureBeat » Search Wikia gets slammed, here’s our review Tue, 08 Jan 2008 04:06:51 +0000 http://venturebeat.com/2008/01/04/wikia-to-launch-new-search-engine-jan-7/#comment-720330 [...] covered some of its business and social aspects and an initial look when it first launched, but have since had more time to [...] [...] covered some of its business and social aspects and an initial look when it first launched, but have since had more time to [...]

]]> By: VentureBeat » Search Wikia gets slammed, here’s our review http://venturebeat.com/2008/01/04/wikia-to-launch-new-search-engine-jan-7/comment-page-1/#comment-720344 VentureBeat » Search Wikia gets slammed, here’s our review Tue, 08 Jan 2008 04:06:51 +0000 http://venturebeat.com/2008/01/04/wikia-to-launch-new-search-engine-jan-7/#comment-720344 [...] covered some of its business and social aspects and an initial look when it first launched, but have since had more time to [...] [...] covered some of its business and social aspects and an initial look when it first launched, but have since had more time to [...]

]]> By: Search Engine Optimization Direct » Blog Archive » Wikia to launch new search engine January 7 http://venturebeat.com/2008/01/04/wikia-to-launch-new-search-engine-jan-7/comment-page-1/#comment-719084 Search Engine Optimization Direct » Blog Archive » Wikia to launch new search engine January 7 Mon, 07 Jan 2008 06:58:16 +0000 http://venturebeat.com/2008/01/04/wikia-to-launch-new-search-engine-jan-7/#comment-719084 [...] Per and Susanne Koch article is brought to you using rss feeds.Here are some of the top articles on search engine optimization.Search Wikia, the highly anticipated search engine by Wikia Inc., Jimmy Wales’ (Wikipedia’s co-founder) for-profit company, will launch publicly on January 7 and is currently in private testing mode. We tested the service’s distributed … [...] [...] Per and Susanne Koch article is brought to you using rss feeds.Here are some of the top articles on search engine optimization.Search Wikia, the highly anticipated search engine by Wikia Inc., Jimmy Wales’ (Wikipedia’s co-founder) for-profit company, will launch publicly on January 7 and is currently in private testing mode. We tested the service’s distributed … [...]

]]> By: VentureBeat » Search Wikia launches: Will it threaten Google? http://venturebeat.com/2008/01/04/wikia-to-launch-new-search-engine-jan-7/comment-page-1/#comment-718972 VentureBeat » Search Wikia launches: Will it threaten Google? Mon, 07 Jan 2008 05:03:43 +0000 http://venturebeat.com/2008/01/04/wikia-to-launch-new-search-engine-jan-7/#comment-718972 [...] Jimmy Wales, who has led online encyclopedia Wikipedia to considerable success (see our previous coverage). We spoke with Wales (pictured here) under embargo last week about his [...] [...] Jimmy Wales, who has led online encyclopedia Wikipedia to considerable success (see our previous coverage). We spoke with Wales (pictured here) under embargo last week about his [...]

]]> By: Saumil Mehta http://venturebeat.com/2008/01/04/wikia-to-launch-new-search-engine-jan-7/comment-page-1/#comment-718468 Saumil Mehta Sun, 06 Jan 2008 18:35:08 +0000 http://venturebeat.com/2008/01/04/wikia-to-launch-new-search-engine-jan-7/#comment-718468 Chris, The need to index the entire web stems not from number of results, but from the ability to serve an essentially infinite number of queries. For example, you can index a 100 million pages and just serve queries related to Entertainment or Sports. But then you can't compete with Google anymore! Chris,

The need to index the entire web stems not from number of results, but from the ability to serve an essentially infinite number of queries. For example, you can index a 100 million pages and just serve queries related to Entertainment or Sports. But then you can’t compete with Google anymore!

]]> By: Chris http://venturebeat.com/2008/01/04/wikia-to-launch-new-search-engine-jan-7/comment-page-1/#comment-718440 Chris Sun, 06 Jan 2008 17:29:55 +0000 http://venturebeat.com/2008/01/04/wikia-to-launch-new-search-engine-jan-7/#comment-718440 How many people search beyond the first 2-5 pages? What is this need to index everything? How many people search beyond the first 2-5 pages? What is this need to index everything?

]]> By: Saumil Mehta http://venturebeat.com/2008/01/04/wikia-to-launch-new-search-engine-jan-7/comment-page-1/#comment-718062 Saumil Mehta Sun, 06 Jan 2008 00:40:59 +0000 http://venturebeat.com/2008/01/04/wikia-to-launch-new-search-engine-jan-7/#comment-718062 Well, I do understand how Grub fits into Wikia from a high level standpoint. They need to get a substantial crawl going to build a real search engine and they don't have Microsoft's and Yahoo's millions to start from scratch. As to your other question, the grub website is fairly light on docs about how they actually do the crawl and doesn't have the source code posted for people to tinker around. I expect that to change relatively soon. Well, I do understand how Grub fits into Wikia from a high level standpoint. They need to get a substantial crawl going to build a real search engine and they don’t have Microsoft’s and Yahoo’s millions to start from scratch.

As to your other question, the grub website is fairly light on docs about how they actually do the crawl and doesn’t have the source code posted for people to tinker around. I expect that to change relatively soon.

]]> By: TS http://venturebeat.com/2008/01/04/wikia-to-launch-new-search-engine-jan-7/comment-page-1/#comment-718034 TS Sun, 06 Jan 2008 00:06:08 +0000 http://venturebeat.com/2008/01/04/wikia-to-launch-new-search-engine-jan-7/#comment-718034 Actually, access to local I/O is usually NOT the main bottleneck in crawling. Assuming of course that one does not write out each retrieved page in a separate random write, and this depends on the crawler architecture that is used. Of course, once you crawl tens of thousands of URLs per second, almost everything becomes a bottleneck, but there are only a few dozen players that have a need for that kind of speed. The main bottlenecks in large-scale crawling (in this order) are probably crawl management (i.e., the human/software complexity side of managing and scaling a crawl to millions of hosts) and the bandwidth. The CPU power is not a major issue. Grub basically harvests bandwidth from clients. But I would be concerned about the crawl management part of the grub approach - are they using a fairly brute-force approach to recrawling that wastes (other people's) bandwidth, as opposed to the smarter recrawling strategies used by the major engines? How do they deal, e.g., with requests by sites to immediately cease crawling a domain (due to possible or perceived misbehavior of the crawler or local problems at the site)? And it is not clear how grub is really fitting into the whole wikia approach. Actually, access to local I/O is usually NOT the main bottleneck in crawling. Assuming of course that one does not write out each retrieved page in a separate random write, and this depends on the crawler architecture that is used. Of course, once you crawl tens of thousands of URLs per second, almost everything becomes a bottleneck, but there are only a few dozen players that have a need for that kind of speed.

The main bottlenecks in large-scale crawling (in this order) are probably crawl management (i.e., the human/software complexity side of managing and scaling a crawl to millions of hosts) and the bandwidth. The CPU power is not a major issue. Grub basically harvests bandwidth from clients.

But I would be concerned about the crawl management part of the grub approach – are they using a fairly brute-force approach to recrawling that wastes (other people’s) bandwidth, as opposed to the smarter recrawling strategies used by the major engines? How do they deal, e.g., with requests by sites to immediately cease crawling a domain (due to possible or perceived misbehavior of the crawler or local problems at the site)? And it is not clear how grub is really fitting into the whole wikia approach.

]]> By: Saumil Mehta http://venturebeat.com/2008/01/04/wikia-to-launch-new-search-engine-jan-7/comment-page-1/#comment-717250 Saumil Mehta Sat, 05 Jan 2008 03:00:54 +0000 http://venturebeat.com/2008/01/04/wikia-to-launch-new-search-engine-jan-7/#comment-717250 Actually, we should clarify "distributed search crawler" some more. I would assume that anyone that wants to crawl splits the crawl across n machines. I wasn't aware of your point around local IO being a bottleneck. I am unable to find the Grub source code or a whole lot of technical literature on the system but that is going to be corrected very soon, I'm told. Actually, we should clarify “distributed search crawler” some more. I would assume that anyone that wants to crawl splits the crawl across n machines. I wasn’t aware of your point around local IO being a bottleneck.

I am unable to find the Grub source code or a whole lot of technical literature on the system but that is going to be corrected very soon, I’m told.

]]> By: Kevin Burton http://venturebeat.com/2008/01/04/wikia-to-launch-new-search-engine-jan-7/comment-page-1/#comment-717194 Kevin Burton Sat, 05 Jan 2008 02:03:50 +0000 http://venturebeat.com/2008/01/04/wikia-to-launch-new-search-engine-jan-7/#comment-717194 The biggest issue with building a crawler is not the bandwidth it's scaling and having fast access to local IO. With Spinn3r: http://spinn3r.com We have a distributed crawler but we run it within our own cluster because having 10k clients wouldn't really buy us anything. Of course maybe from Wikia's perspective this is just a blind HTTP fetch task and they then aggregate it locally within their cluster. The comparison to Wikipedia might fall down here. Wikipedia has about 1.5M english pages. The net has billions. You can't just rely on humans for this stuff. Kevin The biggest issue with building a crawler is not the bandwidth it’s scaling and having fast access to local IO.

With Spinn3r:

http://spinn3r.com

We have a distributed crawler but we run it within our own cluster because having 10k clients wouldn’t really buy us anything.

Of course maybe from Wikia’s perspective this is just a blind HTTP fetch task and they then aggregate it locally within their cluster.

The comparison to Wikipedia might fall down here. Wikipedia has about 1.5M english pages. The net has billions.

You can’t just rely on humans for this stuff.

Kevin

]]> By: Wikia on VentureBeat « Life in the Bit Bubble http://venturebeat.com/2008/01/04/wikia-to-launch-new-search-engine-jan-7/comment-page-1/#comment-716956 Wikia on VentureBeat « Life in the Bit Bubble Fri, 04 Jan 2008 22:41:56 +0000 http://venturebeat.com/2008/01/04/wikia-to-launch-new-search-engine-jan-7/#comment-716956 [...] Here it is if you’d like to follow the link and read it: http://venturebeat.com/2008/01/04/wikia-to-launch-new-search-engine-jan-7/ [...] [...] Here it is if you’d like to follow the link and read it: http://venturebeat.com/2008/01/04/wikia-to-launch-new-search-engine-jan-7/ [...]

]]>