A company called uTest solicited 1,100 software developers and set them loose on the four major search engines of the day: Google, Yahoo, Bing, and Google's new Caffeine update. Google had the fewest number of bugs and the least severe bugs among the competition, while Bing amassed the most bugs yet still scored well in the accuracy of its results.
uTest solicits bug testers with cash rewards for the quality of the bugs they discover, and pitches the service to companies as an outsourced QA department. It turned participants loose on the search engines for a week in August and only accepted bugs that were judged as new and unknown to those outside the company that developed the search engine.
Some interesting tidbits highlighted by the study:
• Google was targeted by 85 percent of the bug hunters but held up to the scrutiny, producing the fewest number of severe bugs and a relatively low total overall. Nearly half the bugs reported with Google were functional, as opposed to technical or GUI (graphical user interface) related.
• As the newest kid on the block it might not be a surprise that Microsoft's Bing had the largest number of bugs reported, accounting for more than half the total bugs reported during the survey. Perhaps more importantly for Microsoft, despite the bugs, survey testers were largely impressed with Bing and delivered high praise for its user interface.
• Yahoo's gradual exit from the search market is under way, but it was the least-buggy search engine in the mix probed by uTest's army. Still, testers ranked Yahoo third behind Google and Bing in page load speed, real-time relevance, and overall accuracy.
• Google's Caffeine update is not even a production search engine, but uTest gave it a run anyway just to see what they could find. Caffeine actually had fewer unknown bugs than Yahoo, but that's a bit deceptive since the list of known bugs is long and because most testers chose not to examine Caffeine. Still, testers were very impressed with the speed of the Caffeine update, which was Google's main priority with the overhaul.