The Seattle Times Company

NWjobs | NWautos | NWhomes | NWsource | Free Classifieds |

Business / Technology

Our network sites | Advanced

Originally published May 2, 2005 at 12:00 AM | Page modified May 2, 2005 at 10:10 AM

E-mail E-mail article      Print Print      Share Share

Microsoft learns to crawl

The software giant set out in 2003 to build a Web crawler — an online robot capable of indexing billions of sites — a task that proved trickier and more frustrating than imagined.

Seattle Times technology reporter

Bill Gates knows when he's been beaten.

Last year, the Microsoft chairman candidly credited Google with winning the first round of the search wars. "Google kicked our butts," he said.

A few months ago, when asked why the company had used technology from rivals to run its search engine, he admitted something rarely heard from Microsoft's top brass: "We were stupid as hell."

Microsoft didn't just miss the boat in search technology. It missed the dock, the pier and the turnoff to the marina. The company has relentlessly pursued new businesses over the years, hoping to find another cash cow that could churn out revenue like its Windows and Office products. But while Microsoft pumped money into video games, cellphones and other ventures, the search business it had ignored quietly became a star.

That hit home for Microsoft in 2002. Google's revenues would reach $348 million that year and $962 million the next. Yahoo! was so hot on search that it soon would shell out $2 billion to buy search-engine companies Inktomi and Overture Services.

And Microsoft? Its revenue from selling search-based advertising soared, but it was paying Inktomi and Overture to power its engine — an unacceptable arrangement now that search was becoming a big-money business.

Search: The engine for change

Part 1

We have invited search into our lives, and search engine companies have responded by giving us more of what we want — and taking some information in return.

Part 2

Microsoft's MSN division went from zero to search engine in less than two years — a remarkable achievement. The division did it in typical Microsoft fashion, devoting enormous resources and effort to catch up. The engine still hasn't beaten Google, though. Will it ever?

Part 3

The Puget Sound has been involved in search development since nearly the beginning. A look at the work being done in search by local people and companies.

Related links

So the MSN division set out to build a new engine from scratch, one that would have to be better and more useful than anything Google and Yahoo had to offer.

Would MSN grab a come-from-behind victory? That depended almost entirely on what the search team could produce.

Odd answers earn loyalty

The search project took root in fall 2002, when MSN executives sat down to look at how they were spending money across their Internet properties.

"It was clear that we weren't really growing the search business, and yet the search industry was starting to take off," said Yusuf Mehdi, a senior vice president in charge of MSN.

Even more dire: What market share Microsoft had with its early search efforts was starting to erode. To that point, MSN had based its strategy on the 80-20 rule on search: 80 percent of searches are for 20 percent of terms. In other words, people search mainly for the same things, and MSN wanted to perform well on those common searches.

But it learned that the arcane searches were the make-or-break moments for Web searchers. People weren't just happy when a search engine could find answers to their most bizarre, obscure and difficult queries. They would switch loyalties.


Mehdi handed the search project to Christopher Payne, who had been running the MSN home page and other Web properties. Payne recruited Ken Moss, a colleague from the home-page team, to help lead the new group.

In early 2003, Moss and another developer set out to build a Web crawler — the online robot that would collect Web pages for gleaning by the future search engine. In his 16 years at Microsoft, Moss had never seen such a challenge.

"It was the hardest thing I'd ever looked at, technologywise," he said.

MSN knew very little about building a search engine. Moss and the developer, Keith Bernie, set an ambitious goal of indexing 5 billion Web sites. They unleashed their primitive crawler on the Internet, hoping to build a vast collection of Web pages that could be searched by an engine. But after a week, they were stuck at 24 documents.

"It was driving me nuts," Moss said.


Microsoft Senior Vice President Yusuf Mehdi says it was clear in 2002 that "we weren't really growing the search business, and yet the search industry was starting to take off."

Google, meanwhile, had been perfecting search for five years and already had indexed more than 4 billion documents. Yahoo had just acquired Inktomi, which had been developing search two years longer than Google.

Microsoft had none of that experience. It didn't know the demands that would come with building a search data center with thousands of computers. It didn't know how to crawl the Web quickly or weed out unwanted pages from its results.

Time and again, the company would learn search the hard way.

Money, machines, brains

A month after Moss and Bernie began their work, the team faced its first big test: presenting the project to Gates, Chief Executive Steve Ballmer and several other executives. The meeting lasted a few hours, and those who were there said Gates and Ballmer were extremely engaged.

The search team asked for a significant amount of money, machines, employees and other resources. It asked to work more closely with Microsoft researchers than nearly any other team had before. It got all it asked for, and more: Gates and Ballmer wanted to be in on the project at every step.

The level of commitment didn't fully set in for some until Microsoft cleared out a large chunk of office space for the project in the northeast part of the main Redmond campus. The team went from cramped quarters to a sprawling floor with two or three times the space needed.

The move spoke volumes at a company where space is so scarce that some employees have to double up or even triple up in offices.

"We had all these empty offices in our hall," Moss said, laughing. "A lot of people hated us for it."

With a growing team and new offices in place, MSN set out to write software fast. Rivals announced new innovations and features almost every week, and everyone felt pressure to catch up.

MSN's Web crawler had been fixed and was scurrying through the Web, boosting the index to 1,100 documents, and then 24,000. By the summer of 2003, it had reached 500,000 pages.

Search was no longer a mystery to Microsoft. The team knew how to index pages and perform a basic search. The system was up and running, and everyone was excited. But finding results that users actually wanted was proving difficult.

When the index soared to a billion pages, everyone expected the search engine's results to be far more relevant than before. The opposite occurred. MSN's Web crawler had picked up Web pages touting coupons, Viagra and other products, and the spam sites were polluting the pool of results. The developers hunkered down again and built technology to help the crawler weed out the junk.

The search engine grew so voluminous that it began slowing down Microsoft's entire corporate network — a feat in itself. In fall 2003, the team moved its data center to Silicon Valley and hit more snags: The network ran into technical problems, and the facility wasn't prepared for the kind of power that MSN Search would require.

Competition grows fierce

The search team took each setback with geeky gusto, however, even laughing at times as new issues popped up. Every glitch was a software problem, after all, and MSN had cherry-picked developers across Microsoft who lived for the hard stuff.

The search team grew in status on campus as it became clear that Microsoft was gunning for the biggest names in one of technology's hottest businesses. Gates and Ballmer were giving MSN an extraordinary amount of attention and support. The search team was the place to be.

"MSN has become kind of the hot place in the company," said Mehdi, who in March was promoted from vice president to senior vice president at Microsoft, in large part because of his work on search.

Yet all the while, it was growing apparent that Microsoft wasn't going to leap out of third place in the industry very easily. Google and Yahoo! were continuing to innovate, refusing to slow down and let MSN catch up.

For Microsoft, third place was unfamiliar territory, but MSN seemed to relish its status. To call up the search engine on Microsoft's internal network, employees had to type one word in the address box: Underdog.

Financial hit self-inflicted

Meanwhile, MSN also tinkered with how it handled search advertising. It had let Web sites pay to be included in its main search results — a controversial program called "paid inclusion" that Google had rejected.

The program was so popular that in more than half of MSN's searches you would have to scroll down the page to see a link that wasn't paid for by an advertiser. MSN executives thought they needed to remove paid-inclusion ads from the main results.

"Customers needed to understand from us when we were showing something that was paid and when we were showing something that wasn't paid," said Richard Chin, a senior director for product planning in MSN's search division.

Microsoft chairman Bill Gates

So MSN dropped paid inclusion altogether in mid-2004, a strategy shift code-named "CIA," for "clarity in advertising."

The financial hit was harsh — MSN's revenue dropped by about $50 million the following quarter, and at least $100 million for the year.

Along with shifting the approach, MSN unveiled a new page last July that was nearly stripped clean of links and other clutter. The page looked strikingly like Google's.

Playing the Encarta card

A few months after the fixes, MSN was ready to present the guts of its search. It unveiled a test version of the search engine last November and moved it to final release Jan. 31. MSN went from zero to search engine in two years, and the search results were often — but not always — as good as those of Google and Yahoo.

The index had finally reached the goal Moss and Bernie set of 5 billion pages. The engine also gave the first hints at Microsoft's developing search strategy. MSN began using its Encarta encyclopedia to give factual query results, and could explain global warming or tell you the name of the largest city in South Africa without having to send you to any Web site.

The feature went right to the heart of an idea that Gates had been touting since 1990: information at your fingertips. And no rival had a weapon like that in its arsenal.

Critics seemed to like MSN's offering, but most acknowledged that it would take more from MSN to topple Google from its perch.

"It's a good product, but it is not at the same level as Google or even Yahoo yet," said Safa Rashtchy, an analyst covering the search industry for Piper Jaffray. MSN needs to boost the relevancy of its search results even more, he said.

So far, MSN's new search engine hasn't caused a vast market shift, but it appears to be growing in use faster than Google. From January to March, according to research firm Nielsen/NetRatings, Google's market share went from 47.1 percent to 47.3 percent while MSN's went from 12.8 percent to 13.6 percent.

Put aside for a moment the features and technical competency of each, and MSN has another barrier to hurdle: People are deeply devoted to Google.

Google has what Rashtchy calls the Kleenex factor: It is so well known that people say "Google" when they mean "search."

"What has Coke done that Pepsi hasn't?" Rashtchy asked. "When you create an image, it's yours to keep and yours to lose. I think Google has done that."

Looking beyond Google

MSN's search team is hardly bothered. The division holds steadfast to a belief that came up repeatedly in the green-light meeting with Gates and Ballmer: Search is in its early stages. That's part of the reason why MSN decided to build a new search engine from scratch instead of buying its way into the industry. It has time.

And, in many ways, the search battle is bigger than beating Google. Microsoft is carving its path in the next generation of computing — one in which search becomes a platform, not a feature.

Microsoft is in a new version of the 1990s browser wars against Netscape Communications, said Joe Wilcox, an analyst covering Microsoft at Jupiter Research. Back then, Microsoft faced a new universe — the World Wide Web — that could be navigated without Windows. It responded by quickly developing its own browser.

It won't be long before people regularly search for information from cellphones and other mobile devices that don't require Windows. That increases the risk to Microsoft, Wilcox said.

"While Google is a target, the real target is much bigger than that," he said. "It's what search represents."

Microsoft's search team has grown to 500 engineers and marketers, and the Microsoft Research division is so involved that 25 percent of its high-powered Beijing lab focuses solely on search.

Some of those employees undoubtedly are eyeing perhaps the biggest weapon in Microsoft's arsenal: the operating system. The company is building search into its upcoming operating system, code-named Longhorn and expected next year, and likely will give it a prominent spot in front of the user.

Wilcox and other analysts say that doesn't guarantee Microsoft an advantage over its rivals. Longhorn will search the Web, but Microsoft had to postpone plans to include an even more powerful feature that searches inside a computer. MSN has had to fill that void with its own limited desktop search system.

Still, the search team says much of its focus is on the future. Payne said he was reminded of that recently while driving with his wife and daughter in his car, which is equipped with a computer system that can respond to certain questions, such as, "What time is it?"

Payne asked his wife if there was a winner in the Kentucky game in the men's college basketball playoffs. From the back seat, his 5-year-old daughter chimed, "Ask the car."

That request captures where Microsoft wants to be, Payne said. People should be able to get the information they want at any time and anywhere. That hasn't happened yet, and Microsoft wants to be the company that accomplishes this — if Google or Yahoo doesn't get there first.

"We knew when we went into this that this would be a long-term commitment, that this was three, five, 10 years of innovation," he said. "Our focus was on the long-term from day one."

Kim Peterson: 206-464-2360 or

E-mail E-mail article      Print Print      Share Share

More Business & Technology

UPDATE - 09:46 AM
Exxon Mobil wins ruling in Alaska oil spill case

UPDATE - 09:32 AM
Bank stocks push indexes higher; oil prices dip

UPDATE - 08:04 AM
Ford CEO Mulally gets $56.5M in stock award

UPDATE - 07:54 AM
Underwater mortgages rise as home prices fall

NEW - 09:43 AM
Warner Bros. to offer movie rentals on Facebook

More Business & Technology headlines...

Get home delivery today!



AP Video

Entertainment | Top Video | World | Offbeat Video | Sci-Tech