Articles: Inside Google

In 1999, Amin Singhal’s mate Krisna started hassling him to come and work for a little company he thought was going to be big news. Singhal, who describes himself as a “thoroughbred academic”, was working in a think-tank at the US telecommunications giant AT&T, and told his crazy friend Krisna that he had a wife and kids to think about.

“I said, ‘Look man, I work for a big company, it’s stable. AT&T will be there forever. Google are all going to die’.”

Against his better judgement, Singhal followed his pal to join a couple of guys who had until recently been working out of their garage. Ten years later, Singhal is a ‘Google Fellow’, with a team of hundreds of engineers working under him on a product that’s synonymous with the verb search. He admits he owes Krisna a beer.

Google takes about four of every 10 advertising dollars. While the US newspaper market made around US$30 billion (NZ$42.25 billion) last year, Google’s made more than two-thirds of that, its revenues around US$22 billion. But no matter how many other irons the ambitious media company puts in the fire, search was still responsible for 98 per cent of that.

The scale of Google’s search service is almost literally astronomical. It handles hundreds of millions of queries, indexes tens of billions of sites in 112 languages every day. Back in 2008 it was processing 20 petabytes (or 20,000 terabytes) of data every day, and it’s grown significantly since then. Google’s not saying by how much, but according to a recent article in the New Yorker, the company handles about three billion searches a day. That accounts for more than 65 per cent of all online search queries.

Singhal, who holds a PhD, is one of the pioneers in a field that until last decade didn’t really exist.

Before the unfolding of the web proper, Stanford was a hothouse for the early development of search engines. In 1994 two students from the Californian university came up with Yahoo!, a service whose ambition was to write a ‘table of contents’ for the internet by automatically creating an index using keywords.

At the same time, another Stanford bright spark, Larry Page, had the idea that the internet was essentially a collection that people had actively chosen to compile. Neatly referred to as ‘PageRank’, by ranking a website’s importance based on the number of times it had been linked elsewhere it was possible to create an accurate picture of what people really wanted to see online. Page was joined by graduate Sergey Brin, and together than ran a prototype out of their student offices. They called their service BackRub, referring to its analysis of ‘back links’. Thankfully they abandoned the name when they misspelled ‘googolplex’ while registering a domain name. In 1998, the pair lashed out and moved into that garage.

NEW APPROACHES

These days, the methods Google uses to assign the level of importance to any given website have, like the web itself, multiplied exponentially.

“Expectations are higher now,” Udi Manber, the man who oversees Google’s entire search-quality group, told the New York Times. “When search first started, if you searched for something and you found it, it was a miracle. Now, if you don’t get exactly what you want in the first three results, something is wrong.”

“When Google started out about 12 or 14 years back as a Stanford project, clearly PageRank was one big innovation that made Google far superior to all search engines at the time,” explains Singhal. “PageRank even today is a very important piece of our system. But as the web has evolved, many more signals have come into our search system that make it so relevant.”

Instead of relying solely on the BackRub system of indexing via links, more than 200 separate types of information are taken into account when the engine assigns a rank. Words, links and images play a part, as does the page’s history over a period of time. Recently, because of the vast array of queries it handles, Google has started using itself as a reference point in order to establish trending topics. Singhal, however, isn’t spilling the specifics.

“Last year in 2009, we launched over 570 changes to our search,” he boasts, not without justification. “It understands far more than when I started out. Our search is not just keyword based. It has moved waaaay beyond keywords.”

According to Singhal, the explosive increase in the sheer volume of information has made the concept of relevance the most critical factor in determining good search from bad.

Keywords are fine for dealing with smaller sample sizes, but when billions of articles feature trillions of words that all have multiple meanings, finding the single piece a user is after is mind-bogglingly complex.

A PHILOSOPHICAL BENT

The problem of language remains the greatest barrier to Google understanding what the user wants.

“The same words mean different things given the context of other words,” Singhal explains, offering the example that ‘GM Cars’ means ‘General Motors Cars’, but ‘GM Food’ means ‘Genetically Modified Food.  “The truth is that natural language processing technology doesn’t really understand language as humans do. Computers still do not understand language like you and I.”

In an effort to help Google’s technology take baby steps towards understanding what people want, Singhal says he looked to the philosopher Wittgenstein.

“I’m a student of language, being an academic, and the key Wittgenstein idea – and I’m telling you this for the first time – is that Wittgenstein proposed that words in isolation do not mean anything. A word means what its context is. If you read Wittgenstein’s writings, that’s one idea that as a language/computer scientist you come away with,” he says. “Using those ideas of context we have moved our search system toward understanding.”

The search quality team has begun to integrate a huge number of contextual criteria like geographical location into its search algorithm. Users who log in to Google receive personalised results, with the engine mining their history to individualise their search.

“If you are interested in the game of cricket, and someone else is interested in the insect, and their past behaviour is telling us that, then the results for that query for those individuals would be different,” he says.

Singhal noticed, however, that Google was favouring older, established pages that had time to gather links. Given the burgeoning popularity of social media, users were growing used to knowing exactly what was going on right now.

“The world was moving forward, people were publishing news every few hours and blogs were publishing every few hours, and as time progressed companies like Twitter and MySpace and Facebook have built services where people are publishing information every second from around the world.”

‘Freshness’ is the latest contextual factor the team has worked into the way it delivers results. Ironically, the ‘freshest’ topic on the web the day we spoke was Google itself. Forty minutes beforehand, the company announced its decision to pull out of mainland China. Singhal used the announcement as a test case for Google Real-Time Search, a system that aggregates information from a range of sources including news and blog posts as well as MySpace, Facebook and Twitter, among others.

“As soon as we published the Google blog at 12.03 California time, I started watching when Real-Time Search will show that something is happening with Google China,” he says. “It happened within seconds. Within a minute, that blog was available to me within Real-Time Search.”

CHROME PLATED

Search remains Google’s bread and butter, but its wild success has allowed the company to launch an incredible array of new products. Most people would be familiar with Google Maps, Gmail, and Google Docs, but there are also services like Google Health, which organises personal medical records, Google TV, Audio and Print Ads, which sell cheap advertising on the respective platforms. Recent reports have indicated the company is considering significant investment in the raw materials of the internet: energy and underwater cables.

Ben Goodger, an AUT graduate who’s had a hand in the development of such defining web browsers as Netscape and Mozilla Firefox, is one of the minds behind the company’s own browser, Google Chrome.

Goodger took responsibility for the user interface, the way people interact with the software. Contrary to much software design, the philosophy guiding Chrome’s development was to include fewer features, paring down the interface to its bare necessities, and focusing on speed, security and stability.

“We actually seek to minimise it as much as possible. A lot of my work is figuring out how to add features and functionality to the product without creating more user interface,” says Goodger. To this end, the team behind Chrome employs the ‘omnibox’, which combines the address and search bars, a move which, Goodger says, “reduces friction”.

The browser found respectable market share almost immediately, with 30 million users taking it up in its first year alone. It’s now the third most popular browser (behind Internet Explorer and Mozilla Firefox) with a share of just under six per cent.

Google announced late last year it had a Chrome operating system in development, and if all things go according to plan, it should be appearing in netbooks in the next couple of months.

Like the browser, Chrome OS is designed to be light on its feet. It assumes what users really need is a framework that better navigates the internet – because in the near future the net will be where most of our software is based.

“With the Chrome operating system, fundamentally the way we think about it is like a browser with a keyboard,” reveals Goodger. In his view, the future of computing is in the cloud, rather than on the desktop. Everything an operating system needs will be online. “The web is really the application development platform of the future… I think it’s possible that every program that the average person uses might live online. You look at what people are using now, that’s already very close to being true today.

BOTTOMS UP

Whether or not Google’s vision of a future online comes to pass, the company seems committed to diversifying beyond its core services.  Both Goodger and Singhal describe a bottom-up approach to innovation at the company’s headquarters, the Googleplex, in Mountain View, California. Its corporate culture has become something of a myth, particularly among those of us in partitioned basements: the replica of SpaceShipOne in the foyer, foosball tables, footballs, pooltables and classic arcaders, a famed café and Segways up the yingang. Apparently it’s really like that, too.

“I cannot emphasise enough how important our management structure has been to the success of Google,” says Singhal. He describes Google as a big company that works like a small one, with all of its employees technically proficient enough to be able to understand the algorithms.

“It’s a totally bottom-up approach to innovation. We have hundreds of people thinking about search algorithms every hour,” he says. “We don’t have a magic bullet to come up with ideas that would succeed 90 per cent of the time. It’s true that half of our ideas don’t work out. However, the fact is that many do, because we’ve built such a healthy bottom-up innovation environment.”

Singhal doesn’t envision Google, or himself, for that matter, resting on laurels.

“I don’t see us slowing down,” he says. “If anything, it’s going to go faster and faster.”

Don’t Be Evil

Google’s corporate motto, ‘Don’t Be Evil’, probably seemed like a great idea at first. But in practice, not being evil is harder said than done.

Last year, Google began using its personalisation technology for ‘behavioural targetting’, which shows ads based on users’ previous online activities. Users are tracked via cookies whenever they visit a site displaying ads from Google’s AdSense program, and are then assigned to a category, say, somebody in the market for a new car. Google will then display relevant advertising to these catalogued users, wherever they travel online.

Unsurprisingly, the move had its critics. Privacy campaigners were about as pleased as traditional media publishers by the new targetted advertising methods.

Likewise, Google was widely perceived to have dropped the ball when it launched the microblogging application Buzz, which automatically assigned ‘friends’ based on who users had been corresponding with in Gmail. It turned out many Gmail users weren’t thrilled about having their little black books made public.

While it probably should have anticipated the Buzz debacle, on the whole the move was consistent with the company’s commitment to freedom of information.

“Google strongly believes in an open ecosystem,” says Singhal. In his opinion, Google can only succeed when it is supported by an uninhibited flow of information online. “I think it’s incredibly exciting when I get to read the Israeli point of view next to the Palestinian point of view, or Indian point of view next to the Pakistani point of view,” he says. “It’s not governed by Google, it’s the open ecosystem of the web.”

A real commitment to not being evil may go at least some of the way to explaining the company’s recent decision to pull out of China over the country’s mandatory censorship laws.

TIM GREY