Rumors that Google is building a new social network have persisted since late June, when Kevin Rose, CEO of Digg.com, posted on Twitter that the Web giant was working on a challenger to Facebook. The company’s recent actions–its reported investment in Zynga, a social gaming company, and its acquisition of Slide, a company that makes various applications for social networks–have fanned the flames.

Google already owns several products that encourage online social interaction–including YouTube, Google Talk, Google Reader, and Blogger. But it has struggled to deliver a successful dedicated social networking service. Its existing social network, Orkut, has far fewer users than Facebook (around 100 million, compared to 500 million), and is mainly popular in Brazil and India. And the launch of Buzz, a social network built into Gmail, was botched after users complained that their privacy had been invaded. Google has acquired several promising social services, including the microblogging site Jaiku and the location service Dodgeball, only to hold back on investing in them.

Some argue that Google has failed to deliver the kind of overall experience people expect from a social network. “Google has never come out with any [social networking product] where the experience drove it,” says Jared Spool, founding principal of User Interface Engineering, a consulting firm based in North Andover, MA. “It was always the technology and the engineering that drove it–the experience was sort-of layered on afterward.”

Spool notes that other failed social offerings from Google, such as Lively, its foray into virtual worlds, and Wave, an experiment in online communication and collaboration, originated as side projects for the company’s engineers. Spool says that it is hard for side projects to be expansive enough to become a fully featured social network.

Nick O’Neill, a social-networking industry expert who runs the blogs The Social Times and All Facebook, says Google is desperate to get more involved in social networking because Facebook is collecting commercially valuable information that Google can’t access.

O’Neill says that sharing content with friends provides important data on users’ interests and behavior–useful both for providing better search results and delivering more effectively targeted advertising. To maintain its dominance in both fields, O’Neill says, Google needs to hone its search results by considering a user’s social connections and the information shared with friends. Google may believe it needs its own social network to get the best social information, he says.

Google’s existing social offerings are scattered, and it will take a focused effort to pull them all together, Spool says. He thinks users will expect nothing less than a spectacular new product from the company. “Google has way too much baggage,” Spool says. While users might forgive a startup social network for lacking features, they’ll want any offering from Google to have full integration with Gmail, Docs, and its other products.

Google already has popular communication tools, and plenty of content being shared on sites like YouTube, Picasa, and Google Reader. It is also involved with OpenSocial, a system for adding third-party applications to social networks, and has FriendConnect, a service that lets websites add social features that allow users to interact with each other and pull in content from social networks.

But Google will have to tread carefully as it tries to gain traction against Facebook. “A social network only works with a social graph in place,” says Spool, referring to the connections between users on a social network site. With Buzz, Google tried to populate its social graph automatically, using links between Gmail users. But the resulting backlash–as users felt their privacy had been violated–shows that Google cannot easily exploit the user data it already holds.

Spool compares Google’s Facebook problem to trying to compete with a popular frat house party. Another group can try to get a better keg and a better band, he says, but if most people are still at the frat house, there’s not much that can be done. Users need a good reason to switch to a new social service. Google may have been hoping that an innovative social service, such as the now-canceled Wave, which offered a completely new approach to online communication and collaboration, could draw users away from Facebook, he notes.

Facebook, meanwhile, has its own problems, and some of these could turn out to be opportunities for Google. Ben Gross, an expert in online identity, notes that Facebook and other social networks don’t accurately differentiate between people’s social connections, making their social graph information less valuable to users and advertisers. For example, social networks tend to put all of a user’s connections into a single group of “friends,” and expect users to manage complex privacy settings to sort out family, work connections, and bar buddies. “Social network services should not assume that networks are flat, or that people are willing to put in the effort to articulate these networks or that they even want to,” he says.