Google’s layoff of 4,000 Motorola Mobility workers today—just three months after buying the company for $12.5 billion—is the first step in a high-stakes bet that it can reinvent Motorola as a competitor to Apple and Samsung using cutting-edge new technology.

But it will be a challenge to actually come up with a smartphone that makes a dent in a marketplace so heavily dominated by Apple and Samsung—all while convincing the handset makers that use Google’s Android operating system that Google won’t favor its own device.

In addition to cutting 20 percent of Motorola’s workforce, Google fired 40 percent of Motorola’s vice presidents and installed a management team led by new CEO Dennis Woodside, a longtime Google advertising executive. Google has also created a department within Motorola—Advanced Technology and Projects—comprised of researchers charged with finding cutting-edge technologies that could give Motorola’s products an edge. And the executive refresh includes a new senior vice president, Regina Dugan, a former director of the Defense Advanced Research Projects Agency, the Pentagon’s long-term research arm.

As smartphone usage has exploded, the mobile market has become a big strategic battleground for technology companies. Apple has established a daunting lead, with the iPhone and iPad leaving everyone—from long-standing incumbents like Microsoft, to relative upstarts like Facebook—scrambling to catch up (see “Can This Man Work Magic?” and “If Facebook Made a Phone, Would You Friend It?”).

Dugan’s DARPA experience could help drive a research mission at Motorola, says Wade Trappe, a professor at the Rutgers University Winlab. “She is very good at driving a larger research program. And Motorola and the Android platform need to think out of the box to compete against the smooth interfaces and other features that the iPhone has,” he says.

A spokeswoman said Google was not making its Motorola executives available for comment beyond embargoed interviews given for a story in Monday’s New York Times. But the cuts were described in this Google legal filing.

According to the Times report, Woodside plans to reduce the number of devices Motorola makes from the 27 it introduced last year down to just a few, and wants those devices to have super-long battery life, improved cameras, and possibly even new features such as voice recognition technology that can recognize people chatting in a room. Dugan is reportedly hiring metal scientists, acoustics engineers, and artificial intelligence experts, too.

But whether the DARPA research model can work in the fast-evolving world of smartphones is unclear, says Chetan Sharma, a wireless analyst in Seattle. “Regina does bring in outside perspective specially related to projects that are leaps, versus incremental steps,” he says. “However, this will need to be executed under the constraints of competition, time, and money.”

While DARPA has had some storied successes—such as the precursor to the Internet—it also freely admits that it often fails. And it has pursued some odd projects, such as setting up a research program to figure out how to reassemble shredded documents.

Sharma says a big task will be convincing other handset makers that use Android that Google won’t give Motorola preferential treatment. “The biggest challenge for them is to keep the Android ecosystem together while launching their own Google branded devices,” Sharma says. “It is a tough battle to attract the ecosystem and effectively compete against them at the same time.”

Google closed on the Motorola deal in May. It announced the planned purchase last year, saying it was partly interested in acquiring Motorola’s 17,000 patents (see “Why Google Wants Motorola”) to help a relatively patent-poor company fend off challengers.

Among the hard problems that need solving in smartphones, Trappe says, is coming up with smoother ways to collect revenue from mobile apps. “That has been a big hard challenge for everyone,” Trappe notes. “Apple has the app store and iTunes—those are instant moneymakers for Apple. But Google has not been able to replicate that.”