Skip to Content

Big Data Will Keep the Shale Boom Rolling

Don’t believe the doomsayers proclaiming the end of the shale oil boom. It’s just getting more efficient.

The number of active oil rigs in the United States continued to fall in May, as low prices pushed oil companies to temporarily shut down some of their production facilities. Since the end of May 2014, the U.S. rig count has fallen from 1,536 to 646, according to the energy analysis firm Platts—a 58 percent drop.

Low prices and plummeting rig counts have prompted a gusher of headlines claiming that the shale oil revolution, which by early this year boosted American oil production to nearly 10 million barrels a day, is grinding to a halt. The doomsayers, however, are missing a key parallel trend: lower prices are prompting unprecedented innovation in the oil fields, increasing production per well and slashing costs.

That’s the main reason that even as rig counts have fallen, total production has held steady or continued to rise. In the Eagle Ford, a major shale formation in South Texas, production in April was 22 percent higher than the same month in 2014, according to Platts. 

In fact, some observers expect a second wave of technological innovation in shale oil production that will equal or surpass the first one, which was based on horizontal drilling and hydraulic fracturing, or fracking. Fueled by rapid advances in data analytics—aka big data—this new wave promises to usher in a second American oil renaissance: “Shale 2.0,” according to a May 2015 report by Mark Mills, a senior fellow at the Manhattan Institute, a free-market think tank. 

Much of the new technological innovation in shale comes from a simple fact: practice makes perfect. Tapping hydrocarbons in “tight,” geologically complex formations means drilling lots and lots of wells—many more than in conventional oil fields. Drilling thousands of wells since the shale revolution began in 2006 has enabled producers—many of them relatively small and nimble—to apply lessons learned at a much higher rate than their counterparts in the conventional oil industry.

This “high iteration learning,” as Judson Jacobs, senior director for upstream analysis at energy research firm IHS, describes it, includes a shift to “walking rigs,” which can move from one location to another on a drilling pad, allowing for the simultaneous exploitation of multiple holes. Advances in drill bits, the blend of water, sand, and chemicals used to frack shale formations, and remote, real-time control of drilling and production equipment are all contributing to efficiency gains.

At the same time, producers have learned when to pause: more than half the cost of shale oil wells comes in the fracking phase, when it’s time to pump pressurized fluids underground to crack open the rock. This is known as well completion, and hundreds of wells in the U.S. are now completion-ready, awaiting a rise in oil prices that will make them economical to pump. Several oil company executives in recent weeks have said that once oil prices rebound to around $65 a barrel (the price was at $64.92 per barrel as of June 1), another wave of production will be unleashed.

This could help the U.S. to replace Saudi Arabia as the top swing producer—able to quickly ramp up (or down) production in response to price shifts. The real revolution on the horizon, however, is not in drilling equipment or practices: it’s in big data.

Thanks to new sensing capabilities, the volume of data produced by a modern unconventional drilling operation is immense—up to one megabyte per foot drilled, according to Mills’s “Shale 2.0” report, or between one and 15 terabytes per well, depending on the length of the underground pipes. That flood of data can be used to optimize drill bit location, enhance subterranean mapping, improve overall production and transportation efficiencies—and predict where the next promising formation lies. Many oil companies are now investing as much in information technology and data analytics as in old-school exploration and production.

At the same time, a raft of petroleum data startups, such as Ayata, FracKnowledge, and Blade Energy Partners, is offering 21st century analytics to oil companies, which have not been known for rapid, data-based innovation. Early efforts to bring modern data analytics into the oil and gas industry faltered, Jacobs says: “The oil companies tried to hire a bunch of data scientists, and teach them to be petroleum engineers. That didn’t go so well. The approach now is to take petroleum engineers and pair them up with technical experts who can supply the analytic horsepower, and try to marry these two groups.”

U.K.-based BP, for example, established a “decision analytics network” in 2012 that now employs more than 200 people “to examine ways to advance use of data and to help BP’s businesses harness these opportunities.”

If these initiatives succeed, big data could not only prolong the shale boom in the U.S., but also launch similar revolutions overseas. Applying the lessons from North America to low-producing oil fields elsewhere could unlock 141 billion barrels of oil in countries like China, Iran, Russia, and Mexico, IHS forecast in a report released last month.  

Keep Reading

Most Popular

Large language models can do jaw-dropping things. But nobody knows exactly why.

And that's a problem. Figuring it out is one of the biggest scientific puzzles of our time and a crucial step towards controlling more powerful future models.

The problem with plug-in hybrids? Their drivers.

Plug-in hybrids are often sold as a transition to EVs, but new data from Europe shows we’re still underestimating the emissions they produce.

Google DeepMind’s new generative model makes Super Mario–like games from scratch

Genie learns how to control games by watching hours and hours of video. It could help train next-gen robots too.

How scientists traced a mysterious covid case back to six toilets

When wastewater surveillance turns into a hunt for a single infected individual, the ethics get tricky.

Stay connected

Illustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at with a list of newsletters you’d like to receive.