Solar power was the fastest-growing form of electricity generation in the United States in 2014. As renewable energy continues to expand, demand is growing for a better way to predict just how much power from these intermittent sources will be available for the grid.

IBM shared new details last week on its program to harness powerful computers to forecast weather and other factors that determine the output of solar and wind installations. Using machine learning and advanced data analytics, IBM is making an aggressive push to give utilities, plant managers, and grid operators clearer guidance on what their arrays will put out today, tomorrow, next week, and even months from now.

At last week’s European Control Conference in Linz, Austria, scientists from IBM and the National Renewable Energy Laboratory (NREL) said they will make the forecasts available, free of charge, to users across the continental United States.

Solar and wind forecasts produced by IBM’s technology are as much as 30 percent more accurate than conventional forecasts, according to Hendrik Hamann, a research manager at IBM. Such precision could make it possible to avoid generating hundreds of megawatts of excess power every year and reduce the need for new “peaker” plants to supply power in times of peak demand, potentially lowering carbon emissions and saving utilities and ratepayers millions of dollars. An NREL study of the independent system operator for New England found that making solar forecasting 25 percent more accurate would offer potential cost savings of $46.5 million a year across the region.

“What we are doing is combining multiple models together into one ‘supermodel,’” says Hamann. That meta-forecasting program can weigh various weather predictions according to historical performance data associated with various atmospheric conditions, locations, and circumstances. The output can be tailored to different users—Midwest utilities, Nevada solar operators, wind-farm managers, and so on.

Solar producers are particularly susceptible to sudden swings in output; a solar array that’s partially shaded can lose 70 percent of its generating capacity within seconds. Utilities serving large numbers of households or businesses with distributed generation capacity also need to anticipate how much power will be flowing into the grid from rooftop solar installations.

As a result, demand for better forecasting is soaring. A clutch of startups and established companies have moved to serve that demand in the last several years (see “10 Breakthrough Technologies 2014: Smart Wind and Solar Power”). “Operators in California need constant updates to feed into their load forecasting system, so every 15 minutes we’re simulating 200,000 solar PV systems, modeling every single location on its own along with the irradiation forecasts,” says Jeff Ressler, head of the software services group for Clean Power Research, which provides forecasting tools for utilities such as the Los Angeles Department of Water and Power, Southern California Edison, and the Salt River Project in Arizona.

 Clean Power Research’s technology is based on the work of Richard Perez, a professor at the State University of New York’s Atmospheric Research Center, in Albany, who pioneered the use of satellite imagery to measure irradiance, i.e., the amount of sunlight reaching the Earth. For longer-term  (days ahead) forecasts, Clean Power Research also includes inputs from numerical weather prediction (NWP), which uses information from a global (though unevenly distributed) system of sensing stations and runs computer simulations to extrapolate from them. The U.S. National Weather Service uses the technique to build its own weather models, including the Global Forecast System and the North American Mesoscale Forecast System. According to John Zack, director of forecasting at AWS Truepower, which provides forecasts for wind and solar in North America, the private-sector forecasters provide technology that works on top of NWP models. “The game for day-ahead forecasting is how well your technology corrects for the systematic errors in the NWP models,” says Zack. The IBM system detailed last week brings the most computing firepower to that problem so far.

The eventual goal is to integrate highly accurate real-time forecasting with generation resources linked by cloud-based computing networks. The resulting virtual power plants would automatically dispatch power in the most efficient manner from the most cost-effective (and cleanest) resource, avoiding the use of expensive and dirty reserve power.

As sophisticated as this technology is becoming, though, no model is or ever will be perfect. Solar forecasting is an archetypal example of the butterfly effect, where tiny changes can have large-scale effects across time and space. “You can’t model every raindrop and turbulent eddy out there,” says Zack. “You have to make approximations.”