This week, IBM announced that it has smashed the speed record for data-sharing across a network within a supercomputer. Previously, files could zip from a supercomputer's memory devices to its processors at about 15 gigabytes per second, a record that IBM had established, says Rama Govindaraju, an IBM distinguished engineer who worked on the recent project. Now, with advanced file-management software and some system tweaks, his group has boosted that rate to 102 gigabytes per second (the equivalent of downloading about 25,000 songs from the Web).

This file-managing feat, which they call "Project Fastball," was conducted using IBM's General Parallel File System software on the world's third-fastest supercomputer, ASC Purple, which resides at Lawrence Livermore National Laboratory (LLNL) in Livermore, CA. ASC Purple is a giant collection of thousands of storage devices and processors that house and crunch 1.6 petabytes of data (1.6 million gigabytes). It's used to run data-intensive simulations that model nuclear reactions and to assess the safety of nuclear stockpiles.

But scientific research is not the only field that can benefit from faster-flowing data, Govindaraju says. "What this does is enable a new class of applications to become possible," he says. Indeed, the breakthrough could be used with any large system of data that needs to be accessed or processed. Thus, it could be highly desirable to an increasing number of organizations that store data on remote, networked server farms throughout the world. In particular, it might benefit the health-care industry, video-on-demand providers, and homeland security workers.

At the heart of the breakthrough is the General Parallel File System (GPFS), IBM software that has been available since 2001. A supercomputer's file system -- GPFS is one of a handful of options -- is, in essence, similar to the folders system on a home computer that allows one to type a file name or pathway and locate a particular file, says Steve Scott, CTO at Cray, the Seattle-based supercomputing company.

Unlike most PCs, however, a supercomputer has multiple processors, performing tasks simultaneously (in parallel), and which need to be constantly fed data from files located throughout the thousands of storage devices in the system. GPFS efficiently manages the flow of data at the supercomputer scale.

The chore of directing the data traffic to and from processors and storage devices is challenging in such large systems -- and is also one of the limiting factors in computational efficiency, says Eng Lim Goh, senior vice president and CTO of Mountain View, CA-based Silicon Graphics, Inc. "The thickness of the connection between the processors will decide how long it takes for a huge piece of data to get through," he says. If the data gets backed up because the bandwidth isn't large enough, "you are not computing, and efficiency keeps dropping."