The biggest success story in the hardware world – the 20-to-40 percent gains in hard-drive capacity that manufacturers have achieved each year since the early 1990s – has threatened to end abruptly as engineers run up against a physical limit on the number of individual bits that can fit on a magnetic disk.

Now a new hard-drive technology, “perpendicular magnetic recording,” being rolled out by Toshiba, Hitachi, and some other companies, promises to buy the industry at least a few more years of progress. It will soon allow users of laptops and handhelds to tote around unheard-of amounts of data, while at the same time hastening the demise of “longitudinal” recording, the method used on hard drives since their inception in the 1950s.

In May 2005, Toshiba introduced the first consumer drive incorporating perpendicular magnetic recording: a 1.8-inch, 40-gigabyte drive used in its Gigabeat MP3 player. Now the company is upping the ante with a 2.5-inch drive that holds five times as much data – a full 200 gigabytes. Introduced at this week’s Computex convention in Taipei, Taiwan, the drive is intended for laptop computers, which today typically come with hard drives no larger than 100 gigabytes. Meanwhile, Hitachi unveiled a 2.5-inch, 160-gigabyte drive with perpendicular recording a few weeks earlier, on May 15.

“We’ve been working on perpendicular recording for a very long time,” says John Best, chief technologist at the main laboratory of Hitachi’s Global Storage Technology Division in San Jose, CA. “We saw it as the thing that could keep [hard drive] density moving forward as longitudinal recording was nearing the end of its extendibility.”

On a traditional longitudinal drive, individual bits – clusters of metal grains that encode a 1 if magnetized in one direction, a 0 in the other – are laid down flat on the disk surface. The problem with this approach is that the only way to increase the density of bits on a drive is to make the bits themselves – and the crystalline magnetic grains that make them up – smaller. Below a certain size, however, a grain’s magnetic charge becomes unstable, and can be “flipped” simply by small changes in temperature (a phenomenon called the “superparamagnetic effect”). And if enough grains flip, a 0 might change to a 1 or vice versa, endangering data integrity.

In perpendicular magnetic recording, by contrast, the bits are stacked on end, and therefore can be crammed much more tightly. (For a goofy but entertaining animated cartoon from Hitachi explaining the concept, click here.) That means engineers not only can fit more bits into the same space, but, for a number of technical reasons, can also continue to make bits smaller.

Standing bits on end, for example, requires the use of a soft underlayer; this layer, which guides the magnetic flux from the drive’s read-write head as bits pass below it, makes the head more efficient, which means it can use a stronger magnetic field, which, in turn, means the grains can be made from a magnetically “stronger” material that’s less vulnerable to the superparamagnetic effect.

Perpendicular magnetic recording has been a long time coming – in part, because it involves arranging new materials in tricky new configurations, and in part because manufacturers wanted to test the technology thoroughly before putting it in front of consumers. “When you introduce perpendicular recording there is a fairly significant change in the material structure of the disk,” explains Best. “There’s a soft underlayer which carries the current from the read/write head, and a media layer on top, and when you introduce these materials you have to worry about things like surface roughness and susceptibility to corrosion from humidity. We built over 20,000 drives and put 5,000 of them into extended reliability tests before we even shipped the product. We wanted to make sure it was super-reliable.”

“Perpendicular recording is not an easy technology to bring to market,” confirms Maciek Brzeski, vice president of marketing for Toshiba’s Storage Device Division. “All of the prior technological improvements on hard drives have been somewhat incremental. But with perpendicular recording, it’s not a slam dunk. It takes some time and persistence.”

Manufacturers estimate that perpendicular recording will allow them to keep shrinking bits until hard drives reach 500 gigabytes, or perhaps 1,000 gigabytes (one terabyte). Eventually, however, the superparamagnetic effect will come back into play. “It’s just bought us a little bit of time – a few generations of hard drive models – and then we get into the same problem,” says Best. He says Hitachi is already studying “patterned media” –the idea of arranging individual grains in specific patterns to reinforce their magnetic stability–as a way to break the 1-terabyte barrier.

Toshiba will begin mass production of its 2.5-inch drive in August, and expects to see them in high-end consumer laptops in 2007. Hitachi says its 160-gigabyte, 2.5-inch drive will show up in a broader range of laptops; it plans to release a 1.8-inch drive for handheld devices next year.

Auspiciously for manufacturers, the new perpendicular-recording drives are arriving at just the moment when digital video – which consumes large amounts of disk space – is booming. Consumers can now use their computers to download network TV shows from iTunes, YouTube, Google Video, and other sources; to record directly from cable or satellite connections, as TiVo and other digital video recorders do; and to upload video they’ve captured themselves on the newest digital video cams, which can eat up 15 gigabytes of space per hour of video.

“For years we’ve dreamed about the consumer applications starting to provide the growth engine for hard drives, and now it’s actually here,” says Hitachi’s Best. “So it’s a pretty exciting time.”