Archiving E-mail Effectively

The White House's recent problems archiving e-mail could be solved by emerging technologies.

Losing e-mail can be a serious problem for both the public and private sector. Recently, the White House came under fire for failing to keep archives of its e-mail, even after the Clinton Administration instituted an archiving system due to a similar scandal. The Bush Administration set aside the existing archiving system as part of a move from Lotus Notes to Microsoft Exchange, and its current system reportedly relies on a combination of backup tapes and hand sorting. Earlier this week, according to the Washington Post, court documents revealed that the White House has not been able to find e-mails sent during a period in 2003 that encompasses the Iraq invasion.

Credit: Technology Review

E-mail can be hard to archive effectively because it gets sent in such vast quantities, which makes it difficult to store in a form that's easy to search and demonstrably tamper proof. Both of these requirements are particularly important for e-mail that's necessary for court cases and other legal situations. Fortunately, experts say that there are many good technologies for proper e-mail archiving and that the capabilities have only gotten better in recent years.

"Backup tapes make a lousy method for archiving e-mail," says Mark Diamond of Contoural, a data and storage consulting firm. Tape systems take snapshots of data at set daily or weekly intervals, leaving open the possibility that data could be both created and deleted before a copy can be made. Consequently, tape backups are really only good as insurance against system failure. A decent archiving system, Diamond says, "includes the ability to know what you have and the ability to . . . find e-mails and retrieve them easily, which is very difficult on backup tapes."

Story continues below

There's also a growing recognition that archiving systems need to be automated in order to work properly, in contrast to the laborious and error-prone manual sorting system currently in place at the White House. Even many digital archiving schemes store files in multiple locations. A complete search for all files related to a particular lawsuit might require searching many machines at different locations. Diamond says that a good archiving system makes a record of the e-mail as soon as it hits the server. Other automated tools aim to protect e-mail that relates to existing litigation, and to make it easy to sort through existing stored messages.

Diamond says that newer archiving systems go beyond simply storing e-mail, making it possible to recover in a matter of minutes data that might otherwise take weeks or months to get in hand. Such improvements are important, he says, because "every e-mail somebody sends, whether it's from the White House e-mail server or the company e-mail server, is a business document." Courts and regulators have been very clear, he says, that this means e-mail needs to be properly preserved. The costs can be high when good systems aren't in place. Beyond worries about the White House, Diamond points to a case between Intel and AMD in which the former reported spending more than $25 million recovering lost e-mail. The typical Fortune 500 company, Diamond says, has more than 150 legal actions pending at any given time. At least 50 percent of the cost of that litigation, he adds, lies in recovering needed documents, most of which are electronic.