As military and security agencies do everything from coordinating action in Iraq to sifting through telephone and banking records on the home front, they collect mountains of data that are difficult to organize and manipulate, and in which intelligence connections can get lost. Now, database tricks borrowed from the world of particle physics are helping intelligence and law enforcement agencies-and any other organizations that manage large, complex databases-to act on their data more nimbly.
The Stanford Linear Accelerator in Palo Alto, CA, is home to a particle collider that has generated the world’s largest database. The company that designed the database, Objectivity of Mountain View, CA, is now attracting a wave of new customers in the areas of defense, telecommunications, and intelligence systems, says Leon Guzenda, the company’s chief technology officer. TRW of Redondo Beach, CA (recently acquired by Northrop Grumman), says it uses Objectivity’s technology in a data analysis system built for an unnamed U.S. government agency that collects and analyzes scientific data. And SYColeman, a defense contractor in Sherman Oaks, CA, employs the technology in a system for controlling battlefield simulations. While Guzenda says security restrictions prevent him from describing specifics, he says the technology was used to help manage complex military operations in the Iraq war.
These national-security players are all attracted by the same thing: the system’s ability to keep complex relationships straight, even as databases swell to mammoth proportions. The underlying architecture of the database-in which data are stored in the form of “objects” that inherit their properties from parent objects in a vast family tree-can bring important connections to light more efficiently than traditional “relational” approaches, where data are stored in the rows and columns of interlinked tables, says Richard Winter, president of Winter Corporation, a database consulting firm in Waltham, MA. The new system has helped scientists keep track of Stanford’s particle collision data, which collectively takes up some 700 terabytes (700,000 gigabytes) of storage space, or 35 times as much as the contents of the Library of Congress (see “Surveillance Nation, Part Two,” TR May 2003). The “object-oriented” nature of the database makes it easy for scientists to create generations of objects from which more and more of the clutter has been extracted.
The same qualities make the database tool equally well suited to boosting national security. “Let’s say you are monitoring suspected terrorists’ telephone calls, e-mails, and the like. The database grows very large, very fast,” Winter says. In a relational database, the tables storing this information would form an abstract tangle, but a database like Stanford’s would more closely resemble the suspected terrorist network itself. “It can provide a groundbreaking kind of capability,” he says. Which means that, as government databases accumulate video surveillance data, communications intercepts, or battlefield intelligence, they’ll be able to detect where the data collides.
A quick guide to the most important AI law you’ve never heard of
The European Union is planning new legislation aimed at curbing the worst harms associated with artificial intelligence.
It will soon be easy for self-driving cars to hide in plain sight. We shouldn’t let them.
If they ever hit our roads for real, other drivers need to know exactly what they are.
Crypto is weathering a bitter storm. Some still hold on for dear life.
When a cryptocurrency’s value is theoretical, what happens if people quit believing?
Artificial intelligence is creating a new colonial world order
An MIT Technology Review series investigates how AI is enriching a powerful few by dispossessing communities that have been dispossessed before.
Get the latest updates from
MIT Technology Review
Discover special offers, top stories, upcoming events, and more.