Skip to Content

Data Mining Reveals Historical Events in Government Archive Records

The course of history is often hidden in government archives. Now statisticians have worked out how to extract the most significant events using data-mining techniques.

The study of political history is undergoing a revolution. The driving force behind this change is the availability of electronic records of government activity and the ability to number-crunch this data in ways that have never before been possible.

Historians have immediate access to contemporary news reports of important world events. But these reports need to be calibrated by comparing them to government records of the time, which may differ in important and unknown ways. But these records are not usually made public until some years later. The sheer number of government documents makes this process difficult since it is all too easy for researchers to overlook something significant.

So an automated way of searching these government records for significant events would be hugely useful.

Data mining reveals the spike in government communications during fall of Saigon (top) but little of significance related to Finland (bottom).

Today, Yuanjun Gao at Columbia University in New York and a few pals say they have developed a set of statistical tools that can do this job. They have tested the technique on U.S. government electronic records released by the National Archives and compared the results with contemporary news records.

The work is made possible by the fact that the U.S. State Department has been storing classified communications electronically since 1973. Many of the records dating from 1973 to 1977 are now publicly available and consist of 1.4 million declassified cables and the metadata associated with 0.4 million documents delivered by diplomatic pouch.

These electronic records are all assigned tags related to their topic. So communications related to South Vietnam are tagged VS, those related to the United Nationals General Assembly tagged UNGA, to Finland tagged FI, and so on.

That allows researchers to track communications related to specific topics without knowing the details of the messages themselves.

Plotting the number of messages over time reveals some interesting patterns. For example, it reveals an increase in communications relating to South Vietnam in April 1975 at the time of the fall of Saigon.

Communications related to the UN general assembly spike at regular intervals corresponding to the dates of the assemblies which occur every year. This pattern includes an additional spike in April-May 1974 which corresponds to a special UN session called for by Algeria to discuss its demands for support for a “New International Economic Order.”

By contrast, the tags associated with Finland between 1973 and 1977 show no spikes or special pattern, reflecting the stability of the nation at that time.

Major spikes are relatively easy to see visually, but Gao and co have developed tools to spot them automatically and rank their significance by comparing them to background level of activity at the time.

That allows the team to produce a ranking of the top 30 most significant “events” during this period.

Not all of these spikes correspond to important world events. For example, the top two most significant spikes relate to administrative issues such as transport and changes to the visa records system.

But the ranking finds a wide range of other important events, such as the Carter administration’s prioritization of human rights, the president of Egypt, Anwar Sadat’s surprise visit to Israel in 1977, the Southeast Asian “Boat People” crisis of 1975-76, the 1973 Yom Kippur War and Portugal’s withdrawal from Angola in 1975-76, and so on.

That’s interesting work that should change the way historians and social scientists do their work. And this is just the beginning. These kind of automated methods can also show how humans used the tagging process at the time, how often mistakes were made, and whether certain types of messages may have gone missing.

And as more detailed electronic records are released, researchers will be able to develop more advanced ways of mining them. Political history will never be the same.

Ref: Mining Events with Declassified Diplomatic Documents

Keep Reading

Most Popular

A Roomba recorded a woman on the toilet. How did screenshots end up on Facebook?

Robot vacuum companies say your images are safe, but a sprawling global supply chain for data from our devices creates risk.

A startup says it’s begun releasing particles into the atmosphere, in an effort to tweak the climate

Make Sunsets is already attempting to earn revenue for geoengineering, a move likely to provoke widespread criticism.

10 Breakthrough Technologies 2023

Every year, we pick the 10 technologies that matter the most right now. We look for advances that will have a big impact on our lives and break down why they matter.

These exclusive satellite images show that Saudi Arabia’s sci-fi megacity is well underway

Weirdly, any recent work on The Line doesn’t show up on Google Maps. But we got the images anyway.

Stay connected

Illustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at with a list of newsletters you’d like to receive.