Skip to Content
MIT Technology Review

The Human Screenome Project will capture everything we do on our phones

It proposes taking screenshots of your smartphone activity every five seconds to better understand our digital lives. Is it worth it?

January 15, 2020
screenshots of device activityscreenshots of device activity
screenshots of device activityMS Tech

If Byron Reeves has his way, the concept of “screen time” will be a relic. Instead, it will be your “screenome” that’s important.

Reeves, a professor at Stanford University, and two colleagues announced the launch of the Human Screenome Project today in Nature. The project aims to more accurately capture our digital footprint using an eyebrow-raising technique: background software that screenshots a volunteer’s phone every five seconds while it’s activated. The idea is that just as the Human Genome Project and genomics changed our understanding of diseases, big data could give us a better idea of how technology is linked with social problems.

Screen time is certainly due for a reckoning. The metric has long been used to measure how much children and teens interact with their digital devices. Because kids have only recently had widespread access to technology, research has been slow—and confounding. Screen time has been associated with negative outcomes like depression and lower literacy skills in kids. At the same time, it has been deemed helpful by some educational advocates (so long as a caregiver is monitoring use), and as any parent can attest, it can serve as a temporary relief from tantrums.

“Screen time is a popular carry-over measure from the context of a TV-centered era, developed around health and parenting concerns,” says Mimi Ito, a cultural anthropologist who studies technology use at the University of California. Even the American Association of Pediatrics, which first popularized the term, has moved away from screen time as a core measure, she says.

The concept might also overlook finer sociological data. In the paper, Reeves describes two 14-year-old boys living in the same Northern California town who had their activity logged from the moment they woke up until they went to bed. At first glance, the boys’ activity might seem exactly the same, but Reeves’s screenome analysis suggests otherwise: one boy had 186 sessions lasting about a minute each, while the other boy had only 26 daily sessions, each lasting almost three minutes each. The first spent a lot of time messaging on Snapchat and Instagram, while the second spent half his time on YouTube and screenshotting food.

As well as giving us better insight into their differing media diets (one consumes more content; the other produces it), the example shows how two people who might typically be clumped into the same data point actually use their devices in very different ways. Every user has an “idiosyncratic” digital fingerprint, Reeves’s team suspects.

To Reeves, identifying specific sequences of smartphone activity is important to understanding how people actually use their digital devices to communicate and lead their off-screen lives.

“It’s about how people connect fragments that seem like they’re unrelated,” he says. “They can go from reading Facebook friends’ posts to looking at presidential campaign news to banking, all in the same minute ... That really has nothing to do with total amount of time [spent].”

These pathways from app to app, post to post, errand to activity, reflect and affect our analog lives, Reeves argues. A screenome would offer a way to study smartphones and tablets for patterns of use linked to issues such as social-media addiction and mental health problems. The project is only in its beginning stages, but thus far, Reeves’s team has found that the ways people flit about on their devices can help explain the proliferation of fake news and act as an epidemiological indicator for diabetes

Still, Ito isn’t sure a “screenome” will answer the key question of how screens shape our activities, interests, and social engagements. It won’t include all the offline information that is driving the online behavior, she says. 

One of the biggest obstacles to the project’s success is likely to be that it raises fears around privacy. Having an app quietly record your activity every five seconds is a hard sell. If the past few years have shown anything, it’s that even the most inane activities online are tracked. That information is sold to advertisers at best, or to hackers and disinformation campaigns at worst. The Cambridge Analytica scandal highlighted how personality tests shared between acquaintances on Facebook were weaponized by Russians in the 2016 American election, for example.

And consider what passes across our phone screens every day: bank account information; emails carrying personal data; car-sharing routes detailing addresses of destinations; meal delivery orders; texts with our loved ones; photos and videos of children; even pornography, cryptocurrency exchanges, and illicit activity.

“It’s a lot of sensitive information,” Reeves concedes. His team has amassed around 30 million screenshots from volunteers in the US as well as China and Myanmar. He says that he’s sensitive to privacy concerns, and that compressed images are sent through an encrypted, secure server at Stanford. 

Andrew Przybylski, the director of research at the Oxford Internet Institute, thinks these privacy issues could hold the project back: “I think it’s a novel approach but doomed to failure if the research isn’t grounded in robust, transparent, open science,” he says.

While the Screenome Project is only in its beginning stages, the data collected so far from hundreds of volunteers is “jaw-dropping.” When asked what it’s like to see screenshot frames, Reeves said it can be tempting to “watch voyeuristically.”

“It’s fascinating,” he says. “You feel like you get to know a person.”