Select your localized edition:

Close ×

More Ways to Connect

Discover one of our 28 local entrepreneurial communities »

Be the first to know as we launch in new countries and markets around the globe.

Interested in bringing MIT Technology Review to your local market?

MIT Technology ReviewMIT Technology Review - logo

 

Unsupported browser: Your browser does not meet modern web standards. See how it scores »

Your Twitter stream (and, soon, your history of Facebook wall posts) constitutes a rich source of information about you, or, considering that most Tweets are public, just about anyone else. It includes everything from your speech patterns and the topics you obsess over to the identity of your “real” friends–at least in the tweet-o-sphere. The trick is unlocking it.

The first step to jacking into your subconscious is to download a representative sample of your tweets. Fortunately, Twitter stores the last 3,200 of them, which is probably more than you’ve ever spouted, unless you’re especially prolific.

One easy way to download all those tweets is to sign up for a service like BackupMy.Net, which will grab all available tweets (up to that 3,200 limit) and allow you to download them in a number of different formats.

Step two, get yourself a free copy of TextWrangler (if you’re on a Mac) or its Windows equivalent. If you’re on Linux, vi or emacs can do everything I’m about to describe.

Step three, start filtering your corpus of tweets. Think of it as a mound of wet clay. You can ask it any questions you like. Here’s what I asked mine; if you have other ideas, leave them in the comments.

i) Delete every other line in order to eliminate the noise of all those time and date stamps. TextWrangler has a powerful “Process Lines Containing…” feature under the “Text” menu. It allows you to, in this case, delete every line containing the string “+0000” which I’m guessing is the blank variable where geo-location data would normally go. If your Tweets are geo-located, just have it search for every year in which you tweeted, assuming you don’t often drop years into your tweets.

This:

Becomes this:

Now paste the resulting text into Wordle, the word cloud generator, because word clouds are just pretty ways to visualize word frequency.

For a first approximation, it’s not bad - some of your interests and friends can be seen trying to peek out around garbage words we all use on Twitter, like “RT”

ii) Using a simple find and replace command, get rid of RT, via and any other words that don’t tell you anything. Now you can see what you’re obsessed with. I happened to be obsessed with climate change, news buzzwords (“new,” “now”) and, apparently, signaling sarcasm and ironic distance by beginning sentence fragments with the word “apparently.”

iii) For a purer distillate of your own speech patterns and neuroses, use the “Process Lines Containing…” command or its equivalent to remove all lines that are retweets.

iv) If you want to find out who your “real friends” are on Twitter (and, to a lesser extent, what you talk to them about) extract all the lines that have an @ in them.

If you want to know what you say when you’re talking to your friends, simply delete every word beginning with an @. Or sort the document so you can grab only every conversation you’ve ever had with a particular person, and turn *that* into a word cloud.

v) If you want to know you talk when you’re talking about yourself, simply extract every line with the (case-sensitive) string “I “

There are countless tools on the web for analyzing your twitter stream, from frequency analysis to emotional content, but so far as I know, downloading all your tweets and parsing them yourself is the only way to visualize their actual content with this level of specificity. I’m sure there are dozens of queries I haven’t even thought of that are possible with this method - feel free to leave your ideas in the comments.

Follow Mims on Twitter or contact him via email.

4 comments. Share your thoughts »

Tagged: Computing

Reprints and Permissions | Send feedback to the editor

From the Archives

Close

Introducing MIT Technology Review Insider.

Already a Magazine subscriber?

You're automatically an Insider. It's easy to activate or upgrade your account.

Activate Your Account

Become an Insider

It's the new way to subscribe. Get even more of the tech news, research, and discoveries you crave.

Sign Up

Learn More

Find out why MIT Technology Review Insider is for you and explore your options.

Show Me