"Reality Mining" the Organization

Data mining is a start, but it misses the critical pieces of information that are transmitted by word of mouth. Social networking systems can foster collaboration.

By Alex Pentland

March 31, 2004

Who are the experts within your organization? Who has the most decision-making influence? Recently, managers have started mining data from e-mail, Web pages, and other digital media for clues that will help answer such questions. That's a start, but it misses the real action: studies of office interactions indicate that as much as 80 percent of work time is spent in spoken conversation, and that critical pieces of information are transmitted by word of mouth in a serendipitous fashion. Fortunately, the data infrastructure for mining real-world interactions is already in place. Most working professionals already carry microphones (cell phones), and many also carry PDAs with ample computational horsepower. This foundation of mobile communications and processing power will support an exciting new suite of business applications: reality mining.

The MIT Media Lab's Human Design research group is demonstrating that commonplace wearable technology can be used to characterize the face-to-face interactions of employees-and to map out a company's de facto organization chart. This capability can be an extraordinary resource for team formation and knowledge management.

The new reality-mined data allow us to cluster people on the basis of profiles generated from an aggregate of conversation, e-mail, location, and Web data. This clustering, in turn, enables us to identify collaboration or lack thereof. For instance, if two groups working on mobile commerce never talk face-to-face, that is a clear sign that they are not coordinating their efforts.

By leveraging recent advances in machine learning, we can build computational models that simulate the effects of organizational disruptions in existing social networks. We could, for example, predict the organizational effects of merging two departments. Such data-driven models help us transcend the traditional "org" chart, allowing organizations to form groups on the basis of communication behavior rather than hierarchy.

Two approaches we are experimenting with could improve organizational performance:

Expert and Collaborator Locator. With speech recognition technology, we can generate profiles of individuals based on the words they use in conversations. These profiles help identify the people within an organization who have particular expertise. By querying profiles-perhaps by picking people who socialize during lunch or group activities-a manager can form a team of employees with harmonious social behavior and skills. This technology can also minimize redundant work by helping managers identify clusters of people who work on similar projects within the larger organization. 

Collaboration Tools. Although standard data-mining operations can analyze existing corporate information, the results reflect a limited and static view of an organization's human and social capital. Augmenting knowledge management with information gathered by unobtrusive wearable sensors that measure such qualities as tone of voice, or "prosody," and body language could be enormously helpful to organizational collaboration. The results can help managers understand who is working with whom and infer the relationships between colleagues. A database of employee profiles that responds to changes in e-mail and oral-conversation behavior and content can lend insight into the sources of an organization's in-house expertise. Querying this database for interests, skills, or even recently used vocabulary would be an efficient way to find people who might work well together.

It's our belief that active analysis of interactions within the workplace can radically improve the functioning of an organization. We expect that by aggregating this information, interpreting it in terms of work tasks, and modeling the dynamics of the interactions, we will be better able to understand and manage complex organizations. We must also, however, provide for the protection of privacy (for instance, by giving employees control over their information) and create an atmosphere of transparency by including the manager's interactions as part of the initiative. If employees can scrutinize their bosses' behavior, they're less likely to object to making their own interactions visible.

Story continues below

To test out this technology, we have focused first on innovation within research groups at the Media Lab, improving the dynamics of small meetings and increasing success of one-on-one business negotiations. We are now validating the approach with larger and more complex organizations beyond the university environment, and we hope to see serious commercial applications within the year.

This article originally appeared in the MIT Technology Insider, a monthly newsletter covering MIT research and commercial spinoff activity.