“Data are the lifeblood of decision-making and the raw material for accountability,” states the UN Data Revolution report. New technologies are leading to an exponential increase in the volume and types of data available, creating unprecedented possibilities for informing and transforming society and protecting the environment. Governments, companies, researchers and citizen groups are in a ferment of experimentation, innovation and adaptation to the data revolution. How will the social science and computer architecture built up around this mountain of data, allow it to be safely used to help adaptation to the new world, a world that is more fair, efficient, and inclusive, and that provides greater opportunities than ever before?

Many of the traditional ideas we have about ourselves and how society works are wrong. It is not simply the brightest who have the best ideas; it is those who are best at harvesting ideas from others. It is not only the most determined who drive change; it is those who most fully engage with like-minded people. And it is not wealth or prestige that best motivates people; it is respect and help from peers. Humans, as a species, are on a continual search for new opportunities, for new ideas, and their social networks serve as a major, and perhaps the greatest, resource for finding opportunities. It is the flow of ideas and opportunities between people that drives society, providing quantitative results at scales ranging from small groups, to companies, cities, and even entire countries.

The disconnect between traditional ideas about our society and the current reality has grown into a yawning chasm because of the effects of digital social networks and similar technology. Much of humanity now has a two-way digital connection that can send voice, text, and, most recently, images and digital sensor data because cellphone networks have spread nearly everywhere. Information is suddenly something that is potentially available to everyone. The data revolution combines this enormous new stream of data about human life and behavior with traditional data sources, enabling a new science of social physics that can let us detect and monitor changes in the human condition, and to provide precise, non-traditional interventions to aid human development.

To understand our new, hyper-connected world we must extend familiar economic and political ideas to include the effects of these millions of digital citizens learning from each other and influencing each other’s opinions. We can no longer think of ourselves as only rational individuals reaching carefully considered decisions; we must include the dynamic social networking effects that influence our individual decisions and drive economic bubbles, political revolutions, and the internet economy. Key to this are strong and rapid reputation mechanisms, and inclusiveness in both planning and execution.

Data is the key to rapid societal change

Today it is hard to even imagine a world where we have reliable, up-to-the-minute data about how government policies are working, and about problems as they begin to develop. Perhaps the most promising uses for big data are in systems like OPAL (Open Algorithm), which allow statisticians in government statistics departments around the world to make a more accurate, real-time census and more timely and accurate social surveys.

OPAL brings stakeholders together to determine what data—both private and public—should be made accessible and used for which purpose and by whom. Indeed, it provides a forum and mechanism for deciding what levels of transparency and accountability are best for society. It also provides a natural mechanism for developing evidence-based policies and a continuous monitoring of the various dimensions of societal well-being, thus offering the possibility of building a much deeper and more effective science of public policy.

The keys to successful rapid change are granular data about what other people are doing and how it is working out for them, and the realization that we cannot actually act independently because every action we take affects others and feeds back to ourselves as others adapt to our actions. In prehistoric times people knew quite accurately what other members of their village did and how well it worked, and this allowed them to quickly develop compatible norms of behavior. Today we need digital mechanisms to help us know about what works and what does not work. This sort of reputation mechanism is exactly the benefit of using an architecture like OPAL.

OPAL is a key contribution to the UN Data Revolution—it is developing and testing a practical system that ensures that every person counts, their voices are heard, their needs are met, their potentials are realized, and their rights are respected.

Better public data can allow both the government and private sectors to function better, and with these sorts of improvements in transparency and accountability, we can hope to build a world in which our government and social institutions work for everyone.

Historically we have always been blind to the living conditions of the rest of humanity; violence or disease could spread to pandemic proportions before the news would make it to the ears of central authorities. We are now beginning to be able to see the condition of all of humanity with unprecedented clarity. Never again should it be possible to say: “We didn’t know.”