Hello,

We noticed you're browsing in private or incognito mode.

To continue reading this article, please exit incognito mode or log in.

Not a subscriber? Subscribe now for unlimited access to online articles.

Intelligent Machines

Why We Should Expect Algorithms to Be Biased

We seem to be idolizing algorithms, imagining they are more objective than their creators.

Technologies driven by algorithms and artificial intelligence are increasingly present in our lives, and we are now regularly bumping up against a thorny question: can these programs be neutral actors? Or will they always reflect some degree of human bias?

The dustup over Facebook’s “trending topics” list and its possible liberal bias hit such a nerve that the U.S. Senate called on the company to come up with an official explanation, and this week COO Sheryl Sandberg said the company will begin training employees to identify and control their political leanings.

This is just one result, however, of a broader trend that Fred Benenson, Kickstarter’s former data chief, calls “mathwashing”: our tendency to idolize programs like Facebook’s as entirely objective because they have mathematics at their core.

One area of potential bias comes from the fact that so many of the programmers creating these programs, especially machine-learning experts, are male. In a recent Bloomberg article, Margaret Mitchell, a researcher at Microsoft, is quoted lamenting the dangers of a “sea of dudes” asking the questions central to creating these programs.

Concern has been building over this issue for some time, as studies found evidence of bias in online advertising, recruiting, and pricing strategies driven by presumably neutral algorithms.

In one study, Harvard professor Latanya Sweeney looked at the Google AdSense ads that came up during searches of names associated with white babies (Geoffrey, Jill, Emma) and names associated with black babies (DeShawn, Darnell, Jermaine). She found that ads containing the word “arrest” were shown next to more than 80 percent of “black” name searches but fewer than 30 percent of “white” name searches.

Sweeney worries that the ways Google’s advertising technology perpetuates racial bias could undermine a black person’s chances in a competition, whether it’s for an award, a date, or a job.

Areas such as lending and credit that have traditionally suffered from well-known human discrimination must be especially careful.

ZestFinance, an online lender founded on the idea that machine-learning programs can expand the number of people deemed creditworthy by looking at tens of thousands of data points, maintains that it is well attuned to the dangers of discriminatory lending. To guard against discrimination, ZestFinance has built tools to test its own results.

But the danger remains that unrecognized bias, not just in the programming of an algorithm but even in the data flowing into it, could inadvertently turn any program into a discriminator. For consumers who are unable to unpack the complexities of these programs, it will be hard to know whether they have been treated fairly.

“Algorithm and data-driven products will always reflect the design choices of the humans who built them,” Benenson explained in a recent Q&A with Technical.ly Brooklyn, “and it’s irresponsible to assume otherwise.”

(Read more: Wall Street Journal, Technical.ly Brooklyn, Bloomberg, “AI Takes Off”[PDF])

Keep up with the latest in machine learning at EmTech Digital.

The Countdown has begun.
March 25-26, 2019
San Francisco, CA

Register now
Next in Top Stories

Your guide to what matters today

Want more award-winning journalism? Subscribe to MIT Technology Review.
  • Print + All Access Digital {! insider.prices.print_digital !}* Best Value

    {! insider.display.menuOptionsLabel !}

    The best of MIT Technology Review in print and online, plus unlimited access to our online archive, an ad-free web experience, discounts to MIT Technology Review events, and The Download delivered to your email in-box each weekday.

    See details+

    12-month subscription

    Unlimited access to all our daily online news and feature stories

    6 bi-monthly issues of print + digital magazine

    10% discount to MIT Technology Review events

    Access to entire PDF magazine archive dating back to 1899

    Ad-free website experience

    The Download: newsletter delivery each weekday to your inbox

    The MIT Technology Review App

  • All Access Digital {! insider.prices.digital !}*

    {! insider.display.menuOptionsLabel !}

    The digital magazine, plus unlimited site access, our online archive, and The Download delivered to your email in-box each weekday.

    See details+

    12-month subscription

    Unlimited access to all our daily online news and feature stories

    Digital magazine (6 bi-monthly issues)

    Access to entire PDF magazine archive dating back to 1899

    The Download: newsletter delivery each weekday to your inbox

  • Print Subscription {! insider.prices.print_only !}*

    {! insider.display.menuOptionsLabel !}

    Six print issues per year plus The Download delivered to your email in-box each weekday.

    See details+

    12-month subscription

    Print magazine (6 bi-monthly issues)

    The Download: newsletter delivery each weekday to your inbox

/3
You've read of three free articles this month. for unlimited online access. You've read of three free articles this month. for unlimited online access. This is your last free article this month. for unlimited online access. You've read all your free articles this month. for unlimited online access. You've read of three free articles this month. for more, or for unlimited online access. for two more free articles, or for unlimited online access.