U of T undergraduate co-wins prestigious research award at AIES Conference

Inioluwa Deborah Raji awarded best paper for detecting facial recognition bias in Amazon technology

U of T undergraduate co-wins prestigious research award at AIES Conference

Amazon’s facial recognition technology may be misidentifying dark-skinned women, according to U of T Engineering Science undergraduate Inioluwa Deborah Raji and Massachusetts Institute of Technology Media Lab research assistant Joy Buolamwini. This finding helped Raji and Buolamwini win “best student paper” at the Artificial Intelligence, Ethics, and Society (AIES) Conference in Honolulu, Hawaii. Held in January, the prestigious conference was sponsored by Google, Facebook, Amazon, and the like.

Their paper, which caught the Toronto Star’s attention, was a follow-up on an earlier audit by Buolamwini on technology from Microsoft, IBM, and Face++, a facial recognition startup based in China.

Origins of the research

Buolamwini’s earlier study, “Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification,” investigated the accuracy of artificial intelligence (AI) systems used by the three technology firms for facial recognition. Then-Microsoft Research computer scientist Timnit Gebru co-authored the paper.

Raji wrote that after reading about Buolamwini’s experiences “as a black woman being regularly misgendered by these models,” she wondered if her personal experience would hold true for a larger dataset containing samples of other dark-skinned women. This proved to be the case in the final analysis.

According to Raji, “Gender Shades” uncovered “serious performance disparities” in software systems used by the three firms. The results showed that the software misidentified darker-skinned women far more frequently than lighter-skinned men.

In an email to The Varsity, Raji wrote that since the release of Buolamwini and Gebru’s study, all three audited firms have updated their software to address these concerns.

For the paper submitted to the AIES Conference, Raji and Buolamwini tested the updated software to examine the extent of the change. They also audited Amazon and Karios, a small technology startup, to see how the companies’ adjusted performance “compared to the performance of companies not initially targeted by the initial study.”

At the time of Raji and Buolamwini’s follow-up study in July, Raji wrote that “the ACLU [American Civil Liberties Union] had recently reported that Amazon’s technology was being used by police departments in sensitive contexts.”

Amazon denied that bias was an issue, saying that it should not be a concern for their “partners, clients, or the public.”

Raji and Buolamwini’s study showed evidence to the contrary. “We found that they actually had quite a large performance disparity between darker females and lighter males, not working equally for all the different intersectional subgroups,” wrote Raji.

Amazon’s response to the study

In a statement sent by Amazon’s Press Center to The Varsity, a representative wrote that the results of Raji and Boulamwini’s study would not be applicable to technologies used by law enforcement.

Amazon wrote that the study’s results “are based on facial analysis and not facial recognition,” and clarified that “analysis can spot faces in videos or images and assign generic attributes such as wearing glasses,” while “recognition is a different technique by which an individual face is matched to faces in videos and images.”

“It’s not possible to draw a conclusion on the accuracy of facial recognition for any use case – including law enforcement – based on results obtained using facial analysis,” continued Amazon. “The results in the paper also do not use the latest version of Rekognition and do not represent how a customer would use the service today.”

In a self-study using an “up-to-date version of Amazon Rekognition with similar data downloaded from parliamentary websites and the Megaface dataset of 1M images,” explained Amazon, “we found exactly zero false positive matches with the recommended 99% confidence threshold.”

However, Amazon noted that it continues “to seek input and feedback to constantly improve this technology, and support the creation of third party evaluations, datasets, and benchmarks.” Furthermore, Amazon is “grateful to customers and academics who contribute to improving these technologies.”

The pair’s research could inform policy

Raji wrote that while it’s tempting for the media to focus on the flaw in Amazon’s software, she thinks that the major contribution of her paper is in helping to uncover how researchers can effectively conduct and present an audit of an algorithmic software system to prompt corporate action.

“Gender Shades introduced the idea of a model-level audit target, a user-presentative test set, a method for releasing results to companies called Coordinated Bias Disclosure,” wrote Raji.

In other words, Raji and Buolamwini’s research showed an effective way for companies and policymakers to investigate and communicate a problem in software systems and take action.

Most importantly, wrote Raji, the studies highlight the need for researchers to evaluate similar software models “with an intersectional breakdown of the population being served.”

Toronto on Amazon’s shortlist for second headquarters

“U of T is a big part of our success story,” says Mayor Tory

Toronto on Amazon’s shortlist for second headquarters

The Toronto region has earned a spot on Amazon’s shortlist for a second headquarters, to be launched by the end of this year. The addition would bring $5 billion in investments and create as many as 50,000 jobs for its host city.

The 20-city shortlist, released January 18, named Toronto as the only non-American city in the running to host the tech giant’s newest project. The project, called Amazon HQ2, received proposals from 238 North American cities when the company called for submissions in September 2017.

Finalists are to be informed of next steps within the coming weeks. Toby Lennox, CEO of Toronto Global — which worked in shaping Toronto’s HQ2 bid — indicated that the Toronto region is anxiously waiting to hear back from Amazon.

Lennox said that Toronto’s proposal to Amazon, made in October 2017, highlighted Toronto as a hub for high quality talent, stemming from its universities. The proposal was “an opportunity for us to hold up a mirror to ourselves,” he told The Varsity.

“We are now a really big, complicated, dynamic, and vibrant city. We’re a global player,” said Lennox. “It’s all the work that all the people have put in to Toronto to make it the region that it is now. And it’s great to see us for what we are.”

While Amazon provided no explicit rationale for selecting the top 20, Lennox and Toronto Mayor John Tory cited Canadian openness to immigration and the city’s innovative growth as major assets attracting corporate investment.

In a statement to The Varsity, Tory emphasized the significance of educational capital, which he said was incorporated as a key selling point of Toronto in the bid to Amazon.

“Our pitch to Amazon highlights the fact we are building a Future-Proofed Talent Pipeline, including a 25% boost in the number of STEM graduates in Ontario,” reads Tory’s statement. “U of T is a big part of our success story.”

Tory said that the university-affiliated Vector Institute “is helping build our reputation in technology and around the world.”

Lennox noted Amazon’s interest in working with universities engaged in research and development. The company hires university interns and full-time employees in the United States, and Lennox hopes that HQ2 would carry that pattern north.

There would still be drawbacks to HQ2 in Toronto, said Shauna Brail, Director of the Urban Studies Program at U of T and a Senior Associate at the Munk School’s Innovation Policy Lab.

Among those drawbacks would be Amazon shifting the job market away from the service and manufacturing industries with augmented automation. Brail also illuminated the consequences HQ2 could impose on Toronto’s already oversaturated and expensive housing market.

Even if the Toronto region is not Amazon’s ultimate choice, Tory, Lennox, and Brail all agree that this sort of investment interest reflects Toronto’s already flourishing tech sector and economy at large.

If Toronto doesn’t win, “it doesn’t change the city’s fundamental attractiveness to other companies,” said Lennox. “This is our opportunity. It’s like we’ve already won this thing. Let’s keep going at it. And if Amazon comes here, great. We’re still going to keep doing the same things.”