School facial recognition software misidentifies black women 16 times more often than white men, report

Racial bias in AI in the spotlight as vendor allegedly lied about accuracy to school district, and leading researcher Timnit Gebru exits Google

Security vendor SN Technologies allegedly lied to officials at New York's Lockport City School District about the accuracy of its facial recognition system which was installed in the district's schools last year.

That's according to Vice, which claims that it recently obtained a copy of a report from the New York Civil Liberties Union (NYCLU), which indicates that Lockport's AEGIS facial recognition system is marred with various performance issues and has failed to give satisfactory results.

Freed Maxick, an accountancy firm, was hired by the district earlier this year to audit SN Technologies' claims. In its report, the company found that Lockport's system misidentifies black students more frequently than claimed by its supplier.

The system misidentifies black women 16 times more often than white men and black men four times more often. Not only that, AEGIS' weapon-detecting feature also generates false alerts a number of times, after mistaking broom handles for guns.

Lockport School District had switched on the system earlier this year, after the New York State Education Department (NYSED) reversed its previous decision to block the system, pending a set of policy changes. The system is supposed to identify people (unauthorised school staff, sex offenders, or other criminals) whose photographs are stored in a database maintained by the district.

According to Vice, SN Technologies' CEO KC Flynn lied to district officials last year when he claimed that the id3 Technologies algorithm used by the AEGIS system had been verified by the National Institute of Standards and Technology (NIST) and was ranked 49th out of 139 algorithms in tests for racial bias.

"Those numbers don't tally with our numbers. They're not even close," NIST biometric testing lead Patrick Grother told Vice.

According to Grother, NIST had tested a different algorithm. Meanwhile, parents have sued the NYSED for its decision to approve use of facial recognition technology at Lockport City Schools.

The news comes as AI bias and the ethics of technology firms are coming under increasing scrutiny. Last week more than a thousand researchers and academics voiced their concern at the apparent ouster by Google of leading AI ethics researcher Timnit Gebru.

Gebru is the co-founder of the Black in AI group, which promotes black employment and leadership in the field. Gebru, in her landmark study in 2018, found significant gender and racial bias in facial recognition software. She has criticised systems that fail to recognise black faces.

More recently, Gebru had been working on a paper that examined threats posed by computer systems that are able to analyse a human language database and use that to create their own human-like text.

According to media reports, the paper also criticised bias in Google's search technology, as well as that developed by other tech firms.

Gebru announced on Twitter that she had been sacked by the Google after sending an internal email that accused the firm of "silencing marginalised voices".

Google, however, told its employees that she had resigned from her job.

Edit: Responding to this story, CEO KC Flynn denied that SN Technologies had misled Lockport City School District.

"It is not uncommon for Facial Recognition algorithms to have differences in accuracy based on race and gender. Our AEGIS system performs at an accuracy level greater than 99.9% regardless of race or gender of an individual, as verified by the independent report conducted by Freed Maxick, one of the Top 100 CPA firms in the United States," Flynn said in a letter.

Flynn also claimed that the reported level of inaccuracy was not correct and that "a false match rate difference for Black Females to White Males is .01% higher for Black Females to White Males."

He said the reported problems with firearm detection were from a testing phase, and that the system is now working as planned.