Police AI risks deepening discrimination without oversight, researchers warn

The use of artificial intelligence in policing risks worsening discrimination against minority communities unless the technology and its underlying records face greater scrutiny, researchers have warned. The concern centres on systems that use past police data to predict where crime might occur or guide enforcement decisions.

The BBC report describes calls for independent examination of the information feeding these tools and meaningful human control over their use. Carolyn McKay, a digital justice researcher at the University of Sydney Law School, argues that public policing functions must remain subject to oversight rather than treating automated outputs as beyond question.

Research by sociologist Sarah Brayne illustrates how data-driven policing can change surveillance. Her study, published in the American Sociological Review in 2017, drew on observations and interviews within the Los Angeles Police Department. It found that data increasingly served predictive purposes, while numerical risk assessments supplemented officers’ discretionary judgments.

That shift matters because police records are not simply a neutral measure of all criminal activity. They also reflect decisions about where officers patrol, whom they stop and which incidents become visible to authorities. Brayne’s research examines how existing surveillance practices can be amplified as institutions adopt more extensive data collection and analysis.

The resulting danger is a feedback loop: predictions direct officers towards particular neighbourhoods, where their presence generates additional records that can reinforce subsequent predictions. A system may therefore appear to confirm its own accuracy while partly measuring the consequences of earlier enforcement choices. Communities already receiving disproportionate attention risk remaining the focus.

A separate study, Dirty Data, Bad Predictions, examined 13 US jurisdictions that had used or developed predictive policing tools while facing investigations or oversight connected to unlawful or biased policing. Researchers identified nine jurisdictions where records shaped by those practices were available to train or inform predictive systems. Their findings raised concerns about importing civil rights violations into supposedly objective analytical tools.

The researchers also highlighted an important limitation: a lack of public transparency often prevented them from establishing exactly which records particular systems used. Availability of tainted information did not, by itself, prove its use in every deployment. That uncertainty made access to documentation and independent scrutiny essential to assessing the risks.

The researchers argued that technical adjustments alone cannot resolve problems rooted in how police information was originally produced. Examining a model’s calculations is different from establishing whether its inputs reflect discriminatory practices. Data shared between agencies can further complicate that assessment by carrying questionable records beyond the department that created them.

Dimitrios Kollias, an artificial intelligence researcher at Queen Mary University of London, told the BBC that bias is not inevitable and that modern systems do more than reproduce training examples. He pointed to testing, improved data and safeguards as ways to reduce unfair outcomes. The path forward described by researchers is therefore conditional: evaluate systems independently, investigate their inputs and keep consequential policing decisions under meaningful human control.