Can AI Screening Be Audited for Bias? NYC Law 144 Explained

Yes, if you're covered by New York City's Local Law 144: any employer using an automated tool to screen or score candidates or employees based in NYC must have it independently bias-audited every year, publish a summary, and give candidates ten business days' notice. But “audited” means one narrow thing — a statistical comparison of scoring rates across race, ethnicity and sex — and two recent, independent reviews found that even that check is rarely done properly.

What Local Law 144 Actually Requires

Local Law 144 covers any “automated employment decision tool” — legally, anything using machine learning, statistical modeling, data analytics or AI that “substantially assists” a hiring or promotion decision, per the NYC Department of Consumer and Worker Protection. That definition catches resume screeners, video-interview scoring, chatbots and skills assessments — even when a human still signs off on the final call. It applies based on where the job or the candidate sits, not where the employer is headquartered, which is how remote-first and out-of-state companies keep discovering it applies to them after the fact.

Employers using a covered tool must get it independently bias-audited within the past year, publish a summary of the results and the distribution date on their website, and notify candidates at least ten business days before the tool is used. Violations run $375 to $1,500 per day of non-compliance, with each day potentially counted as its own violation.

What a Compliant Bias Audit Actually Measures

A Local Law 144 audit is a specific statistical test, not a general safety check. An independent auditor — one with no financial stake in the employer or the vendor — calculates the selection or scoring rate for each sex, race/ethnicity and intersectional category, then compares each group's rate to the highest-scoring group's rate as an “impact ratio.” A ratio below 0.8, the long-standing four-fifths rule used in US employment discrimination law, is the conventional threshold for concern. The audit has to run on real historical application data or representative test data — never inferred demographics — and it has to be redone every twelve months for the tool to keep operating legally.

That is the entire scope of what the law asks an auditor to check.

What the Audit Requirement Doesn't Check

It doesn't check whether the tool is any good at its actual job. Researchers from the ACLU, the University of Pennsylvania, Stanford and Harvard examined every publicly posted Local Law 144 audit they could find — 116 audits across 44 audit reports, covering July 2023 through November 2024 — and published the results at the 2025 ACM Conference on Fairness, Accountability and Transparency. Their finding: 83% of the audits were missing demographic data on some share of applicants, and once that missing data was accounted for, as many as 70% of the reported impact ratios could plausibly have actually fallen below the 0.8 threshold the law is built around. Some reports showed what the researchers called “silent duplicates” — patterns suggesting data had been pooled across employers or tools without saying so. Their conclusion, in their own words: Local Law 144 audits are “incomplete evaluations of algorithmic bias” — not fabricated, but built on data too thin to support the pass/fail claim printed at the top of them.

None of that measures whether the underlying tool identifies who can actually do the job. A tool can clear a Local Law 144 audit and still be bad at predicting performance, because the audit was never designed to ask that question.

New York's Own Watchdog Found the Enforcement Doesn't Work Either

In December 2025, the New York State Comptroller's office published its own audit — not of employers, but of the city agency responsible for enforcing the law. Covering July 2023 through June 2025, it found the Department of Consumer and Worker Protection had received just two AEDT-related complaints in two years, and that most test calls to the city's 311 hotline about AEDT issues never reached DCWP at all. When DCWP reviewed 32 companies' published bias-audit disclosures, it found a single compliance issue. When the Comptroller's office reviewed the same 32 companies, it identified at least 17 potential violations DCWP had missed. DCWP had also skipped its own enforcement workbook and never looped in the city's Office of Technology and Innovation, despite an agreement to do so.

DCWP reviewed 32 companies' bias-audit disclosures and found one problem. New York's own state auditor reviewed the same 32 and found at least seventeen.

DCWP has said it will implement most of the audit's recommendations. As of this writing, none of that changes what's true today: the agency built to check whether “audited” AI hiring tools are actually compliant is, by its own overseer's account, barely checking.

Where Buyers Get This Wrong

A vendor's “Local Law 144 compliant” badge means one dataset cleared one statistical threshold on one occasion — not that the tool is fair in general, not that it's accurate, and, per the FAccT researchers, not always even that the underlying data was complete enough to trust the result. Four mistakes show up repeatedly in how buyers read that badge.

Assuming the law doesn't apply because the company isn't headquartered in New York. Coverage follows the candidate and the job, not the employer's address — a company hiring remote NYC-based staff from anywhere in the world can be in scope.

Assuming a passing audit is a legal shield. It's a compliance requirement, not a defense to a discrimination claim, and a “clean” audit built on the kind of incomplete data the FAccT paper found doesn't strengthen that shield much anyway.

Assuming a bias audit and an accuracy check are the same question. A bias audit asks whether groups are scored at different rates. It says nothing about whether the tool's scores track who can actually do the job, or whether anything on the application it scored was even true.

Assuming small teams are exempt. The law has no headcount carve-out — it applies based on tool use, not company size.

Where AgentR Fits — and Where It Doesn't

AgentR's screening runs on a rubric written and locked before applications open, scored the same way against every candidate in the pool, with the evidence behind each score attached to the shortlist a human reviews. That structure works against the specific failure a bias audit is designed to catch — criteria that quietly shift per candidate — because the criteria don't shift.

What it doesn't do: replace the audit itself. If Local Law 144 covers your hiring, you still need an independent third-party auditor with no financial relationship to AgentR or to you — that's a specific legal requirement, and no vendor can satisfy it on your behalf. AgentR is also currently in private beta and hasn't published bias-audit data of its own at the scale the law expects; that's a limit worth naming rather than glossing over. And AgentR's verification work — checking what a candidate claims against the public record — answers a different question than a bias audit does. One is about whether the process treats groups evenly. The other is about whether what's on the resume is true. Buying a tool that does one well doesn't answer the other.

Questions People Ask

Does Local Law 144 apply if my company isn't based in New York?

Yes, if the job is based in New York City or a candidate for it is — coverage follows where the position or applicant sits, not where the employer's offices are, which is why some remote-first companies learn they're covered only after a complaint or a candidate's notice request.

What happens if I skip the bias audit?

Civil penalties run from $375 to $1,500, with each day a covered tool operates without a current audit or notice potentially counted as a separate violation. Enforcement has been rare so far — DCWP fielded only two complaints in two years — but the Comptroller's December 2025 findings are explicitly meant to push enforcement to tighten.

Does passing a bias audit mean the AI isn't biased?

It means the specific dataset the auditor tested didn't show an impact ratio below 0.8 for the groups measured, using the auditor's chosen scoring threshold. The FAccT researchers found most published audits had incomplete demographic data, which means some “passing” results may not hold up if the missing data were filled in.

Is a bias audit the same as checking whether the tool works?

No. A bias audit measures disparate impact across demographic groups. It doesn't test whether the tool's scores predict job performance, whether it favors the right criteria, or whether it's evaluating true information. That's a validity question, and it sits almost entirely outside what the law requires.

Can a small company afford a Local Law 144 audit?

The law sets no size exemption, only a tool-use trigger, so a small team using a covered tool on NYC-based hiring is in scope regardless of headcount. Independent audit firms scope engagements to applicant volume, so cost tracks data size more than company size.

The practical fix costs nothing and doesn't wait for DCWP to catch up: check coverage by where your candidates sit, not your own address; ask any AI hiring vendor for the raw impact-ratio numbers behind their compliance badge instead of the badge itself; and put the bias-audit question and the accuracy question on two separate lines of your vendor checklist, because right now almost nothing is verifying either one for you.

More on where AI hiring trust actually breaks down: Only 16% of HR Teams Trust AI Screening Alone, 56% of Recruiters Ignore Your AI Match Score, and how AgentR's fairer-interviews approach is built around a rubric written before anyone applies.