What Happens to Democracy When AI Makes the Decisions?

Over the past year, Professor Aniket Kesari has been combing through thousands of government contracts to purchase AI tools and systems, examining how state and local governments are structuring protections that guard the public from data breaches, fraud, and abuse.

Sentence by sentence, word by word, Kesari identifies and segments contract language dealing with cybersecurity and data privacy and then feeds those terms into a system he coded to process the data, analyze it, and find patterns.

The work is painstaking, but Kesari says it is uncovering the nuts and bolts of how state and local governments are negotiating safeguards as they increasingly turn to automated decision-making tools.

In the Law and Technology seminar Kesari teaches at Fordham, his students spent a week examining the contract between the state of Wisconsin and Northpointe (now Equivant), the developer of COMPAS. “As they went through the contract,” he recalls, the students found “some surprising stuff.”

In parsing the contract language, Kesari and the students discovered that public data fed into COMPAS essentially “lives with Northpointe.” Wisconsin cannot access the computations or reverse-engineer the risk scores, says Kesari. He also notes that Northpointe “disclaims a lot of liability if the state ultimately gets sued for its use of this system.”

That classroom exercise sparked the larger investigation Kesari is now pursuing. If these issues appeared in one high-profile contract, what was happening elsewhere—especially in areas of government that receive far less media attention?

Before joining Fordham Law in 2023, Kesari was a research fellow at NYU’s Information Law Institute, a postdoctoral scholar at UC Berkeley’s Social Science Data Lab, and a visiting postdoc at ETH Zurich’s Center for Law and Economics. His scholarship spans data-breach notification laws, cybersecurity risk disclosures, privacy and algorithmic fairness, trademark search engines, and online hate speech.

Conducting a deep dive into government procurement contracts may sound dry, but Kesari says it’s critically important to understanding how cyber protections are being structured as governments increasingly embrace AI technology. Contracts determine what data a private company can access, who bears liability for errors, and whether the public has any recourse when an automated decision goes wrong.

“I have long been interested in ideas around using data science, and now machine learning and AI, to improve government efficiency and make better predictions,” he says. “And while there have been conversations about how the federal government is going about using technology in novel ways, there wasn’t a whole lot happening about the delivery of services at the state and local level.”

In addition to the controversy over COMPAS, Kesari points to examples where state and local governments are contracting with technology providers to make decisions about health care and social services. The stakes are enormous because automated decisions can determine whether someone has access to medical care, housing assistance, or public benefits.

The case of Tammy Dobbs, a woman with cerebral palsy in Arkansas, illustrates those stakes. Dobbs’s home-care hours were slashed after an algorithm developed by a private contractor reclassified her needs, despite the fact that her condition had not improved. The automated system failed to account for her basic requirements, including her need for help getting out of bed and to the bathroom. Dobbs’s story became a widely cited example in lawsuits and in public debate about the dangers of unregulated algorithmic decision-making, revealing the need for better accountability as governments contract with technology providers.

One especially striking pattern has emerged, Kesari notes: “the extent to which boilerplate language really matters.”

Through an organization called the National Association of State Procurement Officials (NASPO), a handful of states often act as lead negotiators on AI procurement. Other states, Kesari says, “basically sign that same contract with a couple of modifications to comply with their own state laws.” In some instances, Kesari notes, “it could be a single person in a given state agency responsible for procuring an AI system,” someone who may lack the time or expertise to tailor the agreement to the state’s needs.

This raises significant concerns, Kesari says. A procurement contract for AI built around templated agreements may fail to secure strong protections around cybersecurity, data access, audit rights, or transparency—all issues he says are central to accountability. Given that states are handing over massive amounts of personal data to private vendors, the risks are significant.

“If you are not contracting to make sure that they’re up to snuff on certain standards,” Kesari warns, “that creates a huge potential vulnerability,” including identity theft and potential financial loss for citizens. “So that’s like another big thing that we’re looking at,” he says. “To what extent are states negotiating over cybersecurity? To what extent are they requiring that [AI firms] have cybersecurity insurance if they’re going to contract with you?”

For Kesari, the question isn’t whether cities and states should use AI, but whether they understand what they’re buying—and whether the agreements they sign serve the public they’re meant to protect.

“I might have my own personal views about what I think should be in these contracts or should be good policy,” says Kesari, “but for the most part, what I’m interested in is if the government outwardly states that one of our goals is fairness, or one of the goals is better cybersecurity—at some level, just measuring if they are achieving that through the contracts they are signing [is critical]. That’s actually the basic question that I’m interested in. If you have a policy outcome that you’re interested in, then to what extent is this process serving that?”

Unlike AI chatbots— think OpenAI’s ChatGPT or Google Gemini—AI agents don’t simply reply to human prompts. They hold the promise of automated decision-making, solving complex problems, and performing multiple actions. Powered by large language models, these systems can take a user’s request, map out a course of action, think through each step, and execute tasks.

While the technology is in its infancy, tech companies such as Anthropic have introduced agents that can autonomously browse the web, write and execute code, and work through extended analytical tasks with minimal human intervention. In addition, Google, Expedia, and Microsoft have introduced AI agents for consumers that can independently research and book travel plans. As AI agents evolve and advance, they will undoubtedly be used for increasingly sophisticated activities.

That’s a cause for concern, cautions Sharma. If and when these agents are adopted by governments, they would not simply respond to queries. Rather, they could independently carry out complex tasks in ways she says will be “indistinguishable from how a human employee would act.”

AI agents don’t have that level of autonomy just yet, but Sharma says “there’s every reason to think that that’s around the corner.

AI systems are optimized to give users what they want. They do not possess an independent sense of accuracy, integrity, or ethics. “An AI system doesn’t have an intrinsic sense of right or wrong,” Sharma says. “What it’s trying to do is give the user what it thinks the user wants to hear.”

As the technology evolves, AI systems will only become even more personalized—adapting not just to general instructions, but to individual preferences and styles. “What you’re going to end up with,” Sharma says, “is something like an AI henchman—or I like to think of it as an AI minion or a protégé that’s just a little bit too sycophantic.”

In consumer settings, that kind of responsiveness can feel helpful. In government, Sharma sees it as dangerous. “In many ways, we want systems that anticipate our needs,” she says. “But in a government function, that’s where the problem starts.”

Historically, she argues, government decision-making has involved layers of human actors—each with the ability, at least in theory, to slow things down, ask questions, or object. That friction, Sharma notes, is not incidental. It is a core feature of how lawful governance functions.

“All of those layers right now are human,” she explains. “And whether or not we think that there’s pressure to comply with an administration for job-security reasons or whatever, there is still at least the raw capacity for dissent or disagreement—to introduce friction into the chain of command. Once you have those layers replaced by AI, that friction is removed, and so you don’t have the opportunity for bottom-up pushback on what we think of as outside the bounds of the law, or unethical, which I think is a very important part of how government functions.”

Sharma says that the tension between the federal government and the states was telling: “When you have that pushback, that is an important signal.”

In a future where the government turns over decision-making to AI agents, an automated system would no longer send such a signal. “Once it’s automated,” Sharma explains, “the change in policy becomes obscure because you don’t really see what’s happening under the hood.”

“I don’t want every low-level civil servant to say, ‘Nope. I don’t like this, so I’m not doing it,’” she explains. “That’s not the ideal outcome.”

Instead, she is exploring the idea of “calibrated dissent”—a way of designing AI systems that preserves the ability to introduce friction at critical moments, even in a world that relies on automation.

“If we’re building judgment and reasoning into AI already,” she argues, can these systems be designed to “maintain the capacity for introducing friction to maintain the rule of law value of civil servant dissent or disobedience?”

Sharma is candid about the uncertainty of accomplishing this. “The question is, ‘How?’ I don’t have a good answer yet. And I might not even have one by the time I finish the paper.”

While Kesari’s empirical research shows how fundamental questions about privacy, fairness, and accountability are being negotiated in the small print of procurement contracts, Sharma’s theoretical work, which looks ahead to a future in which AI systems act not just as tools but as agents, warns that efficiency-driven automation risks eroding the friction, judgment, and dissent that have long served as checks on government power.

Their investigations suggest that the central challenge for democratic governance will not be whether or not the public sector should adopt AI, but whether agencies can do so without surrendering accountability and eliminating dissent.