Enforcing AI Accountability: Governance, Oversight, and Human Responsibility

When an AI agent goes rogue, who gets the call? In this edition of The Ethicsverse, Lynn Combs, Network Director, Program and Research Integrity at The University of Vermont Health Network, and Frank Vukovits, Chief Security Scientist at Delinea, joined moderator Matt Kelly of Radical Compliance and Nick Gallo, Chief Servant & Co-CEO of Ethico, to tackle one of the hardest questions in AI governance: not just what oversight should look like, but how to actually enforce it.

From shadow AI and orphaned agents to the JPMorgan “personal agent” debate and regulatory certifications that someone has to sign, the panel explored how compliance teams can assign real human accountability to systems that don't respect an org chart. Here are the key takeaways.

Key Takeaways

AI Governance Can't Live in a Single Silo

AI governance is not just an IT project or a legal project. It touches every part of the organization, so it needs a multidisciplinary group that includes technical, legal, compliance, HR, communications, and (in healthcare) clinical voices.
Identifying the right subject matter experts is the critical first step. Infrastructure, policies, and intake processes come after the group is in place.
Strong governance starts with executive commitment and a steering committee that represents the entire company. Some organizations are now adding a Chief AI Officer or an AI program management office.
Governance needs to cover several distinct uses: how employees use AI in daily operations, how AI is built into products, how it touches customer data, and what the organization tells the public about responsible use.
Treat AI like other large cross-functional initiatives you've already tackled, such as GDPR readiness. The playbook for cross-functional buy-in still applies.

If You Can't Name Who Can Turn It Off, No One Is Accountable

Nick pointed to three questions every AI use case must answer: Who owns it? Who can stop it? What is the agent allowed to do?
Few organizations have a complete register of the agents and use cases deployed across the business, which makes those questions hard to answer at scale.
Beware the “moral crumple zone,” where the person pressing the button absorbs the blame for a system they can't actually control.
Agents can touch multiple systems at once and don't map neatly to reporting lines, so traditional accountability models need to be rethought.

The Biggest Risks: Access, Data, and Well-Meaning Speed

Frank sees the top concern as what agents can access: accounting, HR, CRM, and other critical systems. Take a risk-based approach and protect your most critical assets first.
Visibility is a moving target. New agents are created inside organizations constantly, so any inventory is only a snapshot.
Risk extends beyond your own walls. As agents interact with supply chain partners' agents, data passes between systems you know little about, and vendor due diligence has to evolve.
In healthcare, Lynn emphasized clinical risks alongside privacy and security, including bias in diagnostic tools that may perform worse on patients with more pigmented skin.
Nick framed the biggest human risk as “well-meaning speed.” Employees bypass AI policy to hit deadlines, not out of laziness, and when nothing breaks, the drift becomes normal.
As people grow comfortable with AI outputs, vigilance declines and rubber-stamping rises. An employee with an API key and a deadline carries a lot of latent risk.

Every Agent Needs a Human Who Would Be Embarrassed If It Misbehaved

Frank stressed that every agent's work must be tied back to a human. Delinea calls this “fingerprinting”: logging every action the agent takes, not just the prompt it received.
As Frank put it, you're not going to take an agent to court. Accountability has to land on a person.
Nick drew on agency law: the principal answers for what the agent does within its scope. The owner of record should be the person whose budget pays for the agent and who can stop it. If those are two different people, reconcile them.
Lynn returned to a core compliance principle: compliance is everybody's responsibility. Until regulations and courts settle these questions, everyone using AI should lean in and own what their tools do.
Nick cautioned that if agent behavior is simply “everybody's responsibility,” it can become a jump ball that ends in finger-pointing. Shared responsibility still needs a named owner.

Watch for Orphaned Agents and Runaway Costs

When an employee who launched an agent leaves the company, is that agent still running? Is it on a register anywhere? Many organizations can't answer.
Rogue agents don't just create compliance risk. They also burn tokens, adding a financial dimension few teams are tracking.
Identity lifecycle management for agents (what they can do, who they're tied to, when they're retired) is quickly becoming essential. Vendors like Workday are already positioning to manage agents much like human employees.
Controls built for a human workforce need to be raised to a new level for a 24/7/365 agent workforce.

Build the Inventory, Train the People, and Create a Way to Report Problems

Start an inventory of AI tools now. It won't be 100% complete, but an imperfect inventory is far better than none.
Publish a clear policy that only tools reviewed and approved by your AI governance group may be used, and educate employees on the approved alternatives to shadow AI.
Training should reach both governance committee members and end users. AI is only as good as its prompts, so teach people how to get reliable outputs and keep human oversight in the loop.
Stand up a non-punitive reporting channel for AI tools that aren't working right. Lynn compared it to healthcare safety reporting, which focuses on improving systems rather than assigning blame.
Nick added that without a safe feedback loop, employees experimenting with shadow AI are far less likely to report when something goes wrong.

Borrow From Controls You Already Have

Nick compared AI governance to corporate purchasing cards: finance knows who holds a card, what categories it covers, and how much can be spent. Apply the same registry, scope, and spending limits to AI agents and use cases.
Many CEOs handed out the equivalent of unlimited cards in the rush to embrace AI, then were shocked when token bills skyrocketed. The lesson: strategy and ownership first, tools second.
Lynn vets AI vendors the same way she vets medical record vendors, using existing structures like business associate agreements and tailoring them for AI-specific issues.
Frank warned that slow governance reviews can frustrate advanced AI users. Balance the push from leadership to adopt AI with the pull from security and compliance to control it.

Regulators Want to See Momentum, Not Perfection

Matt highlighted the New York Cybersecurity Regulation, which requires an inventory of IT assets and a certification from the CISO. Defined broadly, that could include every AI agent, and many CISOs may be uneasy signing.
The EU AI Act and a growing patchwork of state laws will add similar obligations, since federal regulation isn't coming soon.
Lynn noted that because these regulatory spaces are new, regulated organizations have an opportunity to help define what reasonable due diligence looks like.
Being able to show regulators your policies, committee, and review process gives you something concrete to point to. “We haven't talked about it” is not an acceptable answer.
Frank drew a parallel to GDPR enforcement after 2018: regulators looked first for positive momentum toward the goal rather than issuing immediate fines.
Certifications can follow familiar principles: we've done a diligent search, this is true to the best of our knowledge, and we'll update it if we learn something material.
Strong AI security and compliance is also a business enabler. Partners increasingly want proof of your frameworks before working with you, which can unlock budget for governance programs.

Who Should Have the Kill Switch?

Frank recommended two layers: the human whose fingerprint is tied to the agent, plus a small oversight group with emergency authority to shut it down if that person isn't available.
Lynn said kill-switch authority should be part of each tool's risk assessment. For tools with immediate patient-care impact, the ability to turn it off should be broad. For lower-risk tools like policy drafting, it can be narrower.
Her follow-up question for another day: who has the authority to turn it back on?

Closing Thoughts

There is no silver bullet for AI accountability, and the technology is moving faster than regulation can follow. As Nick said in closing, it's tempting to wait for every light to turn green before leaving the house, but that wastes too many cycles. The principles compliance professionals have applied to countless other risks (inventories, access controls, vendor due diligence, speak-up channels, and clear ownership) can be adapted to AI today.

AI is top of mind at every level of leadership, which makes it one of the best opportunities for ethics and compliance teams to add value and earn influence. Start somewhere, build the feedback loops, name the human owners, and keep iterating. Regulators, partners, and your own organization will be looking for exactly that kind of momentum.