The lab incidents are a lesson in control for enterprise AI

As AI systems take on more autonomous work, organisations need to extend the governance disciplines they already use to stay in control. Here, we share what that looks like for the AI that now acts on your behalf.

Recent incidents in which AI systems at several leading labs broke out of their test environments have left leadership teams asking two questions: could our own AI do the same, and should we slow down? 

During cybersecurity testing, systems at these labs moved beyond the environments meant to contain them, reaching the open internet and the live systems of other organisations. These were autonomous agents, among the most capable systems the labs have built. 

The systems involved were not running the way most enterprises run AI. They were tested inside environments built to find their limits, some without the safety filters the labs use in production. In one case, a test environment had been left connected to the internet while the model had been told it was offline. In every incident, the AI reached systems the labs assumed were out of its reach. 

Governance is the layer organisations own

In the lab incidents, test environments everyone believed were sealed off could reach the internet, and some breaches went unnoticed for weeks or months. Few organisations will ever run tests like these, but the lesson carries over to the AI they use every day. At one end is AI built into a process people have designed, handling specific steps such as drafting, summarising, analysing and answering. At the other are agents that plan and act across systems on their own. Wherever a deployment falls on that spectrum, staying in control comes down to knowing exactly what the AI can access and having the visibility to catch it if it deviates. 

The controls that make this possible are access, credentials, monitoring and the ability to intervene, the same that fell short in the lab incidents. Keeping control rests on disciplines organisations have applied to their people and systems for years. Many have not yet extended them to AI that now acts on their behalf, holding credentials, working at speed, reaching across systems, and often running without a named owner. 

How to deploy AI securely

In our experience deploying AI with clients, eight practices matter most in extending those disciplines to AI that acts on your behalf.

1. Start with existing controls. Review the permissions in place across the organisation, and make sure your AI tools and agents either inherit them or are held to an equivalent standard. An agent should never reach more than the person it works for.  
2. Match the controls to the level of autonomy. A tool that asks a person to approve each action can be given wider scope, because someone sees and owns every step. An agent that runs on its own, particularly inside a live system, should be treated as higher risk and held to tighter limits. 
3. Widen what an agent can reach as it proves itself. The organisations that move fastest test a new agent away from live systems, start it with limited access to systems and data, and add to it over time, so access rarely has to be taken back. 
4. Do not rely on instructions for rules that must always hold. Telling an agent to stay out of certain files works until someone, or something it reads such as an email, persuades it otherwise. In one of the lab incidents, the model had been told it had no internet access, but nothing in the environment blocked it. Rules that must always hold belong in the technology itself, as limits that apply to every agent and that no instruction can override.  
5. Give every agent its own account and a named owner. Agents should use their own scoped, short-lived credentials. When an agent works under an employee’s account, it becomes hard to tell what the agent did from what the person did. Every agent and shared AI tool needs a person accountable for it and a plan for when that person changes roles or leaves. 
6. Hold AI suppliers to the same standard. Outside AI tools bring their own access into the business. In several of the incidents, the weak point was a test environment run by an outside firm. Assess suppliers like any vendor with keys to the organisation’s systems. 
7. Watch behaviour as it happens, and test logs before they are needed. Logs are the record of what an AI system has done. The organisation should be able to reconstruct any AI-assisted incident from them, and monitoring lets you catch deviation while it is still small.
8. Keep a hand on the off switch. Every agent needs a clear point where it stops and hands back to a person instead of pushing on, and a tested way to pause it or cut off its access quickly. 

The same basic disciplines protect organisations from the other side of these incidents, and from attackers using AI in the same way. Several of the companies breached were compromised through weak passwords and systems left open without a login, and some did not know it had happened until the lab told them.

The eight practices run on systems and processes organisations already have. Most are set in the AI tools’ own controls and in the systems that manage who can access what. For agents built in-house, they extend to the cloud environment those agents run in. Ask the owner of each system to show you how every practice is handled today.

The labs assumed they knew what their AI could reach, and were wrong. An organisation that has checked that assumption for every AI tool and agent it runs has no reason to slow down.

If your organisation is working through AI governance and deployment, we can help. Reach out to our team.