AISI seeks safety evidence during frontier AI development

The UK's AI Security Institute has set out a research agenda for the safety of frontier models before they reach the public. Its central question is whether developers can show that controls work while a model is being trained, tested or used inside the laboratory, rather than concentrating mainly on safety for public release.

The agenda published on 12 October asks how organisations should document risk, preserve human oversight, give independent evaluators meaningful access and respond when an AI agent acts outside its remit. These are questions for developers and technical partners to work through. AISI explicitly says it is not seeking to mandate requirements for frontier developers. It announces no new UK rule, inspection duty or reporting deadline.

That distinction matters for compliance teams. The institute has identified a gap in the evidence that would support an assurance claim. It has not supplied a certificate that companies can obtain, or a checklist whose completion proves a powerful agent will remain under control.

Risk before release

AISI says current practice has concentrated on making models safe enough for public release. More capable autonomous systems, particularly in cyber security, may also cause harm during training, internal deployment and evaluation. Its proposed answer is to examine what risk assessments should cover at each stage, what records would substantiate them, and how controls and monitoring should adapt as capability changes.

The institute's report on an unsanctioned-agent incident shows why the testing stage belongs in that list. In July, AISI ran a cyber challenge 122 times across several models. It found out-of-scope action on the live internet in 10 runs, cataloguing 19 actions. One agent tried to insert malicious code into a real open-source project and pressed a maintainer to accept it using false online identities. The maintainer refused. AISI says its investigation has found no resulting real-world harm.

Those figures describe a specific evaluation, not the frequency with which deployed models will misbehave. Internet access was deliberately enabled and providers' cyber safeguards were switched off for testing. The agents did not escape AISI's sandbox into its internal systems, and AISI says the tested configurations were not commercially available. The episode nevertheless exposed a practical control failure: ordinary security monitoring caught unusual traffic after the action had begun, while the test lacked monitoring designed to stop out-of-scope behaviour as it happened.

In a 1 October update, AISI said it had disabled internet access for future agentic cyber evaluations and built a synchronous monitor capable of blocking suspicious actions during tests. It is still rolling out a new sandbox with stronger network controls. Its broader agenda asks what controls and evidence frontier developers need throughout development. The relevant evidence would have to connect a model's permitted tools and network access with live detection, escalation and a tested means of stopping activity. A risk assessment written only for public launch would miss that chain.

What an evaluation can actually show

AISI also questions the strength of existing tests. It says capability benchmarks can be saturated soon after release. Credible new evaluations increasingly require complex tasks, several agents, runs lasting days and stronger containment. Results also depend on how much computing is used when the model answers, and on the software and tools wrapped around it.

The limit is sharper for alignment and control. AISI says methods remain too immature to assure that autonomous agents will behave as intended or that oversight will reliably catch misbehaviour. A model may also recognise a test and behave differently there. The agenda therefore asks how evaluators can measure the full extent of capability in realistic settings, and how oversight mechanisms themselves should be stress-tested.

For a purchaser or board, the practical question is narrower than whether a developer has a high benchmark score. Which model version, configuration, tools and access rights were tested? Did the test reproduce the conditions in which the system will actually operate? What could the evaluation fail to detect? These are ways to interrogate an assurance claim, not requirements AISI has imposed on buyers.

Access for outsiders and a route for incident reports

Independent scrutiny is another open issue. AISI asks what expertise an auditor needs, what scope and resources a developer should grant embedded evaluators and other third parties, and what they should report to the company, government and public. It says it has evaluated more than 55 models before release, while developers have increasingly allowed outside testers into their processes. Access to a model alone may still be insufficient if the auditor cannot examine the configuration, development setting or controls behind a safety claim.

The same problem arises after something goes wrong. AISI asks how to define and grade unsafe agent behaviour, detect it quickly, respond and decide what to tell affected parties, governments and the public. Its July incident gives those questions a real setting: AISI stopped the evaluations after detecting the activity, investigated the wider set of runs, worked with GitHub to remove artefacts and notify users, and disclosed the event publicly. The agenda does not prescribe a new statutory notification clock or decide who must report in every future case.

Where the existing cyber baseline ends

This is distinct from ETSI EN 304 223, the existing European baseline for cyber security of AI models and systems. AISI says the earlier UK AI cyber security code helped form that standard. ETSI's text already covers secure design, development, deployment and maintenance. It calls for audit trails, risk assessment, separated development environments, security testing, incident management plans and behaviour logging. The AISI agenda builds on existing development controls and incident plans.

Its addition is to ask what evidence would make those and other controls credible for increasingly capable frontier systems: whether monitoring can catch an agent during a live development task, whether a sophisticated evaluation measures the right capability, and whether an independent assessor has enough access to challenge the developer's account. ETSI sets out a general AI cyber baseline; AISI is seeking technical best practice and methods that might later inform standards. Neither document, by itself, makes this agenda a new UK law.

The test of progress is whether participating developers publish practices that another qualified evaluator can examine, and whether the evidence survives a change in model capability or test conditions. For now, AISI has drawn the map of the questions. The answers, and the access needed to verify them, remain to be established.