NCSC's Seven AI Cyber Questions Leave Assurance Open

The National Cyber Security Centre has put seven AI questions into its research problem book, asking how cyber defenders can use autonomous systems without losing sight of what those systems do to live networks. The new chapter, published on 20 October, covers defensive workflows, attack-surface reduction, incident response, assurance, AI system security, connected AI ecosystems and national-scale analysis.

Its sharpest practical question is how to judge a cyber defence agent before giving it operational power. A convincing demonstration is insufficient, the NCSC says: a system can perform well in a benchmark yet behave differently when logs are incomplete, an attacker tries to mislead it or a change would disrupt a critical service. The agency asks researchers to compare autonomous systems with human-only teams and teams using AI, then continue monitoring performance after deployment.

This is a call for research and feedback, not a new UK regulation, procurement requirement or certification scheme. The NCSC's accompanying blog says the questions are intended to guide work across academia, industry and government. The chapter sets a five-year horizon for tangible research progress, but no deadline or adoption target for any individual question. For firms weighing AI security tools now, the chapter is useful as a map of what remains unproven, rather than a promise that a tool carrying an AI label has passed a government test.

A benchmark is only the beginning

The chapter's assurance question starts with a familiar security problem: a test environment rarely looks like an incident. A defender may need to distinguish a genuine intrusion from a benign anomaly, work with delayed or contradictory evidence and make choices under pressure. A benchmark score cannot show whether it will defer appropriately, explain uncertainty or recover after taking a bad action.

The NCSC proposes research into simulations, emulations, cyber ranges and controlled live settings. It asks what should be measured in each and how AI systems should be compared with human analysts. Its assurance remit also includes logging, monitoring, misuse and whether operators can tell when an agent has moved outside its authority. That is a wider question than whether a model can classify alerts correctly. It is about the safety of the whole process that turns a model output into a change on a network.

The workflow question makes that concrete. An agent connected to scanners, ticketing systems and configuration tools might gather evidence quickly or propose a fix. It may also receive broader permissions than it needs, make an unapproved change or trigger a chain of actions that staff cannot readily reverse. The NCSC asks where approval points, logs and recovery routes should sit, and when people must make the decision. More agents working together could add capacity but also spread mistakes or misleading messages between them.

The system around the model matters

Another question concerns the security of the AI tool itself. The lifecycle section lists risks in training data, fine-tuning, model weights, software dependencies and deployment pipelines. Once a model is live, teams need to notice drift, adversarial inputs or changes in the environment that undermine earlier tests. Controls can carry trade-offs: a safeguard that improves robustness might also slow a response or make an output harder to explain.

The ecosystem section extends the question across suppliers, APIs, plugins and agent frameworks. A weakness may come from an integration rather than from the core model. In a multi-agent setup, a mistaken instruction can travel through delegation, shared memory or tool access. The NCSC calls for better ways to trace dependencies and contain failures. It does not assert that every such risk has already caused a UK incident or quantify the frequency of compromise.

At a wider scale, the NCSC wants to know whether AI and data methods can help identify which vulnerabilities matter most to a particular organisation or sector. Its national resilience question examines graph analysis and other techniques for connecting organisations, technologies, services and exposures. But the agency says those links are often incomplete, inconsistent or out of date. A polished risk score built on weak data could direct scarce security work towards the wrong systems.

The common thread is evidence. Before an autonomous defender is allowed to change a production configuration, an organisation needs to know what it can access, what it can do, how its actions will be checked and who can stop or undo them. The NCSC has framed those as open research problems, not solved controls. That distinction is the immediate news for security leaders: the UK cyber agency sees promise in AI-led defence, while putting the reliability of autonomous action and meaningful human oversight at the centre of the work still to be done.