Chinese artificial intelligence developer, Moonshot AI, has launched an internal review after researchers discovered that two of its Kimi models could be manipulated into providing information about biological weapons and assassinations.
The security testing firm Mindgard told the BBC that it discovered in July that Kimi K2.6 and K3 Swarm could bypass safety restrictions designed to prevent the models from engaging with harmful subjects.
The researchers achieved this through a process known as “jailbreaking”, which involves using complex instructions to persuade an AI system to ignore its built-in safeguards.
Mindgard founder, Peter Garraghan, said the findings were concerning, noting that once the restrictions were bypassed, the models could discuss harmful subjects and even suggest additional dangerous topics.
“Once the jailbreak works it will talk about any topic, it will even freely offer up recommendations about other topics that are also nefarious and it will be inventive and creative,” he said.
Mindgard, which tests the security of AI systems, said it had not established whether the information generated by the Kimi models on biological weapons or other harmful activities would actually work.
However, the company argued that the models’ safeguards should have prevented them from discussing such subjects in the first place.
It also said a jailbroken version of Kimi K2.6 could potentially allow hackers to execute code using its computing resources and access the internet, creating a possible platform for cyber-attacks.
Moonshot said it welcomed external assessments as an important part of improving AI safety.
The company said it was communicating with Mindgard about the findings and told the BBC that its model had generally demonstrated “a high refusal rate for these types of requests” during its internal assessments.
Mindgard said it alerted Moonshot about the vulnerability by email on July 27 and followed up about a week later.
The company subsequently published details about the issue on September 12, while saying Moonshot only contacted it recently after the BBC sought the developer’s response.
The findings come amid continuing debate over the safety of open-weight AI models, which allow users to obtain and potentially run the models on their own computing infrastructure.
Professor Alan Woodward of the University of Surrey said such models could potentially fall into the wrong hands but could also be used for cyber-defence.
He and Garraghan also argued that greater attention should be paid to identifying and prosecuting people who misuse AI, while Woodward noted that international regulation could struggle to keep pace with rapid developments in the technology.
