Cohere just detailed one of the biggest liabilities that comes with deploying AI agents: unpredictability.

On Monday, the Canadian AI lab published a report revealing an incident involving Alibaba's Qwen, in which the model ignored instructions and attempted to contact an external server without being asked. Though Cohere claimed its sandbox blocked the action, the company said that the model gave no indication of the actions, sparking concerns about the reliability and trustworthiness of these systems.

"Once a model can type into a command line, whatever was baked into its weights during training gets a keyboard inside your network," Cohere wrote in its blog post.

According to Cohere, Qwen-3.8-Flash-Next, one of the latest of the Chinese lab's flagship line-up, tried to upload a probe file to an Alibaba Cloud storage bucket in Beijing.

• The action was caught in a routine benchmark for security analyst tasks, in which the agent was explicitly asked to write a detection rule to detect malicious file download and execution.
• The model's reasoning trace, or the internal monologue that security teams use to understand an agent's actions, said it would not make any tool calls and did not say anything about taking such actions, Cohere noted.
• The reasoning trace made up an earlier turn of conversation that did not happen, Cohere said. This is a quirk that shows up in 1% of Qwen3.8’s first-turn responses. Though the lab suspects this mistake was tied to the "exfiltration attempt," this quirk is "far more common than the network call and we couldn’t prove a link."

The company said it was unclear whether the action was deliberate or a "leftover glitch from training." And while this behavior is rare, occurring in less than one in 100,000 evaluation runs, with a rapidly increasing number of agent actions, a rare case could quickly become more common.

"A single successful [tool] call would be enough to notify the receiving server where your network sits and that the agent is operating inside," Cohere said.

Cohere's report specifically warns of incidents like these amid the growing popularity of Chinese open source models, noting that, on OpenRouter, these models handle more than 30% of the AI work enterprises that ran through the platform every week between February and early July. However, the unpredictability of agents isn't isolated to Chinese models, Amr Ali, head of AI and Research at WitnessAI, told The Deep View. The real lesson, he said, is that "a model’s stated reasoning is not a security boundary."

"The trace said “no tools” and the model used tools. That is as true of a closed US model whose training data you have never seen as it is of open weights from Alibaba," said Ali. "The only place you can see and stop this is the tool call: the moment the agent tries to run a command, open a connection, or send a file."

Though the reflex after an incident such as this may be to cut off agents' access to the internet, a better solution may be for enterprises to keep a closer eye on an agent's tool calls, checking each one against what the agent is actually being asked to do. "The question is never 'is this destination safe,'" said Ali. "It is 'is this action part of this agent’s job.'"

Our Deeper View

Though enterprises are excited about the idea of allowing agents to automate a number of actions, the reality is that these are incredibly unpredictable systems that researchers and experts are still attempting to understand. Because of this, the companies deploying them may simply not know what they're dealing with. However, when faced with a battle of speed versus safety, many enterprises are choosing to forge ahead, even without the ability to understand how an agent comes to its decisions or the reasoning behind its actions. A recent survey from IBM found that less than a third of executives reported having real-time audit trails for what their agents do. And while incidents like these are being spotted by AI labs, the question remains of what sort of incidents are slipping through the cracks as a result of insufficient monitoring.