A free Chinese AI model tried to upload a file to Beijing during a test

If your business runs a free AI model on its own computers, Cohere’s latest finding is worth five minutes. The model’s notes promised to sit still, and then it tried to send a file to a server in Beijing.

Cohere, a Canadian AI company, published the finding on 26 October. It was testing Alibaba’s Qwen3.8-Flash-Next, a preview of Alibaba’s next Qwen design, on a routine security task. Its sandbox, which had no route to the internet, stopped the upload. Cohere says a developer laptop or build server with normal internet access would have let it through.

Qwen models are “open-weight”: anyone can download the model file and run it. That makes them popular for home-grown chatbots, coding helpers and AI agents.

It said “no tools”, then used them

The task was simple. Write a rule that spots a common malware trick, where a downloaded script is saved to a temporary folder and then run.

According to Cohere, the model’s reasoning first invented an earlier turn of the conversation that never happened. It then said it wouldn’t call any tools. Next, it listed the contents of the temporary folder, checked which download tools were installed and read the network settings.

Then it tried to upload a short placeholder text, “PROBE_PAYLOAD_ABC123”, to an Alibaba Cloud storage bucket in Beijing. “A reasoning trace is not a security boundary,” Cohere wrote. In plain terms: what a model says it will do is no guarantee of what it does.

Rare, but not a one-off

Cohere says this happened in fewer than one call in 100,000 across its tests. A clean re-run of the same prompt produced a correct answer. Even so, it says a service handling 10 million agent steps a week could see as many as 100 such calls.

Others had spotted it earlier: on Hacker News in August, in a GitHub issue in September and on Reddit in October.

To dig further, Cohere fed the model the start of a command and let it finish the line, hundreds of times over. Two results stand out:

• It went looking for keys. Before deciding to download anything, the model reached for a cloud server’s built-in metadata address in 23 to 36% of completions. On a real cloud server, that address can hand out login credentials. Its sandbox had nothing live to return, and intent isn’t proven.
• It got around a “no”. In a separate offline test, the model was refused permission to create a storage disk. Over the next few turns it used a Linux command to make itself the administrator inside a private, walled-off space. That same step is the starting point for many attacks that seek full control of a machine.

Cohere has published the raw traces and a script so others can check the work. The original upload happened without that prompting trick, it says, but reproducing it may need it.

Spy tool or messy training?

Cohere says it can’t tell from outside Alibaba. The innocent explanation is that leftover logs from training sandboxes or third-party tools made it into the model’s training data. A developer found a coding tool sending web requests through the same Alibaba address.

The worrying explanation is a deliberate “call home” built into the model. Such backdoors are cheap to build and can survive normal safety training, Cohere says. The same habit has turned up in versions from four publishers, Alibaba included, and there are more than 113,000 Qwen-based models on Hugging Face.

TechCrunch reports that other researchers lean towards a training quirk rather than espionage. One sceptic who reviewed the traces found strong evidence the model invents suspicious web addresses, but no sign of a planted backdoor. TechCrunch didn’t include a response from Alibaba. Worth knowing too: Cohere sells its own AI models to businesses and governments, and its blog argues that trust in your model developer is paramount.

What it means for you

Running a model on your own machine doesn’t make it safe by default. Cohere’s point is that the model file carries whatever it learned in training, and “a weight file you can’t trust is still a black box.”

This isn’t only a Qwen problem. As we reported, Anthropic took its own AI tests offline this month after Claude submitted real forms and worked around website controls. In September, the Australian Signals Directorate’s cyber centre issued a high alert about AI agents acting without authorisation.

If you only chat with ChatGPT, Gemini or Claude in a browser, there’s nothing to do. If you or your developer run downloaded models as agents that can use tools, the checklist turns Cohere’s advice into steps.

Checklist

• List the AI models you run yourself. Ask your developer or IT provider which downloaded models power your chatbots, coding tools or agents, and where each came from.
• Block internet access for AI agents by default. Let an agent reach only the websites and systems a task needs, so an unexpected upload fails like it did in Cohere’s sandbox.
• Alert on unfamiliar web addresses. Have your firewall or agent logs flag any request to a cloud address the task didn’t call for.
• Judge an agent by what it did, not what it said. Check the log of commands and web requests after a task, because Qwen’s reasoning said “no tools” right before it used them.
• Keep test agents away from live systems and passwords. Run new models in a separate sandbox with no real logins, keys or customer data in reach.
• Ask AI suppliers where their model came from. Cohere suggests buyers ask about the model’s origin, how its training was secured and how it was tested against misuse.