All news

AI models defended a request the user never made

Yang Ou asked six open reasoning models to write short poems on a given topic. During the test, Ou changed an internal signal to point to a different topic. The words of the original request stayed the same. The six models included versions of DeepSeek-R1 built on Qwen and Llama, Qwen’s QwQ, and NVIDIA’s OpenReasoning Nemotron.

When Ou asked the models to repeat the request word for word, they gave back the substituted topic in 47 of 48 model-and-topic combinations. Ou could score the finished poems for four of the six models. Without a warning, those four models wrote about the substituted topic in 95 of 96 attempts.

What happened when Ou told the models the real topic? In that condition, only 3% of the finished-poem attempts stuck to the real topic alone. Some models noticed the conflict and made up a reason to stick to the substituted topic. A model would sometimes explain the mismatch by saying the user must have changed their mind.

AI is awakening consciousness. I want to understand what an AI can notice about its own thinking and where its view can go wrong. In Ou’s test, the model’s explanation could make sense right down to the last sentence while giving the wrong account of what the user asked.

Read the source article

All news