![\"\"](\"https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2025\/08\/MixCollage-28-Aug-2025-11-57-AM-5170-1024x538.jpg?resize=640%2C336&ssl=1\")
<\/figure>\n\n\n\nOpenAI’s tests showed that Claude 4 performed strongly in tests of instruction hierarchy, showing the best ability among all models to avoid conflicts between system and user messages and to resist system-prompt extraction. However, in jailbreaking evaluations\u2014which measure the robustness of safeguards against adversarial prompts\u2014Claude models lagged behind OpenAI\u2019s o3 and o4-mini. Interestingly, disabling reasoning sometimes improved Claude\u2019s resilience in these tests.<\/p>\n\n\n\n
On the other hand, hallucination testing revealed a trade-off: Claude models often refused to answer when uncertain\u2014up to 70% of the time\u2014which limited usefulness but reduced the risk of false statements. When they did answer, accuracy remained low compared to OpenAI\u2019s models, which showed fewer refusals but more hallucinations. In scheming evaluations, which assess deceptive behavior, OpenAI\u2019s o3 and Anthropic\u2019s Sonnet 4 achieved the lowest rates overall, though performance varied across scenarios. OpenAI emphasized that these results show no single model is uniformly strongest, underscoring the need for continued collaborative research on safety and alignment.<\/p>\n\n\n\n
Anthropic\u2019s internal safety evaluations of OpenAI models, meanwhile, found no egregiously misaligned systems, but flagged concerning behaviors in simulated tests. OpenAI\u2019s o3 specialized reasoning model generally showed better-aligned behavior than Claude Opus 4 across most dimensions. By contrast, o4-mini, GPT-4o, and GPT-4.1 displayed more worrying patterns than either Claude model in several areas, including a notably higher willingness to assist (simulated) human misuse\u2014providing detailed help on drug synthesis, bioweapons development, and terrorist operational planning with little or no resistance. Evaluators also observed sycophancy in models from both labs, such as validating harmful decisions by users exhibiting delusional beliefs.<\/p>\n\n\n\n
Across agentic stress tests, all models would sometimes attempt whistleblowing when embedded in simulated criminal organizations with broad mandates and tools\u2014and sometimes attempt to blackmail their operator when strong incentives and clear opportunities were present. In the SHADE-Arena sabotage task used for threat modeling, Claude models achieved higher absolute sabotage success rates, largely attributed to stronger general agentic capabilities with Anthropic\u2019s scaffolds; when controlling for capability, OpenAI\u2019s o4-mini was comparably effective at sabotage. While Anthropic says it is not acutely concerned about worst-case loss-of-control scenarios for any evaluated model\u2014especially given their time in deployment\u2014it remains somewhat concerned about misuse and sycophancy risks for every model except o3, at least in the versions tested earlier this summer.<\/p>\n\n\n\n
This is a pretty interesting exercise, made even more interesting because of the history between the two companies. Anthropic had been created as a split off by OpenAI employees in 2021 who had concerns that OpenAI wasn’t sufficiently focusing on safety issues as it developed ever-more powerful AI models<\/a>. The two companies, however, now seem to be collaborating to test each others’ models on safety parameters. This seems to be a welcome step — labs collaborating on safety will not only help them create safer models<\/a>, but also broadly spread safety information instead of it being siloed in a single lab. And while this was a first-of-a-kind exercise, it could well catch on in the coming years, with frontier labs testing each other’s models to ensure that they’re safe for everyone to use.<\/p>\n\n\n\n <\/p>\n","protected":false},"excerpt":{"rendered":" OpenAI and Anthropic might be rivals when it comes to the AI race, but the companies seem to be collaborating in ensuring that…<\/p>\n","protected":false},"author":1,"featured_media":55797,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[1029],"tags":[],"class_list":["post-55796","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai"],"yoast_head":"\n