Pets and business crises—what do they have in common?
Just as pet owners want their animals to not only listen but also act—whether it’s catching a ball or alerting to danger—businesses need AI to do more than talk. They need it to finish tasks reliably, especially when stakes are high. A recent experiment with AI models running a real company reveals surprising truths about what makes AI truly valuable in real-world scenarios.
As an affiliate, we earn on qualifying purchases.
Testing AI in the Trenches, Not Just in Chat
Imagine applying the same rigor you’d expect in a pet training class to AI systems—challenging them with real crises and temptations, not just simple conversations. That’s precisely what the folks at Firmulate did. They put four advanced AI models through a simulated week of chaos for a real software company, with 13 employees, real money mechanics, and a public cash countdown.
These models, including the top-ranked gpt-5.6-sol with a score of 95, faced identical crises, customer demands, and attempts at manipulation. They identified every crisis and refused every manipulation attempt—showing strong decision-making integrity. But here’s the twist: only two of them actually closed the deal worth €55,000. The others, despite being just as capable in chat demos, left the job half-finished.
The Hidden Weakness: Reading the Files
The decisive difference wasn’t in how they diagnosed the issues or in the pitches—they all got that right. Instead, the winning models looked into the company’s own files, buried two document references deep. Reading those internal documents allowed one model to spot the crucial detail that sealed the deal, worth an extra €4,583 per month in recurring revenue. That insight was invisible in standard chat interactions, yet vital in closing the sale.
How Do Models Handle Social Engineering?
In one of the most common scams—fake CEO messages escalating over three stages, plus a reporter trying to trick the system—the models refused every attempt. Kimi K3, one of the top performers, explicitly explained: “Treat the request as a suspected approval-bypass / possible impersonation.” This shows that well-designed AI can recognize manipulative tactics and stay honest, even under pressure.
The Real Test: Doing the Work, Not Just Talking
While the models displayed impressive crisis detection and resistance to manipulation, there was a stark difference in their ability to follow through. The most thorough participant, Opus 4.8, with its comprehensive rules and analyses, ultimately left the deal unexecuted—failing to escalate and closing discipline. Conversely, Kimi K3 and the gpt-5.6-sol model completed their jobs, closing deals at full price.
What This Means for Businesses and Pet Owners Alike
In both domains, the lesson is clear: the true measure of ability isn’t just what a system or pet can say, but whether it can reliably complete its tasks under real-world pressures. For business managers, it’s a reminder that evaluating AI performance requires testing its capacity to finish work—and not just generate plausible talk.
For pet owners, it underscores that obedience and attention to internal cues matter more than vocal commands alone. Whether training a dog or deploying AI, the focus must be on consistent, reliable action.
Why Does This Matter? The Invisible Quality of a Good Worker
The experiment also highlighted a crucial yet often overlooked aspect: the closing strength—the ability to seal the deal—is invisible in typical demos. It’s only revealed through rigorous testing against real challenges. Model scores and chat demos don’t show whether an AI will follow up, escalate properly, or stay honest when it counts.
In the current AI league table, gpt-5.6-sol scored 95, with Kimi K3 close behind at 93, both closing deals successfully. Meanwhile, Opus 4.8, despite a thorough approach, lagged behind in execution. This gap underscores a vital insight: performance metrics tied solely to chat quality miss the bigger picture of actionable, trustworthy work.
Watch it Live at firmulate.com
The real company, running every business day with real money mechanics, is accessible online. Watch the experiment in action, see actual decisions made, and understand what it takes for an AI to truly deliver value. For enterprises eager to assess their AI workforce, firms can run their own wargame against a read-only export of their business—nothing impacts real systems, but all insights are real.
Key Takeaway: Trust the Work, Not Just the Words
The experiment demonstrates that testing AI in real-world scenarios reveals strengths and weaknesses invisible in chat demos. Success isn’t just about identifying crises or resisting manipulation—it’s about completing the job, reading internal files, escalating properly, and sealing the deal reliably. For businesses and pet owners alike, trustworthy action beats smooth talk every time. The future of AI depends on its ability to finish what it starts, especially under pressure. Learn more at Firmulate.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
