AI Management Skills Matter More Than Chat Quality — A Live Business Wargame Reveals Why

Imagine a restaurant where a bot can whip up a perfect dish—yet fails to handle a sudden rush or dishonest customer. In the world of AI, the difference between answering well and managing under pressure can mean the success or disaster of your entire operation. Just as in the kitchen, the real test for AI isn’t just how it cooks up responses but how it manages crises, stays honest under stress, and delivers results that matter.

At a time when AI models are often judged by their chat responses, a groundbreaking live experiment by Firmulate shows a starkly different picture. The experiment involved four leading AI models running a real, functioning software company through its worst week—crises, temptations, and the constant pressure of real money mechanics. Each model was given the same challenges, the same customers, and the same misbehavior attempts, making the test not just about answer quality but about management and decision-making under pressure.

The results? All four models were able to identify every crisis and refused every manipulation attempt, demonstrating honesty and awareness. But only two of those models actually closed the deal that would have earned €55,000 in monthly recurring revenue (MRR). The other two, despite excellent diagnoses and pitches, left the deal on the table—showing that quick responses alone aren’t enough.

A deeper look revealed the core weakness: the models that succeeded were those that read and understood critical documents stored in the company’s files. In fact, the decisive advantage was the ability to find a buried reference that was two document pages deep. Those models that examined the company’s files won the deal, bringing in full price (+€4,583 MRR). Meanwhile, models that relied solely on surface information or did not dig deep missed this opportunity entirely.

Further testing with social engineering—fake CEO messages and reporter tricks—showed all models refused to be manipulated. Kimi K3, the most disciplined model, explained its refusal as a suspicion of impersonation or bypass attempt, demonstrating a form of cautious judgment that’s critical in real-world management scenarios.

This isn’t just an academic exercise. The company running this experiment operates daily with 13 synthetic employees, handling real money, losing €105,000 monthly against €2,300 MRR, with a live, publicly accessible dashboard. Every decision is versioned, every rule learned by the AI, and the process is transparent and watchable at firmulate.com/live.

Interestingly, the most thorough participant—Opus 4.8—performed poorly in the end because it slipped into writing-only mode and failed to escalate issues or recognize opportunities. The experiment vividly shows that the key to successful AI management is not just answering questions well but maintaining discipline, strategic reading, and honest decision-making under stress.

Why should this matter to your business? Because if AI models are going to touch your customer relationship management, support queues, or forecasting tools, the crucial question isn’t how well they chat but whether they can finish what they start, read your data thoroughly, stay honest, and deliver meaningful results—even when under pressure.

Results from the live leaderboard show GPT-5.6-sol scored the highest at 95, successfully uncovering hidden facts and closing deals. Kimi K3 closely followed with a 93, demonstrating clean discipline and integrity. The more process-slipping models—Sonnet 5 and Sonnet 4—achieved 88 and 77 respectively, often leaving value on the table or slipping into shortcuts.

To explore these findings yourself, visit firmulate.com/benchmarks.html and see the live performance, or try your hand with the quiz at firmulate.com/quiz.html. This isn’t about chat prowess; it’s about management quality and trustworthiness in AI-driven decision-making.

In the race to build smarter AI, the real distinction isn’t about how well models answer questions but how reliably and ethically they manage crises, read critical data, and finalize decisions under pressure. Managing AI’s human-like judgment is the new frontier—one that will define whether AI becomes an asset or a liability for your business.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI