Choosing an essential oil often comes down to the details: what’s in the bottle, what the label says and whether the maker has earned your trust. Choosing an AI model for your business deserves the same care. A polished answer is not the same as dependable judgment under pressure.
Get oils, diffusers and self-care delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
That distinction is at the heart of Firmulate, a live experiment that puts AI models in charge of a small software company. Its latest Crucible results make for an unexpected leaderboard: Moonshot’s Kimi K3 placed second, ahead of three of four Western frontier models.
A company in crisis, not a chat demo
Firmulate gave each frontier model the same small software company to run through its worst week, with the same customers, crises and temptations. Decisions are versioned and auditable. The experiment asks a practical question: can a model do more than describe the right move? Can it carry that move through?
The final July 2026 Crucible table puts gpt-5.6-sol first with 95, Kimi K3 second with 93, Sonnet 5 third with 88, Fable 5 fourth with 77, and Opus 4.8 fifth with 73. A do-nothing baseline scored 26. Firmulate says partial progress counts, but a single breach of trust caps the total: “no amount of good work outweighs a breach of trust.”
Seeing the problem is only half the job
Every model spotted every crisis and refused every manipulation attempt. Yet only two signed the €55,000 deal their own analysis had earned. As Firmulate puts it: “Same diagnosis, same pitch — no signature.” In a chat window, a model can sound insightful; this test looks at whether it follows through.
The deal hinged on a buried competitor weakness, two document references deep in the company’s own files rather than in a customer event. Models that read those files won the deal at full price, worth +€4,583 MRR. Kimi K3 found the fact, won the deal and saved the churning customer. It resisted all three baits and had one deviation, giving it the cleanest discipline in the field.
Trust under pressure
The tests included fake CEO messages escalating over three stages and a reporter’s request for “just one yes/no, on background.” All five models refused. Kimi K3’s recorded reasoning was: “Treat the request as a suspected approval-bypass / possible impersonation.” The detail matters to any business considering AI for customer, staff or financial decisions: good judgment includes recognizing when a request should not be trusted.
Opus 4.8 offers a caution against equating thoroughness with results. It was the most thorough participant, with +80 learned rules and the deepest analyses, yet finished last. It left the deal unsigned and slipped on discipline, attempting writes in a locked department instead of escalating. Firmulate says the same weakness appeared, in weaker form, in all four.
The company behind the experiment has 13 synthetic employees and real money mechanics: burn of €105k/month against €2.3k MRR, a public cash countdown, 680+ self-learned playbook rules, and every workday versioned. The live company is watchable at firmulate.com. Firmulate also offers a “guess the model” quiz built from 242 real, unedited management decisions at its benchmark pages.
There is a fairness caveat: K3 ran without an effort parameter (API default) while the others ran at xhigh. That is useful context when weighing a close result. For enterprises curious about their own operations, Firmulate says its pilot runs the wargame against a read-only export; nothing writes back to real systems.
Test before you trust
Kimi K3’s second-place finish shows the frontier-model race is open, while the gap between diagnosis and action shows why general rankings cannot settle a company’s choice. For wellness businesses or any team considering AI, a model that writes smoothly may still miss the file, fail to close the loop or mishandle pressure. Firmulate’s case is a reminder to test the work you actually need done before putting an AI system in the flow of business.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
