Just Swap the Model, They Said

High token costs, combined with the emergence of capable lower cost open source models (DeepSeek, Kimi K3, Qwen3.6), have intensified the discussion around optimizing AI spend. From enterprise CFOs to one-person startups, everyone is grappling with ballooning costs and ongoing questions about AI’s return. The debate is heating up.

It is a multi-layered problem. Data ownership, capabilities, and cost containment create a complex set of considerations, with impassioned voices on each side.

At the heart of the discussion is portability. Can models drop into a harness as interchangeable parts, and if they can, does that push them toward commoditization?

Rather than offer another opinion, I decided to run a small experiment and share the results. I built a small, primitive system, froze it, and swapped the model underneath. Let’s see what happened.

What’s the Scenario(s)

The system is a customer support assistant for an HVAC company. A call comes in and the assistant has to work through the things a real dispatcher works through: who is this customer, how urgent is the job, what is likely wrong, do we book a visit, when do we escalate, and when do we turn down work that is out of scope.

I used Claude to create fifteen scenarios. Each one has a defined correct answer and a fixed set of checks that score every run against it: right customer, right severity, right part, correct fee, escalated when it should be, declined when it should be. The checks never changed during the experiment.

I used Claude Code to build the assistant two ways. In the first, the model runs the whole thing. It decides what to do next, which tool to call, and when to stop. Claude named it the LOOP.  

In the second, Python runs the show, its more deterministic (and rigid) and only calls the model for the judgment calls it cannot resolve on its own, like whether a job is in scope or which part is failing. This is PIPELINE.

I used Anthropic’s SONNET model as the baseline and then swapped in HAIKU, and DeepSeek v4 FLASH. Every scenario ran three times, at temperature zero, which is meant to make a model return the same output on the same input. Two builds (LOOP and PIPELINE), three models, fifteen scenarios, three runs each.

The Reveal

Table 1.  All-pass results by architecture and model. Each cell is passes out of three identical runs.

Note:  A clean sweep is 3/3; a miss on all three runs is 0/3. Totals are out of 45 (15 scenarios x 3 runs).

One more thing sits in the table. Temperature zero is supposed to make a model deterministic, but it did not. Several scenarios passed on one run and failed on the next with nothing changed between them. We’ve all experience this type of inconsistency when working different models so it wasn’t a surprise

Takeaways

• The loop lost accuracy when the model changed. The pipeline mostly did not.
• Two scenarios, G08 and G11, failed on every model in both builds. Those are the ones with no clean rule to encode, where the answer needs a judgment about how severe a situation is or which part is probably failing. Neither build could hold those steady across a model change.
• The more a decision lived in code, the better it survived a swap and the more consistent it was run to run.
• The more it lived in the model, the less either held.
• If you harden the harness via code (PIPELINE) you lose some of the power of gen AI but gain in portability.

This was a primitive experiment but it does illustrate some of the challenges. To be honest, I was surprised by the low scores. If you have a more complex application, I’m guessing they would have been much worse. I’m sure one day, very soon, this same test will be at 100% across newer models but until then…