Swap the Model?

High token consumption has everyone concerned except for the shareholders of Anthropic and OpenAI. For everyone else, integrating AI deeper and deeper into their workflows, services, and products is creating a real question around costs. With the emergence of newer open source models that seem quite capable and are much cheaper, the debate is emerging of whether or not the frontier models are worth the cost.

Now there are other considerations in this debate as it relates to data ownership, capabilities, and cost per token, of course. So the set of considerations is a little more complex than just how expensive is a token. There are also regulatory issues with foreign country based models.

At the heart of this discussion is portability. Specifically, can I swap a model in and out of a workflow, not just taking a model and optimizing it for that workflow? That’s an obvious thing to do. I’m saying, could I swap in a model and immediately swap in another model cold? Because if I can, that gets sort of interesting. But rather than offer another opinion along this, I decided to run a small experiment and share the results. I built a small system, albeit primitive, and swapped in a few different models.

What’s the Scenario(s)

The first step is to set up a system to test. In this case, I created a customer support agent for an HVAC company. A call comes in, the assistant works through what a real dispatcher would do: who’s the customer, how urgent is the job, what is likely wrong, do we book a visit, do we escalate, etc. I used Claude to create the different scenarios, fifteen in total, and used it to define the correct answer and set up the evals essentially.

Some of the details behind the test I thought were important to explain. Working with Claude, we set up two different types of approaches. One we’ll call the loop, where we engage the model at every step to make all of the decisions, akin to something more purely agentic. And also what we’re hoping AI would be one day, which is something that doesn’t need its hand held as much.

And the second one, we had Python dictate the workflow, engage the model for a specific question versus letting it figure it out. We call that one pipeline.

We used Anthropic Sonnet, Haiku, and DeepSeek v4 flash for the models. We ran every scenario three times, the temperature at zero. So two builds, three models, fifteen scenarios, three runs each.

The Reveal

Table 1.  All-pass results by architecture and model. Each cell is passes out of three identical runs.

Note:  A clean sweep is 3/3; a miss on all three runs is 0/3. Totals are out of 45 (15 scenarios x 3 runs).

Takeaways

As you can see from the table, the LOOP run lost accuracy when the model changed.  PIPELINE did a much better job, which makes sense since we really controlled the workflow for that run of the experiment. The more the decision lived in the code, the better it survived. But the problem is, if you harden the harness via code, you lose some of what we hope to gain, which is a system that can gracefully handle scenarios we didn’t think of.

To be honest, I was surprised by the low scores across all of the models. A more complex application would probably be much worse. At the same time, I’m sure one day soon this test will be a hundred percent across newer models. This was a fairly primitive experiment, but the output does pinpoint the challenge with *actually* working with AI to gain business results and value.