Getting Better Results with Open Models

Your software requirements do not rise every time a lab releases a better model. If yesterday’s frontier was good enough for production, today’s open model matching its relevant capabilities should be good enough.

Because a new frontier model can tackle harder problems, it doesn’t mean your existing workflow automatically requires it. Open models can keep trailing the frontier while replacing more of the work people actually need to accomplish. 1

Spend the savings on better review

Coding agents consume tokens throughout a job: reading files, editing code, running tests, and reviewing results. A cheaper model can often do this work well and escalate if it runs into trouble. 2

It can also buy more scrutiny. At a 10-to-1 or 20-to-1 cost advantage, four passes still cost much less than one expensive pass, and often yield similar or better results.

In my own work, repeated adversarial reviews have brought cheaper models to results I would accept from a frontier model. Reviewers challenge the code and each other’s findings, often using different LLMs and with a different focus; the agent repairs what survives. This is better than the same model with the same blind spots making one additional pass. 3

Open weights change the economics

Cheap closed models can do this too, but open weights give teams additional options:

Change providers: competing hosts can serve the same model. 4
Specialize: LoRA weights can adapt the model to recurring work. 5
Run it yourself: trade a provider’s token bill for hardware and operating costs. 6

At work, we fine-tuned a 7B Qwen model to call stable internal APIs without carrying the full tool descriptions in every prompt. A narrow, repeated task lets specialization pay for itself.

Self-hosting changes the calculation further. Predictable workloads could mean dedicated boxes can run at capacity and shut down when not in use. With your own router, you can send traffic to on-prem models when they’re available, and a 3rd-party provider when they’re not.

What remains for closed models

It depends.

At a minimum, compare the total cost per PR accepted (this assumes PRs only merge after heavy testing and quality checks). Include the token cost and wall-clock timein the accessment.

For a more direct approach, have different models write the code from the same plan, then have the models review each other’s work. However, remember that the cost difference between models can be large. It may be fairer to compare a 1-shot from the expensive model to an N-shot (N depending on cost and time) from the cheap model.

Use what works best for your workloads.

I bet that open models will take most repeatable software work. Closed models’ distinct role will likely concentrate on the hard cases beyond that threshold. As established tasks become cheaper to handle, closed labs must keep finding new work that justifies the difference.

Those are the margins I expect them to own. Today’s revenue share doesn’t tell us how large, durable, or profitable that remaining business will be. 7

The evidence and economics

Evidence 1: Catching up to production requirements

Epoch AI measured an average four-month lag between the best open and closed models on its composite capability index from January through May 2026. Requiring the open model’s point estimate to exceed the earlier frontier’s score, rather than allowing statistical uncertainty, stretches the estimate to six months. Epoch AI, 2026.

That supports the catch-up argument without establishing equivalence on every workload. Epoch also flags possible overstatement of open-model capabilities from public benchmarks.

Use your production acceptance criteria to test a replacement. The best open model may also require substantial hardware; open weights do not imply laptop-sized deployment.

Evidence 2: The cost of a completed agent run

Bai et al. studied eight models on SWE-bench Verified and found that input tokens drove agentic coding costs even with caching. Agents repeatedly ingest accumulated context, so costs extend throughout the working loop. Bai et al., 2026.

SWE-Router’s authors ran 500 SWE-bench Verified issues through mini-SWE-agent with a 75-step budget:

Model	Issues resolved	Reported total inference cost
GPT-5 mini	55.6%	$16.60
DeepSeek V3.2	62.4%	$22.60
Gemini 3 Pro Preview	70.4%	$391.30

DeepSeek solved a majority at about one-seventeenth of Gemini’s spend. Each attempt was a multistep agent run. These results compare the study’s configurations, not the best workflows available at equal dollar budgets. SWE-Router, Appendix A.

The researchers also let a cheaper model explore before deciding whether to escalate, counting that exploration in escalation costs. This improved the cost–resolution tradeoff over routing from the issue description alone on a held-out SWE-bench slice of 100 issues. Results on unfamiliar SWE-Smith repositories were less consistent, leaving generalization as a practical concern. SWE-Router, 2026.

Evidence 3: More attempts and adversarial review

Four cheap passes cost:

40% of one expensive pass at a 10-to-1 cost advantage.
20% of one expensive pass at a 20-to-1 cost advantage.

That assumes comparable work per pass. Count growing context, tests, repairs, and the cost of selecting or combining results.

Hassid et al. compared repeated generations from smaller coding models against a larger model under matched compute or runtime budgets, including five outputs from a 13B model versus one from a 70B model. Smaller models could outperform when unit tests selected the answer; ranking candidates without tests fell short. Identifying a good result matters as much as generating more candidates. The Larger the Better?, COLM 2024.

Qiu and Gill paired a reviewer with a critic who challenged its findings. Their Adversarial Review protocol improved results in tested coding settings, but also exposed false consensus: agents agreeing without sufficient evidence. Adversarial Review, 2026.

All roles used Sonnet 4.5. The study supports structured review, but does not establish that four cheap passes equal one frontier pass. My experience with cheaper models is a practical observation, not a controlled benchmark.

Evidence 4: Competition within a model

Where the license permits independent hosting, several providers can serve the same weights. OpenRouter lets you select providers by price, throughput, or latency, but quantization and serving differences still require quality checks. OpenRouter’s routing documentation.

My economic inference is that competition among hosts makes a durable premium harder to defend for widely available capability. Closed suppliers can still win through efficiency, reliability, or better products. Efficiency and reliability can be objectively accessed with test on your own use cases. Better products may be objective also, but could be more subjective on ease of use and preferences.

Evidence 5: Specializing for repeated work

Agentica and Together AI’s DeepSWE improved Qwen3-32B from roughly 23% to 42% single-attempt success on SWE-bench Verified through reinforcement learning. The team released the training code, data, and logs. Training took six days on 64 H100s: specialization costs money, but builds on an existing foundation. DeepSWE, 2025.

Our internal API example uses a LoRA adapter on a 7B Qwen model. Its economics depend on call volume, accuracy, latency, and how often API changes require retraining. A fair baseline includes a general model with cached tool descriptions.

Evidence 6: The economics of owned capacity

If you own your own your own hardware the comparison calculations may be tricky. Amortized hardware, power, and support operations need accounting. However you handle that, a way to compare is by merged PRs (again, assuming merged PRs are heavily vetted). Include additional human review and other workflow costs in the broader comparison.

Utilization affects the comparison:

Spare capacity: You’re leaving money on the table, but it still could beat API token usage.
Full capacity: You’re using what you have, could be perfect or could mean work is queuing and your losing time.

On-prem or off, tokens still affect memory, throughput, and capacity. Self-hosting changes how that consumption becomes an expense. A hybrid approach is dedicated machines with a cloud provider potentially spinning up or down with demand.

OpenRouter’s September 2026 report says open-weight models passed 50% of token volume on a sustained basis in April and stayed above 62% after their July peak. Closed models still captured 73% of spending in the second week of September, down from 89% in the week of January 5. These figures describe one intermediary’s traffic across many uses, not all software development. OpenRouter, The Token State of Play.

An illustrative million-token workload shows how volume and revenue diverge:

All premium: at a blended $15 per million tokens, the bill is $15.
95% cheaper execution: routing that share to a model charging $0.50 per million brings the total to $1.225, about 92% less.
Premium revenue: the expensive model still collects 61% of the remaining bill.

This assumes equal outcomes and no routing overhead. It describes API spending, not self-hosting economics or supplier profitability.

Sustaining frontier capabilities costs money. Today’s concentration of spending does not establish tomorrow’s durable business.