Sovereign AI: running an LLM entirely within a client’s own infrastructure, behind their firewall, with no data ever leaving their network, has moved from a niche compliance requirement to a mainstream enterprise conversation. Three forces are driving it.

First, a genuine and reasonable fear: enterprises worry that routing their proprietary processes, codebases, and workflows through a frontier model’s API is effectively teaching that provider their business with no guarantee that knowledge doesn’t eventually inform a future product, or worse, a future competitor.

Second, cost predictability; As frontier model providers adjust pricing, enterprises building AI deeply into core workflows are waking up to a real concern: token costs at production scale are not a rounding error, and losing control of that cost line is a legitimate risk to underwrite against.

Third, and this is the one I find most interesting is a growing consensus that most enterprise workflows simply don’t need frontier-level capability. A well-scoped business process automation task may not require the most capable model in the world. It requires a model that is good enough, reliably, at a fraction of the cost, running entirely within the client’s own walls.

We decided to test that third premise directly. This article summarises what we found.

Our iBEAM platform runs a suite of agents; BA, Dev, and QA, across legacy application modernization projects. We chose the BA Agent as our sovereign evaluation reference for a specific reason: it is our most mature agent, with the richest ground truth data, and its workflow is sufficiently complex to be a meaningful test. The BA Agent’s job is to parse legacy application source code, Oracle Forms, PowerBuilder and generate the business requirement documentation that downstream Dev and QA agents depend on. If the BA Agent’s output degrades, everything built on top of it degrades with it.

We evaluated candidate models on five dimensions: structured output fidelity, instruction adherence, domain content accuracy, completeness, and consistency across repeated runs. Each scored 0 to 4, for a maximum of 20 per artefact. Our threshold for a “sovereign ready” claim was 14/20 (70%) on simple-to-moderate complexity artefacts.

Our first evaluation round, run on a reasonably sized PowerBuilder application modernisation intent, produced results that would have ended the sovereign conversation if we’d stopped there.

Qwen3 235B’s collapse was particularly stark and on closer inspection, it wasn’t a capability problem. On repeat runs of the identical prompt, the model would sometimes abandon the required BRD template entirely and revert to a generic technical specification format. That is a scored failure under any evaluation framework. But the root cause mattered: our production prompt didn’t sufficiently reinforce the BRD schema across calls. Qwen performed plausibly on structural fidelity in its first run. The second-run collapse pointed to a prompt reinforcement gap, not an incapable model.

We found a second, more systemic issue. The consistency dimension, scoring whether identical prompts produce structurally similar outputs was weak across every model: Mistral and GLM both scored 1.8 out of 4, Qwen scored 0. This wasn’t a ceiling problem either. All three evaluations had been run at default sampling temperature. For a BA Agent that needs deterministic, repeatable output in a production pipeline, that’s the wrong setting. We reran with temperature at or near zero.

With the prompt reinforcement fix and corrected sampling settings, we ran a second evaluation; substituting Devstral 2 for GLM-5 to avoid over-concentrating the candidate pool in Chinese-origin models.

Mistral Large 3 crossed the 70% threshold. A meaningful jump from a setup-and-instruction fix, not a different model.

We then tested Qwen3 Coder Next, a variant purpose-built for coding and structured technical output and it performed strongly:

To calibrate what “good” actually looks like, we ran the identical evaluation against Claude Sonnet 4.6, our production baseline and against Claude Haiku 4.5, out of curiosity, since we’d seen Haiku perform comparably to Sonnet on a separate, less complex use case.

This mattered more than it might look. Sonnet at 81%, not 100% tells you something important: even a frontier model has a realistic variance ceiling on a complex artefact generation task. Sovereign models don’t need to hit a perfect score to be viable. They need to close the gap to a realistic ceiling, not an idealised one. And Haiku landing at 65%, below several open-weight sovereign candidates, was a useful reminder that raw brand or scale is not a reliable predictor of fit for a specific structured task.

With the corrected methodology in hand, we ran a wider comparison across seven open-weight candidates.

Four of seven open-weight models cleared the full-pass threshold. That is a materially different conclusion than our first round suggested.

A model’s benchmark score alone doesn’t answer whether it’s the right sovereign choice. Infrastructure and running costs matter a marginally better score from a much larger, more expensive-to-host model is not automatically the better business decision. We compiled size, context window, token pricing, and licensing across all seven candidates.

A few things stood out. Pricing across the board is a fraction of typical frontier pricing. Licensing is mixed, four candidates are clean Apache 2.0, three carry Modified MIT terms that likely come with additional conditions worth reviewing carefully, though all are available on Hugging Face. And architecture varies enormously: total parameters range from 80B to 1.04T, with most candidates using mixture-of-experts designs optimised for cheap inference — Devstral 2 is the one dense, compute-heavy outlier in the set.

We are now moving to a larger, real-world side-by-side evaluation: our production BA Agent running on a broader dataset of Oracle Forms and PowerBuilder applications, comparing our current frontier default, Claude Sonnet directly against Qwen3 Coder Next, with Mistral Large 3 as the standby candidate. This moves beyond synthetic benchmark artefacts into the kind of legacy complexity our clients actually bring us.

A few things are worth carrying forward for any CTO or engineering team running a similar exercise.

Weak first-round sovereign results are not evidence that sovereign deployment isn’t viable. They are frequently evidence that the evaluation setup, prompt reinforcement, sampling temperature, model selection at the wrong scale needs fixing before you draw a capability conclusion. Diagnose before you disqualify.

Benchmark against your own frontier model’s realistic ceiling, not against a theoretical 100%. Even Claude Sonnet did not score a perfect result on our own evaluation. That gap is the honest target for sovereign models to close, not an impossible one.

And score alone is an incomplete decision input. Model size, active parameters, context window, licensing, and token economics all belong in the same table as the accuracy score. The best model for a sovereign deployment is not necessarily the highest scorer, but it is the best fit across capability, cost, and operational footprint.

We ran this evaluation on a synthetic legacy codebase specifically so we could publish the results without any client confidentiality concern. If you are running a similar evaluation, I hope this gives your team a starting point rather than a blank page.