# What CFOs, CIOs and CTOs Weigh When Choosing an LLM

Enterprises buy LLMs on outcomes, not leaderboards or tokens: 85% test on their own data, 60% judge cost by ROI, yet 53% have no cost-per-outcome metric. n=102.

**Analyst:** Anubha Garg

## Key findings

- Public benchmarks are leaderboards that rank LLMs on general tasks, not on any one company’s data.
- Token economics is paying per million tokens; cost per outcome is the price of one finished task.
- 85% treated public benchmarks as background or let a test on their own workloads decide their last model choice.
- 60% compare models on downstream ROI rather than price per token.
- 53% have no formal cost-per-outcome metric.

## Report structure

- What do enterprise buyers actually weigh when they choose an LLM?
- Is cost per token how enterprises measure LLM spend?
- Do enterprise buyers actually use LLM benchmarks to choose a model?
- Is token cost the ceiling on enterprise LLM scale?
- What does “working well in production” mean to LLM buyers?
- Which governance and admin controls decide whether an LLM gets deployed?
- Are non-US LLMs such as DeepSeek and Qwen getting into the enterprise at all?
- How do enterprises route work across LLM providers, and what triggers a switch?
- 6 voices on what decides an LLM, in the buyers' own words.
- 5 moves for LLM providers and vendors selling to the enterprise scorecard
- Test it, price it by outcome, keep it predictable

## FAQ

### Do enterprises actually use LLM benchmarks to choose a model?
Rarely as the deciding factor. Among the 93 leaders who described where benchmarks sat in their last model decision, 44% called them background context or ignored them and 41% said real-workload testing was the decisive evidence. Only 15% raised vendor claims failing in production as a benchmark-related concern. Asked how much benchmark discourse helps, 55% of 92 said it has limited or background value and 42% prefer pilots and direct testing.

### Are DeepSeek and other non-US LLMs replacing US models in enterprises?
No, not in this sample. 75% of 101 leaders said non-US models such as DeepSeek, Kimi and Qwen are excluded or have never been tested in their organization, and 93% have limited or no comparative production evidence. Only 27 of the 102 interviewed have run one on a real workload. Where they did, 5% found lower cost and adequate performance on simple workloads and 2% preferred US frontier models for complex, consistent work.

### How do enterprises evaluate LLMs before buying?
By testing them on their own workloads. 61% of 100 leaders are involved in hands-on model evaluation and selection, and 41% of those who discussed evidence named real-workload testing as decisive. The criteria weighed most were task capability, output quality and accuracy (42% of 96), then security, privacy and enterprise integration (29%) and cost, speed and hands-on workflow testing (29%). Security and data exposure was the dealbreaker for 59% of the 75 who named one.

### What percentage of companies use more than one LLM provider?
63% of 101 enterprises interviewed run multi-provider environments spanning several frontier labs. A further 20% are actively evaluating Claude, Gemini, open-weight or embedded tools alongside their current provider, and 17% run OpenAI or Microsoft Copilot alone. Routing between providers is manual for 80% of the 84 who described it, and 46% of 72 are building for portability rather than leaning on one provider.

### Is cost per token the right way to measure LLM spend?
Buyers say no, but few have replaced it. 60% of 98 leaders compare models on downstream ROI and labor savings rather than sticker price per million tokens, and only 23% track aggregate spend and token efficiency. Yet 53% of 96 have no formal cost-per-outcome metric; 29% measure per case, ticket, document or task and 18% count time saved. Budget owners are twice as likely as hands-on evaluators to have a unit (50% of 18 against 25% of 61).

### Why do companies switch LLM providers?
Performance first, cost second. Among the 84 leaders who described their last model change, 57% said it was driven by performance or capability, 27% by an IT, engineering, business or executive decision, and 15% by cost, pricing or redundant spend. Of the 49 who weighed switching overhead, 55% said meaningful performance, cost or security gains justify a switch, while 33% said incumbent ecosystems, contracts and licensing keep them where they are.

### What blocks enterprises from adopting a new LLM?
Security and data exposure. 59% of the 75 leaders who named a near-dealbreaker cited security, privacy or data-exposure risk, ahead of failure on real-world task quality (28%) and unfavorable pricing (13%). Day to day, 43% of 98 said access, permissions and sensitive-data governance is the main work of running a model, and security, privacy, legal and compliance teams hold the veto for 44% of 77.

### Do LLM updates break production systems?
Sometimes, and quietly. 40% of 97 leaders said their last bad surprise in production was an output regression, hallucination or silent model change, and 34% cited outages, integration failures or unexpected cost spikes; 26% reported no material incident. Asked directly about provider updates, 21% of 78 said silent updates or version changes caused regressions, 72% reported no material incident, and 8% have testing, monitoring, rollback or fallback protections in place.

## Methodology

Research methodology How was this research conducted? This research draws on 102 in-depth interviews with enterprise leaders across the United States involved in choosing LLMs or the budget for model spend, conducted in August and September 2026 as conversational, AI-led, open-ended interviews. Interviews ran up to 34 minutes, with an average duration of 23 minutes. Discussions covered where public benchmarks sit in model decisions; how the model bill is judged and whether a cost-per-outcome unit exists; what working well in production means and the last bad surprise; the day-to-day work of access, integration and governance; routing and switching across providers; and the evaluation of and trust boundaries around non-US models such as DeepSeek, Kimi and Qwen. Who we interviewed Participants were senior technology, finance, security and procurement leaders and the committees they sit on, including CEOs, CIOs, CTOs, CFOs, finance directors, heads of AI and information security, and directors of IT and engineering. 61% are hands-on in model evaluation and selection, 21% sit in a cross-functional governance or recommendation role and 18% own the budget or approve spend. Of the 93 who named an industry, 49% work in healthcare, education, public sector and industrial organizations, 33% in technology, software and professional services and 17% in financial services, insurance and banking. Company names are withheld. How we analyzed it Each interview response was coded to one of three or four labels per question by AI semantic analysis of the transcript, followed by multi-iteration validation and cross-verification, and every transcript was independently reviewed by G2’s AI Custom Research team. Percentages are calculated on the respondents who addressed each question, and the base is stated with every figure. Every quotation is verbatim from the interview transcript, with light cleanup of transcription stutters and misheard product names only. Limits Not every leader answered every question, so each percentage is calculated only on the people who did, and that number is stated with every figure; it ranges from 28 to 101. Where a figure rests on fewer than 30 people, such as the 27 who have run a non-US model, the report says so and treats it as an indication rather than a precise measurement. All respondents are in the United States and most are hands-on evaluators, so the findings describe these buyers rather than the whole market.

**Topics covered:** Software Buying