Output length is often the quietest driver of AI spend because it scales with every successful request. If a workflow runs 10,000 times a day and the default answer is 700 generated tokens longer than needed, that is 210 million extra generated tokens in a 30-day month before model tier, caching, or batch routing even enter the discussion.
This is for AI engineers, platform engineers, AI product managers, and startup CTOs deciding whether a workflow should use a cheaper model tier, a batch endpoint, or a shorter default response. The hidden cost question is not only “which model?” It is also “how much text are we asking the model to generate every time?”
A simple operating formula catches the issue early: monthly completion cost equals calls per month times average generated tokens times the provider’s output price per token. In a recent review pattern, cutting a support-routing explanation from roughly 900 tokens to 320 tokens did not change the routing decision, but it removed about 64% of the response body the system had been paying to create and discard.
Most AI cost reviews start with input tokens, model tier, and context window size. Response length deserves the same treatment. OpenAI’s pricing page separates input, cached input, and output pricing, and also notes that reasoning tokens are billed as output tokens even when they are not visible through the API.[1] That makes every extra paragraph, table row, JSON field, tool-call explanation, or regenerated answer a product decision, not just a writing style choice.
The practical mistake is letting a demo prompt become a production default. “Write a comprehensive answer” may look good in a sales call. In a support triage queue, an internal code-review assistant, or a nightly document-classification job, it can turn one useful answer into hundreds of unnecessary tokens per request. If usage volume stays flat while the answer template gets longer, spend still rises.
1. Detect verbosity
Output length creeps up when teams optimize for perceived completeness instead of task completion. Common examples are support bots that always include a greeting, diagnosis, steps, warning, and recap; code assistants that explain every line after producing a patch; and extraction prompts that ask for optional JSON fields even when the downstream system only reads three keys.
It also grows when the model is compensating for weak instructions. If users regenerate because the first answer is vague, the product may pay twice: once for the bad answer and once for the retry. In structured workflows, verbosity can create another failure mode. OpenAI’s function-calling docs say function definitions count against context and are billed as input tokens, while the generated arguments still come back in the model response.[2] Anthropic’s tool-use docs make the same broad point: tools, tool-use blocks, and tool results add tokens that have to be counted.[3]
- Track average, p90, and p95 generated tokens by workflow, not only by model. Separate “support answer,” “JSON extraction,” “code review,” “batch classification,” and “chat follow-up” so one verbose feature does not hide inside account-level spend.
- Compare completion length with task completion. If a support answer moves from 450 tokens to 1,200 tokens but the resolution rate does not improve, the extra 750 tokens are probably product debt.
- Review prompts that contain “comprehensive,” “detailed,” “exhaustive,” or “include everything.” Replace them with a target such as “answer in 5 bullets,” “return only valid JSON,” or “give the short answer first, then include optional detail only if requested.”
- Limit default verbosity for high-volume features. A nightly classifier, lead enricher, or catalog-tagging job should usually return labels, confidence, and a short reason, not a full natural-language explanation for every row.
A useful operating threshold: investigate any workflow where p95 completion length is more than 2 times the median for the same task. That pattern often means the prompt is allowing runaway answers, the model is over-explaining edge cases, or retries are being logged as normal completions instead of failures.
2. Measure before routing
A long answer can hide weak reasoning, slow review, and make the UI feel heavier than the task requires. For structured tasks, extra prose can also break parsers. If a pipeline expects one JSON object and the model returns a paragraph before the object, the product may pay for text that the application discards and then pay again for a retry.
Benchmarks can help decide which models deserve testing, but they do not tell you whether your onboarding assistant should return 80 words or 800 words. Keep benchmark snapshot dates in the model-routing review, and keep response-length decisions in your product telemetry.
Use a before/after test before changing model tier. Suppose a product-description workflow currently returns 1,200 generated tokens per item: headline, long description, feature bullets, SEO keywords, social captions, and rationale. If the downstream CMS only uses the headline, 120-word description, and 5 bullets, redesign the response to target 500 tokens. That removes 700 generated tokens per item, a 58% reduction in completion length for that workflow before changing provider, model, or endpoint.
| Step | Current behavior | Length-controlled behavior |
|---|---|---|
| 1. Define required fields | Prompt asks for every useful marketing asset. | Prompt returns only fields the CMS stores: headline, description, 5 bullets. |
| 2. Set a ceiling | No explicit length cap beyond the API maximum. | Template says “120 words max” for the description and “5 bullets max.” |
| 3. Measure output | Track total tokens after the invoice arrives. | Log generated tokens per item and alert when p95 exceeds 700 tokens. |
| 4. Route | Use the same synchronous model path for every item. | Use synchronous calls for interactive edits and batch endpoints for nightly catalog runs when provider limits fit. |
3. Cap it through product design
Start with the product surface, not the provider bill. For each workflow, write down the user-visible answer shape, the machine-readable fields, the maximum useful length, and the retry rule. A required field should be read by code, displayed to a user, stored for audit, or used in routing. If no system consumes it, remove it.
For customer-facing chat, use progressive disclosure. Return the direct answer first. Put supporting detail behind “show more,” a follow-up question, or a separate expansion action. For internal tools, make the short path the default and let power users request a longer explanation only when they need it for review, audit, or handoff.
For structured outputs, treat every field as a cost line. If a field is useful only for debugging, sample it on a small percentage of requests instead of generating it for every call. If a field is useful only for humans, show it only in human review paths.
Resource: Use AI Models after the workflow has a length target. Compare model choices by pricing per million input and output tokens, context window sizes, modalities, public benchmark scores, and the cost estimator before deciding whether model routing also needs to change.
4. Then consider batch or model routing
Batch endpoints matter when the user does not need an immediate answer. They do not fix verbosity, but they can make large offline jobs cheaper if the job fits the provider’s rules. OpenAI documents 50% lower costs than synchronous APIs, a 24-hour turnaround, 50,000 requests per batch, and a 200 MB input-file limit.[4] Azure OpenAI Global Batch documents 50% less cost than global standard, a 24-hour target turnaround, a 200 MB maximum input file size, and 100,000 requests per file.[5]
Anthropic documents batch processing at 50% of standard API prices, with Message Batches that can contain up to 100,000 requests and 256 MB and can take up to 24 hours to complete.[6] Google documents a 50% discounted batch rate for Gemini, up to 200,000 requests, a 1 GB Cloud Storage input-file limit, queueing for up to 72 hours, SLO exclusion, and cache discount precedence over the batch discount.[7] Amazon Bedrock documents asynchronous batch inference through Amazon S3 and notes that batch inference is not supported for provisioned models.[8]
| Decision | Use when | Watch for |
|---|---|---|
| Cap the response | The workflow generates more text than the user reads or the system stores. | Parser failures, support macros, long rationales, unused JSON fields. |
| Move to batch | The job is offline, repeatable, and fits provider file, request, quota, and turnaround limits. | Queue time, model availability, partial results, enqueued-token quotas. |
| Change model tier | The capped workflow still misses quality, latency, or cost targets. | Benchmark relevance, retry rate, context size, reasoning-token behavior. |
The decision path is simple: detect verbosity, measure it by workflow, cap the answer shape, and only then compare batch limits or model tiers. If a workflow is still expensive after the cap and does not need an immediate response, routing may help. If the answer is still too long, routing just moves the same waste to a cheaper lane.
Operational questions
How do we set a cap without truncating useful answers?
Start from the consuming surface. If the UI shows 5 bullets, cap the answer at 5 bullets. If the database stores a 160-character summary, ask for that size explicitly. For high-risk workflows such as legal review, incident analysis, or migration planning, set a higher ceiling and measure whether readers actually use the extra detail.
What if longer answers improve trust?
Separate confidence from length. A short answer can include evidence, caveats, and next actions without becoming a memo. When auditability matters, store a concise rationale or trace field and let reviewers expand the full explanation only on sampled or escalated requests.
Should we remove explanations from tool and JSON workflows?
Usually yes for the default path. Return the object the system needs, plus a short reason only if another system or reviewer consumes it. Keep verbose explanations in debug mode, evaluation datasets, or review queues where the extra tokens have a defined purpose.
Sources
- OpenAI API pricing: input, cached input, output pricing, and reasoning-token billing. https://platform.openai.com/docs/pricing/
- OpenAI function calling guide: function definitions count against context and are billed as input tokens. https://platform.openai.com/docs/guides/function-calling/function-calling%20.webm
- Anthropic tool use pricing and token accounting. https://docs.anthropic.com/en/docs/tool-use-pricing-and-tokens
- OpenAI Batch API guide: discount, turnaround, request cap, and input-file limit. https://platform.openai.com/docs/guides/batch/
- Azure OpenAI Global Batch documentation: cost, turnaround, file limit, and requests per file. https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/batch
- Anthropic batch processing and Message Batches limits. https://docs.anthropic.com/en/docs/build-with-claude/batch-processing
- Google Vertex AI batch inference for Gemini: discount, limits, queueing, SLO exclusion, and cache precedence. https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/batch-prediction-gemini
- Amazon Bedrock batch inference: asynchronous S3 workflow and provisioned-model limitation. https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference.html
- OpenAI Help Center on ChatGPT Search and OAI-SearchBot access. https://help.openai.com/en/articles/9237897-chatgpt-search
- OpenAI product discovery guidance for ChatGPT Search visibility. https://openai.com/chatgpt/search-product-discovery/