Models chat well; so do invoices—price each successful job into the product spec before launch
In pilots nobody watches the meter; once assistants, summaries, or RAG sit inside a paid client product, tokens are margin. When KSX ships AI from Shanghai, cost control is a discovery constraint—not a post-launch ops footnote: who triggers calls, what a successful job may cost, and how the product degrades on overage instead of burning quietly. B2B workflows that auto-process docs and assist pricing every day will scare finance faster than any feature list if unit economics are missing. This piece is for agencies and client product teams: cost structure in client products, engineering levers that actually move the bill, quotas that match business rules, and how to treat predictable cost as an acceptance criterion across discover→design→build→launch—before accounting knocks.
Price a successful job before you pick a model
In client products the cost unit is not “one API call” but a business-readable successful job: a support summary that passes review, a cited knowledge answer, a schema-valid extraction. In discovery define the job, then estimate average in/out tokens, retries, retrieved span length, and tool calls. Backsolve allowed model tier and context from “target cost per success.” If the business will pay ¥0.5–2 per decision-assist turn but you run a top multimodal model for classification, the spec is wrong. Writing that unit price into the PRD and SOW beats arguing “why is AI so expensive?” after the invoice.
Where the bill comes from: generate, embed, retry, hidden multipliers
Visible generation tokens are only part of the story. Embedding refreshes, re-stuffing long system prompts, auto-retries, second requests after aborted streams, and full log replay for “quality sampling” all multiply the real bill. Agency delivery should default to: meter by feature flag and tenant; separate user-visible traffic from batch jobs; chart P50/P95 tokens and cost—not just call counts. Multi-tenant products must not let one heavy trial client drain a shared pool. Without labels you cannot explain invoices or apply fair throttles.
Route: small models triage, large models synthesize
Most client-product requests do not need the flagship model. Typical tiers: rules/small models for intent, safety refuse, language detect; mid models for drafts and structured extract; escalate only on high-value or low-confidence paths. For RAG, retrieve first—if sources are thin, abstain or hand off instead of generating long ungrounded answers. We keep routing tables as config in build so price or provider swaps do not touch business logic. Route policies belong in the eval set: thrifty paths must not quietly degrade faithfulness on red-line jobs.
Context and cache: fewer repeated bytes, more useful ones
Shipping the full system prompt and tool docs on every call scales cost with QPS. Prefer: compressed versioned prompts; provider prompt cache on stable prefixes; append-only session deltas; hard top-k with dedupe on retrieval. For client docs, never stuff whole PDFs—chunk and cite. Cap max tokens on output and offer “continue” in UI instead of default long essays. Set length expectations in design: “enough with expand” usually costs less and feels more trustworthy than “write the full report in one shot.”
Product quotas: let business rules drive the throttle
Raw QPS limits protect systems, not margin. Client products should map plans: daily job counts, per-seat pools, workspace budget alerts. Design overage UX: queue, degrade to templates/keyword search, read-only history, or upgrade prompts—never silent fail or unlimited overdraft. B2B contracts that state included AI allowance and overage math explain better than invoices full of model SKUs. Get sales and CS to sign the allowance assumptions in discovery so delivery does not inherit a fight.
Observability: every call must be reconstructable
Cost work without logs is mysticism. Minimum fields: tenant, feature, model, in/out tokens, latency, cache hit, route decision, retry flag, business outcome (success/refuse/error). Sample prompts/responses for audit under privacy rules (redact, retention, no default training use). Weekly review: costly features, retry-heavy pairs, outlier tenants. We put cost regression in release checks: when prompts or retrieval depth change, re-run a cost baseline beside quality evals so a tiny quality bump cannot double the bill.
Vendors and contracts: swappable beats “cheapest”
One vendor locks price and quota risk. Keep an adapter for message shape, retry, timeout, streaming, and billing field maps. Evaluate residency, training policy, SLA, cache features, and whether China/global routing matches compliance. Price sheets change quarterly; product logic should not marry one SDK. Explain to clients with “job price bands + degrade strategy,” not model marketing names. If private deploy is required, fold hardware and ops labor into TCO—do not only compare API list prices.
Bake cost gates into each phase—not a post-launch catch-up
Discover: freeze job unit price, plan allowance assumptions, and unacceptable cost spikes. Design: default context length, degrade UI, usage cues. Build: routing, cache, metering tags, evals with cost assertions. Launch: production boards, alert thresholds, a one-week cost review. On retainer, monthly work hits high-ROI levers (cache hit rate, over-wide retrieval, retry storms)—not endless model shopping. When you need KSX to put AI in a real product, manage cost controls on the same sheet as scope—or you ship demo temperature and the client receives a production invoice.
Checklist
- 1Define target cost on a business successful job and write it into PRD/SOW
- 2Meter tokens/cost by tenant and feature; keep cache and routing configurable
- 3Design overage degrade UX and plan allowances—no silent overdraft
- 4Run quality evals and a cost baseline on every release to catch bill regressions
- 5Provider adapter plus written confirmation of residency and training policy
Key takeaways
- LLM cost control in client products is unit economics per successful job—not a raw API price table.
- Routing, cache, context caps, and product quotas stabilize margin better than blindly chasing a cheaper model.
- Without labeled metering and release cost gates, optimization is guesswork and client invoices stay unexplainable.
FAQ
- Do we need full cost infrastructure during the demo phase?
- Demos can be thin, but log estimated job price and volume assumptions. Before client staging or paid pilot, metering tags, routing, and overage policy should be live—or a “successful” pilot becomes a finance incident.
- Will caching hurt answer freshness?
- It can—so tier it: long-cache stable system prompts; short TTL or version-bust retrieval; never cache personalized context across users. Invalidate on version and business events instead of turning cache off entirely.
- How do we explain model fees to non-technical clients?
- Speak in “about ¥X per completed job, Y jobs included, overage = bill or degrade,” with one week of real usage. Avoid token unit prices alone—they buy job outcomes, not tokens.