LISTEN TO THIS ARTICLE
An agent can make a cheap model call and still be expensive to run. Planning, searches, tool execution, retries and human corrections all belong to the user task. Start by connecting those costs to an outcome you can actually accept.
This guide provides a measurement plan and a small accounting example. It does not promise a percentage saving.
Define success before dividing the bill
An accepted task meets your application's requirements. For a research assistant, that might mean answering the question with supporting sources. For an extraction workflow, it might mean producing correct fields that pass validation. A successful HTTP response is not enough.
Keep the acceptance rule stable while comparing changes. Report task success alongside cost per accepted task: rejecting more requests should not masquerade as an optimisation.
Count the work that failed as well as the answer you kept.
Use a task identifier to connect every attempt, including fallbacks and abandoned runs. Count the accepted task once, after the final decision. Keep failed-task costs in the numerator. Otherwise the dashboard makes unreliable workflows look cheaper than they are.
Collect enough detail to explain the cost
Record the model and version, input and output usage, cached usage where reported, provider request identifier, task identifier, attempt outcome and elapsed time. Keep a pricing version or billing date so an estimate can be reproduced later.
Collect tool, hosting and human review costs separately. Avoid logging confidential prompts merely to count tokens. Reconcile estimated usage charges with the provider's invoice and investigate unexplained differences.
Define whether a cost report includes only marginal API charges or allocated infrastructure and staff time. Both can be useful, but comparing them without stating the scope is misleading. Do not allocate the same invoice to attempts and add it again as monthly overhead.
A runnable whole-task example
The following Python program uses only the standard library. Save it as cost.py. The example and recorded test results are available to inspect. It accepts a JSON file of tasks and all their attempt costs, expressed in a single currency. The costs are already calculated: obtain them from billing records or an explicitly versioned usage estimate.
#!/usr/bin/env python3
"""Illustrative cost accounting. Input: {"tasks": [...]} JSON; no external services."""
import argparse
import json
from decimal import Decimal
def summarise(tasks):
total = Decimal("0")
accepted = 0
seen = set()
for task in tasks:
if task["id"] in seen:
raise ValueError("Duplicate task id")
seen.add(task["id"])
if not isinstance(task["accepted"], bool):
raise ValueError("accepted must be a boolean")
accepted += int(task["accepted"])
for attempt in task["attempts"]:
value = Decimal(str(attempt["cost"]))
if not value.is_finite() or value < 0:
raise ValueError("cost must be finite and non-negative")
total += value
return {"total_cost": str(total), "accepted_tasks": accepted,
"cost_per_accepted_task": str(total / accepted) if accepted else None}
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("input", help="JSON file containing tasks and all attempt costs")
args = parser.parse_args()
with open(args.input, encoding="utf-8") as handle:
data = json.load(handle)
print(json.dumps(summarise(data["tasks"]), indent=2))
if __name__ == "__main__":
main()
Save this deliberately illustrative input as example.json. These are invented currency units for explaining arithmetic, not provider prices or benchmark measurements:
{"tasks": [
{"id": "research-a", "accepted": true,
"attempts": [{"cost": "0.04"}, {"cost": "0.06"}]},
{"id": "research-b", "accepted": false,
"attempts": [{"cost": "0.03"}]}
]}
Run python3 cost.py example.json. The output includes the unsuccessful task and both attempts on the accepted task. If no tasks are accepted, the ratio is null; reporting a zero cost per accepted task would conceal that the system produced no accepted work.
This is a teaching example, not a billing integration. It assumes each task appears once and all costs use the same currency and accounting scope. Add your own export step to map provider usage into attempt costs. Keep cancellations and timeouts if they incurred charges.
Pick the next change from the traces
Sort tasks by total cost and inspect representative expensive cases. Look for repeated searches, retries that repeat the same failure, large tool responses, unnecessary model calls and answers that need manual repair. Do not assume the largest model is the main problem.
A practical experiment record contains the baseline, proposed change, acceptance test, total cost, success rate and completion latency. Keep the raw task outcomes so someone can examine a saving that looks suspicious.
OpenAI's model optimisation guide places evaluations around prompt and model changes. Apply that discipline to cost work: a lower bill matters only if the resulting application still does the job.
Test routing and caching carefully
A cheaper model may handle a particular operation adequately. Compare it on held-out examples before changing the route. Include the routing decision and fallback attempt in cost and latency totals. A model's claim that it is confident is not an acceptance check.
Prompt caching and answer caching are different. Provider prompt caching can reuse eligible shared input processing while still generating a new answer. OpenAI's caching documentation describes prefix matching and usage reporting. Use the selected provider's actual rules and observed cache usage rather than assuming a discount.
An answer cache returns an earlier result. Its key and invalidation rules must account for permissions, source versions and request context. Similar wording does not establish that an old answer is correct for a new user. Start with exact repeated requests whose reuse you can justify.
Keep a saving only when the accepted result survives the change.
Reduce unnecessary work
Shorten tool responses to the fields the next step needs, while preserving evidence required for verification. Remove repeated instructions only after testing the change. If you summarise history, test whether the summary retains decisions and constraints; include the summarisation call in the bill.
Use output limits appropriate to the operation and inspect truncated responses. A limit that causes retries may increase the total cost. Replace a model step with ordinary code when the task has a well-defined deterministic solution, such as validating a schema.
Batching is worth investigating for work that can wait, using current provider terms. It is not suitable merely because the unit price is lower: deadlines, retries and result delivery still matter.
Put the numbers to work
Choose a bounded group of real tasks, measure the baseline and change one contributor. Keep an unsuccessful experiment in the record so it does not get repeated later. Review costly failures separately from healthy but inherently expensive work.
Use the evaluation checklist to define acceptance. If memory or serving utilisation is driving costs, continue with the MoE and dense deployment comparison. If large document inputs dominate, use the retrieval and long-context test matrix.