Agent cost and latency come from the entire task, not only one model response. Calls, retrieval, tools, retries and continuing operation can produce expense and delay.
Unit price and total cost
A cheap call may require repetition or human correction. A fast first token does not guarantee faster completion.
Throughput and latency
Throughput concerns work processed over time; latency concerns a request's or task's waiting time. Improvements can involve model services, context and task organization. One metric does not describe every experience.
Compare outcomes and resources
State scale, quality requirements and conditions. Original vLLM research illustrates service-layer optimization, not fixed performance in every current environment.
Are fewer tokens always cheaper overall? No. Removing irrelevant material helps, but omitting necessary information may cause failure or repeated retrieval.
Continue with Tokens and tokenizers: why a token is not a character to compare related ideas.
For a related project perspective, explore LingoAI’s agents work.