Holon.liveHOLON OF HOLONS

Tokens and tokenizers: why a token is not a character

English · Compare Chinese

A token is a representation unit used to process model inputs or outputs. For text it may represent a word, part of a word, a character or another fragment. A tokenizer applies particular rules to produce these units. There is no universal conversion to Chinese characters or English words.

Different segmentation

Vocabularies, encoding and writing systems affect token counts. Use the relevant model's documentation or tools rather than a fixed ratio.

Different token meanings

Model tokens represent content; tokens in financial networks may represent rights or value. Matching names do not establish automatic exchange.

Agent tasks

Instructions, material and tool outputs may enter context. Token count affects capacity and cost, but more tokens do not guarantee better results. Retrieval and organization help select what matters.

Is Chinese always more expensive? Not independently of the tokenizer, prices and task length.

Continue with What are AI models and large language models? to compare related ideas.

For a related project perspective, explore LingoAI’s AI work.

Sources and further reading