What are AI tokens? A retailer’s guide

AI tokens measure the text a model processes and produces. See how a product question, catalogue context and reply contribute to usage and billing.

AI tokens are the units a model processes when it reads a request and generates a response. In a shopping conversation, usage includes the product information supplied to the model as well as the customer’s question and the reply. The words visible in the chat show only part of that usage.

Token-based charges depend on how much content the assistant processes and the rates applied to each usage category. Understanding those categories lets you compare quotes without treating every short question as an equally small job.

What an AI token represents

An AI token is a model-specific unit of processed content. For text, it can be a whole word, part of a word or punctuation. Characters are individual text elements, such as letters and spaces. A tokeniser groups them into the units the model uses.

NVIDIA’s explanation of AI tokens describes tokenisation across text and other data, including images. This guide uses text examples. A language-model token is a processing unit, separate from a cryptocurrency token someone can own or trade.

For a concrete example, the product request “Show navy raincoats under $150.” produces nine tokens with GPT-4o’s o200k_base encoding. Running OpenAI’s tiktoken tokeniser gives this split, with each code span showing one token:

Show / navy / rain / co / ats / under / $ / 150 / .

The leading spaces belong to their tokens. “raincoats” splits across three tokens, while the full stop occupies one. This is a count of that exact text with that encoding. Another model’s tokeniser can divide it differently.

OpenAI’s tokenisation guide explains how spacing and capitalisation affect the split. A fixed conversion from characters or words gives only an estimate. The nine-token example also excludes the surrounding request structure used to send a chat message.

Input, output and cached tokens

Input tokens are the content supplied to a model for a request. Output tokens are what the model generates. Cached input is previously processed context that the provider can reuse under its caching rules.

Suppose a customer asks, “Does the navy coat come in medium?” The input can include the store’s instructions and the earlier conversation. Retrieved product details supply the medium variant’s stock and price. A delivery-policy passage adds further input if the question also asks about shipping.

The generated answer counts as output. A short reply can therefore follow a large input. Reasoning models can also generate internal reasoning tokens that aren’t shown in the chat, so visible answer length doesn’t measure all output usage.

When an application supplies earlier turns again, those messages become input to the next request. An earlier answer can count once as output when generated and again as input when the conversation continues. Applications can instead shorten history or retain selected facts, which changes that later input.

OpenAI’s prompt-caching documentation describes reusing an unchanged beginning of a request, called a prefix. Cache reuse depends on a matching eligible prefix and an available cache entry. Repeating an idea in different wording doesn’t establish a cache hit.

Cached tokens still belong to the input used for that request. Providers can apply different rates to uncached input, cache reads and cache writes. Treating all repeated text as free would understate the bill.

Estimate one shopping conversation

Estimate a conversation by adding the usage from every model request, then keeping the billing categories separate. A single customer message can trigger several requests if the assistant searches for products before composing its answer.

The following hypothetical conversation assumes one model request per row. The counts are illustrative totals, not measurements of these messages or a forecast for any provider. Input includes the instructions and product information needed for each request. Output includes all generated tokens counted by this example.

Shopper’s actionUncached inputCached inputOutputTotal tokens
Ask for raincoats under $1501,20001201,320
Narrow the results to navy6001,0241401,764
Compare two coats7501,4082002,358
Ask for the second coat in medium4502,048802,578
Conversation total3,0004,4805408,020

The conversation uses 7,480 input tokens, including 4,480 cached tokens. Adding 540 output tokens gives 8,020 tokens across the four requests. These are cumulative usage figures. They don’t mean that 8,020 tokens were present in one request.

To calculate a charge, apply the quoted uncached-input rate to 3,000 tokens and the cached-input rate to 4,480 tokens. Calculate the output charge on 540 tokens. Keep the supplier’s billing unit consistent, whether its rates are per thousand or per million tokens. Account for cache-write charges separately when the supplier’s terms require them.

A real assistant that makes extra tool-related requests needs those requests added too. Use its reported usage for a representative conversation before extending an estimate to your store’s traffic.

Why token usage changes

Token usage changes with the material each request includes and the work required to produce an answer. Two shoppers can send equally short messages that cause different amounts of processing.

Concise catalogue fields can supply the facts needed to compare two coats. Sending every product’s marketing description adds text even when most products don’t match. A lengthy returns policy adds more input than the specific passage answering a question about sale items.

Tool responses are another source of input. A search tool can return a few selected records or many detailed records. If the application retains those results, later requests can include them again. Caching can change their price, while shortening the retained history changes the amount of input.

Reducing text too far can remove the evidence needed for an accurate answer. A coat’s name and price don’t establish whether its fabric is waterproof. Keep the relevant specification and the correct variant record. Remove unrelated descriptions before removing facts that decide the shopper’s choice.

For a complete request, use the provider’s input counter or returned usage record. OpenAI’s token-counting documentation explains that message structure and tool definitions contribute tokens beyond a plain-text tokeniser’s count.

Read token-based pricing

Read token-based pricing as a set of usage categories and rates, with any included allowance stated separately. A token allowance is the amount included in a billing period. A context window is the capacity available to a model for an individual response. Request limits govern how many requests can run within a period.

Per-session billing uses a different unit: the supplier’s defined conversation or session. Compare how that session starts and ends, alongside any usage restrictions. The number of sessions alone doesn’t tell you how many model tokens they consume.

shoppa has per-token pricing for retailers who want to pay for the agent’s usage and per-session pricing for predictable conversation costs. It is an AI shopping agent on your own site. Customers discover and compare products, then buy in the chat using your catalogue and store policies. It also handles order support and returns. The pricing choice covers how you pay for that shopping assistance.

Ask for a quote that identifies what counts towards usage and how cached input is charged. Include a product comparison and a longer policy question in your expected conversation mix. Compare the resulting cost for that mix, alongside the supplier’s session definition, before choosing your billing model.

Give your store its own shopping agent

Customers ask in their own words. Your agent finds, compares and checks out from your catalogue, right on your site.

See it in actionFree trial · Simple setup