Tokens

Tokens are the small pieces of text AI models read and write: words or parts of words. In English, a token is about three-quarters of a word.

AI searchUpdated 1 min read

Also called token · AI tokens

What they are

AI models don't see letters or whole words. They split text into tokens: common words are one token, rare words are split into several. "Recommendation" might be two tokens; "the" is one.

A rough rule for English: 100 tokens is about 75 words.

Why they matter

  • Cost: AI companies charge developers per token read and written.
  • Limits: each model can only handle a certain number of tokens at once, its context window.
  • Speed: more tokens take longer.

What it means for your pages

When an AI search tool reads pages, it usually only keeps parts of each page, to save tokens. Long pages with the key facts buried deep may lose them. Put the important facts early and in clear sections.

Example

A pricing page is 6,000 words, with prices near the bottom after long feature descriptions. An AI tool reading it keeps the first part and misses the prices.

After moving a short price table to the top, AI answers start quoting the right prices.

Frequently asked questions

Do brand names use more tokens?

Made-up or unusual names are often split into several tokens. It doesn't stop AI from learning them.

How many tokens is a typical web page?

A few thousand. A long article can be 10,000 or more.

Do I need to count tokens?

No. Just keep key facts near the top and in clear sections.

Go deeper

All posts →

Make AI search your next revenue channel

Start free trial