How AI models actually work: tokens, transformers, training and inference
A plain-English tour of how AI models work — tokens, embeddings, attention, training and inference — with diagrams and charts. No maths degree needed.
Published , updated 8 min read
Key takeaways
- AI models are prediction machines: they guess the next token, one at a time.
- Tokens, embeddings and attention turn text into numbers and context into meaning.
- Training adjusts the model once; inference runs the prediction loop for every prompt.
- Fluent does not mean true — give the model sources, tools and checks.
When someone asks me how an AI model works, they usually expect a maths lecture. The real answer is simpler, and more useful: a modern AI model is a prediction machine. It has read an enormous amount of text, learned the patterns in it, and now guesses what should come next — one small piece at a time.
That one idea explains almost everything else: why models are fluent, why they sometimes invent facts, and why the same prompt can give you a different answer twice.
The whole pipeline in one diagram
Here is the full journey, from your prompt to an answer:
Your prompt
|
v
Tokenizer — text is cut into tokens
|
v
Embeddings — each token becomes a list of numbers
|
v
Transformer layers — attention + feed-forward, repeated many times
|
v
Probability distribution — a score for every possible next token
|
v
Pick the next token (and repeat)
|
v
Your answer, one token at a time
Nothing in that pipeline "understands" the way a person does. It is pattern matching at a scale that starts to look like understanding. Keep that diagram in mind — every section below is one box in it.
1. Tokens: the model does not read words
Models do not read words. They read tokens: small chunks of text, usually parts of words.
For example, this sentence:
The capital of France is
might be split roughly like this:
| Text | Tokens |
|---|---|
| The capital of France is | The, capital, of, France, is |
Longer or rarer words get split further. "Tokenization" is a good example — a tokenizer might split it into Token, ization, or similar pieces. The exact split depends on the model, but the principle is the same: everything becomes a sequence of token IDs, and those IDs are all the model ever sees.
Why does this matter to you? Two practical reasons:
- Cost and limits are counted in tokens, not words. A long prompt costs more and leaves less room for the answer.
- Weird splits explain weird mistakes. A model can struggle with spelling or counting letters because it never saw the letters — it saw the tokens.
2. Embeddings: turning tokens into numbers
A token ID on its own means nothing to a computer. So each token is turned into an embedding: a long list of numbers that places the token in a kind of map.
On that map, words used in similar ways sit close together. "King" and "queen" are near each other. "Paris" is near "France" and "capital". "Dog" is far from "spreadsheet".
The model does not store a dictionary definition for each word. It stores a position. Meaning, for a model, is mostly about where a token sits relative to all the others — and that position is learned from how the token was actually used in training text.
3. Attention: how the model reads context
This is the part that changed everything, and it comes from the transformer architecture.
When the model processes a sentence, each token is allowed to "look at" the other tokens and decide which ones matter most for it. That is called attention.
Take a classic example:
The animal didn't cross the street because it was too tired.
What does "it" refer to? A person knows it is the animal. A model works it out because, during training, the token "it" learned to pay strong attention to "animal" in sentences shaped like this one, and weaker attention to "street".
That is the whole trick, repeated at massive scale:
- Each token asks: which other tokens change my meaning?
- It collects information from those tokens, weighted by relevance.
- It updates its own representation with what it found.
A transformer stacks this process in many layers — dozens of them in a large model. Early layers pick up simple patterns like grammar and word order. Later layers pick up longer-range ideas like topic, tone and intent. No single layer "knows" the answer. The answer emerges from all of them together.
4. Prediction: the model scores every next token
After all those layers, the model produces a score for every token in its vocabulary: how likely is this token to come next?
Chart: an illustrative example, not a measurement from a real model.
| Next token | Illustrative probability |
|---|---|
| Paris | 92% |
| London | 3% |
| Berlin | 2% |
| Rome | 2% |
| Madrid | 1% |
Then it picks one — usually the most likely one, but not always. A setting called temperature controls how adventurous that pick is:
- Low temperature: the model almost always picks the top token. Answers are predictable and consistent.
- High temperature: the model sometimes picks a lower-ranked token. Answers are more varied and creative — and more likely to go off track.
Then the chosen token is added to the text, and the whole process runs again to predict the next token. An answer that looks like one smooth paragraph is really hundreds of small predictions chained together.
This is why a model can write fluent text and still be wrong. Fluency is not the same as truth. The model is optimising for "what usually comes next in text like this", not "what is true in the world".
5. Training vs inference: learning once, answering many times
People mix these two up, so here is the clean split:
| Training | Inference | |
|---|---|---|
| What happens | The model learns from a huge amount of text | The trained model answers your prompt |
| How often | Once (plus later fine-tuning) | Every time you send a message |
| What changes | The model's internal numbers (its weights) are adjusted | Nothing in the model changes |
| Cost | Very high: weeks of computing on specialised hardware | Low per request, but it adds up at scale |
| Goal | Reduce prediction error on the training text | Produce the most useful next tokens for your prompt |
Training, in one paragraph: show the model a piece of text, hide the next token, ask it to predict it, and measure how wrong it was. That error is called the loss. Then adjust the model's numbers slightly in the direction that would have made the prediction better — a process called backpropagation and gradient descent. Repeat that billions of times, over trillions of tokens, and the model gradually gets very good at predicting what comes next.
Inference, in one paragraph: freeze those numbers, give the model your prompt, and let it predict token after token until it decides to stop. The model is not learning from your conversation (unless a product is specifically built to do that). It is just running the prediction loop.
6. Not all models work the same way
Language models are the most famous kind, but the same idea — learn patterns from data, then generate something new — shows up everywhere:
| Model type | What it learns | What it generates |
|---|---|---|
| Language models (LLMs) | Patterns in text | Text, code, answers |
| Image models | Patterns in images, often with text descriptions | Images from a prompt |
| Speech models | Patterns in audio | Spoken words, transcription |
| Multimodal models | Patterns across text, image and audio together | Answers that mix text, image and sound |
Image models work differently under the hood — many use a process called diffusion, starting from noise and gradually cleaning it up into an image that matches the prompt — but the high-level story is the same: learn the patterns, then generate new examples that follow them.
7. What models are bad at (and why it matters)
If you build with AI, the limitations matter more than the demos:
- Hallucinations. A model can state a wrong fact confidently, because confident-sounding text is a common pattern. It does not check a source unless you give it one — that is what retrieval (RAG) and tools are for.
- No live knowledge. A model only knows what was in its training data, up to its cutoff. Ask about something newer and it will guess, not admit it does not know, unless it is built to search.
- No real memory of you. Each conversation starts fresh unless the product stores context. The model itself does not remember yesterday.
- Bias in, bias out. Training data reflects the world that wrote it, including its biases. A model will reproduce patterns it saw, good and bad.
- Reasoning is still prediction. A model can work through a maths problem step by step and get it right, or confidently get it wrong halfway through. For anything that must be exactly right, verify the result or give the model a tool (a calculator, a database, a code runner) to check with.
None of this makes models useless. It makes them a component — a very powerful one — inside a system that also has sources, tools, checks and a human in the loop.
What this means if you are building with AI
At CodeMyPixel, this is how I think about it on real client work:
- Start with the job, not the model. What should the user get, and how will you know it worked?
- Give the model something true to work from. Your documents, your database, your rules — not just its training memory.
- Let it use tools. Search, calculation, sending a message, updating a record. A model that can act is far more useful than one that can only talk.
- Evaluate like a product, not a demo. The same prompt, run many times, with real inputs. Measure what matters and watch where it fails.
- Keep a human in the loop for anything irreversible. Publishing, paying, deleting, promising a customer something. The model drafts; a person decides.
A model is not a colleague who understands your business. It is a prediction engine that, used well, can do a remarkable amount of the drafting, summarising, classifying and generating — if you build the system around it properly.
If you are working on something like that, or you want to build with us, send me a message.
Frequently asked questions
Do AI models understand what they write?
Not the way a person does. They learn patterns in text and predict what usually comes next. That can look like understanding, but there is no checking against the real world unless the system around the model adds sources and tools.
What is a token?
A token is a small chunk of text, usually part of a word. Models read tokens, not words, and costs and context limits are counted in tokens.
What is the difference between training and inference?
Training is when the model learns: its internal numbers are adjusted over billions of examples. Inference is when the trained model answers a prompt. Nothing in the model changes during inference.
Why does the same prompt sometimes give a different answer?
Because the model picks each next token from a probability distribution. A setting called temperature controls how often it picks a lower-ranked token, so higher temperature means more variety — and more risk of going off track.
