1. What Is a Language Model?

A language model is, at its core, a probability engine for text. Given a sequence of words, it estimates how likely each possible next word is to appear.

That sounds simple, and the core idea genuinely is. A language model answers one question, over and over: given the text so far, what comes next? Everything you see from ChatGPT or Claude, whether it is writing code, summarising a document, or translating Hindi to English, is built on this single mechanism repeated billions of times.

That does not mean an LLM understands language in the way a human does. A model has never "had a thought" or experienced the world it is describing. What it has is a statistical sense of which words, ideas, and patterns tend to follow one another, learned from an enormous amount of text. That statistical sense turns out to be surprisingly powerful, powerful enough that the output can often look remarkably like understanding.

To see how this works, consider an incomplete sentence:

An application using this model can pick the highest-probability option (cook soup), or it can sample from the distribution, choosing randomly among the candidates weighted by their probabilities. Sampling is what gives models their variability. Two conversations with the same model rarely produce identical answers because the model is rolling weighted dice at each step.

Estimating what fills a blank is the primitive operation. From that single capability you get text generation (keep filling blanks), translation (fill in the foreign-language word), summarisation (fill in the short version), and code completion (fill in the next line of code). The task changes, the core mechanism does not.

2. Tokens and Probabilities

Before a language model can work with text, it needs to break that text into units called tokens. A token can be a whole word, a piece of a word, or sometimes a single character. The word tokenisation refers to the process of slicing text into these pieces.

English text tends to produce roughly one token per three-quarters of a word. The sentence 'I love cooking' might become three tokens: I, love, cooking. A longer word like uncharacteristically might be split into several subword tokens: un, character, istic, ally. Languages with non-Latin scripts often use more tokens per word, which is one reason inference costs can be higher for those languages.

Once text is tokenised, the model's job is to assign probabilities over the vocabulary for the next token. The vocabulary of a modern model can be 50,000 to 200,000 tokens. At each step, the model computes a score for every token in that vocabulary and converts those scores into a probability distribution. The token with the highest probability is the model's top guess, but as mentioned, applications often sample from the distribution instead.

Parameters are the internal numbers the model uses to compute those probabilities. A 7-billion-parameter model has 7 billion learnable weights, each one a small dial that was tuned during training. More parameters generally means the model can represent more nuanced patterns, but it also means more compute, more memory, and more cost. Parameters alone do not determine quality, the training data and architecture matter just as much, but they are a rough proxy for a model's capacity.

3. N-grams and the Context Problem

The earliest language models counted word sequences. An n-gram is simply an ordered sequence of N words. When N is 2, you get a bigram, two words in a row. When N is 3, a trigram, three words in a row. The model builds a table of how often each n-gram appeared in its training corpus, then uses that table to guess the next word.

Given the phrase "you are very nice", the resulting bigrams are "you are", "are very", and "very nice". The trigrams from the same phrase are "you are very" and "are very nice". A bigram model that sees "orange is" looks up every bigram starting with "orange" and picks the most frequent follower. Similarly, a trigram model that sees "orange is" looks up trigrams starting with "orange is".

The problem with n-grams is that they capture very little context. Given "orange is", a trigram model can only look two words back. Is "orange" the fruit or the colour? The two-word window is not enough to tell. The model might predict "ripe" (fruit) or "cheerful" (colour), and it has no way to know which is correct without more surrounding text.

You could increase N to capture more context. A 5-gram looks back four words, a 7-gram looks back six. But as N grows, the number of possible sequences explodes. A 7-gram model trained on a large corpus will have seen most specific 7-word sequences only once, if at all. When every sequence is unique, the model has no statistical basis for prediction. The counts become too sparse to be useful.

This is the fundamental tension: short n-grams lack context, long n-grams lack data. Language models spent decades stuck in this trade-off.

4. Neural Language Models

Recurrent neural networks, or RNNs, were the first serious attempt to break out of the n-gram trap. An RNN processes text token by token, and at each step it carries forward a hidden state, a kind of running summary of everything it has seen so far. The easiest way to understand it is to think of it as a reader who forms an impression as they go, updating their understanding with each new word.

This was a real improvement over n-grams. An RNN could, in principle, look back across an entire sentence or even a short paragraph. It could learn that "orange" followed by "is" followed by "ripe" is probably about fruit, and carry that forward to influence later predictions. The context window was no longer fixed at N minus one words.

But RNNs had two stubborn problems. First, they processed tokens sequentially. To predict the hundredth token, the model had to compute the first, then the second, then the third, all the way through. There was no way to parallelise this, which meant training on large datasets was slow. Second, and more fundamental, the hidden state tended to degrade over long sequences. Information from the start of a long passage would fade or become garbled by the time the model reached the end. This is the vanishing gradient problem, and it put a hard ceiling on how much context an RNN could usefully retain.

RNNs learn more context than n-grams, but the amount of useful context they can hold is still relatively limited.

LSTM and GRU architectures, popular in the 2010s, were clever workarounds that helped preserve information over longer distances. They helped, but they did not solve the core issue. Sequential processing was still the bottleneck, and truly long-range context remained out of reach.

What was needed was an architecture that could look at the entire input at once, not token by token, and weigh the relevance of each part to every other part. That architecture arrived in 2017.

5. Transformers and Self-Attention

In 2017, a team at Google published a paper called "Attention Is All You Need." The architecture it introduced, the Transformer, discarded the sequential processing of RNNs entirely. Instead of reading token by token, a Transformer looks at the entire sequence at once and decides which parts are relevant to which.

The mechanism at the heart of this is called self-attention and the name is apt: the model attends to itself, meaning every token in the input can look at every other token and decide how much weight to give each one. There is no fixed window or a sequential decay. The first word of a document can directly influence the last word, and vice versa.

In the sentence "the orange is ripe and sweet," the word "orange" is ambiguous on its own. But self-attention lets the model link "orange" directly to "ripe" and "sweet" without anything in between degrading the signal. The model learns to assign high attention weight between words that are relevant to each other, regardless of how far apart they sit in the sequence.

In an n-gram, the only context available is the last N minus one words. In an RNN, context has to survive a chain of hidden states. In a Transformer, every token has a direct line to every other token. That is why Transformers handle long-range dependencies so much better, and why they scale so well: the entire computation can be parallelised across the sequence, which makes training on massive datasets feasible.

The "large" in large language model refers to scale: hundreds of millions to hundreds of billions of parameters, trained on trillions of tokens of text. The Transformer architecture is what made that scale possible, because it could be trained efficiently on the parallel hardware, GPUs and TPUs, that became available in the late 2010s. The combination of a better architecture and more compute is what produced the capability jump from early neural language models to the systems we use today.

6. Training an LLM

Building a large language model is a multi-stage process and each stage serves a different purpose, and understanding the pipeline helps explain why models behave the way they do.

7. Key Problems

Large language models are powerful, but they are not perfect. Three problems in particular are worth understanding, because they shape how these models can and cannot be used.

8. Fine-Tuning and Distillation

Once a base model exists, there are ways to make it better for specific tasks without training from scratch. Two of the most important are fine-tuning and distillation.

Fine-tuning means taking a pre-trained model and training it further on a smaller, task-specific dataset. If you want a model that is good at answering customer support tickets, you fine-tune it on thousands of example ticket-and-response pairs. The model already knows language and general knowledge from pre-training; fine-tuning just nudges it toward the patterns, vocabulary, and format your use case needs.

This is far cheaper than training from scratch. You are starting from a model that already works and making targeted adjustments. A full pre-training run might cost millions of dollars in compute; a fine-tuning run can cost a few hundred. This is also why open-weight models like Llama are valuable: anyone can fine-tune them for their own domain without needing the resources to build a foundation model.

Distillation is about efficiency. A large model, say 70 billion parameters, might be very capable but too expensive to run for every query. Distillation trains a smaller model, say 7 billion parameters, to mimic the larger one's outputs. The smaller model will not match the large one on everything, but for many tasks it can get close enough while being an order of magnitude cheaper and faster to run.

The way it works is straightforward in concept: you generate outputs from the large model, then train the small model on those input-output pairs. The small model learns to approximate the large model's behaviour rather than learning from scratch. The result is a model that is cheaper to deploy, faster to serve, and often good enough for production use.

9. Model Families Compared

The major LLM families share the same Transformer foundation but differ in how they are trained, what data they use, how they are aligned, and whether their weights are openly available. The table below summarises the key distinctions as of mid-2026.

The closed versus open distinction matters a lot in practice. With GPT, Claude, and Gemini, you interact with the model through an API and you cannot see or modify the weights. With Llama, you can download the model, fine-tune it on your own data, run it on your own hardware, and audit its behaviour. That freedom comes with the cost of hosting and maintaining the model yourself.

The reasoning-focused models, like OpenAI's o-series and Claude's extended-thinking variants, represent a newer development. These models are trained to spend more compute at inference time, producing intermediate reasoning steps before giving an answer. They tend to be stronger on maths, coding, and multi-step problems, but slower and more expensive per query.

No single model is best for everything. Claude might be better for careful writing, GPT for general versatility, Gemini for multimodal tasks involving images, and a fine-tuned Llama for cost-sensitive deployments where you need control. The right choice depends on the task, the budget, and the constraints.

10. Frequently Asked Questions